Production failures: the things that page you · 1. The three in the morning page

Production failures: the things that page you · Chapter 1 of 24 · Reliability · new chapter every Friday morning By the end of this chapter: Read an unfamiliar dashboard under pressure and decide what to look at first. Why this matters When a pager goes off at three in the morning, adrenaline floods your system. Heart rate increases, cognitive focus narrows, and the brain defaults to pattern-matching rather than analytical reasoning. If you do not have a structured method for reading system state, this physiological response will force you down a rabbit hole. You will spend forty minutes investigating a minor CPU spike on a background worker while the primary database is completely unreachable. Dashboards are almost always built by the engineers who wrote the application, which means they are designed to confirm the system is working, not to diagnose it when it is broken. They are dense with implementation details: garbage collection pauses, cache hit rates, queue depths, and thread counts. Reading an unfamiliar dashboard under pressure requires ignoring ninety percent of what is on the screen. You must know exactly which three or four charts to look at first to isolate the fault domain. Without this discipline, you will guess at the root cause based on whichever chart looks the most erratic, prolonging the outage and exhausting yourself in the process. Before you start To execute the worked example and follow the technical reasoning in this chapter, you must have: - Docker and Docker Compose installed on your local machine. - Python 3.10 or higher installed. - A basic understanding of HTTP status codes (specifically the difference between 200, 400, and 500 series). - Familiarity with the concept of a time-series metric (a value recorded at a specific timestamp). Bounding the problem from the outside in The most common mistake in incident response is starting the investigation at the component you understand best, rather than the component closest to the user. If you are a database engineer, you look at database locks. If you are a frontend engineer, you look at the CDN. You must evaluate the system from the outside in. The "outside" is the absolute edge of your infrastructure-usually a load balancer, an ingress controller, or an API gateway. The "inside" is the deepest layer of persistence. When you open a dashboard, locate the metrics for the edge first. You are looking to answer a single, binary question: is traffic reaching our infrastructure at all? If the load balancer shows zero incoming requests, there is no point looking at application logs or database queries. The failure is upstream-perhaps a DNS failure, a revoked TLS certificate, or a severed transit link. If traffic is reaching the edge, you move one layer inward to the application. If the application is receiving traffic but failing, you move to its dependencies. You only look at internal component metrics (like memory usage or thread counts) once you have proven that the layer above it is receiving requests and generating errors. The three signals that dictate the investigation Once you have identified the layer you are investigating, you must ignore custom business metrics and focus entirely on the RED signals: Rate, Errors, and Duration. These three metrics describe the experience of the consumer calling that layer. Rate is the volume of requests per second. You are looking for sudden, sharp changes. A gradual increase is organic load; a vertical spike is a retry storm or a malicious attack. A sudden drop to zero means a pipe has broken upstream. Errors are the rate of failed requests. In HTTP systems, this means 5xx status codes. You must explicitly filter out 4xx codes during initial triage. A spike in 4xx errors usually means a client deployed a bug and is sending malformed requests. A spike in 5xx errors means your infrastructure is failing to handle valid requests. Duration is latency. You must never look at average latency. Averages hide catastrophic failures. If an endpoint serves 99 requests in 10 milliseconds, and one request hangs for 10,000 milliseconds before timing out, the average latency is 109 milliseconds. On a dashboard, 109ms looks like a healthy system. You must look at the 99th percentile (p99) latency. In that same scenario, the p99 latency is 10,000ms, which immediately tells you that a subset of users is experiencing total failure. Correlating the signals to isolate the fault Reading a dashboard is not about looking at one chart; it is about observing how Rate, Errors, and Duration move in relation to one another at the exact minute the alert fired. Their correlation tells you where to look next. Scenario A: Rate remains flat, Duration spikes, Errors spike. Traffic did not change, but suddenly the system got slow, and then it started throwing errors. This is the signature of resource exhaustion. A downstream dependency (like a database or a third-party API) slowed down. Your application threads waited for that dependency, holding open connections until they hit a timeout limit, at which point they returned 500 errors to the user. Do not look at application code; look at the downstream dependencies. Scenario B: Rate spikes, Duration spikes, Errors spike. This is a capacity failure. The system was overwhelmed by a sudden surge in traffic. The application ran out of CPU, memory, or network bandwidth trying to serve the load. The fix is usually to shed load, scale up, or block the offending traffic source. Scenario C: Rate drops, Errors remain flat, Duration remains flat. The system is fast and error-free, but nobody is using it. During peak hours, this is impossible. This means clients are failing to reach you. The fault domain is strictly upstream: DNS, BGP routing, or a misconfigured external firewall. To see these correlations, you must control the time window of the dashboard. By default, dashboards often show the last 24 hours. A critical failure that began five minutes ago is invisible on a 24-hour graph; it looks like a single, tiny pixel. When you open the dashboard, immediately change the time window to start ten minutes before the alert fired, and end at the current time. This maximizes the visual contrast between "healthy" and "broken." A worked example This example simulates a production service that suddenly experiences a downstream dependency failure. We will spin up a Python application and a Prometheus instance to scrape its metrics. We will then trigger the failure and use PromQL (Prometheus Query Language) to triage the symptoms exactly as you would on a dashboard. Create a directory named triage-example and create three files inside it. 1. app.py This is our web server. It exposes metrics and a /simulate_outage endpoint. import time import random import threading from http.server import HTTPServer, BaseHTTPRequestHandler from prometheus_client import start_http_server, Counter, Histogram # RED Metrics REQUEST_COUNT = Counter('http_requests_total', 'Total HTTP Requests', ['method', 'endpoint', 'status']) REQUEST_LATENCY = Histogram('http_request_duration_seconds', 'HTTP Request Latency', ['endpoint']) outage_active = False class MetricsHandler(BaseHTTPRequestHandler): def do_GET(self): global outage_active start_time = time.time() if self.path == '/simulate_outage': outage_active = True self.send_response(200) self.end_headers() self.wfile.write(b"Outage simulation started.") return # Simulate normal traffic vs outage traffic if outage_active: # Downstream dependency is hanging, causing latency spikes and timeouts time.sleep(random.uniform(2.0, 5.0)) status = 500 else: # Normal operation time.sleep(random.uniform(0.01, 0.05)) status = 200 REQUEST_COUNT.labels(method='GET', endpoint='/api/data', status=status).inc() REQUEST_LATENCY.labels(endpoint='/api/data').observe(time.time() - start_time) self.send_response(status) self.end_headers() self.wfile.write(b"Response") def generate_background_traffic(): """Simulates users constantly hitting the API.""" import urllib.request while True: try: urllib.request.urlopen('http://localhost:8080/api/data', timeout=10) except Exception: pass time.sleep(0.5) if name == 'main': # Start Prometheus metrics server on port 8000 start_http_server(8000) # Start background traffic generator threading.Thread(target=generate_background_traffic, daemon=True).start() # Start main application server on port 8080 server = HTTPServer(('0.0.0.0', 8080), MetricsHandler) print("Server running. Metrics on port 8000, App on port 8080.") server.serve_forever() 2. prometheus.yml This configures Prometheus to scrape our Python application. global: scrape_interval: 2s scrape_configs: - job_name: 'python_app' static_configs: - targets: ['app:8000'] 3. docker-compose.yml This wires the application and Prometheus together. version: '3.8' services: app: build: context: . dockerfile_inline: | FROM python:3.10-slim RUN pip install prometheus_client COPY app.py . CMD ["python", "app.py"] ports: - "8080:8080" - "8000:8000" prometheus: image: prom/prometheus:v2.45.0 volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml ports: - "9090:9090" Execution and Triage: - Open your terminal in the triage-example directory and run:docker-compose up -d - Wait 30 seconds for background traffic to generate baseline metrics. - Open Prometheus in your browser at http://localhost:9090 . - In the query bar, type the following to check the Rate and click "Execute", then switch to the "Graph" tab: rate(http_requests_total[10s]) You will see a steady line representing normal traffic. - In a new terminal window, trigger the failure: curl http://localhost:8080/simulate_outage - Wait 15 seconds. - Now, triage the system using the RED method by running these three queries in Prometheus: Query 1: Errors rate(http_requests_total{status="500"}[10s]) You will see errors spiking from zero to a high rate. Query 2: Rate (Total Traffic) sum(rate(http_requests_total[10s])) You will notice the total rate of requests has