Chapter 10 — Production Observability

From "the Alert Fired" to "This Is the Root Cause"

Part 2: Instruments. Prerequisites: Chapters 1–9. Concept background: SD QA Q54 — Metrics, logs and traces and Q55 — Distributed tracing.

Opening

Every instrument so far looks at one layer up close: the browser, the wire, the logs, the process, the database. In production you almost never start there. You start with an alert that says "error rate above 5%" or "p99 latency above 2 s," a dashboard full of lines, and a Slack channel asking what is going on.

Observability is the ability to answer new questions about your system from the outside, using the signals it emits — without shipping new code to ask them. This chapter teaches you to read those signals without being fooled by them, and gives you a repeatable path from the first alert to the root cause. It closes Part 2: after this, you have an instrument for every layer and a way to choose between them.

Part 1: The Signals and What Each One Is For

The fourth row is the one teams forget. Most incidents are caused by a change, and the single most useful thing on a dashboard is a deploy marker — a vertical line at the moment each release went out.

The investigation moves from aggregate to specific: metrics to see the shape of the problem, traces to find where it lives, logs to see exactly what happened.

Part 2: Metrics Literacy — How Not to Be Fooled

The metric types

Averages lie; percentiles tell the truth

Suppose 99 requests take 50 ms and one takes 10 seconds. The average is about 150 ms — it looks fine, and describes no request that actually happened. The percentiles describe the experience:

• p50 (the median) — half of requests are faster than this. The typical experience.
• p95 — 95% are faster. The experience of a meaningful minority.
• p99 — 99% are faster. The tail — your slowest regular users, often your biggest customers with the most data.

Watch p50 and p99 together. p50 rising means everyone got slower. p99 rising while p50 stays flat means some requests got much slower — usually one endpoint, one customer, one dependency or one host.

Two percentile traps

1. You cannot average percentiles. The p99 across ten servers is not the average of the ten servers' p99s. Correct percentiles come from combining the underlying histograms.
2. Tail latency multiplies. If a page calls 10 services in parallel and waits for all of them, and each is slow 1% of the time, the page is slow whenever any of them is: 1 − 0.99¹⁰ ≈ 9.6% of page loads. A tiny tail in each service becomes a large tail for users.

Resolution hides spikes

A graph that shows one point per minute, averaged, smooths a 5-second total outage into a small bump. When the shape matters, zoom in to the finest resolution available, and prefer max or high percentiles over averages for spiky problems.

Cardinality

Each distinct combination of labels on a metric is a separate time series. Labelling a request metric with route and status is fine — there are a few hundred combinations. Labelling it with user_id creates millions, which is slow, expensive, and often rejected by the metrics system. Per-user detail belongs in logs and traces, not metric labels.

Part 3: What to Watch — RED, USE and Saturation

Two short checklists cover almost everything.

RED — for every service or endpoint:

• Rate — requests per second.
• Errors — failed requests per second (or the error percentage).
• Duration — latency percentiles.

USE — for every resource (CPU, memory, disk, connection pool, thread pool, queue):

• Utilisation — how busy it is (CPU at 70%, 18 of 20 pool connections in use).
• Saturation — how much work is waiting for it (requests waiting for a connection, run-queue length, queue depth, event-loop delay).
• Errors — errors from the resource itself (pool timeouts, disk errors).

Saturation is the early warning. Utilisation can sit at 100% while things are fine. The moment work starts waiting — pool waitingCount above zero (Chapter 9), event-loop delay rising (Chapter 8), queue depth climbing — latency is about to rise. Many outages are visible in a saturation metric minutes before they are visible in error rates.

Part 4: SLOs and Alerts That Mean Something

• An SLI (service level indicator) is a measurement of what users experience: "the fraction of checkout requests that succeed in under 1 second."
• An SLO (service level objective) is the target for it: "99.9% over 30 days."
• The error budget is what the SLO allows to fail: 0.1% of requests over 30 days. Spending it fast is an emergency; spending it slowly is normal.

Alert on symptoms, not causes. Page someone when users are hurting — the SLI is bad, or the error budget is burning fast. Do not page someone because CPU is at 85%; if users are fine, CPU at 85% is a dashboard note, not a 3 a.m. phone call. Cause-based alerts (CPU, memory, disk) are useful as warnings and as clues once an investigation starts.

Too many alerts that do not need action lead to alert fatigue (Chapter 7) — people stop trusting pages, and the real one gets ignored.

Part 5: Reading a Trace

A trace is the full journey of one request. It is made of spans — one span per unit of work (an incoming request, a database query, an outgoing HTTP call) — each with a start time, a duration, a parent, a status and attributes. The tracing UI draws them as a waterfall.

POST /checkout                      ████████████████████████████████████  2,340 ms
├─ auth.verify                      █                                        18 ms
├─ db: SELECT cart                   █                                       12 ms
├─ db: SELECT product  (×42)          ▌▌▌▌▌▌▌▌▌▌▌▌▌▌▌▌▌▌▌▌                   420 ms
├─ pricing.quote                                         ███                160 ms
├─ (gap)                                                    ░░░░░░░░        610 ms
└─ payments.charge                                                   ████  1,080 ms

What to read in a waterfall:

Sampling — the trace you need may not exist

Recording every trace is expensive, so systems sample:

• Head sampling decides at the start of the request (for example, keep 5%). Cheap, but it decides before anyone knows whether the request will be slow or fail — so 95% of your errors are thrown away.
• Tail sampling decides after the request finishes, so it can keep all errors and all slow requests plus a sample of normal ones. Much better for debugging, more complex to run.

If you cannot find a trace for the failing request, check the sampling before concluding it did not happen.

Exemplars — from a spike to an example

Many metrics systems can attach exemplars to a histogram: trace IDs of example requests in each latency bucket. On a latency graph, you click the spike and jump straight to a trace of a request that was actually that slow. This is the fastest bridge from "something is slow" to "this is why."

Part 6: From Alert to Root Cause — the Workflow

This is the method from Chapter 1, with the instruments from Part 2 plugged in.

Step 4 is where most of the narrowing happens. Break the failing metric down by every dimension you have, one at a time:

"It's everywhere" is as informative as "it's only here." Many services failing at the same moment almost always share one dependency.

Part 7: How Observability Lies

Missing deploy markers. Without change events on the graph, the most likely cause is invisible.

Clock skew. Spans and logs from different machines with slightly different clocks can appear out of order, or a child span can seem to start before its parent. Order within a trace by the parent-child relationships, not by raw timestamps.

Survivorship in latency metrics. If a request times out at the load balancer, your app may never finish it — so it never records a duration. The app's latency graph can look fine while users get 504s. Compare app-side metrics with load-balancer or edge metrics, which see every request.

Sampling dropped the evidence. See Part 5.

Monitoring that shares the failure. If your monitoring runs in the same cluster or depends on the same database as the system it watches, it can go dark exactly when you need it. Keep at least one external check — a simple probe from outside your infrastructure.

Dashboards built for the last incident. A dashboard shows what someone decided to graph. When the current problem is new, go to the raw signals and break them down yourself (Part 6, step 4).

The Worked Investigation — Step Through It

Here is a full investigation as a timeline. At each pause, decide what you would check next before revealing it.

Notice how few things were actually examined. Every check was chosen to cut the space in half: tail vs everyone, one endpoint vs all, deploy vs not, one host vs all, one tenant vs all. Four checks went from "the API is slow" to one query on one tenant's data.

The Causal Chains

The Rule You Keep Forever

Go from aggregate to specific: metrics show that something is wrong and when, change events show what changed, breakdowns by dimension show where, a trace shows which hop, and logs show exactly what happened. Every step should cut the space of possible causes in half.

Vocabulary Map

Practice Problems

Problem 1 — Read the shape. Error rate is normal, p50 latency doubled from 80 ms to 160 ms, and p99 is unchanged. What kind of cause does this suggest, and what is the first thing you check?

Problem 2 — The averaged dashboard. A dashboard shows "average p99 across hosts" at 300 ms. One host's p99 is 4 s and the other nine are at 100 ms. Explain why the dashboard number is misleading, and how the real p99 should be computed.

Problem 3 — Alert design. Your team pages on-call when any server's CPU exceeds 80% for 5 minutes. On-call is paged three times a week, and users were unaffected each time. Redesign the alerts: what should page, and what should only warn?

Problem 4 — Read the trace. A trace of a 1.8 s request shows: auth 20 ms, a 900 ms gap with no spans, then three sequential HTTP calls of 250 ms each. Give a hypothesis for the gap, a way to confirm it, and a change that would cut the total time without making any single call faster.

Problem 5 — Everywhere at once. At 10:42, error rates rise on six unrelated services simultaneously, with no deploys in the last two hours. Walk through your first five checks in order.

What's Coming Next

That completes Part 2: an instrument for every layer, and a workflow that connects them. Part 3 puts everything together in the Casebook — complete investigations of the bugs engineers meet again and again. Chapter 11 starts with the family behind "it works on my machine": configuration and build differences, time zones, sessions across instances, version skew after deploys, and CORS that breaks only in production.