Chapter 13 — Casebook III: Slow, Flaky, and Falling Over

Tail Latency, 502s and 504s, Retry Storms, Leaks, Double Writes and Cascading Failures

Part 3: The Casebook. Prerequisites: Parts 1 and 2 (Chapters 1–10), especially Chapters 3, 4, 8, 9 and 10. Background: SD QA Q29 — Retry systems, Q30 — Circuit breaker and Q33 — Timeouts, deadlines and retry budgets.

How to Read This Chapter

Chapter 11's bugs lived in differences between environments. This chapter's bugs need what only production has: load, concurrency and time.

They rarely throw a clear error. They show up as a creeping percentile, a 502 rate of 0.1%, a memory graph that only rises, or a customer charged twice. The cases follow Chapter 11's shape, and three are playable. Flaky tests and time-based bugs are in Chapter 14; cache stampedes are in Chapter 12.

The one piece of physics behind most of them

Little's Law: the number of requests in flight equals the arrival rate multiplied by the time each one takes.

200 req/s × 0.05 s =    10 requests in flight   (normal)
200 req/s × 5 s    = 1,000 requests in flight   (a dependency got slow)

Every request in flight holds memory, a socket, perhaps a pooled connection. So latency turns into concurrency, and concurrency fills fixed-size resources. Once a resource is full, new work waits — saturation (Chapter 10). Waiting adds latency, which adds requests in flight, which fills more resources. That feedback loop is how "a bit slow" becomes "falling over", and it runs through every case below.

Case 1 — "The API Is Slow Sometimes"

Tail latency is the latency of your slowest requests — p99 and above. "Slow sometimes" is almost always a tail problem. Chapter 10 gave the first move: if p50 is flat and p99 rose, some requests got slower, and you find which ones by breaking the tail down by dimension. With fan-out — one request calling many services or queries and waiting for all of them — a small tail in each part becomes a large tail for users.

N+1 fixes are covered in PostgreSQL Ch 11. Keep the dead ends' tells. Pool waiting: a gap before the database spans and waitingCount above zero (Chapter 9). GC pauses: every endpoint on one pod slow at the same instants — V8 collects mostly concurrently, but with large heaps its pauses can reach tens or hundreds of milliseconds (Chapter 8). Cold starts: the slow requests are each new instance's first; on AWS Lambda the REPORT log line includes Init Duration only on a cold start. One slow dependency: one long leaf span in the trace. A noisy neighbour — another workload on the same physical machine taking CPU, disk or network — or container CPU throttling: one host only.

What an interviewer is testing: that you narrow the tail by dimension before opening code, and know that "no query is slow" and "this request is slow because of the database" can both be true.

Case 2 — "Random 502s From the Load Balancer"

Symptom. Between 0.05% and 0.2% of requests through an AWS Application Load Balancer (ALB) return 502 Bad Gateway, on every endpoint, all day. Retrying always works.

First five minutes. Find out who sent it (Chapter 4). In CloudWatch, HTTPCode_ELB_502_Count rises while HTTPCode_Target_5XX_Count stays at zero: the load balancer generated the 502s, and its access logs show target_status_code as - — no response came from the app. They arrive within milliseconds, so they are not timeouts.

Hypotheses.

Root cause. The load balancer keeps keep-alive connections open to your app and reuses them. An ALB's idle timeout defaults to 60 seconds. Node's HTTP server closes an idle keep-alive socket after server.keepAliveTimeout, 5 seconds by default. So Node always closes idle connections first — and a request the load balancer sends at the instant Node closes the socket is lost.

This is the server-side twin of the client-pool race in Chapter 3: same rule, roles swapped.

Fix.

const server = app.listen(port)
server.keepAliveTimeout = 65_000  // longer than the load balancer's 60 s idle timeout
server.headersTimeout = 66_000    // keep this higher than keepAliveTimeout

Some Node versions could close a waiting keep-alive socket via headersTimeout first, so keep it higher; defaults changed across major versions, so check your version's docs. Behind nginx with upstream keep-alive, the same mismatch logs upstream prematurely closed connection while reading response header from upstream.

The other classic source: deploys. If the 502s cluster at deploy markers, processes are being stopped with requests in flight, or the load balancer is still sending to a target that stopped listening. The fix is graceful shutdown (Chapter 8): on SIGTERM, fail readiness, keep serving until the load balancer stops routing to you, then close and drain. On Kubernetes, removal from the load balancer and SIGTERM happen in parallel, so a short preStop delay is common.

What an interviewer is testing: that you prove which hop generated the 502 before debugging the app, and can explain a keep-alive race as "who closes the socket first."

Case 3 — "504s, and the Work Happened Anyway"

Symptom. Users clicking "Generate invoice" see "Request timed out" and click again. The proxy logs 504s. Finance reports customers emailed the same invoice two or three times.

First five minutes. Take one request ID and read it at every hop. The three views disagree:

browser   POST /invoices   aborted after 10.0 s     (fetch timeout)
proxy     POST /invoices   504 after 30.0 s         (proxy timeout)
app       POST /invoices   201 Created in 41.8 s    (invoice saved, email sent)

Round numbers are configured timeouts (Chapter 4). The app says the request succeeded, long after everyone stopped waiting. (An ALB logs a client that left early as the non-standard 460; nginx logs 499.)

Hypotheses.

Root cause. Each timeout stops only the layer that owns it. When the browser gives up at 10 s, nothing tells the proxy; when the proxy sends 504 at 30 s, nothing tells the app; the app's tax call waits up to 60 s, then saves the invoice and sends the email. Outer layers give up while inner work continues. That work is wasted — nobody is waiting for it — and with a retry from outside, it runs twice. Chapter 4's 504 timeline is the payment version.

Fix — a timeout budget that shrinks inwards. A timeout budget is the total time a request may take, divided among its hops. Each layer's timeout must be shorter than its caller's, so the innermost work gives up first and the error reaches someone still listening:

browser 15 s  >  proxy 12 s  >  app request 10 s  >  tax call 4 s  >  statement_timeout 3 s

Better still is deadline propagation: the edge sets one deadline for the request, and each hop passes the remaining time downstream instead of a fixed number. gRPC does this natively; over HTTP, services agree on a deadline header. In Node, tie outbound calls to the budget and to the caller's connection, so work stops when nobody is waiting:

const left = new AbortController()
res.on('close', () => { if (!res.writableFinished) left.abort() })   // the caller went away
const signal = AbortSignal.any([AbortSignal.timeout(10_000), left.signal])
const tax = await fetch(TAX_URL, { method: 'POST', body, signal: AbortSignal.any([signal, AbortSignal.timeout(4_000)]) })

AbortSignal.any needs Node 20 or later. And make the write safe to repeat with an idempotency key (Case 6), because no timeout order removes every retry. Concepts: SD QA Q33.

What an interviewer is testing: that timeouts must shrink inwards, and cancellation does not travel downstream unless you build it.

Case 4 — "A Small Blip Became a Full Outage"

Symptom. At 14:02 the inventory database fails over to its replica, which takes about 20 seconds. The site is down for 25 minutes. The database is healthy at 14:03 — but receiving about nine times its normal query rate, and it stays overloaded until someone blocks traffic at the edge.

First five minutes. Compare the request rate at each layer with real user traffic. The edge sees 2,000 requests per second, the same as yesterday; inside, the rate grows at every hop. The logs' attempt field is mostly 2 and 3. The mobile app, the backend-for-frontend (BFF) and the orders service each retry up to 3 times.

Hypotheses.

Root cause — a retry storm. A retry storm is load created by retries, large enough to keep the system failing after the original cause has gone. Retries at every layer multiply: 3 × 3 × 3 = 27 queries at the bottom for one user action. Worse, they were synchronised: every layer waited exactly one second, so requests that failed together came back together, in waves. The database recovered into a load it could not serve; slow responses caused timeouts, timeouts caused retries, and the system stayed down with no trigger left — a metastable failure. Many clients acting at the same instant are a thundering herd; the caching version is in Chapter 12.

Fix.

• Retry at one layer — usually the one closest to the failing dependency. Every other layer fails fast.
• Exponential backoff with jitter. Exponential backoff doubles the wait after each failure; jitter randomises it so clients that failed together do not retry together. "Full jitter": delay = Math.random() * Math.min(10_000, 100 * 2 ** attempt).
• A retry budget — cap retries at a fraction of normal traffic (say 10%). When the budget is spent, stop retrying.
• Retry only what is retryable — connection failures, 503, 429 (after Retry-After), timeouts on idempotent requests. Never validation errors, never a POST without an idempotency key.
• A circuit breaker tracks a dependency's failures. Closed: calls flow, failures are counted. Open: after too many failures, calls fail immediately for a cool-down without touching the dependency. Half-open: a few trial calls go through; success closes it, failure reopens it.

Mitigation: shed load at the edge, let the database recover, ramp traffic back. Background: SD QA Q29, Q30.

What an interviewer is testing: that you see retries as load and can do the multiplication.

Case 5 — "Memory Climbs Until the Container Restarts"

Chapter 8's memory instruments, applied end to end.

Leak or warm-up? A bounded cache filling up makes the floor rise, then level off; a leak never does. The other usual leaks are in Chapter 8's table; the snapshot technique is the browser's (FE-07).

What an interviewer is testing: that you tell a leak by the sawtooth's floor, know a bigger limit only postpones it, and compare snapshots to find it.

Case 6 — "A Customer Was Charged Twice"

Symptom. A customer has two identical charges for one order; a few similar cases appear each week.

First five minutes. The payment provider's dashboard shows two charges 15.4 s apart. The app's logs, searched by user ID, show two POST /checkout requests with different request IDs from the iOS app. The first took 16.1 s and returned 200; the second took 1.9 s. The app's HTTP client has a 15-second timeout and an automatic retry.

Hypotheses. The gap between the duplicates is the fastest tell.

Root cause. Two requests, two charges: nothing linked the retry to the first attempt. Chapter 4's 504 story, triggered by a client timeout.

Where duplicates come from.

At-least-once delivery means a message is guaranteed to arrive, possibly more than once — what nearly every queue and webhook system provides. At-least-once plus idempotent processing gives effectively once.

Fix. An idempotency key is a unique ID the client generates once per intent and sends with every attempt. The server records it in a table keyed by it (with a status and the stored response): INSERT … ON CONFLICT (key) DO NOTHING. If the row is new, do the work and store the response. If it exists, return the stored response — or 409 Conflict while the first attempt is still running. Pass the same key to the payment provider too.

The safety net. Put a unique constraint where a duplicate would be a real-world error, such as UNIQUE (checkout_session_id) on orders. A duplicate that slips past everything else then fails with SQLSTATE 23505 (Chapter 3): treat it as "already done", return the existing order, and never retry it. Background: SD QA Q26, Q43 — Delivery guarantees, Q72 — Webhook delivery.

What an interviewer is testing: that you can list every source of duplicates, and answer with idempotency keys plus a database constraint — not a disabled button.

Case 7 — "One Slow Dependency Took Everything Down"

Symptom. At 19:40 the recommendations service — the "you might also like" strip on the home page — slows from 80 ms to 30 s. By 19:46 the whole storefront, checkout included, returns 503.

First five minutes. Put every signal on one timeline and ask what moved first (Chapter 10): recommendations latency at 19:40, the pool's waitingCount at 19:41, health-check failures at 19:43, instances removed from 19:44. Traces of GET /home show 30 s inside recommendations.get with a database connection checked out throughout.

Hypotheses.

Root cause — a cascading failure: one part's failure increases the load or waiting on others until they fail too.

Little's Law explains the speed: 40 /home requests per second × 30 s ≈ 1,200 in flight per instance, against a pool of 20.

Fix and prevention.

• Timeouts on every outbound call, inside the budget from Case 3.
• Bulkheads. A bulkhead gives each dependency its own pool or concurrency limit, like a ship's watertight compartments, so one cannot drain the rest. Never hold a database connection across a network call (Chapter 9).
• Circuit breakers (Case 4) with a fallback: render the page without the optional part.
• Load shedding — when in-flight work exceeds capacity, reject the excess immediately with 503 and Retry-After instead of queueing work that will time out.
• Backpressure — bounded queues and limits that push "slow down" back towards the sender (SD QA Q32).
• Shallow health checks. One that depends on a shared dependency removes every instance at once when it is slow.

What an interviewer is testing: that you can trace how waiting spreads through shared resources, and how to contain it.

Case 8 — "Jobs Are Stuck and the Queue Keeps Growing"

The signals for any queue. Depth alone is ambiguous; what matters to users is the age of the oldest message — how late the work is. On SQS: ApproximateNumberOfMessagesVisible and ApproximateAgeOfOldestMessage. In Kafka: consumer lag, how far each consumer group is behind the newest offset, per partition.

Kafka has no built-in DLQ; consumers publish failures to a dead-letter topic themselves. Background: SD QA Q42, Q43, Q67 — Job scheduler.

What an interviewer is testing: that you watch the oldest message's age, spot a poison message in a crash loop, and know at-least-once delivery needs idempotent consumers.

The Pattern Behind Every Case

Every case was a wait that spread until the waiting became the failure — and every one was found with an instrument from Part 2, not by reading code first.

The Causal Chains

The Rule You Keep Forever

Slowness spreads: every request that waits holds something, so latency becomes concurrency and concurrency fills every shared resource. Give every wait a limit that shrinks inwards, retry in one place with backoff and jitter, isolate what each dependency can consume, and make every write safe to happen twice — because under load, it will.

Vocabulary Map

Practice Problems

Problem 1 — Read the tail. p99 on GET /search doubled; p50 is unchanged. Slow requests are spread across hosts, versions and tenants, but all fall in the first 30 seconds after a pod starts. Give your leading hypothesis and its check.

Problem 2 — The 502 trickle. Behind nginx with upstream keep-alive, the error log shows upstream prematurely closed connection while reading response header from upstream a few hundred times a day. Explain the mechanism and say which side's timeout must be longer.

Problem 3 — Do the multiplication. A web app retries failed API calls twice. The API retries calls to search three times. Search retries its database queries twice. What is the worst-case number of database queries per user click? Propose a retry policy for the chain.

Problem 4 — Order the timeouts. A request passes through a CDN (100 s), a load balancer (60 s), a Node API (no timeout), a pricing call (30 s) and a Postgres query (no statement_timeout). Rewrite the timeouts so the system fails in the right order, and say what happens to the work when the user closes the tab.

Problem 5 — Make it once. A "Send invoice" job comes from an at-least-once queue, calls an email API and writes a row to sent_invoices. How do you make each invoice go out effectively once — including when the worker crashes between sending and writing the row?

What's Coming Next

Every case here had a trigger you could point at. Chapter 14 is about bugs where nothing obvious changed — expiring certificates, daylight saving time, dependency drift, third-party changes, flaky tests — and about git bisect, which finds the change nobody remembers making.