Drill 11 — One Downstream Takes 10 Seconds

What the sim shows. p99 went from 100 ms to 3.5 s and was still climbing while p50 stayed at 90 ms. Round robin kept sending B one request in four, about 15/s, but at 5× slower B could finish only about 5/s, so its queue grew without limit. Under least connections, p99 held at about 540 ms: B got only 6% of traffic and its queue stayed empty, but requests that did land on B were still slow.

The same in this design. A recommendations instance normally finishes about 1,000 calls/s and gets 750. At 5× slower it can finish about 200/s, so its queue grows by 550 calls every second. Every home load that lands on it waits for the queue. Because Promise.all waits for the slowest call, a quarter of all home loads inherit that wait. That is why p50 is fine and p99 is unbounded.

Detect. Per-instance latency for recommendations (one instance far above the others); the home endpoint's p99 diverging from its p50; traces where the recs span is the long pole (Q55 — Distributed tracing). A health check won't catch it: the instance is up.

Contain. In the load balancer: least-requests or power of two choices (pick two instances at random, send to the one with fewer requests in flight), plus outlier ejection, which removes an instance whose latency or error rate stands far out from its peers. In the BFF: a timeout on recommendations of 250 ms (its p99 is 200 ms), so no home load waits longer than that for it.

Degrade. Calls that hit the timeout render without live recommendations (Break 2 builds the fallback).

Recover. The recommendations team drains the instance and reschedules it on another host.

Prevent. For a read like this you can also hedge: if no answer has arrived after 150 ms (about the p95), send a second copy to another instance and use whichever answers first. Normally about 5% of calls get hedged, so 5% extra load. During this incident a quarter would, so cap hedges at 10% of calls.

Because the timeout is 250 ms and the whole screen has 400 ms. A retry after the timeout starts at 250 ms and needs up to another 200 ms: it finishes after the deadline, so the user never sees its result, and the slow instance still does the first call's work. A hedge sends the second copy before the deadline is spent, at 150 ms, while there's still time to use it. Retries suit failures that are fast; hedges suit latency.

The arithmetic. Each pod gets 100 home loads/s, and each waits 10 s, so 1,000 home loads are open on every pod instead of about 20, each holding the four finished results in memory and a socket to the sidecar for recommendations. Across 30 pods that is 30,000 concurrent calls into a service that is already slow, which makes it slower. The app gives up at its own 10 s timeout and the user sees an error. Nothing has failed, and the product is down.

Detect. Home p50 jumping to 10 s; in-flight requests per pod; recommendations p50 at 10 s on the client side; event-loop delay and heap growing on the BFF.

Contain. Timeouts from the p99, and a deadline for the whole request.

const BUDGET = { profile: 80, posts: 150, recs: 250, ads: 120, badge: 50 };

app.get('/home', async (req, res) => {
  const deadline = AbortSignal.timeout(400);               // the whole screen
  const call = (name, path) =>
    get(path, AbortSignal.any([deadline, AbortSignal.timeout(BUDGET[name])]));
  const u = req.user.id;
  const [profile, posts, recs, ads, badge] = await Promise.allSettled([
    call('profile', `/profile/${u}`), call('posts', `/posts/feed/${u}`),
    call('recs', `/recs/${u}`), call('ads', `/ads/slot?u=${u}`),
    call('badge', `/notifications/count/${u}`),
  ]);
  if (posts.status === 'rejected') return res.status(503).json({ retryAfterMs: 2000 });
  res.json({
    profile: value(profile) ?? (await cachedProfile(u)),
    posts: posts.value,
    recs: value(recs) ?? (await lastGoodRecs(u)),           // degraded, not broken
    ads: value(ads), badge: value(badge),
  });
});
const value = (r) => (r.status === 'fulfilled' ? r.value : undefined);
const get = (path, signal) =>
  fetch(`http://localhost:15001${path}`, { signal }).then((r) => r.json());

allSettled instead of all, because one optional failure must not reject the whole screen. The per-call timeouts come from each service's p99 plus a margin; the 400 ms deadline caps the total. Now a home load holds recommendations for at most 250 ms: 100 × 0.25 = 25 calls in flight per pod.

Degrade. Decide in advance which sections are core (posts) and which are optional (recommendations, ads, badge), and what each optional one falls back to: the user's last good recommendations, written to Redis whenever a call succeeds (TTL 24 h), then a global "popular now" list kept in process memory and refreshed every 5 minutes. The screen shows up in about 260 ms with slightly older recommendations.

Recover. Recommendations rolls back the model. Nothing to do on our side: the next successful calls overwrite the last-good copies.

Prevent. Set an explicit connection cap so the in-flight count can't run away: one keep-alive agent to the sidecar with maxSockets: 256, about 13× the normal 20. Remember this cap: it comes back in Break 5. And pass the deadline downstream (a header such as x-deadline-ms with the time left), so recommendations can stop work nobody will wait for (Chapter 1).

Then the fallback has become a second slow dependency, and the request waits for both. A fallback must be cheaper and more reliable than the thing it replaces: give the Redis read its own 20 ms timeout, and if that fails, use the in-process "popular now" list, which can't be slow because it is already in memory. Never chain remote fallbacks without a timeout on each one, and check the total against the deadline: 250 ms for recommendations plus 20 ms for Redis still fits in 400 ms.

What the user sees now. Half the screens have live recommendations, half have last-good ones. With the Break 2 fallback this is already a degraded-but-working state, which is why the fallback comes before the breaker.

What kind of 50%? That decides it.

• Overload (crash-looping pods, so the survivors are swamped): errors are spread over all calls, and the 503s often carry Retry-After. Here a breaker helps: while it's open, recommendations gets 30 pods' worth of load (3,000 calls/s) removed, which may be what lets its pods start. Users lose the live half of recommendations for a while, a small price.
• A deterministic bug (say, every user with an empty history gets a 500): the same users fail every time, and the other half succeed every time. A breaker throws away the good half and helps nobody. Keep calling, fall back per request, and page the owning team.

Detect. Error rate per dependency, split by status code and by error type; the breaker's state as a metric and an alert when it opens.

Contain. One breaker per dependency (not one for everything): open when errors exceed 40% over a 10 s window with at least 100 calls, so a handful of errors at low traffic can't trip it; stay open 10 s; then half-open, letting 5 probe calls through, and close only if they succeed. The load balancer's outlier ejection works one level down: it removes individual bad pods, which in the crash-loop case is the better fix. Breakers are covered in depth in Q30 and Chapter 3.

Degrade. Last-good recommendations, as in Break 2.

Recover. When recommendations recovers, the half-open probes succeed and the breaker closes by itself. Watch for flapping (the timeline's last rows): if a breaker closes and reopens every 20 s, the errors are not overload, or the probe count is too small to tell.

Prevent. The fallback, the per-dependency breaker, and an alert on "breaker open for more than 5 minutes", which is a degraded product someone should know about.

At a 50% random error rate, five failures in a row happen with probability 0.5⁵ ≈ 3% per run of five calls; at 100 calls/s that's several times a second, so a consecutive-failure breaker opens almost at once and, after each half-open, opens again. At 10% errors, which may be normal background noise for some services, it still opens now and then (0.1⁵ is tiny per run, but there are millions of runs a day). A rate over a window measures what you actually care about, and the minimum call count stops it reacting to three errors at 4 am. Consecutive-failure breakers are fine for a dependency that is either fully up or fully down.

What the sim shows. With exponential backoff and no jitter, the clients that failed together stayed in lockstep: each retry wave landed in one 100 ms tick, up to 32 requests against a capacity of 8, and the provider had still not recovered by the end of the 24 s window. With fixed 1 s retries it sent 855 requests and also never recovered. With full jitter (sleep(random() × backoff)), the peak was 7 and it recovered 4.5 s after coming back.

The amplification here. Each layer multiplies the one below it. When calls fail, the app tries /home up to 1 + 2 = 3 times; the BFF tries recommendations 1 + 3 = 4 times per home load; recommendations tries the feature store 1 + 3 = 4 times per call.

A service at 50% errors because it's overloaded doesn't get better at 12× the load. It goes to 100% errors and stays there even after its pods are fixed, because the retries alone exceed its capacity. This is the retry storm: the outage now sustains itself.

And the BFF's retries can't even help. The first attempt times out at 250 ms; a retry 100 ms later starts at 350 ms and needs up to 200 ms more. It finishes after the 400 ms deadline, so its result is thrown away. These retries add load and nothing else.

Detect. Retry rate per client as its own metric (retries ÷ first attempts); a dependency's incoming rate rising while the user-facing rate is flat.

Contain. Turn off the BFF's retries to recommendations now, with a flag: it has a fallback, and for optional data a fallback is better than a retry. Tell the app to stop: return a 200 with a partial screen instead of a 5xx (an app doesn't retry a success), and honour Retry-After when you do fail.

Degrade. Last-good recommendations, as before.

Recover. Once the extra load stops, recommendations recovers on its own, which you can watch in the sim's jitter run.

Prevent.

• Retry at one layer only, the one that knows the deadline and the idempotency of the call. Here: the app retries /home once on network errors; the BFF doesn't retry optional sections; recommendations retries the feature store once.
• A retry budget: retries may be at most 10% of calls per client (a token bucket where each success adds 0.1 token and each retry costs 1). In the worst case each layer then multiplies by 1.1, so three layers give 1.1³ ≈ 1.33× instead of 48×.
• Exponential backoff with full jitter, and no retry when the remaining deadline is shorter than the call's p99.

You can't coordinate them, so you control what they see. Ship the app with jittered exponential backoff and a cap of one automatic retry. Then steer it from the server: return 200 with whatever sections you have, so most failures never look like failures to the app; send Retry-After on real errors and make the app obey it; and keep a server-driven switch (a field in the response, or remote config) that tells the app to stop retrying altogether during an incident. The phones' retries are the largest multiplier in the table above, and the only way to cut them during an incident is a switch you built beforehand.

The arithmetic. At 9 pm each pod sees 150 home loads/s, so 150 recommendation calls/s, each now allowed to take 2 s: 150 × 2 = 300 sockets wanted, against a cap of 256 that every call from every endpoint shares. Once recommendations holds all 256, a posts call, a profile call or a /search call waits in the agent's queue for a socket. Its own 150 ms timeout expires while it is still queued, before a byte is sent. /home loses its core section and returns 503; /search fails although it has nothing to do with recommendations. The 2 s timeout looked like a product decision about one feature; through Little's law it was a decision about the whole pod: raising the timeout from 250 ms to 2 s multiplied recommendations' worst-case socket use by 8.

Detect. Pool saturation per pod (sockets in use = cap, queued requests > 0); errors on endpoints that don't call the slow service, which is the signature of a shared resource; traces where spans spend their time before the request is even sent.

Contain. Cut recommendations' timeout back to 250 ms with a flag, or switch /home/recs to the fallback for everyone. Either frees the sockets within seconds.

Degrade. Last-good recommendations in the lazy-loaded section; everything else is back to normal.

Recover. Restore the 2 s timeout only after the bulkhead below is in place.

Prevent. Bulkheads: give each downstream its own pool, sized from its own Little's law at the 9 pm peak, so one can't take another's sockets (Chapter 3). Separate Agent objects keep separate pools even to the same sidecar address.

Recommendations' pool is deliberately smaller than its worst case: when it is slow, 60 sockets serve 30 calls/s per pod and the other 120 get the fallback immediately instead of queueing. Put a semaphore in front of it that rejects when 60 calls are in flight, so the rejection takes microseconds instead of a 2 s wait. The total, 165 sockets, is below the old shared 256, and no service can starve another.

Because the pool was also protecting recommendations. With 2,000 sockets each pod would hold 300 calls open, 9,000 across the fleet, against a service that is slow because it's overloaded: more open calls make it slower, which makes each call hold its socket longer. You'd also move the limit somewhere you can't see: the sidecar has its own per-upstream limits (Envoy, for example, defaults to 1,024), and the process has file-descriptor limits. A bigger shared pool delays the exhaustion; separate pools stop it spreading. The fix is isolation plus a limit that rejects fast, not a bigger number.