Chapter 4 — Overload Protection

Rate limits, backpressure and load shedding: staying useful when demand is bigger than capacity

Prerequisites: Chapter 1 — Timeouts and Deadlines, Chapter 3 — Isolation and Blast Radius. Deep chapters this one summarises: SD 2 Chapter 17 — Rate Limiting, SD QA Q31 — Rate limiting algorithms, Q32 — Backpressure, Q63 — Design a rate limiter. Retries and jitter: JS Chapter 7, Part 6.

How to read this chapter

Overload means requests arrive faster than you can finish them. It happens on good days (a launch, a flash sale, a viral post) and bad days (a retry storm, a client bug, a dependency at half capacity). The interviewer's version is short: "traffic just tripled". A design that only scales out has no answer for the minutes before the new capacity arrives, or for the database that doesn't scale out at all.

There are three tools, and they act in different places:

Rate limiting and backpressure are covered deeply elsewhere, so Parts 2 and 3 are summaries with a sim each. Load shedding is thin elsewhere, so it gets most of the chapter, with a Node experiment that measures goodput under eight different queue policies. All numbers below come from that run (Node 22, one machine).

Part 1: What overload does to a queue

Take a server with 5 database connections and 50 ms per query. Its capacity is 5 ÷ 0.05 = 100 requests a second. Send it 200 a second. Each second, 100 more requests join the queue than leave it. After 5 seconds there are 500 waiting, and a new request waits 500 ÷ 100 = 5 s before its query starts.

Now add a client timeout of 1 s, which every real client has. Once the queue holds more than 100 requests, every request waits more than 1 s, so every client gives up before its answer is ready. The server doesn't know. It keeps taking requests off the front of the queue, running their queries and writing responses to closed sockets. The database is 100% busy, throughput is 100 a second, and goodput is zero.

Measured, with 80 req/s for 5 s, then 200 req/s for 10 s, then 80 req/s again, and a plain FIFO queue with no limit:

77% of all queries the server ran (1,944 of 2,535) were for clients who had already left. And look at the last two rows: load dropped below capacity at 15 s, and goodput stayed at zero. The queue held about 1,000 abandoned requests, and spare capacity was only 20 a second, so it would take nearly a minute just to reach the live requests. That is a metastable failure: the overload has ended, and the backlog it created keeps the system down. Add client retries, and the backlog never drains at all.

Part 2: Rate limiting (summary)

SD 2 Chapter 17 and Q31 cover the algorithms and the distributed implementation. In the room, be ready to say:

• Why: abuse, fairness between clients, and protecting capacity. Each needs a different key: IP, API key, tenant, endpoint.
• Algorithms in one line each. Fixed window: cheap, lets up to 2× through at the boundary. Sliding window log: exact, one entry per request. Sliding window counter: two counters, close enough, what most large systems use. Token bucket: a steady rate with bursts up to the bucket size, the usual choice for APIs. Leaky bucket: a queue that drains at a fixed rate, smooth output with added latency.
• Distributed: per-pod counters let a client through N times the limit. Use a shared counter (Redis INCR/EXPIRE or a Lua token bucket) or limit at the gateway.
• The response: 429 with Retry-After, plus remaining-quota headers on successful responses so well-behaved clients slow down early.
• Fail open or closed when the limiter's store is down. Fail open (allow, with a rough per-process limit as a backstop) for ordinary API traffic, so the limiter can't take the API down. Fail closed for endpoints where abuse costs money or safety: login attempts, OTP sends, LLM generation billed to you.
• A rate limit is not overload protection. A thousand clients each under their limit can still exceed your capacity. Limits protect you from one caller; shedding protects you from all of them together.

Try the burst: a quiet client fires 40 requests in one second against three limiters set to 10/s.

The token bucket lets the saved-up burst through and the windows don't. Which is right depends on whether your backend can absorb a burst of 20; if it can't, the bucket size is the number to change.

Part 3: Backpressure (summary)

Q32 covers it in depth. The points to make:

• Backpressure is the consumer slowing the producer down, so buffers stay small. An unbounded queue hides overload until memory runs out; a bounded one shows it.
• When a bounded queue is full, you either block the producer (backpressure: the producer waits) or reject (load shedding: the work is refused). Blocking only works when the producer can wait and has its own bound, otherwise you have moved the pile-up one stage back.
• In Node: writable.write() returns false when the stream's buffer is past its highWaterMark; stop writing until 'drain'. stream.pipeline() does this for you (JS Chapter 9). For job queues, a worker's concurrency is the backpressure on processing, and the producer should check queue depth (BullMQ's getWaitingCount(), or a count(*) on a Postgres queue) and reject or defer new work past a threshold.
• At the edge, you cannot slow the clients down. HTTP has no way to make a browser wait politely. Backpressure ends at your front door; past it, the only tools are rate limits and shedding.

In the sim, producers send 30 messages a second to consumers that manage 20. Predict, then after the reveal switch When full between No cap, Reject and Block.

With no cap, the age of the oldest message climbs without limit. With a cap, it levels off, and the excess is either rejected or held by the producer. Either way, the queue stops lying about how far behind you are.

Part 4: Load shedding in depth

Load shedding is refusing work you can't finish in time, early and cheaply, so the work you do accept finishes in time. Every policy below was run against the same server (capacity 100 req/s) and the same load (200 req/s for 10 s, client timeout 1 s, a quarter of requests marked critical).

The results first

Steady overload, 10–15 s:

And once load fell back to 80/s (15–20 s): plain FIFO stayed at 0/s; skip-if-hung-up recovered to 76/s; every other policy served 80/s at once.

Each row teaches one idea. The rest of this part goes through them.

Admission control: decide at the door

The cheapest place to refuse a request is before you've done anything for it. Admission control looks at a signal and decides accept or reject on arrival:

Rejection must be much cheaper than service. Reject before authentication lookups, body parsing and database calls; better still, at the load balancer or gateway. A server that spends 30% of its effort on each request it rejects still collapses, just later.

Sizing a bounded queue

A queue is only useful for absorbing bursts. Size it by how long you're willing to make someone wait: queue length = acceptable wait × capacity. With capacity 100/s and a 200 ms budget for waiting, that's 20. The run confirms it: the bounded queue of 20 had a p50 of 249 ms (200 ms waiting plus a 50 ms query), zero timeouts, and its rejections took 2 ms, so a client could retry elsewhere straight away.

A queue of 1,000 at the same capacity means a 10 s wait, which is longer than any client will stay.

Deadlines: drop work that's already too late

Requests should carry a deadline, the time after which the answer is useless (Chapter 1). Then the server can check it when it takes a request off the queue:

function pump() {
  while (active < LIMIT && queue.length) {
    const job = queue.shift();
    if (Date.now() + EXPECTED_MS > job.deadline) { // can't finish in time
      reject(job.res, 503, 'past deadline');
      continue;
    }
    start(job);
  }
}

The two FIFO rows show why the check must be about time left, not "is the client still there". Skipping requests whose client had already hung up still gave zero goodput: the requests at the front of the queue were always about 990 ms old, still connected, and timed out during their 50 ms query. Checking now + expected work > deadline fixed that: 90/s goodput, only 23 wasted queries.

But look at its latency: p50 991 ms. The queue stayed full of requests just young enough to pass the check, so every success waited almost the whole deadline, and the rejected ones also waited about 960 ms before being told no. A deadline check alone keeps goodput up and lets latency sit at the deadline. You want the queue short as well.

Two practical points:

• Node cannot see time spent in the kernel's accept queue or in the load balancer. If you need the true arrival time, have the proxy add it as a header (nginx can set one from $msec; some platforms' routers add one, such as Heroku's X-Request-Start), or propagate the caller's deadline.
• Pass the deadline down. A downstream that knows the caller gives up in 300 ms can refuse work it can't do in time, instead of finishing it for nobody.

LIFO under overload, and CoDel

FIFO is fair when the queue is short and the worst possible order when it's long: it always serves the request that has waited longest, which is the one most likely to have given up. LIFO serves the newest request, which still has its whole deadline. In the run, LIFO took goodput from 0 to 102/s with a p50 of 57 ms.

The cost is in the other columns. Under LIFO the oldest requests are never reached while overload lasts: 484 clients waited the full second and timed out without any answer, and with no deadline check the queue kept 673 dead requests at the end of the run. LIFO decides who loses; it doesn't tell them quickly.

CoDel (Controlled Delay) comes from network routers. Its insight is to measure the queueing delay, not the queue length, and to tell a burst from a standing queue: a burst drains, so the queue is empty at some point; a standing queue never is. Applied to a server (Facebook described this in "Fail at Scale", ACM Queue, 2015, together with adaptive LIFO):

• If the queue has been empty at least once in the last 100 ms, it's absorbing a burst. Use a generous queue timeout (500 ms) and FIFO.
• If it hasn't, it's a standing queue. Switch to a short queue timeout (60 ms) and LIFO.

let lastEmpty = Date.now();
const standing = () => Date.now() - lastEmpty > 100;

setInterval(() => {               // sweep every 5 ms
  const now = Date.now();
  if (queue.length === 0) { lastEmpty = now; return; }
  const maxWait = standing() ? 60 : 500;
  for (let i = queue.length - 1; i >= 0; i--) {
    if (now - queue[i].arrived > maxWait) { reject(queue[i].res, 503, 'queue timeout'); queue.splice(i, 1); }
  }
  if (queue.length === 0) lastEmpty = now;
}, 5);

// in pump(): const job = standing() ? queue.pop() : queue.shift();

Result: 100/s goodput, p50 57 ms, zero timeouts, zero wasted queries, the queue never above 13, and every rejected client heard "no" in about 66 ms. In normal operation it behaves like a plain FIFO queue, so you don't pay for it on good days.

Priority shedding

All of the above treat every request the same. They shouldn't. Under overload, a checkout matters more than a "people also viewed" call, and a user's request matters more than a background sync. Label each request with a criticality and shed from the bottom.

In the run, a quarter of the traffic (50/s) was critical, well inside the capacity of 100/s. The adaptive policy, which ignores priority, served only 97 of 250 critical requests; the bounded queue served 182. The priority policy let sheddable requests wait only while fewer than 5 were queued, and critical ones up to 40:

function admit(job) {
  if (active < LIMIT) return start(job);
  const room = job.critical ? 40 : 5;
  if (queue.length >= room) return reject(job.res, 503, 'overloaded');
  queue.push(job);
}

It served 250 of 250 critical requests, at the same total goodput. Shedding by priority doesn't save more work; it chooses which work survives.

How to do it in a real system:

• Few classes, set by the caller. Google's SRE book describes four levels (CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE). Two or three are enough for most products: user-waiting-critical, user-waiting-optional, background.
• Propagate it. A checkout calling inventory passes "critical" along, so inventory sheds the right work too. A header is enough.
• Decide it before the incident. The list of what's core and what's optional is a product decision (Chapter 6). Writing it during the outage is too late.
• Watch for priority inflation. If every team marks its calls critical, nothing is. Make "critical" need a reason and review it.
• Background work sheds first, and it can wait in a durable queue instead of being dropped.

503 or 429?

Two traps. A shed request must not fail the health check: if the load balancer sees 503s from /health, it removes the instance and moves its load to the rest, which then shed more and get removed in turn (Debugging Chapter 13, Case 7). Keep /health outside the shedder. And don't send 429 for your own overload: clients that read 429 as "you did something wrong" may stop retrying entirely, or page their own on-call for a bug that isn't theirs.

The client's half: don't retry into an overloaded server

Every retry is more load on a server that already has too much. Three client rules (Chapter 2):

• Honour Retry-After, and add jitter on top.
• Retry budget: retries may add at most about 10% to the request rate. When a server is overloaded, most retries fail, so the budget runs out and retries stop.
• Client-side throttling: Google's SRE book describes clients that track, over the last couple of minutes, how many requests they sent and how many the backend accepted, and reject locally with probability max(0, (requests − K × accepts) / (requests + 1)), with K around 2. When the backend is rejecting most requests, the client stops sending most of them, so rejections stop costing the backend anything.

Part 5: Protecting the database

The database is usually the last component to scale and the first to saturate. Overload protection for it has its own list:

• Pool size is admission control. 30 pods × a pool of 20 = 600 connections, and Postgres's default max_connections is 100. Each Postgres connection is a separate server process, and past a point, more connections make total throughput lower, not higher. Size pools from what the database can do: total connections across the fleet ≤ what it handles well, with a reserve for admin and migrations. A pooler such as PgBouncer in transaction mode lets many clients share fewer real connections.
• A pool wait is a queue: give it a timeout. connectionTimeoutMillis in pg defaults to 0 (wait forever). Set it to a few hundred milliseconds, so a saturated database turns into fast 503s instead of an ever-growing wait.
• statement_timeout per role, so one runaway query can't hold a connection for an hour.
• Separate pools per workload (Chapter 3): reporting and backfills never share the API's connections.
• Limit by cost. An export that scans 9 million rows costs as much as ten thousand page loads. Give expensive endpoints their own concurrency limit.
• Reads: cache and serve stale on overload rather than fall through to the database (Chapter 6).
• Writes: smooth bursts through a queue that the database drains at its own pace, with a bound on the queue (Part 3).
• No retries at the database layer when it's saturated. Retry at one layer, higher up, with a budget.

When the Interviewer Breaks It

Say the answer out loud before opening each one.

Break 1: "Traffic just went to 3× for a flash sale"

Break 2: "Your Redis-based rate limiter is down"

Break 3: "The database is at 100% CPU and p99 is 8 seconds"

Break 4: "The spike ended ten minutes ago and the system is still down"

Break 5: "Should your shedder return 429 or 503?"

The Causal Chains

Checklist: overload protection for any design

1. State capacity in numbers (per instance and for the database) and the client timeouts you're designing for.
2. Rate limit per tenant or API key at the edge, with a shared counter; decide fail open or closed per endpoint.
3. A concurrency limit per instance, and a bounded queue sized by acceptable wait × capacity.
4. Requests carry a deadline; the server checks time left against expected work when it dequeues, and passes the deadline downstream.
5. A queue policy for standing overload: queue timeout plus LIFO (CoDel-style), so old requests are dropped quickly.
6. Criticality labels, propagated with each request, shedding sheddable work first.
7. Rejection is cheap and early: before auth lookups, body parsing and database calls, or at the gateway.
8. 503 + Retry-After for overload, 429 for quota; /health stays outside the shedder.
9. Clients: honour Retry-After, jitter, a retry budget, one retry layer.
10. Database: fleet-wide connection total under what it handles well, pool wait timeout, statement_timeout, separate pools per workload, cost-based limits.
11. Dashboards show goodput and queue wait, not only throughput and CPU.

What's next

Overload protection keeps one service useful when demand exceeds capacity. Chapter 5 — Replication and Failover handles a different break: the machine holding your data dies, and something else has to take over without losing or doubling writes.