Chapter 3 — Isolation and Blast Radius

Bulkheads, cells and shuffle sharding: making one failure reach only the users it has to

Prerequisites: Chapter 1 — Timeouts and Deadlines and Chapter 2 — Retries and Idempotency. Deep chapters this one summarises: SD QA Q30 — Circuit breaker, JS Chapter 7, Part 8 — building a breaker, Debugging Chapter 13, Case 7 — one slow dependency took everything down, Q16 — Connection pooling. Node internals: JS Chapter 8 (event loop, libuv threadpool, workers).

How to read this chapter

Every design shares things: a connection pool, an event loop, a queue's workers, a database, a region. Failure spreads through what is shared. When one dependency slows down, one tenant floods you, or one deploy is bad, the question the interviewer is really asking is: who else does this reach? That set of affected users and requests is the blast radius, and isolation is everything you do to make it smaller.

The chapter goes from small to large: one process (bulkheads), then one fleet (tenants, cells, shuffle sharding), then zones and regions. Circuit breakers are covered deeply elsewhere, so Part 2 is a summary. Bulkheads and blast radius are thin elsewhere in the reader, so they get most of the length. The Node bulkhead demo and the Postgres fairness queries in this chapter were run on Node 22 and Postgres 18; the numbers quoted are from those runs.

Part 1: Failure spreads through shared, fixed-size things

Start with Little's Law, because it turns "slow" into "full".

/home receives 50 requests per second and calls the recommendations service. At its normal 80 ms, that is 50 × 0.08 = 4 calls in flight. Recommendations slows to 5 s. Now it is 50 × 5 = 250 calls in flight. Nothing failed, and you need 60 times the slots. Any fixed-size thing those calls hold (a pool, sockets, memory, a worker) fills up, and then every other user of that thing waits too. That is the whole mechanism of a cascading failure: latency becomes concurrency, and concurrency fills shared resources.

So the first step in any isolation answer is to list what is shared. In a typical Node service:

Each row is a place where a bulkhead, a quota or a cell can go. An isolation design names the shared resource and says how much of it each party may hold.

Part 2: Circuit breakers (summary)

The full treatment is in Q30 and JS Chapter 7, Part 8, which builds one. What you must be able to say in the room:

• Three states. Closed: calls pass, failures are counted over a window. Open: calls are rejected immediately, without touching the dependency. Half-open: after a cooldown, a small number of trial calls pass; success closes it, failure reopens it.
• Why. A slow failure holds resources for the whole timeout; a fast failure frees them at once. The breaker turns slow failures into fast ones, and gives the dependency a quiet period to recover.
• Count timeouts as failures, and do not count 4xx from bad input as failures, or one bad client trips it for everyone.
• Pair it with a fallback: the page without the optional strip, a cached value, the next provider.
• Breakers are per process. Thirty pods means thirty breakers, each tripping on its own evidence. That is usually fine; say so.

Breaker vs bulkhead. A breaker reacts after it has seen enough failures, so it needs a failure signal and some seconds. A dependency that is slow but not failing (every call succeeds after 4 s) may never trip a breaker set on errors. A bulkhead caps the damage from the first second, whether the dependency is failing, slow, or merely popular. You want both: the bulkhead limits how much the dependency can take while the breaker is still deciding.

Part 3: Bulkheads in depth

What a bulkhead is

A bulkhead has three parts:

1. A limit: at most N in flight for this dependency (or this class of work, or this tenant).
2. A rule for "full": reject at once, or wait in a small queue with a short timeout.
3. A fallback for the rejection: render without it, serve a cached value, return 503.

Without the second part it is not a bulkhead. A pool with an unbounded wait queue (pg.Pool's connectionTimeoutMillis defaults to 0, meaning wait forever) just moves the pile-up from the dependency to your queue.

A bulkhead in Node

export class BulkheadFullError extends Error {
  constructor(name, why) {
    super(`bulkhead ${name}: ${why}`);
    this.code = 'BULKHEAD_FULL';
  }
}

export class Bulkhead {
  #active = 0;
  #queue = [];
  constructor({ name, limit, maxQueue = Infinity, queueTimeoutMs = Infinity }) {
    Object.assign(this, { name, limit, maxQueue, queueTimeoutMs });
  }

  async run(fn) {
    if (this.#active >= this.limit) {
      if (this.#queue.length >= this.maxQueue) throw new BulkheadFullError(this.name, 'queue full');
      await new Promise((resolve, reject) => {
        const waiter = { resolve, timer: null };
        if (Number.isFinite(this.queueTimeoutMs)) {
          waiter.timer = setTimeout(() => {
            this.#queue.splice(this.#queue.indexOf(waiter), 1);
            reject(new BulkheadFullError(this.name, 'waited too long'));
          }, this.queueTimeoutMs);
        }
        this.#queue.push(waiter);
      });
      // the releaser handed its slot straight to us, so #active already counts it
    } else {
      this.#active++;
    }
    try {
      return await fn();
    } finally {
      const next = this.#queue.shift();
      if (next) { clearTimeout(next.timer); next.resolve(); }
      else this.#active--;
    }
  }
}

It is the semaphore from JS Chapter 7, Part 3 with a bounded, timed queue added. The release hands the slot straight to the next waiter, so a newcomer cannot slip in between, and the release is in finally, so an error cannot leak a slot. One per dependency:

const payments = new Bulkhead({ name: 'payments', limit: 12, maxQueue: 20, queueTimeoutMs: 500 });
const recs = new Bulkhead({ name: 'recs', limit: 8, maxQueue: 0 }); // optional: never wait

app.get('/home', async (req, res) => {
  let strip = null;
  try {
    strip = await recs.run(() => fetch(RECS_URL, { signal: AbortSignal.timeout(2000) }).then((r) => r.json()));
  } catch { /* full, slow or down: render without it */ }
  res.json({ feed: await loadFeed(req), recommendations: strip });
});

Sizing: Little's Law again

Size each bulkhead from the dependency's normal load, not from its worst case: limit ≈ peak rate × p99 latency × 2 (the factor of 2 is headroom).

• Payments: 20 req/s × 50 ms = 1 in flight. A limit of 12 leaves room for a payment-provider slowdown up to about 600 ms before anything is rejected.
• Recommendations: 50 req/s × 120 ms p99 = 6. Limit 8.

When recommendations goes to 5 s, it would need 250 slots. The bulkhead gives it 8. Those 8 calls wait out their 2 s timeout; the other 242 calls a second are rejected in under a millisecond and the page renders without the strip. The bulkhead decides in advance how much of your capacity a dependency may waste when it goes bad.

Measured: one slow dependency, shared pool vs bulkheads

The demo: a Node API with /checkout (calls payments, 20 ms) at 20 req/s and /home (calls recommendations, 80 ms) at 50 req/s. Five seconds in, recommendations starts taking 5 s. The recommendations call has a 2 s timeout, payments 1 s, and the client gives up after 3 s. First, every outbound call shares one limit of 20 with an unbounded wait queue (what you get from one shared pool or one shared concurrency limiter). Then each dependency gets its own bulkhead, as in the code above.

Two things to say from this table:

• Timeouts were set and it still fell over. With a 2 s timeout, recommendations needs 50 × 2 = 100 slots; the shared pool had 20. The wait queue grew by about 40 a second, and checkout requests sat behind it until the client gave up. A timeout bounds how long one call holds a slot. It does not bound how many slots a dependency takes.
• With bulkheads, the cost of the failure was one strip on one page. Checkout's p99 did not move.

Where the bulkheads go in a Node system

Bulkheads by priority, not only by dependency

A bulkhead can also reserve capacity for the work that matters most. With one pool of 20 connections:

const RESERVED = 4; // only critical work may use the last 4 slots
function canStart(priority, active, limit = 20) {
  return priority === 'critical' ? active < limit : active < limit - RESERVED;
}

Checkout and the health check are critical; browsing and search are not. When browsing floods the pool, checkout still finds a slot. The health check deserves its own path entirely: a shallow check (the process is up and its loop is responsive) that uses no shared pool, so a slow dependency cannot make the load balancer remove every instance at once (Debugging Ch 13, Case 7).

The sim: a limit sized to the dependency is also faster

A per-dependency limit also helps when the dependency enforces its own concurrency limit, as LLM providers do. Here a provider serves 5 requests at a time and returns 429 Retry-After: 1 above that. Predict which finishes 30 calls first, then drag Pool size K and Provider limit R.

Promise.all spends its time collecting 429s and sleeping; the pool of 5 never gets one. A bulkhead set at the dependency's real capacity wastes nothing.

Bulkhead mistakes interviewers look for

• Holding a pooled resource across a network call. Taking a DB connection, then calling recommendations, means recommendations' latency now fills the database pool. Release it first.
• Per-process limits multiplied by the fleet. A bulkhead of 8 per pod is 240 concurrent calls at 30 pods. If the dependency can take 100, the limit per pod is 100 ÷ pods, and it has to change when you autoscale (or live in a shared limiter).
• Bulkheads that add up to more than the database can take. Five pools of 20 per pod at 30 pods is 3,000 connections; Postgres's default max_connections is 100. Bulkheads protect your workloads from each other; they do not protect the database from you. Chapter 4 covers that.
• A queue with no timeout. It is a waiting room that never closes.
• One bulkhead for everything. That is just a pool.

Part 4: Blast radius per tenant

Noisy neighbours

In a multi-tenant system, the biggest risk to most tenants is another tenant. One customer imports 2 million rows, runs a report across five years of data, or has a bug that calls you in a loop. Nothing is broken, and everyone else on that shared resource is slower.

The defence is a quota on every shared resource the tenant can fill, not only on request rate:

Concurrency quotas matter more than rate quotas when request costs differ. Ten requests a second is nothing if each takes 20 ms, and is 300 connections if each takes 30 s.

The hot tenant on a shared shard

When data is sharded by tenant (or by key), one very hot tenant lands on one shard, and everyone else on that shard shares its fate. In this sim one key draws 30% of all traffic. Predict, press Play, then add the two shards when the button unlocks. After the reveal, try Cache post:42.

Adding shards does not move the hot key: hashing spreads keys, and this is one key. The fix has to target that one tenant: cache it, give it its own shard, or move it to its own cell. Your biggest tenant sets the size of your shards and cells, so know who it is.

Fair share in a Postgres job queue

A job queue claimed with FOR UPDATE SKIP LOCKED and ORDER BY created_at is first-in-first-out across all tenants. If tenant A enqueues 2 million jobs, tenant B's single job waits behind them. The fix is to choose the next job by tenant, not only by age.

The test: six workers, 100 ms jobs. Tenant A enqueued 40 jobs, then B and C enqueued 5 each. Four ways to claim the next job:

-- Fair share: prefer the tenant with the fewest jobs running, then the oldest job.
WITH next AS (
  SELECT j.id
  FROM jobs j
  WHERE j.status = 'queued'
  ORDER BY (SELECT count(*) FROM jobs r
            WHERE r.tenant_id = j.tenant_id AND r.status = 'running'),
           j.created_at
  LIMIT 1
  FOR UPDATE SKIP LOCKED
)
UPDATE jobs SET status = 'running', started_at = now()
FROM next WHERE jobs.id = next.id
RETURNING jobs.id, jobs.tenant_id;

What the runs show:

• FIFO scales the wait with the other tenant's backlog. 40 jobs ahead cost B 0.9 s. Two million jobs ahead at 60 jobs a second is about 9 hours.
• The soft cap does not hold. Six workers ran the claim at the same moment; each statement counted A's running jobs from its own snapshot, saw fewer than 2, and claimed one. It is check-then-act across concurrent transactions, the same race as in JS Chapter 5. Fine as a rough limit; not a guarantee.
• The hard cap holds, at a cost. Locking the tenant row serialises claims per tenant, so A never exceeded 2. But when B and C finished, four workers sat idle while A waited. A hard cap is not work-conserving: it leaves capacity unused.
• Fair share is usually the right default: contended capacity is split, idle capacity is used. Add a hard cap only where a tenant could hurt a shared downstream (an LLM provider's rate limit, a third-party API).

At scale, the per-row count(*) gets expensive; keep a running count per tenant in its own table (and fix it from a reaper when a worker dies mid-job).

Diagnosing a noisy neighbour

Part 5: Cells — blast radius as a fraction of tenants

A cell is a full copy of the stack (API, workers, queue, database, cache) that serves a fixed subset of tenants. A thin cell router maps tenant to cell.

With 10 cells, anything that goes wrong inside a cell reaches about 10% of tenants: a database failover, a poison tenant, a runaway query, a full disk, a bad deploy (if you deploy one cell at a time and watch it before the next).

The rules that make cells work:

• The router is as simple as possible. It is the one shared part, so it reads a small, cached tenant → cell map and does nothing else. If it needs a database to route, that database is your new single point of failure.
• Cells are the same size and the same shape, so capacity planning, dashboards and runbooks are the same for each.
• The largest tenant sets the cell size. A tenant too big for a cell has to get a cell of its own.
• Deploy cell by cell, starting with a small canary cell, with a soak time between waves.
• Moving a tenant between cells is a data migration, not a config change. Design it once and practise it.

What cells cost and what they don't isolate:

For a startup with 50 tenants, cells are usually overkill; separate pools, queues and quotas give most of the benefit. Say that, and say at what point you would move to cells: when one tenant's incident hurting everyone becomes a contract problem, or when one database can't hold everyone.

Part 6: Shuffle sharding

Cells give each tenant a fixed group. Shuffle sharding gives each tenant a random small set of nodes from the whole fleet. Two tenants rarely get the same set, so a tenant that kills its nodes (a poison request, a flood) rarely takes anyone else down completely.

The numbers, computed and then simulated:

Shuffle sharding only works if three things are true:

1. Clients retry on the other nodes of their own shard, so losing one node is invisible to them.
2. A failing request is not retried across the whole fleet. If the poison request bounces to every node, it kills every node, and the shard boundary means nothing. Cap retries to the shard.
3. Nodes are interchangeable: stateless workers, or replicas of the same data.

It suits stateless request-serving fleets (API front ends, workers pulling from per-tenant queues). For data, you get the same idea from placing each tenant's replicas on different, randomly chosen nodes.

Part 7: Blast radius per zone and region, and per feature

Zones and regions

An availability zone is one or more data centres with independent power and networking, close enough to its siblings for synchronous traffic. A region is a group of zones. Rules that keep a zone failure small:

• Static stability. Run three zones, each with enough capacity to carry the load if one zone disappears: each runs at no more than about 66% at peak. Don't plan to scale up during the failure; the control plane you would scale with may be the thing that is struggling.
• Keep requests in their zone. A request that crosses zones on every hop fails when any zone has trouble. Prefer zone-local replicas and caches where consistency allows.
• The database is the hard part. A primary in one zone with a synchronous standby in another fails over without losing commits, but managed services typically take from tens of seconds to a couple of minutes to do it; Chapter 5 covers failover.
• Regions are a bigger step: data residency, replication across tens of milliseconds, and deciding which region owns each tenant's writes. A region failure's blast radius is every tenant whose writes live there.

Features

One optional feature should not take down the core path. Give optional features their own bulkheads (pools, queues, provider keys), their own deployments if they are heavy, and a kill switch that turns them off without a deploy (Chapter 6). An "AI summary" button that shares checkout's database pool and LLM quota is a design where a feature launch can stop orders.

One slow instance

Blast radius also applies inside one fleet. A single slow server behind round robin still gets its full share of requests, and each of them becomes a p99 sample for someone. Predict, then switch the algorithm.

One server in four was slow, and everyone's p99 rose. Least-connections routing or outlier ejection (remove instances whose latency or errors stand out from the rest) shrinks that blast radius to the few requests already on it.

When the Interviewer Breaks It

Say the answer out loud before opening each one.

Break 1: "Recommendations now takes 10 seconds"

The design: a home feed API calls five services, including recommendations, which is optional.

Break 2: "One tenant just sent 100× their normal traffic"

Break 3: "A bug in today's deploy crashes the process on one tenant's data"

Break 4: "One availability zone just went dark"

Break 5: "You autoscaled from 10 to 60 pods and the payment provider started rejecting you"

The Causal Chains

Checklist: isolation for any design

1. List the shared fixed-size resources on the request path: pools, sockets, event loop, threadpool, workers, provider quotas, the database.
2. Give each external dependency its own bulkhead: a limit from Little's Law, a small queue with a short timeout or none, and a fallback.
3. Never hold a pooled connection across a network call.
4. Reserve capacity for critical work (checkout, health) inside shared pools, and keep the health check shallow.
5. Separate queues and workers for interactive and bulk work; separate deployments for different traffic classes.
6. Per-tenant quotas on rate, concurrency and cost; fair-share scheduling in the job queue.
7. Know your biggest tenant, and what happens to everyone on its shard.
8. Check per-process limits against the fleet size and the downstream's real limit.
9. For larger systems: cells with a thin router, deploys cell by cell; shuffle sharding for stateless fleets; three zones sized for one loss.
10. Optional features get their own resources and a kill switch.

What's next

Isolation decides who a failure reaches. Chapter 4 — Overload Protection handles the case where the failure is simply too much demand: rate limits, backpressure, and load shedding by priority and deadline, with measured goodput for each.