Drill 13 — One Tenant Takes Everyone Down

Why nothing stopped it. For a Pro tenant the credit check is a natural limit: the balance runs out and the API returns 402. Acme is invoiced afterwards, so there is no balance to run out. It has no per-tenant rate limit, no concurrency limit, and no idempotency key, so every resubmission is a new job. Say this first: it shows you know the credit check was protecting you by accident.

Detect. Two signals, both per tenant:

• Submission rate compared with that tenant's own baseline. Acme at 100× its 7-day average pages within a minute.
• Queue wait (p95) by tenant, and the share of running jobs per tenant. "One tenant holds more than 25% of slots" is an alert.

Contain. Three limits, at three layers:

1. A per-tenant rate limit at the API: a token bucket per tenant. Enterprise: 1 job/s sustained, a burst of 60. Above that, 429 with Retry-After. Acme's loop now gets 429s instead of new jobs.

2. A per-tenant concurrency cap in the dequeue: at most 20 running jobs per tenant, 10% of the fleet. It then takes ten tenants at their cap to fill the fleet. The claim query picks the least recently served tenant that is under its cap, then that tenant's oldest job. That's round-robin across tenants, not FIFO across jobs:

   sqlCopy

   WITH t AS (
     SELECT t.tenant_id
     FROM tenants t
     WHERE t.running < t.max_running
       AND EXISTS (SELECT 1 FROM jobs q
                   WHERE q.tenant_id = t.tenant_id AND q.status = 'queued')
     ORDER BY t.last_served_at
     LIMIT 1
     FOR UPDATE OF t SKIP LOCKED
   ), j AS (
     SELECT id FROM jobs
     WHERE tenant_id = (SELECT tenant_id FROM t) AND status = 'queued'
     ORDER BY created_at
     LIMIT 1
     FOR UPDATE SKIP LOCKED
   ), claimed AS (
     UPDATE jobs SET status = 'running', started_at = now()
     FROM j WHERE jobs.id = j.id
     RETURNING jobs.id, jobs.tenant_id
   )
   UPDATE tenants
   SET running = running + 1, last_served_at = clock_timestamp()
   FROM claimed WHERE tenants.tenant_id = claimed.tenant_id
   RETURNING claimed.id, claimed.tenant_id;
   

   The counter tenants.running is incremented in the same statement that claims the job, under the tenant row's lock, and decremented in the transaction that finishes the job. (The reaper that requeues dead jobs must decrement it too.) We tested this on Postgres 18, with partial indexes on jobs (tenant_id, created_at) WHERE status = 'queued' and jobs (tenant_id) WHERE status = 'running'. With 50 tenants and Acme holding 6,000 queued jobs, 200 claims gave every tenant exactly 4 slots, Acme included. With 10 workers claiming concurrently, Acme's running count stopped at exactly 20. A first version that counted running jobs in a subquery instead of keeping a counter overshot to 21: the count ran on a snapshot taken before another worker's claim committed. That is the same check-then-act race as in the credit ledger.

3. An idempotency key on POST /jobs. If Acme's loop resubmits the same job with the same key, the API returns the original job instead of creating a new one (Q26). This is the fix for this bug; the two limits above are the fix for the next one.

Degrade. Acme gets 429s and its queued jobs run 20 at a time. A tenant that submits a new job now gets a slot within one claim cycle, because the round-robin serves it before Acme's next job.

Recover. Two backlogs are left, and they need different answers.

• Everyone else's. At 13:10 about 1,125 of the 2,250 waiting jobs belong to other tenants. With Acme held to 20 slots, they share 180 slots: 180 ÷ 32 s = 5.6 jobs/s, against 4.95 arriving, so the spare is 0.67/s and the backlog takes 1,125 ÷ 0.67 ≈ 28 minutes. Set Acme's cap to 0 until it clears: 200 slots give 6.25 − 4.95 = 1.3 spare, and it takes about 14 minutes. Pausing the tenant who caused the backlog is fair, and it halves everyone else's recovery.
• Acme's. By 13:10 Acme had submitted 3,000 jobs, but they are 60 distinct documents, each resubmitted about 50 times. About 1,875 have already run, at half the fleet, and 1,125 are still queued. Ask Acme, then cancel the queued duplicates and don't invoice the ones that ran. Whatever is left drains at Acme's cap: 20 slots ÷ 32 s = 0.625 jobs/s. If all 1,125 were real, that would be 1,125 ÷ 0.625 = 1,800 s, 30 minutes. That is the right outcome. Acme waits for Acme's own burst, and nobody else does.

Prevent. Per-plan limits in the tenants table: Free 2 running, Pro 5, Enterprise 20, raised by contract. An Enterprise customer that really needs 5 jobs/s needs 160 slots: that's capacity they reserve and pay for, not something they take from everyone else at 13:00 on a Tuesday.

The trade-off: a tenant with a real, urgent burst is now slower than it would have been under FIFO. You are choosing predictable latency for everyone over best-case latency for one.

It limits the blast radius to a quarter of the tenants, but it doesn't help that quarter, and adding groups doesn't move the hot tenant. The hot-shard sim shows why. On the Celebrity key tab, one key draws 30% of all traffic and lives on one of 4 shards. Lock in a guess and press Play. Shard 2 runs at 153% of capacity. After a second, click Add 2 shards (4 → 6): Shard 2 only drops to 127%. A key lives on exactly one shard, so more shards just thin out the keys next to it. Replace "key" with "tenant" and "shard" with "worker group": Acme's 500 neighbours in its group still wait 18 minutes.

The sim's fix for a hot read key is a cache. A tenant's jobs can't be cached, so the fix here is the per-tenant cap. Hash partitioning is still worth having for faults a quota can't catch: a job type that crashes its worker, a 200 MB input that runs the process out of memory. For those, use shuffle sharding. Run 8 worker groups and assign each tenant 2 of them. There are C(8,2) = 28 pairs, so a given other tenant shares both of Acme's groups with probability 1/28 ≈ 3.6%. Even a tenant that shares one group still has its other one working. Quotas protect against too much load. Shuffle sharding limits how many tenants one bad workload can reach.

Detect. The hang is the dangerous minute, because hangs produce no errors. Two signals catch it in seconds, not 60:

• A time-to-first-token timeout of 5 s on streaming calls. A healthy provider starts streaming in about 1 s, so a call that hasn't produced a token in 5 s is counted as failed.
• Calls in flight per provider. In-flight calls climbing while completions drop to zero is a hang, whatever the error rate says.

Then lower the total timeout from 60 s to 30 s, a bit above the p99 of a healthy call (about 20 s). Ch 1 covers choosing timeouts from p99.

Contain. The breaker for A opens on timeouts as well as 5xx. Then failover must respect the fallbacks' limits. Convert each provider's per-minute limit into a concurrency limit with Little's law: B allows 600/min = 10 calls/s, at 8 s each = 80 in flight. C allows 300/min = 40 in flight. A gets 360 (90% of 400). Each provider has its own pool. A worker that can't get a slot waits in a short queue instead of firing a call that will come back as a 429.

Degrade. You can serve 75% of peak demand. Close the gap on purpose instead of letting the queue decide:

• Drop the optional polish step during failover. Three calls per job instead of four: 15 calls/s ÷ 3 = 5 jobs/s, exactly peak demand. The backlog stops growing. Mark those jobs polished: false and offer a free re-polish later. That is a feature switch decided in advance (Ch 6).
• Shed by plan if that's not enough: Free-tier jobs wait (they're about 30% of jobs), and paid jobs go first.
• Tell users: "Generation is slower than usual" on the status page and in the app.

Recover. Jobs that were mid-way resume on B from their last saved step: step 2 was done on A, steps 3 and 4 run on B. That works only if each step's output was saved when it finished, and if each prompt has been tested on every provider. Test the fallback prompts in CI, not during the outage. The tenant pays the same price. Your cost rises from about $0.02 to $0.035 a job for the duration, so 9,000 jobs × $0.015 ≈ $135. Say that number: it shows you know failover has a price.

Failback is its own step. When the breaker's half-open probes succeed, send 10% of traffic to A, then 25%, 50%, 100%, a few minutes per stage, watching A's error rate and latency. Sending everything back at once hits A with your whole load, at the same moment A's other customers are failing back, while A is still recovering.

Prevent. The 30 s timeout and 5 s first-token timeout, per-provider pools sized from their limits, the polish switch, and the polished flag. For the 25% gap itself, either raise B's limit with that provider (it costs little until you use it) or add a fourth provider.

The trade-off: degraded output (no polish) for 40 minutes, against an 83-minute backlog. Decide which you prefer with product, before the outage.

If every process keeps its own breaker and they all opened at about 15:01, they all probe on the same 30 s schedule. At 15:40 all 40 probe together, all succeed, and all close together: 360 calls land on A in the same second. That is the synchronised retry spike that the Retry storm tab of the same sim shows. The fixes: put jitter on the probe interval, so processes probe at different moments; share breaker state (a small table or a Redis key), so one probe speaks for the fleet; and ramp the traffic share in stages, as above, rather than flipping each breaker from open to fully closed. A breaker that closes all at once is just a retry storm with a delay.

Detect. Alert on the projected time to cap, not on the spend itself: (cap − spent) ÷ spend rate over the last hour. At 12:30 that is (8,000 − 3,800) ÷ 1,560 ≈ 2.7 hours, early enough for a person to look. Pair it with a per-tenant spend leaderboard: by 14:00, Lumen accounted for 77% of the last two hours' spend.

Contain. Right now: raise the platform cap deliberately (it's your own number, and finance can approve a one-day raise in minutes) and pause Lumen's remaining jobs, as paused, not failed. Then call Lumen.

Degrade: what the cap should have done. A single cap has the whole platform as its blast radius. Make it a hierarchy, checked in this order:

And at every level: in-flight jobs finish. A job that has done 2 of its 4 steps has spent half its cost already. Failing it wastes that spend and owes a refund. Finishing it costs about $0.01 more.

Recover. Every failed job owes exactly one refund. Four code paths can reach the same failed job: the failure handler, the stale-job reaper, a user's cancel, and the admin tool. A refund is safe to repeat only because the ledger row is unique per job and kind, and the wallet is credited only when that insert succeeds:

WITH r AS (
  INSERT INTO ledger (tenant_id, job_id, kind, amount)
  VALUES ($1, $2, 'refund', $3)
  ON CONFLICT (job_id, kind) DO NOTHING
  RETURNING tenant_id, amount
)
UPDATE wallets w SET balance = balance + r.amount
FROM r WHERE w.tenant_id = r.tenant_id;

The second path to arrive inserts nothing, so it credits nothing. Re-queue the 8,860 failed jobs (8,700 new + 160 mid-way) automatically, and charge them again when they run, so a tenant never pays twice and never loses a job.

Prevent. Estimate before you spend, and reserve, then settle:

-- At submission: check and hold the estimate in one statement.
UPDATE wallets
SET available = available - $2, reserved = reserved + $2
WHERE tenant_id = $1 AND available >= $2
RETURNING available;

-- When the job ends (once per job): charge the actual cost, release the rest.
WITH s AS (
  INSERT INTO ledger (tenant_id, job_id, kind, amount)
  VALUES ($1, $3, 'settle', LEAST($4, $2))
  ON CONFLICT (job_id, kind) DO NOTHING
  RETURNING amount
)
UPDATE wallets
SET reserved = reserved - $2, available = available + ($2 - s.amount)
FROM s WHERE wallets.tenant_id = $1;

Here $2 is the estimate and $4 is the actual cost, capped at the estimate. A failed job settles at its actual spend so far, or at 0 if you absorb it. Tested on Postgres 18: a second reservation larger than the available balance returns no row, and a second settle changes nothing. The estimate comes from the input size: 300 pages × about 500 tokens a page × 4 calls ≈ 600k input tokens, at the provider's price. For Enterprise tenants, the "wallet" is their daily spend limit rather than prepaid credit, and the same two statements apply.

Big batches like Lumen's belong on a provider's batch API: many providers offer one at a discount, with results within a day. "Your batch runs overnight at lower cost" is a better answer than "your batch took down the platform".

The trade-off: an estimate can be wrong both ways. Reserve generously and release the difference, and cap the actual charge at the estimate so a bad estimate costs you, not the customer.

Because the check and the spend are separate steps, and 20 of the tenant's jobs run at once. All 20 read "$10 left" at the same moment, each decides it can afford a $1.50 call, and together they spend $30. That is check-then-act across concurrent workers, the same race as reading a balance and then writing it (JS 5 — Race conditions). The reservation makes the check and the hold one atomic statement, so the 7th job's reservation fails while the first 6 succeed. There's a second cost: a platform-wide spend counter updated before every call is one hot row taking 20 updates a second, every one contending for the same row lock. Reserve per job instead, and keep the platform total as a sum you refresh every few seconds. The backstop can be a few seconds stale, and that's fine.

Detect. pg_stat_activity grouped by query fingerprint and by tenant. Set application_name per workload (api, worker, export) and put the tenant id in a SQL comment, so one query shows the 30 identical scans and who owns them. Database CPU and the p99 of the charge query page.

Contain. Cancel the exports (SELECT pg_cancel_backend(pid) for each), and turn off the export endpoint for Acme with a switch. That takes under a minute.

Degrade. Exports become asynchronous jobs: "Your export is being prepared; we'll email a link." Nobody needs 2 million rows within one HTTP request.

Prevent: bulkheads inside the database.

• Separate connection pools per workload: the API pool, the worker pool, and an export pool of at most 4 connections. The pool size is the bulkhead. Exports can never hold more than 4 connections, however many requests arrive (Ch 3).
• A role per workload with its own timeout: ALTER ROLE api SET statement_timeout = '5s', a longer one for export. An API query that runs away is killed at 5 s instead of holding a connection for minutes.
• Limit expensive endpoints by cost, not by count: one running export per tenant, two platform-wide.
• Move the exports off the primary, to a read replica or to a nightly snapshot in object storage, and page through with keyset pagination (WHERE id > $last ORDER BY id LIMIT 5000), not one 4-minute query.

The trade-off: a replica for exports is a second machine to pay for, and it is stale by its replication lag. For an export, a few seconds stale is fine.

Long queries on a replica can be cancelled when replaying the primary's changes would remove rows those queries still need to see. hot_standby_feedback = on stops the cancellations by reporting the replica's oldest running snapshot to the primary. The primary then keeps every dead row version that snapshot might still need, so vacuum can't remove them. Your job table churns: every job's status changes leave dead versions behind. With a 4-minute export running almost all day, they pile up, and the claim query has to step over the dead index entries at the head of the queue before it reaches a live row.

We reproduced the effect on Postgres 18 with a long transaction on the primary, which holds vacuum back the same way. While a 40-second repeatable-read transaction was open, we inserted and deleted 300,000 job rows. VACUUM reported "300040 are dead but not yet removable". The claim query took 12 ms and read 3,689 buffers. After the transaction ended and vacuum ran, it took 0.05 ms and read 5. That's a 240× slowdown, from rows that no longer exist.

The fixes: run exports on a replica without hot_standby_feedback, with max_standby_streaming_delay = -1, so that replica pauses replay and lags during an export rather than cancelling it (fine, because nothing else reads from it). Alert on the age of the oldest open transaction on the primary and on any replica with feedback. Set idle_in_transaction_session_timeout so a forgotten open transaction can't do the same thing.

With the baseline, there's nothing to fail over to. Your recovery time is the region's recovery time, and your data is only as safe as your backups. If those are stored in the same region, you can't reach them either. Say this plainly. Then describe what you would build, and its cost.

The warm standby (build it before the outage):

• An asynchronous replica in Region 2, typically under a second behind. Alert when the lag exceeds 5 s.
• Backups and the WAL archive copied to Region 2 as well (Ch 7).
• Workers and API deployed in Region 2 at minimum size (4 processes), ready to scale to 40.
• Provider API keys that work from both regions. Provider limits are usually per account, so the same 3,000/min follows you.
• DNS with a 60 s TTL for the API.

Detect. Region-level health checks from outside the region, plus the cloud provider's status. The hard part is the decision, not the detection: failing over has a cost (some data loss and a failback later). Set the rule in advance, for example "fail over if the region is down for 10 minutes and there's no recovery estimate", and name who makes the call.

Contain: fence first. Before you promote, make sure the old primary can't take writes if Region 1 comes back. Otherwise you have two primaries (split brain, Ch 5). Fence it with every lever you have: stop it through the cloud API when it becomes reachable, remove it from service discovery, and point every config at the new primary.

Degrade. About 10–15 minutes of errors, then normal service. In-flight work resumes rather than restarting.

Recover: the last second. At 5 jobs/s, about 5 jobs were committed in Region 1 and never replicated.

This is the payoff of keeping the queue, the ledger and the outbox in one Postgres: everything that must agree commits in one transaction, replicates in one stream, and is lost or kept together. With the queue in Redis and the ledger in Postgres, a failover can leave a job without its charge, or a charge without its job, and someone has to reconcile them by hand.

Prevent. The warm standby above, practised: a planned failover drill every quarter, timed. And consider cells: split tenants into two or more independent stacks (database, workers, queue each), so the loss of a region or a cell affects half the tenants or fewer, not all of them.

The trade-off: a warm standby roughly doubles database cost and adds a cross-region replication stream to watch. You buy an RTO of about 15 minutes instead of "whenever the region comes back". Price it against what a multi-hour outage costs your largest customers.

Not straight away, and never by simply restarting the old primary. It holds about 5 transactions that Region 2 never saw: its history has diverged. First save those transactions for reconciliation: compare its jobs and charges after the last replicated position with what clients resubmitted. Then rebuild Region 1 as a replica of Region 2, using pg_rewind to rewind the old primary onto the new timeline, or a fresh base backup. Then fail back as a planned switchover, in a quiet window: stop writes, wait for replication lag to reach zero, promote, and move DNS. Done that way, failback loses nothing, because you control the timing. Many teams simply stay in Region 2 and make Region 1 the standby. Failback is its own risky operation, with its own runbook (Ch 5).