Drill 12 — The Queue Backs Up

Detect. Alert on the age of the oldest message, per lane, and on the acknowledgement rate dropping to zero. Depth alone misleads: 50,000 waiting is normal at a busy minute and fine if they are 20 seconds old. Age measures what users feel. A sound pair of alerts is "oldest critical message older than 60 s" and "ack rate below 10% of the arrival rate for 2 minutes". The second one would have paged at 09:02, not 09:33.

Contain. Revert the config. Then ask why one push reached all 20 pods at once: worker config should roll out to one canary pod first and stop if it fails its health check.

Degrade. Not every message in the backlog deserves to be sent.

Recover. The baseline has one queue, so the passcodes wait their turn behind 2 million other messages. With a separate critical lane, the 60,000 critical messages drain at the full 2,000/s in 30 seconds. Then decide whether to scale out for the rest:

• 20 pods: 2,000/s capacity, 1,000/s spare → 2M ÷ 1,000 = 33 min.
• 40 pods: 4,000/s capacity, but email is capped at 1,500/s by the provider. Email backlog 1.2M, email arrivals 600/s, so email spare is 900/s → 1.2M ÷ 900 ≈ 22 min. Push and webhooks (800,000) get the other 2,500/s, minus 400/s new arrivals → 800,000 ÷ 2,100 ≈ 6 min.

Doubling the pods cut the drain from 33 minutes to 22, not to 16, because the provider limit became the bottleneck. The drain rate is the smallest limit in the path. Say that sentence in the room.

Prevent.

• Split one queue into lanes by priority: critical, transactional, bulk. Workers always take from critical first.
• Give each message an expiry (expires_at). Workers drop expired messages and count them on a metric instead of sending them.
• Roll out worker config and code with a canary, like any other deploy.
• Alert on age and on ack rate, as above.

The trade-off: priority lanes add a scheduler that someone must own, and expiry means the system deliberately drops messages it accepted. Write down which types may expire, and get the product owner to agree, before the outage, not during it.

Do the sum out loud. To clear 2M in 300 s while 1,000/s keeps arriving, you need 2,000,000 ÷ 300 + 1,000 ≈ 7,700 msg/s: 77 pods. Of that, 60% is email, about 4,600 sends a second, three times the provider's limit. The extra pods would mostly collect 429s.

So the answer is not "more workers". It is: drain what matters fast and decide what not to send. Critical messages recover in 30 seconds with lanes. Transactional mail is limited by the provider, about 22 minutes after a 33-minute outage. Bulk can wait hours. If the business needs faster email recovery, that is a bigger provider contract or a second provider, and it has a monthly cost you can name. Compare it with how often you expect a 30-minute outage.

Detect. Pod restarts climbing, the receive count per message climbing (most managed queues expose how many times a message has been delivered), and ack rate falling while arrivals stay flat. A log line TypeError: Cannot read properties of undefined (reading 'name') repeated with the same template id points at the cause within a minute.

Contain. Roll the template back to v6. Templates are code: they need the same versioning, canary and rollback as code. Until the rollback lands, a switch that makes the worker skip order_shipped (send it to the dead-letter queue) stops the crashes in seconds.

Degrade. Digital-order customers get a fallback template: a plain "Your order #1234 has shipped" with no carrier details. A worse email on purpose is better than none.

Recover. The 230,000 waiting messages drain at the spare 1,000/s: about 4 minutes. Messages parked in the dead-letter queue (DLQ) are replayed after the template is fixed, using a replay tool that re-publishes them at a controlled rate.

Prevent. Three layers, because each catches what the one before misses:

1. Never let one message crash the process. Wrap the handling of each message in try/catch, including the asynchronous parts: await the renderer so its errors reach the catch instead of escaping as an uncaught exception. A message that fails must fail alone.
2. Classify the error. A render error, a 400, or a 404 on a webhook URL is permanent: retrying will never work, so send the message to the DLQ on the first failure. A timeout, a 5xx or a 429 is transient: retry with exponential backoff and jitter, through a delay queue, not an immediate NACK. (Ch 2 has the table of what is retryable.)
3. Cap the attempts in the queue, not in the worker. A counter in the worker's memory dies with the worker. The queue's own delivery count survives crashes. After 5 deliveries, the queue moves the message to the DLQ. SQS calls this a redrive policy with maxReceiveCount, and RabbitMQ quorum queues have a delivery limit.

Then alert on DLQ depth > 0 for critical messages, and on DLQ growth rate for the rest.

The trade-off: a DLQ is a place where messages go to be forgotten unless someone owns it. Give it an owner, an alert, and a replay tool, or you have only swapped a loud failure for a quiet one.

No. Forty new poison messages arrive every second. With a cap of 5, each can still kill up to 5 pods before it is parked, so the fleet faces about 200 crashes a second. That is far more than 20 pods can absorb when each restart takes 10 seconds or more. The delivery count protects you from one bad message. A bad class of messages needs the per-message try/catch and the "permanent error → DLQ at once" rule, so a bad message costs one failed attempt and no crash. Delivery caps are the last line of defence, not the first.

Detect. The 429 rate from the provider, and the oldest-message age on the passcode and reset types breaching 30 s. Both page.

Contain. Pause the campaign: stop consuming bulk messages. If bulk shares the queue, you can't pause it without pausing everything. That is the design flaw. In the baseline the only lever is "stop all email", which hurts more.

Degrade. Marketing goes out late. Nobody is harmed by a newsletter arriving at 11:00 instead of 10:00.

Recover. Once lanes exist, the campaign drains at whatever the provider has left over:

• Run a client-side token bucket per provider at 1,350/s, 90% of the contract. The 10% margin covers clock differences and the provider counting slightly differently from you.
• Critical and transactional email have first claim: about 400/s on average.
• Bulk gets the rest: 1,350 − 400 = 950/s. Three million emails take 3,000,000 ÷ 950 ≈ 3,200 s ≈ 53 minutes. Tell marketing their campaign takes about an hour. They will accept it if they hear it before the launch.

Handle 429 correctly. The limit is per account, not per message, so a 429 means the whole sender must slow down. Honour Retry-After by pausing the provider's token bucket for that long. Don't NACK each rejected message to retry it alone, which only produces more 429s.

Prevent.

• Priority lanes (from Break 1) with a scheduler per provider: it takes tokens from the bucket and fills them from critical, then transactional, then bulk.
• Rate-limit the producer side for bulk: a campaign is published to its own lane, or published gradually. It is never 3M messages into a shared FIFO.
• Many teams send marketing through a separate provider account or stream with its own limit and its own sender reputation. That is a bulkhead on the provider side.

The trade-off: strict priority can starve bulk forever on a busy day. See the follow-up.

Use weighted shares with a floor, not strict priority. Critical always goes first: it's 30/s, too small to matter. Then transactional gets up to 85% of the bucket and bulk is guaranteed at least 15%: 0.15 × 1,350 ≈ 200/s. At the normal 10:00 peak, transactional uses 700 and bulk takes the remaining 650. In a surge to 1,300/s, transactional is held to about 1,150 and bulk keeps its 200/s: the extra transactional mail waits a few minutes, which is within its 5-minute target. Then move the campaign's start to a quieter hour, or buy a higher limit for the marketing stream. Strict priority is right for passcodes. For two kinds of useful mail, a guaranteed floor is fairer and easier to explain.

The three sources of duplicates:

Detect. Count sends per notification_id and alert when the ratio of sends to notifications rises above 1.001. Without that metric you learn about duplicates from support tickets, as here.

Contain. Make the worker's own deadline shorter than the visibility timeout. A send that has not finished in 10 s is abandoned by the worker, with a timeout on the HTTP call, so it never overlaps a redelivery. Raise the visibility timeout to 60 s, and have long jobs extend it with a heartbeat. The rule: worker deadline + processing time < visibility timeout ÷ 2.

Prevent. You cannot make delivery exactly-once. When a call times out, the sender cannot tell "the provider got it and the reply was lost" from "it never arrived" (the "timeout means unknown" problem in Ch 2). What you can build is at-least-once delivery with deduplication, which looks exactly-once to users in all but a tiny crash window.

1. A claim record before sending. One row per notification and channel:

   sqlCopy

   INSERT INTO deliveries (notification_id, channel, state, claimed_at)
   VALUES ($1, $2, 'sending', now())
   ON CONFLICT (notification_id, channel) DO UPDATE
     SET state = 'sending', claimed_at = now()
     WHERE deliveries.state = 'sending'
       AND deliveries.claimed_at < now() - interval '60 seconds'
   RETURNING state;
   

   A row back means you own the send. No row means someone else is sending it or it is already sent, so acknowledge and skip. After a successful send, set state = 'sent'. A claim older than 60 s is taken over, because its worker probably died. Tested on Postgres 18: the second claim returns no row, and a stale claim is taken over.

2. Pass an idempotency key downstream. If the email provider accepts one, send notification_id, so that the remaining crash window (sent, then died before writing sent) is deduplicated on the provider's side. Not every provider supports this. Find out before you rely on it.

3. Make webhook receivers deduplicate. Every webhook carries a stable event id header, the same on every retry, and the documentation says: "We deliver at least once. Use the event id to ignore duplicates." The warehouse that shipped twice was missing exactly this. It's the same contract as in Q26 and Q72.

The trade-off: one extra Postgres write per send, about 1,000 writes a second at average load, plus an index. That is cheap next to a double shipment.

It depends on the type, and you decide per type, in advance. For a passcode or a password reset, send again: a duplicate is harmless and a missing one locks the user out. For a receipt, send again: a duplicate is mildly annoying. For a webhook, send again with the same event id, because the receiver is required to deduplicate. Only skip when a duplicate does real harm and the downstream has no idempotency key, for example an SMS that costs money and confuses users. In that case, mark it unknown and reconcile against the provider's delivery log before resending. The general rule: when the outcome is unknown, the cost of a duplicate and the cost of a loss decide it.

Detect. Per-destination error rate and latency, and the send slots in use per destination. An alert of "one destination holds more than 10% of all send slots" finds this in seconds.

Contain. Two mechanisms, both per destination:

• A concurrency cap per endpoint: at most 10 requests in flight to any one webhook URL. Now Acme can hold 10 slots, not 1,000, and the other 990 keep serving everyone else. This is a bulkhead: separate capacity per downstream, so one sick dependency can only use up its own share (Ch 3).
• A circuit breaker per endpoint: after 20 consecutive failures, stop sending to Acme. Send one probe every 30 s and resume only on success. While the breaker is open, 144,000 messages wait in Acme's backlog, not as 144,000 retry timers each firing on its own schedule.

Degrade. Acme's webhooks are late. Everyone else's are on time. Acme can also fetch current state from the API, and webhooks should be designed so that is always possible.

Recover. Drain Acme at a rate it can take, and increase the rate only while it stays healthy. Start at 2× its normal rate, 40/s. Every minute that the error rate stays under 1%, add 10/s. On any 5xx or timeout, halve it. This is additive increase, multiplicative decrease (AIMD), the same idea TCP uses to find a link's capacity without being told. The arithmetic:

Tell Acme the number and let them choose. Some customers prefer "skip to latest": drop the backlog and resync from the API.

Size the bucket for recovery. The sim's lesson: a token bucket's capacity is the burst it allows after a quiet period, and the endpoint is weakest exactly after a quiet period. Keep the per-destination bucket's capacity at about one second of its rate, and reset it to empty when a breaker closes.

Prevent.

• Per-destination concurrency caps, rate limits and breakers. Keep each destination's backlog as its own queue (or a partition of the webhook queue keyed by destination), so the scheduler can round-robin across destinations, and one customer's backlog never sits in front of another's.
• Apply the same rule to the email provider after Break 1's drain: the provider's bucket, not the worker count, sets the pace.

The trade-off: per-destination state for 5,000 endpoints (a bucket, a breaker and a backlog each) is more moving parts than one queue. It is the price of keeping one customer's bad afternoon from becoming everyone's.

Strict ordering and parallel retries don't mix, so first ask how strict it must be. Usually, per-entity order is enough: all events for order #1234 in order, while different orders can interleave. Partition each destination's queue by entity id, and send one event at a time per partition. A failed event blocks only that order's later events. Put a sequence number and a timestamp in every event, so receivers can drop an event older than what they have already applied. The most robust contract is thin events: the webhook says "order #1234 changed", and the receiver fetches the current state from the API. Then order and duplicates stop mattering, at the cost of one API call per event.