Queues
Turn "do it now, or fail" into "do it soon, for sure". Then handle the duplicates, the backlog and the poison.
The story
At checkout, an e-commerce API does five things in a row: charge the card, save the order, send a confirmation email, generate an invoice PDF and update the analytics dashboard. All synchronous, all inside the request.
On sale day the email provider slows down to 8 seconds per call. Checkout requests now take 9 seconds. The API's worker threads fill up waiting on email, new checkouts queue behind them and start timing out, and customers can't pay because an email was slow.
Of those five steps, only the first two have to happen before the customer sees "Order placed". The other three just have to happen soon and eventually for sure.
Now checkout takes 300 ms whatever the email provider is doing. If email is slow, emails arrive a few minutes late. If it's down, messages wait in the queue and go out when it recovers.
The idea
A queue sits between a producer and a consumer so they don't have to be fast, or even up, at the same time. That gives you:
- Decoupling: checkout doesn't know or care which services handle the order.
- Load levelling: a spike of 10,000 messages/second drains at the rate the workers can handle.
- Retries: a failed job goes back on the queue instead of being lost.
- Fan-out: one event, many independent consumers.
Three shapes
Use a work queue for jobs (send this email). Use a log like Kafka when several systems need the same stream of events, need to replay it, or need ordering per key at high volume.
Delivery: assume duplicates
A worker takes a message, does the work, and acknowledges it. If the ack is lost, the queue can't tell "worker finished" from "worker crashed", so it delivers the message again:
| Guarantee | Meaning | Reality |
|---|---|---|
| At-most-once | Ack before working. A crash loses the message. | Fine for metrics you can afford to drop |
| At-least-once | Ack after working. A crash causes a redelivery. | The practical default |
| Exactly-once | Each message has its effect once | Only end to end: at-least-once + an idempotent consumer |
So every consumer must be idempotent: record the processed message ID (or a business key like order_id + "receipt") and skip duplicates. Fault Tolerance and Idempotency covers how.
When the queue backs up
A queue hides a slow consumer only until it fills up. Then you need a plan:
- Watch the age of the oldest message, not just the count. "10,000 messages" might be 2 seconds of work. "Oldest message is 40 minutes old" is an incident.
- Autoscale consumers on lag, up to what the downstream can take. Fifty workers hammering a database that handles ten is a new outage.
- Bound the queue and apply backpressure (reject with 503, or tell producers to slow down) rather than growing until memory runs out.
- Dead-letter queue: a message that fails N times goes aside. Otherwise one malformed "poison" message is retried forever and blocks the workers.
- Ordering: Kafka keeps order only within a partition. Use the entity ID (
order_id) as the partition key so one order's events stay in order.
Cheat sheet
| Question | Answer |
|---|---|
| What goes in the request | Only what the user must see succeed (charge, save). The rest goes on a queue. |
| Jobs | Work queue: SQS, RabbitMQ, BullMQ on Redis |
| Event stream, many consumers, replay | Kafka / Kinesis / Pub/Sub |
| Delivery | At-least-once + idempotent consumers |
| Ordering | Per partition key only. Choose the entity ID. |
| Failing message | Retry with backoff, then the DLQ after N attempts |
| Main metric | Age of the oldest message (consumer lag) |
| Scaling consumers | On lag, capped at downstream capacity |
| Queue full | Backpressure: reject or slow producers. Never unbounded. |
| Writing to the DB and publishing | Transactional outbox, so you never save without publishing or publish without saving |
Say it in the interview
"Checkout synchronously charges and saves the order, then publishes OrderPlaced. Email, invoice and analytics consume it independently, so a slow email provider delays emails, not payments. Delivery is at-least-once, so each consumer dedupes on order_id. Failures retry with backoff and land in a DLQ after five attempts, and we alert on the age of the oldest message."