Chapter 2 — Retries and Idempotency

A retry is a second request: it adds load to something already failing, and it may do the work twice

Prerequisites: Chapter 1 — Timeouts and Deadlines. Deep chapters: SD QA Q29 — Retry systems, Q26 — Idempotency for write APIs, Q43 — Delivery guarantees, JS 7, Part 6 — Retry with backoff, jitter and a budget, Debugging 13 (Case 4: retry storm, Case 6: charged twice), SD 2 Ch 23 — Payments.

How to read this chapter

Every retry makes two bets: that the same request might succeed if sent again, and that sending it again is safe. The deep chapters cover the first bet well: backoff, jitter, budgets, retry storms. Part 1 summarises them in the form you need in the room. The rest of the chapter is about the second bet, which the reader covers only in pieces: which failures are safe to retry, why a timeout leaves you not knowing whether the work happened, and how idempotency keys and deduplication make "maybe twice" harmless.

The measured runs used Node 22.23 and Postgres 18 on a laptop. The interview version of this topic is usually one line: "The charge request timed out. What does your client do?"

Part 1: The load half, in five sentences

The deep chapters (Q29, JS 7, Debugging 13 Case 4) cover this. What you must be able to say:

1. Retries are load, and they arrive when the dependency can least take it. One retry on an 80% failure rate turns 1,000 req/s into 1,800.
2. Retries multiply across layers. Three layers that each make 3 attempts send 3 × 3 × 3 = 27 calls to the bottom for one user action. Retry at one layer, usually the one closest to the failing call.
3. Use exponential backoff with full jitter: delay = random() × min(cap, base × 2^attempt). Backoff alone spaces the waves out; jitter breaks them up, because clients that failed together otherwise retry together.
4. Cap retries with a budget, e.g. retries may be at most 10% of recent requests. Under a full outage, traffic grows by 10%, not by 3×. When the budget is spent, fail fast.
5. Honour Retry-After and the deadline. Retry only if the remaining budget (Chapter 1) can fit a useful attempt.

In the sim, 40 clients call a provider that is down from 2 s to 6 s and can take 8 requests per 100 ms tick once it is back. Run it with Fixed 1 s, then Exponential, then Exp + jitter, and compare the "Peak after recovery" stat with the capacity.

Who retries without asking you

The multiplication in point 2 usually happens because layers retry by default, and nobody counted them.

The reader's world makes this concrete. An LLM generation job with 3 queue attempts, an SDK with 3 attempts per call, and failover across 3 providers can make up to 27 provider calls for one job during a regional outage. Pick the retry layer on purpose. For a queued job that is usually the queue: set the SDK's maxRetries: 0 inside jobs, keep failover, and let the queue's backoff handle the rest.

Part 2: Safe and unsafe retries

Before any retry, three questions:

1. Will it be different next time? A 400 will not. A dropped connection might.
2. Did the first attempt happen? Sometimes you know it didn't. Often you can't know.
3. Is doing it twice harmless? Only if the operation is idempotent, or you made it so.

What HTTP methods promise

RFC 9110 defines which methods are safe (no side effects intended) and which are idempotent (repeating them has the same effect):

The method is a promise your handler has to keep. Proxies, SDKs, prefetchers and link scanners rely on it. A GET /confirm?token=... that confirms a subscription will be "clicked" by email security scanners that fetch every link to check it. A PUT /wallet/credit that adds to the balance breaks every layer that retries PUT because it is supposed to be idempotent.

Which failures are retryable: sort them by where they failed

Q29 lists retryable status codes. The more useful split is by how far the request got, because that decides question 2:

The rows marked unknown are the hard part, and they are the ones that cause real double charges.

Part 3: A timeout means an unknown outcome

A client timeout ends the client's wait. It sends nothing to the server, and the server's handler keeps going (Chapter 1 measured the same thing across three hops). So after a timeout on a write, there are three possible states, and the client cannot tell them apart:

1. The request never arrived.
2. It arrived and is still running.
3. It finished, and the response was lost or late.

Measured: three "failures", three charges

A POST /charges handler that takes 1,500 ms to commit. The client gives each attempt 1,000 ms and retries twice with jittered backoff.

Read the first row again: the client reported failure to the user, and the user was charged three times. Retrying made it worse, and not retrying would have shown "payment failed" for a payment that went through.

The same thing happens in the browser with a double click. In the sim, set the fix to None (3 stories charged), then Disable (2: the disabled button stops the second click but not the first click's own retry), then Idempotency key (1).

Three ways to resolve an unknown outcome

Cancellation helps but cannot close the gap. If the handler checks the abort signal before committing, a request that timed out early won't commit. But the commit can succeed a millisecond before the client's timeout fires, and then the response is lost on the way back. Only idempotency makes the retry safe in every case.

Part 4: Idempotency keys, beyond the basics

Q26 covers the core: the client creates the key per intent, the server stores the key with a status and the response, a unique constraint settles races, a payload hash catches key reuse with a different body (422), and keys expire after a window (24–72 h). Here is what it leaves out.

When the side effect is in your own database: one transaction

If the work is a write to your own Postgres (debit a wallet, create a job), put the key claim and the work in one transaction:

BEGIN;
-- 1. Claim the key. Exactly one concurrent request gets a row back.
INSERT INTO idempotency_keys (tenant_id, key, request_hash, status)
VALUES ($1, $2, $3, 'processing')
ON CONFLICT (tenant_id, key) DO NOTHING
RETURNING key;
-- No row returned: someone else holds or held this key.
--   SELECT status, response FROM idempotency_keys WHERE tenant_id = $1 AND key = $2;
--   'done' → replay the stored response. Hash differs → 422.

-- 2. The work, in the same transaction.
UPDATE wallets SET balance = balance - $4 WHERE id = $5 AND balance >= $4;
INSERT INTO charges (wallet_id, amount) VALUES ($5, $4);

-- 3. Store the response.
UPDATE idempotency_keys SET status = 'done', response = $6 WHERE tenant_id = $1 AND key = $2;
COMMIT;

Two behaviours make this work, both checked on Postgres 18:

• A concurrent duplicate waits, then sees the result. Session 1 claimed key k1, debited the wallet and held the transaction open for 3 s. Session 2's INSERT … ON CONFLICT DO NOTHING for the same key blocked for 2.5 s until session 1 committed, then returned no row and read done with the stored response. One charge, balance 100 → 20.
• A crash rolls back the key with the work. Session 1 claimed k2, debited the wallet, then failed (division by zero) and rolled back. Session 2, waiting on the same key, then got the row and did the work. Again one debit. No row is ever left stuck in processing.

The wait in the first case holds a pooled connection. If transactions are long, add SET LOCAL lock_timeout = '200ms' before the claim: the duplicate then fails after 200 ms with canceling statement due to lock timeout (SQLSTATE 55P03, measured), and you return 409 so the client retries later.

When the side effect is someone else's API: forward a key, and record progress

You can't put Stripe or an LLM provider inside your transaction. The pattern is to split the work into steps, each committed before the next:

• Derive the provider's key from your own record (order-8812-charge), not from a fresh UUID per attempt. Then any process that resumes the order, the API, a worker, a reconciliation job, sends the same key and the provider deduplicates.
• A row stuck in pending is not an error; it is a recovery point. A job that finds orders pending for more than a few minutes retries the provider call with the same key, or looks the charge up. This is how a crash-resumable generation engine works, and it is the same idea.
• Check the provider's key window. Payment APIs keep keys for a limited time (Stripe, for example, at least 24 hours). A recovery job that retries after that window can charge again.

Natural idempotency is often better than a key table

A generic key table is not the only tool. Many operations can be made idempotent by their own data:

Keep a unique constraint where a duplicate would be a real-world error, even when you also have keys. It is the last net when every other layer has a bug.

Part 5: At-least-once delivery and dedupe

Queues, webhooks and event streams almost all deliver at least once (Q43). A consumer that processes a message and crashes before acknowledging it gets the message again. So every consumer is, in effect, a retried request with an unknown outcome, and the same answer applies: make processing idempotent.

Order of operations in the consumer

BEGIN;
INSERT INTO processed_messages (consumer, message_id) VALUES ('ledger', $1)
ON CONFLICT DO NOTHING RETURNING message_id;
-- no row: already processed. COMMIT, ack, done.
UPDATE wallets SET balance = balance + $2 WHERE id = $3;
COMMIT;
-- then ack the message

Details interviewers push on:

• Dedupe on the producer's event ID, not the broker's delivery ID. If the producer republishes (say, from an outbox after a crash, Q49), a broker assigns a new delivery ID but the event ID is the same.
• The dedupe window must outlast redelivery. If messages can be redelivered for 4 days, keeping IDs for 24 hours lets a late redelivery through.
• Side effects outside your database can't join the transaction. For "send an email", sending then recording risks a duplicate email on a crash in between; recording then sending risks a lost one. Choose per effect. A duplicate receipt is better than a lost one; a duplicate SMS to a customer's phone might not be. Pass a key to the provider where it accepts one.
• Jobs longer than the visibility timeout get redelivered while still running. That's a duplicate with no crash at all (Debugging 13, Case 8). Extend the lease while working, or keep jobs shorter than the timeout.

When the Interviewer Breaks It

Answer each one out loud before opening it.

Break 1: "The charge request timed out. What does your client do?"

Break 2: "The payment provider is down for five minutes. What do your retries do?"

Break 3: "Your queue delivered the same 'credit purchased' message twice. The user got double credits."

Break 4: "Two retries with the same idempotency key arrive in the same millisecond."

Break 5: "Walk me through one user request during an outage. How many calls hit the LLM provider?"

The Causal Chains

Checklist: add this to any design

1. One retry layer per call path, chosen on purpose. Count the defaults: SDKs, proxies, mesh, queue, client.
2. Exponential backoff with full jitter, a retry budget, and a circuit breaker on every retried dependency.
3. Retry only what can succeed: connection failures, 429/503 after Retry-After, timeouts on idempotent or keyed requests. Never 4xx validation errors or a caller's abort.
4. Retry only inside the deadline, and only if a useful attempt still fits.
5. An idempotency key on every write a client or worker may retry, scoped per tenant, with a payload hash and an expiry longer than the retry window.
6. Key claim with INSERT … ON CONFLICT against a unique constraint, in the same transaction as local side effects; a short lock_timeout on the claim.
7. For external side effects, a pending record first and a provider key derived from it, plus a recovery job that resumes pending records with the same key.
8. Unique constraints on business identities (checkout_session_id, (job_id, kind)) as the last net.
9. Consumers that dedupe on the producer's event ID in the same transaction as their effect, and acknowledge after commit.
10. "Processing", not "failed", when the outcome is unknown, and a reconciliation path that settles it.

What's next

Timeouts bound how long you wait and retries recover from transient failures. Neither stops a slow or failing dependency from spreading to the parts of the system that don't need it. Chapter 3 — Isolation and Blast Radius covers circuit breakers, bulkheads (separate pools per dependency and per priority) and keeping one tenant, region or feature from taking down the rest.