Fault Tolerance and Idempotency
Timeouts, retries, circuit breakers and bulkheads keep a failure small. Idempotency makes the retries safe.
The story
A mobile app calls POST /payments to charge ₹2,499. The payment service charges the card successfully, but the response is slow getting back: the database commit took a while under load, and then the mobile network dropped the reply.
After 10 seconds, the app gives up and shows "Something went wrong. Try again?" The user taps Try again. A new POST /payments. A new charge.
The customer has been charged twice. From the app's point of view the first attempt failed. From the payment service's point of view both succeeded. Neither is wrong: after a timeout, the client cannot know whether the request happened.
Every fault-tolerance tool in this page involves trying again or giving up early. Both are only safe if doing something twice has the same effect as doing it once. That property is idempotency, and it's why the two topics share a page.
The idea: four patterns that keep failures small
A dependency that's down is easy to handle: calls fail fast. A dependency that's slow is much more dangerous, because every caller waits, and waiting uses up threads, connections and memory:
1 · Timeouts on every network call
No call without a timeout, ever. Set it from the dependency's normal latency (e.g. p99 is 200 ms → time out at 500 ms), not a library default of 30 seconds or infinity. Pass a deadline down the chain, so a downstream service doesn't keep working on a request whose caller has already given up.
2 · Retries, done carefully
Retries fix transient faults (a dropped packet, a restarting pod). Done carelessly, they multiply load on a service that's already struggling:
- Retry only idempotent operations, or ones that carry an idempotency key.
- Retry only transient errors: timeouts, 503, 429. Never a 400.
- Exponential backoff + jitter: wait 100 ms, 200 ms, 400 ms, each plus a random amount.
- Cap it: 2–3 attempts, and a retry budget (say, retries at most 10% of traffic).
- Retry at one layer only. If the app, gateway and client each retry 3×, one failure becomes 27 calls.
3 · Circuit breakers
If a dependency is failing, stop calling it for a while. That spares it, and your callers get an answer immediately instead of waiting for a timeout.
When the circuit is open, return a fallback: cached data, a default ("recommendations unavailable"), or a queued "we'll confirm shortly".
4 · Bulkheads
Give each dependency its own limited pool of threads or connections, so a slow one can only use up its own pool:
The name comes from ships: a hull divided into watertight compartments stays afloat when one floods.
Idempotency: making "try again" safe
Some operations are naturally idempotent: PUT /users/7 {name: "Asha"}, DELETE /cart/items/3, SET status = 'shipped'. Doing them twice is harmless.
Others aren't: POST /payments, balance = balance - 100, "send an email". For those, use an idempotency key: a unique ID the client generates once per logical operation and sends with every attempt. Stripe's API popularised this as the Idempotency-Key header.
Rules that make it hold up:
- The client generates the key once per operation and reuses it on every retry. A new key per attempt defeats the point.
- The server records the key and the outcome in the same transaction as the effect, with a unique constraint on the key, so two concurrent retries can't both get through.
- A retry that arrives while the first attempt is still running gets
409 Conflict/"in progress", not a second execution. - Keep keys long enough to cover any retry (e.g. 24 hours), then expire them.
- For queue consumers the key is the message ID or a business key (
order_id:receipt). Same idea (Queues).
Cheat sheet
| Pattern | Rule of thumb |
|---|---|
| Timeout | Every network call. About 2–3× the dependency's p99. Propagate deadlines. |
| Retry | Transient errors only, 2–3 attempts, exponential backoff + jitter, one layer, with a budget |
| Circuit breaker | Open on a high error rate, fail fast with a fallback, half-open to probe |
| Bulkhead | Separate pool per dependency so one slow one can't starve the rest |
| Fallback | Cached / default / degraded answer beats an error page |
| Idempotent by nature | PUT, DELETE, "set X to Y" |
| Needs a key | POST that creates, charges, sends or increments |
| Idempotency key | Client-generated once. Stored with a unique constraint, in the same transaction as the effect. |
| Concurrent duplicate | Return 409 / "in progress", never run twice |
| Queue consumers | Dedupe on message ID or business key |
Say it in the interview
"Every outbound call has a timeout around 2–3× the dependency's p99, and we retry only transient errors, twice, with jittered backoff. Calls to the fraud service go through a circuit breaker and their own bulkhead pool, so if it slows down only checkout's fraud step degrades. The payment API takes a client-generated Idempotency-Key, stored under a unique constraint in the same transaction as the charge, so a retry returns the original result instead of charging again."