Q29 — How do you design a retry system that doesn't destroy your own service?

Why do retries exist at all?

Networks are unreliable. Services crash and restart. Databases briefly hit connection limits. A request that fails at 2:00:00.001 might succeed perfectly at 2:00:00.500 if retried. Retries are the mechanism for converting transient failures into eventual successes — automatically, without the user having to click "try again."

Without retries, every transient failure is a user-visible error. With naive retries, every failure amplifies into a flood of repeat requests. The challenge is building a retry system that recovers from transient failures without making degraded situations catastrophically worse.

Why does naive retry immediately make things worse?

Suppose a downstream service starts struggling — perhaps its database is slow, causing it to respond to 80% of requests with a 503 (Service Unavailable). Your service receives 1,000 requests per second from users, and passes them to the downstream service.

Without retries: the downstream sees 1,000 req/s, 800 fail, 200 succeed. The downstream is under load but still processing 200 req/s successfully.

With naive immediate retry (retry once on failure): your service retries each failed request immediately. The downstream now sees:

• 1,000 original requests
• 800 retries of the failed ones
• Total: 1,800 req/s — an 80% increase in load

The downstream was already struggling at 1,000 req/s. At 1,800 req/s it struggles more, fails a higher percentage, which causes more retries, which increases load further. This is retry amplification — retries turn a degraded service into a dead service.

And this is the best case with only one retry. If your service retries 3 times and has 3 layers (A → B → C → D), and each layer retries independently, a single user request can generate 3^3 = 27 requests at the deepest layer. This is called a retry storm, and it's how systems collapse under load that they were handling fine before the first failure occurred.

What is exponential backoff — and why does it help?

Exponential backoff means waiting progressively longer between each retry attempt:

• First failure: wait 1 second, retry
• Second failure: wait 2 seconds, retry
• Third failure: wait 4 seconds, retry
• Fourth failure: wait 8 seconds, retry

Why this helps: it gives the downstream service time to recover. If a server restarted and needs 3 seconds to initialize, immediate retries pile up during that 3-second window. With backoff, retries arrive when the server is more likely to be ready. Backoff also naturally reduces the retry load on the downstream — retries are spread over time rather than concentrated in the moment after failure.

The exponential growth (doubling each time) is important. A service that's been down for 5 minutes needs more recovery space than one that's been down for 500ms. Exponential backoff adapts — long outages result in long waits between retries.

Why does backoff alone fail — and what is jitter?

Suppose your service has 10,000 instances, all making requests to the same downstream service. The downstream fails at exactly 12:00:00. All 10,000 instances record the failure at approximately the same moment. All of them apply exponential backoff. All of them wait exactly 1 second. All of them retry at exactly 12:00:01.

The downstream service, which was just recovering, gets hit by a synchronized wave of 10,000 retries simultaneously. It may fail again. All 10,000 instances wait exactly 2 seconds. Another synchronized wave at 12:00:03.

The backoff didn't help because all clients are perfectly synchronized. The solution is jitter — adding randomness to the backoff delay:

wait = random(0, min(cap, base * 2^attempt))

Instead of all clients waiting exactly 1 second, each waits a random time between 0 and 1 second. The retry load is spread over a 1-second window rather than concentrated at the exact 1-second mark. The downstream service sees a smooth trickle of retries as it recovers, not a synchronized assault.

AWS famously wrote about this: their internal services suffered retry storms until they added jitter. With full jitter, their retry-induced load dropped dramatically.

What should and shouldn't be retried?

Not all failures are retryable. Retrying the wrong errors wastes resources and can cause correctness problems.

Retry these:

• Network timeouts — the connection failed, not the operation itself
• 503 Service Unavailable — the server is temporarily overloaded
• 429 Too Many Requests — you're being rate limited; retry after the indicated delay
• 500 Internal Server Error — only if the operation is idempotent (see below)
• Connection refused / DNS resolution failure — transient infrastructure issues

Do not retry these:

• 400 Bad Request — your request is malformed. Retrying sends the same malformed request again. Will always fail.
• 401 Unauthorized / 403 Forbidden — your credentials are wrong or you lack permission. Retrying with the same credentials achieves nothing.
• 404 Not Found — the resource doesn't exist. It won't exist on retry.
• 422 Unprocessable Entity — the server understood your request but it failed business validation. Won't change on retry.

Retry only idempotent operations. If your operation has side effects — sending an email, charging a card, inserting a new record — retrying it might execute the side effect twice. Always guard retryable write operations with idempotency keys (Q26). A GET request is safe to retry freely. A POST that creates a resource must be protected by idempotency before it can be safely retried.

What is a retry budget?

At the service level, independently configured retries on each request can produce emergent behavior nobody planned for. If every component retries 3 times on failure, and failures are correlated (a downstream service struggling), the load amplification multiplies through the call stack.

A retry budget caps the total fraction of outgoing requests that can be retries — for example, at most 10% of all outgoing requests to a given downstream can be retry attempts. Once the retry budget is exhausted, additional failures are returned to the caller immediately rather than retried.

This bounds the amplification. If the downstream is failing at 50% and your retry budget allows 10% retries, you'll retry a portion of failures but not all of them. The downstream sees at most 10% more traffic than it would without retries — not 3x or 10x.

Google's SRE book recommends retry budgets as the correct tool for managing retry storms in a multi-service system. Each service tracks its retry ratio and backs off on retrying when the budget is exceeded.

What is the maximum retry count — and when do you stop?

Beyond exponential backoff with jitter, every retry system needs a maximum retry count and a maximum total duration. Without these, a service will retry indefinitely for a resource that is permanently gone — wasting resources and hanging goroutines/threads.

Max attempts: 3–5 for most services
Max delay per attempt: cap the exponential growth (e.g., max 32 seconds)
Total deadline: retry only while the overall operation deadline hasn't expired (see Q33)

After exhausting retries, the operation fails and returns an error to the caller. The caller can decide whether to surface the error to the user, queue the operation for later, or trigger an alert.

The mental model

Naive retry → retry storm → amplifies load on struggling service → collapse

Exponential backoff:
  Wait 1s → 2s → 4s → 8s (capped)
  Gives downstream time to recover between retry waves

Jitter:
  Randomize the wait: random(0, 2^n) instead of exactly 2^n
  Desynchronizes retries from multiple clients → smooth load instead of synchronized waves

Don't retry:
  4xx client errors (400, 401, 403, 404, 422) → will always fail, don't waste time
  Non-idempotent operations → could double-execute side effects

Do retry:
  Network timeouts, 503, 429, 500 (if idempotent)

Retry budget:
  Cap total retry % of outgoing requests (e.g., max 10%)
  Prevents emergent amplification across multi-service call stacks

Full retry recipe:
  1. Only retry retryable errors on idempotent operations
  2. Exponential backoff with full jitter
  3. Max attempts (3–5) + max delay cap
  4. Honor the operation's deadline — don't retry after deadline expires
  5. Service-level retry budget to bound amplification

A retry system is a contract with your dependencies: "I will persist through transient failures, but I will not make a degraded situation catastrophically worse." Honoring the second half of that contract requires backoff, jitter, selectivity about what you retry, and budget constraints on how much amplification you tolerate.