Rate Limiting

Decide in advance who gets how much, so one client's bug can't become everyone's outage.

The story

A B2B API serves about a thousand customers. Total traffic is a comfortable 3,000 requests per second.

One Friday evening, a single customer ships an integration with a bug: when a call fails, it retries immediately, forever. One failure sets off the loop. Within minutes that one customer is sending 8,000 requests per second, more than everyone else combined.

The API has capacity for about 6,000. Every server's thread pool fills with that one customer's requests. The other 999 customers, who did nothing wrong, start getting timeouts. The on-call engineer spends 40 minutes working out who is sending the traffic, then blocks them by hand with a firewall rule.

A per-customer limit of, say, 200 requests per second would have turned this into a non-event. The buggy customer gets 429 Too Many Requests on most of their calls, which they deserve. Everyone else never notices.

Rate limiting isn't mainly about attackers. It's about fairness and protecting shared capacity from any single caller, including your own batch jobs and other teams' services.

The idea: the algorithms

Algorithm	How it works	Bursts?	Typical use
Fixed window	Count per calendar minute, reset at :00	Up to 2× at the window boundary	Simple quotas
Sliding window	Count over the last 60 seconds	No	Login attempts, OTPs, payments
Token bucket	Bucket refills at a steady rate. Each request takes a token.	Yes, up to bucket size	Public APIs (the most common default)
Leaky bucket	Requests queue and drain at a constant rate	Smoothed out	Protecting a fragile downstream

Why fixed windows leak

A limit of 100 per minute, enforced per calendar minute:

A sliding window, or a sliding-window counter that weights the previous window, closes that gap.

Token bucket, the usual default

Two numbers describe it: rate (10/s sustained) and burst (50 at once). It allows real usage patterns, like a page load that fires 20 calls together, while capping the average. AWS, Stripe and GitHub all describe their API limits in roughly these terms.

Try a burst against each algorithm:

Loading simulation…

Where to enforce it

Per IP catches crude floods but punishes everyone behind one office NAT or mobile carrier.
Per API key / user / tenant is the fairness layer that fixes the story.
Per endpoint, because one /export can cost as much as 10,000 /get calls.
Outbound: if a payment provider allows you 100 rps, you must limit yourself, or you'll get 429s mid-checkout.

Across many servers

Ten API servers each counting locally will each allow the full limit, which is 10× the intended total. Keep the counter in a shared store:

Redis with an atomic INCR + EXPIRE for windows, or a small Lua script for a token bucket so the read-and-update is atomic.
That's one extra round trip (~0.5 ms) per request. At very high volume, count locally and sync to Redis every few hundred ms, accepting slight over-admission.
If Redis is down: usually fail open (allow traffic) for general API limits, so a limiter outage doesn't become an API outage. Fail closed for login and OTP limits, where letting traffic through means letting brute-forcing through.

Responding well

A rejected client needs to know what to do:

code

HTTP/1.1 429 Too Many Requests
Retry-After: 2
X-RateLimit-Limit: 200
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1729600000

And well-behaved clients back off exponentially with jitter (wait 1s, 2s, 4s… plus a random amount), so a thousand rejected clients don't all retry in the same millisecond.

Cheat sheet

Question	Answer
Default algorithm	Token bucket: rate + burst
No bursts allowed (login, OTP, payments)	Sliding window
Protect a fragile downstream	Leaky bucket / queue at a fixed drain rate
Key to limit on	API key or tenant for fairness. IP only at the edge.
Multiple servers	Shared counter in Redis, atomic via INCR or Lua
Redis down	Fail open for general APIs, fail closed for auth
Setting the number	Above normal peak, below the measured breaking point. Start lenient, tighten with data.
Response	429 + `Retry-After` + remaining-quota headers
Clients	Exponential backoff with jitter
Not a substitute for	WAF, auth, or autoscaling. It buys time and fairness, not security.

Say it in the interview

"Token-bucket limits per API key at the gateway, with the counters in Redis via a Lua script so ten gateway nodes share one count. Expensive endpoints like export get their own tighter limit. Over the limit you get a 429 with Retry-After. If Redis is unavailable the general limiter fails open, but login attempts fail closed."

Go deeper

SD 1 · Rate Limiting: the algorithms in depth.
SD 2 · Chapter 17 — Rate Limiting: distributed counters and their trade-offs.
SD 3 · Q63 — Design a Rate Limiter: the full interview question.
SD 6 · Drill: The Noisy Tenant: the story above, as a drill.