Load Balancing
Spreading requests evenly is easy. Spreading work evenly, and routing around sick servers, is the actual job.
The story
Four API servers sit behind a round-robin load balancer. Each request goes to the next server in turn, so each gets exactly 25% of the traffic. The dashboard says the load is perfectly balanced.
Then one server's disk starts to fail. It doesn't crash. It just answers in 2 seconds instead of 50 ms.
Round robin doesn't notice. It keeps sending that server 25% of all requests, which pile up on it 40 times longer than anywhere else. Its connection pool fills, its queue grows, and soon a quarter of all users are waiting seconds for every request while the other three servers sit mostly idle.
The health check is no help either. It calls /health, which returns 200 OK from memory without touching the disk or the database. The server is sick in every way users care about and healthy in the only way the load balancer checks.
Two fixes would have prevented it: an algorithm that sends work to the server that's finishing work, and a health check that tests what users actually need.
The idea
A load balancer sits in front of identical servers and decides, for each connection or request, which one gets it. It has three jobs:
- Spread the load so no server is overwhelmed while others idle.
- Route around failure by taking unhealthy servers out of rotation.
- Hide the fleet so clients see one address while you add, remove and deploy servers behind it.
Layer 4 vs Layer 7
| L4 (TCP/UDP) | L7 (HTTP) | |
|---|---|---|
| Sees | IP, port | URL, headers, cookies, body |
| Speed | Very fast, very cheap | Slower: has to parse and often decrypt |
| Can route by path or header | No | Yes |
| Balances | Connections | Individual requests |
| Examples | AWS NLB, IPVS, Maglev | NGINX, HAProxy, Envoy, AWS ALB |
A common setup uses both: an L4 layer at the edge for raw speed, spreading connections across a fleet of L7 proxies that do the smart routing.
The algorithms
- Round robin: next server in turn. Blind to how busy each one is. Fine for uniform, cheap requests.
- Least connections: send to the server with the fewest requests in flight. A slow server holds connections longer, so it automatically gets fewer new ones. This is what fixes the story.
- Power of two choices: pick two servers at random and use the less loaded one. Nearly as good as checking every server, without every load balancer needing a perfectly up-to-date view of the whole fleet.
- Hashing: same key → same server. Useful for caches and sticky state. Use consistent hashing so adding one server doesn't reshuffle every key.
Watch round robin feed a slow server, then switch algorithms:
Health checks that tell the truth
Neither extreme is right on its own:
- Shallow only misses servers that are up but useless.
- Deep checks on shared dependencies are dangerous: if the database blips, every server fails its check at the same moment, the load balancer removes them all, and a 2-second database hiccup becomes a full outage.
The usual answer: check what's local to that server (its disk, its thread pool, its own error rate) and let passive health checks do the rest, which means the load balancer ejects a server that's returning lots of 5xx errors or timeouts on real traffic. Envoy calls this outlier detection.
Two more things to get right:
- Connection draining. When a server is removed for a deploy, stop sending it new requests and let in-flight ones finish (say 30 seconds) before killing it.
- Slow start. A freshly started server has cold caches. Ramp its share up over a minute instead of giving it full load at once.
Who balances the load balancer?
The load balancer is itself a single point of failure unless you run several:
Managed cloud load balancers do this for you. If you run your own, run them in pairs with a floating IP (keepalived) or behind DNS/anycast.
Cheat sheet
| Question | Answer |
|---|---|
| L4 or L7? | L7 when you route by path/header or need TLS termination. L4 for raw TCP, very high throughput, or in front of L7. |
| Default algorithm | Least connections (or least response time). Round robin only for uniform requests. |
| Huge fleet | Power of two choices |
| Need affinity | Consistent hashing on the key, not plain hash % N |
| Health check | Local checks + passive ejection on 5xx/timeouts. Don't fail every server on a shared dependency. |
| Deploys | Connection draining + slow start |
| LB as SPOF | Several nodes behind DNS/anycast, or a managed LB |
| Sticky sessions | Avoid. Move session state out instead (Horizontal vs Vertical). |
Say it in the interview
"I'd put an L7 load balancer in front of the stateless API tier, using least-connections so a slow server automatically gets less traffic. Health checks test the server's own resources, and passive checks eject anything returning lots of 5xx. I wouldn't fail health checks on the shared database, or one DB blip would remove every server at once."
Go deeper
- SD 1 · Load Balancing: algorithms and failure modes in depth.
- SD 2 · Chapter 16 — Load Balancing: L4 internals, Maglev, consistent hashing.
- SD 3 · Q08 — What Is a Load Balancer: the chain of whys.