Q08 — What is a load balancer — why can't you just add more servers without one?

What happens when one server isn't enough?

A single server has finite capacity — finite CPU, RAM, network bandwidth, and a maximum number of connections it can handle simultaneously. When your application grows beyond what one server can handle, the natural instinct is to add another server.

But now you have a problem: how does a client know which server to talk to? The client knows your domain name. DNS resolves it to an IP address. If you have two servers, which IP does DNS return?

You might think: put both IPs in DNS. DNS round-robin — alternate between the two IPs. Clients get IP A half the time, IP B half the time.

This partially works. It distributes traffic. But it breaks in important ways.

Why DNS round-robin is not a load balancer

Problem 1: Caching. DNS responses are cached — by browsers, by operating systems, by ISPs — for the duration of the TTL. Once a client resolves your domain to IP A, it keeps hitting IP A for the next N minutes regardless of how loaded server A becomes. DNS round-robin distributes new connections, not ongoing load.

Problem 2: No health checking. If server B crashes, DNS still hands out its IP half the time. Clients that get IP B get connection failures. There is no mechanism in DNS to detect that a server is down and remove it from rotation.

Problem 3: Sticky state. Many applications maintain session state. A user logged in to server A is not logged in to server B. If the client switches IPs mid-session, it loses its session.

Problem 4: Granularity. DNS operates at the IP level. It can't look inside HTTP to see that server A is handling 10 slow database queries while server B is idle.

These problems aren't edge cases — they're fundamental limitations of routing at the DNS layer. You need something that operates at the connection or request level, with real-time knowledge of server health and load.

What does a load balancer actually do?

A load balancer sits in front of your servers. All traffic goes to the load balancer's IP address. The load balancer then decides which backend server each request should go to, based on current load, server health, and a routing algorithm.

From the outside world, your service has one address. Internally, the load balancer manages a pool of servers. When server B crashes, the load balancer stops sending traffic to it — instantly, before any client sees a failure. When you add server C, you register it with the load balancer and traffic flows to it immediately.

What is the difference between L4 and L7 load balancing?

Load balancers operate at different layers of the network stack, and this determines how sophisticated their routing decisions can be.

Layer 4 (transport layer): An L4 load balancer works at the TCP/UDP level. It sees IP addresses, ports, and connection state — but not the content of the data being sent. It routes connections based on which backend is least loaded or which is next in round-robin. It's fast and simple because it doesn't need to parse request content. But it can only route whole connections, not individual requests within a connection.

Layer 7 (application layer): An L7 load balancer understands the application protocol — typically HTTP. It can read request headers, URLs, and cookies. This enables powerful routing:

Route /api/* requests to API servers and /static/* requests to file servers
Route requests with Cookie: session=XYZ always to the same backend (sticky sessions)
Route requests based on the Host header (virtual hosting — multiple domains on one load balancer)
Strip and add headers, terminate TLS, inspect and log request content

L7 load balancing has higher per-request overhead than L4 (it must parse HTTP), but this overhead is negligible compared to the flexibility it provides. Most web application load balancers (Nginx, HAProxy, AWS ALB) operate at L7.

How does the load balancer decide which server to use?

Several algorithms exist, each suited to different workloads:

Round robin: Send request 1 to server A, request 2 to server B, request 3 to server A, and so on. Simple and works well when requests take similar amounts of time. Fails when requests have wildly different processing times — server A could be stuck on a slow database query while server B is idle but still gets every other request.

Least connections: Send the next request to whichever server currently has the fewest active connections. Better for variable-duration requests — slow requests accumulate on their server while fast requests free up capacity quickly. The load balancer tracks connection counts in real time.

Weighted round robin / weighted least connections: Assign weights to servers based on their capacity. A server with 4 CPUs gets weight 4; a server with 2 CPUs gets weight 2. The load balancer routes proportionally. Useful when your servers have different hardware specs.

IP hash / consistent hashing: Hash the client's IP address to always route the same client to the same server. This provides "sticky" routing without cookies. Useful when server-side session state can't be avoided. Breaks if you add or remove servers (consistent hashing mitigates this — covered in Q34).

Least response time: Route to whichever server is responding fastest. Accounts for real performance, not just connection counts. More complex to implement — requires the load balancer to track response time metrics per backend.

How does health checking work?

The load balancer periodically probes each backend server — sending a simple HTTP request to a /health endpoint, or just trying to open a TCP connection. If a server stops responding or returns errors, the load balancer marks it as unhealthy and stops sending traffic to it. When the server recovers, the load balancer detects this and brings it back into rotation.

This is why your service can survive a server dying at 3am with no one paging anyone — the load balancer quietly removes the dead server, the remaining servers absorb its share of traffic, and clients experience nothing beyond perhaps a momentary connection error on inflight requests.

Health checks are configurable: how often to probe, how many consecutive failures before marking unhealthy, how many successes before marking healthy again. These parameters let you tune how quickly the load balancer reacts to failures without false-positiving on transient blips.

Is the load balancer itself a single point of failure?

Yes — and this is a legitimate concern. If your load balancer dies, nothing gets through.

In practice, load balancers are deployed in high-availability pairs: an active load balancer and a standby that monitors it. If the active one fails, the standby takes over. The IP address they share is a "virtual IP" (VIP) — DNS points to the VIP, and whichever load balancer is active owns that VIP. The failover is transparent to clients.

Cloud load balancers (AWS ALB/NLB, Google Cloud Load Balancing, Cloudflare) are managed services where the provider handles this redundancy internally. You get a single endpoint that's actually backed by distributed infrastructure you don't manage.

The mental model

code

Without a load balancer:
  Client → DNS → Server A (or B) → but no health checking, no smart routing, sessions break

With a load balancer:
  Client → load balancer (single stable IP) → Server A, B, C... (pool managed internally)

Load balancer provides:
  ├── Health checking → dead servers removed automatically
  ├── Even distribution → algorithms ensure no single server is overwhelmed
  ├── Single entry point → add/remove backends without changing DNS
  ├── SSL termination → decrypt HTTPS once at the LB, send plain HTTP to backends
  └── Smart routing (L7) → route by URL, header, cookie, host

L4: routes TCP connections (fast, can't inspect content)
L7: routes HTTP requests (flexible, content-aware)

LB itself: deployed in HA pair with a virtual IP to avoid being a single point of failure

A load balancer exists because distributing traffic is not just about sending requests to different servers — it's about knowing which servers are alive, tracking real-time load, maintaining session consistency, and providing a stable address that decouples clients from your ever-changing backend topology.