Chapter 16: Load Balancing — Distributing Traffic Across Servers

Curriculum: Distributed Systems from First Principles Chapter: 16 of 24 Part: 7 — Edge Infrastructure Prerequisites: Chapter 1 (latency numbers, network), Chapter 4 (failure taxonomy), Chapter 8 (consistent hashing), Chapter 9 (caching) Estimated reading time: 75–85 minutes

Opening

You've built a horizontally scalable backend: multiple application server instances, a sharded database, a cache cluster, a message queue. Now consider the front door. A client makes an HTTP request to api.yourapp.com. That resolves to an IP address. Packets travel to that IP. Something at that IP receives the request and decides which of your ten application server instances to send it to. That "something" is a load balancer — and the decision it makes every millisecond, millions of times a day, determines your system's performance, reliability, and behaviour under failure.

Load balancing is not a single technique — it is a family of problems that appear at multiple layers of the network stack, each with different constraints and different correct solutions. The load balancer between the internet and your application tier is solving a different problem than the load balancer between your application and your database replicas. The algorithm that works for stateless HTTP requests is wrong for stateful WebSocket connections. The approach that works at 1,000 requests/second breaks at 1,000,000 requests/second. This chapter builds load balancing from the physical reality of how packets move through a network, up through algorithms, health checking, session affinity, and the specific architecture choices that production systems make.

Part 1: Why Load Balancing Exists — The Physical Bottleneck

From Chapter 1: a single machine has finite resources. A web server handling HTTP requests is typically bounded by one of: CPU (compute-intensive requests — JWT verification, JSON serialisation, business logic), memory (concurrent connection state, in-process caches), network bandwidth (large response payloads, file downloads), or I/O (database calls, disk reads).

A well-optimised Node.js or Go HTTP server on modern hardware can handle roughly 10,000–100,000 simple requests per second (no database, minimal computation). Add a database call per request (5ms typical latency): throughput drops to roughly 1 / 0.005 = 200 concurrent requests per worker thread, times worker count. Even with 100 worker goroutines, you're capped around 20,000 requests/second on one machine — and that's the ceiling, not sustainable steady-state throughput.

At internet scale — 100,000–10,000,000 requests/second — one machine is orders of magnitude insufficient. The solution is horizontal scaling: N machines, each handling 1/N of the traffic. But horizontal scaling requires something to split the traffic: a load balancer.

The load balancer has its own resource constraints. A software load balancer (Nginx, HAProxy) running on a 4-core machine can forward roughly 1–5 million requests/second. A hardware load balancer (F5, Citrix NetScaler) can handle 10–100 million requests/second. The load balancer itself must not become the bottleneck — this is why large systems either run multiple load balancers in parallel or use DNS-based load balancing to distribute across multiple load balancer IPs.

Part 2: Where Load Balancing Happens — Layers of the Stack

Load balancing occurs at multiple layers of a system, solving different problems at each layer.

DNS load balancing (Layer 3/4 — before the packet arrives)

DNS resolves a hostname to one or more IP addresses. By returning different IPs for the same hostname to different clients, DNS distributes traffic before it reaches your infrastructure.

DNS load balancing is simple and inherently distributed — no single load balancer machine is in the critical path. It also enables geographic routing: a client in Europe gets a European IP, a client in Asia gets an Asian IP. AWS Route 53, Cloudflare, and GCP Cloud DNS all support latency-based routing and geo-routing.

The problems: DNS TTL means that when a server goes down, clients may continue sending requests to its IP for seconds to minutes (until the TTL expires and they re-resolve). DNS doesn't know about server health — it can't stop routing to a crashed server instantly. And clients and intermediate DNS resolvers cache DNS responses, so the actual distribution is less predictable than round-robin would suggest.

DNS load balancing is appropriate for: distributing across datacenters or regions (where TTL-based lag is acceptable), geographic routing, and as the outermost layer in a multi-tier load balancing architecture. It is not appropriate as the sole load balancer for fine-grained traffic distribution or fast failover.

L4 load balancing (Transport layer — TCP/UDP)

An L4 load balancer operates at the TCP level. It sees source IP, destination IP, source port, destination port — but not the HTTP content inside the TCP packets. It makes routing decisions based purely on connection-level information.

When a client connects, the L4 load balancer establishes a TCP connection with the client and a separate TCP connection with a backend server, proxying packets between them. Or in Direct Server Return (DSR) mode, it rewrites the destination IP in the packet and forwards it directly — the backend server responds directly to the client, bypassing the load balancer on the return path.

L4 load balancers are extremely fast because they process raw TCP packets without parsing application-layer content. They can handle millions of concurrent connections with minimal CPU. AWS Network Load Balancer (NLB) and Linux's LVS (Linux Virtual Server) are L4 load balancers. Use L4 when: you need maximum throughput, you're load balancing non-HTTP protocols (TCP-based databases, gRPC, custom protocols), or you need static IP addresses (NLB provides stable IPs, ALB doesn't).

L7 load balancing (Application layer — HTTP/HTTPS)

An L7 load balancer terminates the HTTP connection (and TLS, if HTTPS), parses the HTTP request, and makes routing decisions based on application-layer content: URL path, HTTP headers, cookies, query parameters, request body.

L7 load balancers enable sophisticated routing: canary deployments (route 5% of traffic to new version), A/B testing (route users to different variants by user ID), blue-green deployments (instant traffic switch between old and new versions), content-based routing (route /static/ requests to CDN, /api/ to application servers). AWS Application Load Balancer (ALB), Nginx, HAProxy in HTTP mode, and Envoy are L7 load balancers.

The cost: L7 load balancing requires parsing each request — more CPU than L4. TLS termination adds cryptographic overhead. For HTTPS at scale, this is substantial: terminating 1 million TLS connections/second requires significant CPU, which is why many systems offload TLS termination to dedicated hardware or to machines optimised for cryptographic operations.

Part 3: Load Balancing Algorithms — How the Decision Is Made

Given N healthy backend servers, which one gets the next request? The algorithm determines fairness, performance, and behaviour under uneven load.

Round robin

The simplest algorithm: cycle through servers in order. Request 1 → Server A. Request 2 → Server B. Request 3 → Server C. Request 4 → Server A. And so on.

Round robin distributes requests evenly by count — each server gets the same number of requests over time. It works well when all servers are identical and all requests take roughly the same time. It fails when requests have vastly different costs: if Server A gets all the 100ms database-heavy requests and Server B gets all the 1ms cache-hit requests, Server A is overwhelmed while Server B is idle — despite receiving the same request count.

Weighted round robin addresses heterogeneous server capacity: assign weights to servers based on their capacity. A server with weight 3 gets 3 requests for every 1 request the weight-1 server gets. Useful when your server fleet has mixed hardware generations.

Least connections

Route the next request to the server with the fewest active connections (in-flight requests). This naturally adapts to request duration: a server processing many slow requests accumulates connections and receives fewer new ones. Servers processing fast requests quickly drain their connection count and receive more.

Current state:
  Server A: 47 active connections
  Server B: 12 active connections
  Server C: 31 active connections

Next request → Server B (fewest connections)

Least connections is better than round robin when request duration varies significantly. It's the correct algorithm for: long-lived connections (WebSockets, HTTP/2 streams), mixed workloads (some requests are fast, some are slow), and any situation where response time is uneven.

Weighted least connections combines capacity weighting with connection count: score = connections / weight. Lower score wins. A server with twice the capacity and the same connections gets a lower score and receives the next request.

Random

Pick a random server for each request. Mathematically, at high volume, random produces near-uniform distribution (by the law of large numbers). Simpler to implement than round robin. No shared state needed (no connection counter to maintain across load balancer instances).

Power of two random choices (P2C): Pick two servers randomly, send the request to the one with fewer connections. This hybrid of random and least connections achieves nearly optimal load distribution with minimal coordination. At N=100 servers, P2C reduces the maximum load on any server from O(log N / log log N) for pure random to O(log log N) — a dramatic improvement in the worst case. P2C is used by Nginx Plus, Envoy, and many modern load balancers.

IP hash / Consistent hash

Hash the client's IP address (or some other request attribute) to determine the backend. The same client always goes to the same backend. This is session affinity by routing policy — no shared session state needed at the load balancer level.

Using consistent hashing (Chapter 8) instead of hash(IP) % N: adding or removing backends only remaps ~1/N of clients, rather than ~all clients. Used when: each backend has local state for a client (in-process cache, WebSocket connection, machine learning model loaded for a user's preferences) and rerouting would be expensive or disruptive.

The problem: IP-based hashing fails behind NAT (all users in an office share one public IP — they all go to the same backend) and for IPv6 (clients may have multiple addresses). Header-based hashing (hash on session cookie or user ID) is more reliable.

Least response time

Route to the server with the lowest combination of active connections and response latency: score = active_connections × mean_response_time. The load balancer measures how long each backend takes to respond (using probe requests or timing actual request-response cycles) and favours faster servers.

Least response time adapts to server slowness in real time: a backend that's having a GC pause, running hot on CPU, or struggling with a slow database query gets a higher score and receives fewer new requests automatically. This is the most sophisticated algorithm and is used by Envoy's LEAST_REQUEST policy with latency weighting.

Part 4: Health Checking — Knowing Which Servers Are Up

A load balancer routing to a crashed server is worse than no load balancer — it deterministically fails some fraction of requests. Health checking is the mechanism for detecting and removing unhealthy backends.

Passive health checking

The load balancer monitors the success/failure of actual requests it proxies. If a backend returns 5xx errors or times out above a threshold, it's marked unhealthy and removed from the pool. When it starts returning successful responses, it's re-added.

Passive health check:
  Backend A: last 10 requests → 0 errors → healthy ✓
  Backend B: last 10 requests → 8 timeouts → unhealthy ✗ (removed from pool)
  Backend C: last 10 requests → 0 errors → healthy ✓

Passive checking requires no extra traffic — it uses the signal from real requests. The downside: it takes some failed real requests before a backend is marked unhealthy. During the detection window, some clients experience errors. This is the lag inherent in passive detection.

Active health checking

The load balancer sends periodic probe requests to each backend (separate from real traffic) and monitors the responses. If a backend fails to respond to N consecutive probes, it's marked unhealthy. When it passes probes again, it's re-added (optionally with a gradual warm-up).

The /health endpoint should check not just that the server process is running, but that it can actually serve requests: it should verify database connectivity, cache availability, and any critical dependency. A shallow health check (just return 200) gives false confidence — the server process is up but the database connection pool is exhausted.

A good health endpoint checks:

• Can I connect to the database and run a trivial query?
• Can I connect to the cache?
• Is my error rate in the last 30 seconds below a threshold?
• Is my memory within acceptable bounds?

Health checks must be fast (< 100ms), lightweight (don't trigger expensive computation), and not interfere with the server's ability to serve real traffic.

The combination of both: use active health checks to detect failures quickly (before real traffic is affected) and passive health checks to catch degraded-but-not-failed backends (high error rate, elevated latency) that active probes might miss.

Algorithms and health checks only matter when something goes wrong. Slow one server down, then kill one, and watch what p99 and the error rate do.

Part 5: Session Affinity — When Stateful Backends Require Stickiness

Stateless applications (most modern REST APIs) can route any request to any backend — the server has no per-client state. Stateful applications (older session-based web apps, WebSocket connections, in-memory shopping carts) require that a client's requests always reach the same backend that holds their state.

Cookie-based affinity: The load balancer injects a cookie (e.g., SERVERID=backend-3) into the HTTP response. On subsequent requests, the client sends this cookie, and the load balancer reads it to route to backend-3.

The response header in full: Set-Cookie: SERVERID=backend-3; Path=/; HttpOnly.

Cookie-based affinity is transparent to the client (the cookie is just another cookie) and works behind NAT (unlike IP hashing). AWS ALB's "sticky sessions" use this mechanism.

IP hash affinity: As described in Part 3 — hash the client IP to a consistent backend. Simpler (no cookie management) but breaks behind NAT.

Session state in a shared store: The cleanest solution to the stickiness problem is eliminating the need for stickiness — store session state in a shared store (Redis, a database) rather than on individual backends. Any backend can serve any request because session state is fetched from Redis on each request. This is why modern architectures push session state to Redis: it enables true statelessness in the application tier and removes the operational burden of sticky sessions from the load balancer.

When you can't move to shared state (legacy applications, WebSocket connections that maintain in-memory state per connection), stickiness is necessary. But it creates an imbalance problem: if backend-3 receives more sticky connections than others (because one early batch of users was routed there), it handles more load than its peers. The fix: monitor per-backend request rates and use weighted load balancing to compensate.

Part 6: The Thundering Herd at the Load Balancer — Slow Start

A newly-started backend server added to the pool receives its full share of traffic immediately. But it might need time to warm up: JVM JIT compilation needs to run (Java startup penalty), connection pools to the database need to establish connections, in-process caches need to fill, thread pools need to initialise. A cold backend receiving full traffic immediately can spike response latency for the fraction of traffic it handles — worse than if it weren't in the pool at all.

Slow start (supported by Nginx, HAProxy, Envoy): when a new backend is added, ramp its weight from 0 to its target weight over a configured duration (e.g., 30 seconds). Traffic gradually increases from 0% to its fair share, giving the backend time to warm up without overloading it.

Without slow start:
  t=0: Backend D added to pool {A, B, C, D}
  t=0: Backend D immediately receives 25% of traffic
  t=0–30s: Backend D is cold, JIT not warmed, caches empty
  t=0–30s: 25% of requests are slow → noticeable p99 latency spike

With slow start (30s ramp):
  t=0:   Backend D weight = 0% (no traffic)
  t=10s: Backend D weight = 33% of target (receives ~8% of total traffic)
  t=20s: Backend D weight = 66% of target (receives ~17% of total traffic)
  t=30s: Backend D weight = 100% of target (receives ~25% of total traffic)
  p99 latency: unaffected

Slow start is especially important for: JVM applications (Spring Boot, Scala services), applications with large in-memory caches that need to fill, and any service with expensive initialisation.

Part 7: Connection Draining — Graceful Backend Removal

Removing a backend from the pool (for deployment, maintenance, or health failure) should not abruptly terminate in-flight requests. Connection draining (or "deregistration delay" in AWS parlance) is the mechanism for graceful removal.

When a backend is marked for removal, the load balancer:

1. Stops routing new requests to the backend (it's removed from the active pool).
2. Allows existing in-flight requests to complete (waits up to a configured timeout — AWS ALB default: 300 seconds).
3. After the drain timeout, forcibly closes any remaining connections.

Connection draining timeline:

t=0:    Backend A marked for removal (deployment starting)
t=0:    No new requests routed to Backend A
t=0:    Backend A has 47 in-flight requests
t=5s:   45 requests completed, 2 still in flight (long-running)
t=10s:  2 requests complete. Backend A has 0 in-flight.
t=10s:  Backend A connections closed. Deployment proceeds.

(Without draining: t=0 all 47 requests immediately get connection reset → 47 errors)

Connection draining is critical for zero-downtime deployments. Without it, every rolling deploy causes a burst of connection resets for clients whose requests were in-flight on the draining instance. With it, deployments are transparent to clients.

The drain timeout must be set appropriately: longer than your longest legitimate request, but not so long that a deployment is stuck waiting for a single stalled connection for 5 minutes. Typically 30–120 seconds for most web applications.

Part 8: Global Load Balancing — Across Datacenters

Single-datacenter load balancing is well-understood. Cross-datacenter (global) load balancing adds geographic constraints, different latency characteristics, and partial failure modes where an entire datacenter becomes unreachable.

Anycast routing

Anycast is a network technique where the same IP address is announced from multiple datacenters simultaneously. The internet's routing protocol (BGP) routes each packet to the geographically nearest datacenter announcing that IP. From the client's perspective, there is one IP. In reality, the request is automatically routed to the closest datacenter.

Cloudflare and Google Cloud use anycast for their global networks. When a Cloudflare datacenter goes down, BGP reconverges (within seconds to minutes) and traffic is rerouted to the next-nearest datacenter. This is how Cloudflare achieves global failover without DNS TTL lag.

GeoDNS

DNS returns different IP addresses based on the geographic location of the DNS resolver. Similar to anycast but at the DNS level — more predictable routing than BGP, but subject to DNS TTL caching and the lag of DNS propagation.

Client in Tokyo → resolver in Tokyo → GeoDNS → 192.168.1.1 (Asia datacenter)
Client in London → resolver in London → GeoDNS → 10.0.0.1 (EU datacenter)

GeoDNS works well for: routing users to the nearest datacenter for lowest latency, data sovereignty requirements (EU user data must stay in EU), and blue-green deployments at the global level (gradually shifting DNS to a new region).

Active-passive vs active-active

Active-passive: One datacenter handles all traffic (active). Another datacenter is on standby (passive). If the active datacenter fails, DNS or anycast switches traffic to the passive one. Simple to reason about, no cross-datacenter data synchronisation required for writes. The passive datacenter's idle capacity is wasted.

Active-active: Multiple datacenters each handle a portion of traffic simultaneously. Lower latency for global users (requests go to the nearest datacenter). Better capacity utilisation. Requires synchronising state (writes) across datacenters — a significant distributed systems challenge. Each datacenter must have current data; writes to one must propagate to others before they go stale. This is the geo-replication problem — Spanner and CockroachDB are designed specifically for active-active multi-region writes.

Most production systems use active-active for reads (replicas in each region serve reads locally) and active-passive for writes (the primary database is in one region, with failover capability to another). Full active-active for writes is operationally complex and only justified when write latency to a single primary region is unacceptable for global users.

Part 9: The Load Balancer as a Reverse Proxy — Beyond Traffic Distribution

Modern load balancers do far more than distribute traffic. The L7 load balancer sits in the critical path of every request, making it a natural place for cross-cutting concerns.

TLS termination: The load balancer decrypts HTTPS, forwards plain HTTP to backends, and re-encrypts responses. Backends don't need TLS certificates or cryptographic overhead. Certificate management is centralised at the load balancer. AWS ACM (Certificate Manager) integrates with ALB to automate certificate renewal.

Request/response transformation: Rewrite URL paths (/v1/users → /users), add headers (X-Forwarded-For to pass real client IP to backends), strip headers (Authorization before forwarding to untrusted downstream services), decompress/recompress request bodies.

Rate limiting: Reject requests from clients that exceed a threshold. The load balancer is the ideal place for coarse-grained rate limiting (Chapter 17 covers this in depth).

Circuit breaking: If a backend's error rate exceeds a threshold, temporarily stop sending it traffic — give it time to recover before trying again. This is the circuit breaker pattern (from Chapter 4's failure taxonomy) implemented at the load balancer level. Envoy has built-in circuit breaking; Nginx and HAProxy implement it via passive health checking + active marking.

Observability: Log every request (method, path, status code, latency, backend IP). Emit metrics (requests/second, p50/p95/p99 latency, error rate by backend). The load balancer's access log is often the first place to look during an incident — it shows the full picture of what every client sent and what every backend returned.

A/B testing and canary deployments: Route a configurable percentage of traffic to a new version. The load balancer reads a request attribute (user ID hash, specific header, cookie) and routes based on it. Route 1% of traffic to the new version, observe metrics, gradually increase to 5%, 10%, 100%.

Part 10: Service Mesh — Load Balancing Inside the Cluster

So far, load balancing has been about external traffic entering your system. Inside a microservices cluster, services call each other directly — Service A calls Service B, Service B calls Service C. This internal traffic also needs load balancing, health checking, circuit breaking, and observability.

A service mesh puts a sidecar proxy (typically Envoy) next to every service instance. All inter-service traffic flows through these sidecars rather than directly between services. The sidecar handles: load balancing of outgoing calls (to multiple instances of the downstream service), health checking, circuit breaking, retry logic, distributed tracing, mutual TLS (mTLS) for service-to-service encryption, and traffic splitting for canary deployments.

Both sidecars report metrics to the control plane (Istio Pilot/Discovery). The control plane pushes routing rules to all sidecars.

The service mesh control plane (Istio's Pilot, Consul Connect, Linkerd's control plane) maintains the registry of all service instances and pushes routing configuration to all sidecar proxies. When Service B adds a new instance, the control plane pushes the updated endpoint list to all Service A sidecars within seconds.

The cost: every request now involves two extra proxy hops (one sidecar on each end) — adding ~0.5–2ms of latency per call. For services making thousands of inter-service calls per user request, this overhead compounds. The service mesh is appropriate when: the operational benefits (unified observability, mTLS, consistent retry policies) outweigh the latency cost and operational complexity of running Envoy sidecars everywhere.

Kubernetes-native load balancing (Services with ClusterIP and kube-proxy) provides basic L4 load balancing via iptables rules without a service mesh. For many teams, Kubernetes Services are sufficient — the service mesh becomes justified when you need L7 features (path-based routing, traffic splitting), mTLS, or per-call observability across service boundaries.

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

Your system uses an AWS ALB with 5 backend instances. Active health checks run every 10 seconds; a backend is marked unhealthy after 2 consecutive failed checks. Deregistration delay is 60 seconds. The load balancing algorithm is round robin.

Backend C crashes completely at t=0. Trace the timeline: at t=0, what percentage of requests fail? At what time is Backend C first detected as unhealthy (assuming it fails all active probes)? Between t=0 and detection, how many total requests are sent to Backend C (assuming 1,000 requests/second total)? After detection, how does round robin redistribute — what fraction of traffic does each remaining backend handle?

Now Backend D, receiving its increased share, has a latency spike (not a crash — it responds but slowly, taking 8 seconds instead of 0.1 seconds). With round robin, what happens to Backend D's connection count over 30 seconds? Why would least-connections have handled this better? What specific metric would you alert on to detect Backend D's degradation before it causes widespread impact?

Problem 2 — Design from Scratch

Design the complete load balancing architecture for a global streaming video platform (like Netflix or Disney+). Users are in North America, Europe, and Asia-Pacific. The system serves: video manifest files (small, frequent, latency-sensitive), video chunks (large, throughput-sensitive, can be cached), API calls (user authentication, watchlist, recommendations — latency-sensitive, stateful sessions), and WebSocket connections (real-time notifications — long-lived, stateful).

For each traffic type: which load balancing layer (DNS, L4, L7), which algorithm, whether session affinity is needed, what health check strategy, and whether a CDN offloads the load balancer. How do you achieve failover when an entire US datacenter goes offline? What is the user-visible impact during the failover period and why?

Problem 3 — Synthesis

Connect load balancing to Chapter 8's consistent hashing and Chapter 9's caching.

Your API has a computationally expensive endpoint: GET /recommendations/{user_id}. Computing recommendations takes 200ms (ML model inference). You add an in-process LRU cache on each application server (Chapter 9): if the same user's recommendations were computed in the last 5 minutes, serve from cache (0.5ms). You have 10 application servers.

With round robin load balancing: what is the cache hit rate for user_id=42 across 10 servers? (Assume the user makes 10 requests in 5 minutes, uniformly distributed.) What is the expected number of cache misses and thus expensive ML computations? Now switch to consistent hash load balancing on user_id: what is the cache hit rate for user_id=42 now? What is the trade-off — what problem does consistent hashing introduce for load distribution if user request rates are not uniform? (Hint: what happens to the server that "owns" a celebrity user with 1 million requests/minute?)

Design a hybrid approach that gets high cache hit rates without creating hot servers. (Hint: Chapter 9's tiered caching and Chapter 8's virtual nodes are both relevant.)

Problem 4 — Go Deeper

The Power of Two Choices (P2C) algorithm picks 2 random backends and routes to the one with fewer connections. It achieves O(log log N) maximum load on any server vs O(log N / log log N) for pure random.

From first principles: why does picking two and choosing the minimum produce dramatically better balance than picking one randomly? Intuitively, what does the second pick add that the first pick alone doesn't? Now extend this reasoning: what is the optimal number of choices? Would "power of three choices" (pick 3, route to the minimum) be meaningfully better than power of two? At what point does increasing the number of choices yield diminishing returns?

Then: P2C requires knowing the current connection count of two servers before routing. In a single load balancer, this is a local memory read. In a distributed load balancer cluster (multiple LB instances sharing traffic), how does P2C work? Does each LB instance need a consistent global view of all backend connection counts? What is the trade-off between accuracy (global coordination for perfect counts) and performance (local estimates with no coordination)? How do systems like Envoy handle this in practice?

Problem 5 — Connect to Previous Chapters

Connect load balancing failure behaviour to Chapter 4's failure taxonomy and Chapter 6's consistency models.

A service has 3 backend instances behind a load balancer. The load balancer uses passive health checking (marks unhealthy after 5 consecutive errors) and least-connections routing. A network partition splits the system: the load balancer can reach Backend A and Backend B, but not Backend C. Backend C is still running and serving the small number of direct clients that bypass the load balancer.

Part A: From Chapter 4's failure taxonomy, what type of failure is this for Backend C from the load balancer's perspective? What type is it from Backend C's perspective? What type is it for the clients that directly reach Backend C?

Part B: Backend C is a read replica. It continues accepting reads from direct clients during the partition. The primary database (which Backend A and B can reach) accepts writes. Using Chapter 6's consistency vocabulary: what consistency model are direct Backend C clients experiencing? Trace a specific violation: a user updates their profile (write goes to primary via Backend A), then immediately reads their profile (read served by Backend C which hasn't received the replication update yet). What is the staleness window? How does this connect to Chapter 5's replication lag?

Part C: When the partition heals and Backend C reconnects, the load balancer's passive health check has not marked it unhealthy (it never received errors — it was unreachable, not failing). What happens to connection distribution immediately after reconnection? Should Backend C immediately receive its full share of traffic? What mechanism would you use to safely reintegrate it?

What's Coming Next

You now understand how traffic reaches your services: DNS distributes across regions, L4 balancers distribute TCP connections, L7 balancers route by content, and service meshes manage internal service-to-service traffic. Chapter 17 is the complement: rate limiting — protecting your services from receiving more traffic than they can handle. While load balancing distributes traffic fairly across available capacity, rate limiting enforces a ceiling on how much traffic any individual client or tenant can send. The two algorithms at the heart of rate limiting — token bucket and leaky bucket — are elegant mechanisms that appear in network switches, API gateways, and database connection pools. They're also asked in almost every distributed systems interview.

Chapter 16 complete. Problem 3 (consistent hash caching + hot server problem) is the most intellectually rich — it shows why caching and load balancing interact in non-obvious ways, and the hybrid solution requires synthesising two chapters. Problem 1 (timeline trace with specific numbers) grounds the abstract health check configuration in real operational consequences. Work through both before Chapter 17. The specific numbers in Problem 1 — "how many requests fail before detection?" — is exactly what you compute during an incident postmortem.