Phase 4 · Chapter 6

Load Balancing: Algorithms, Health Routing & Zero-Downtime Deployments

What actually happens between the internet and your application servers

Every request to your product passes through a load balancer before it reaches your code. Most engineers treat it as a black box — traffic goes in, traffic comes out, somehow spread across servers. After this chapter it will not be a black box. You will understand the algorithms that decide which server handles which request, what happens at the TCP and HTTP layers when a server goes down, why naive round-robin fails for stateful applications, how health checks work and what they miss, and how to deploy new code across a fleet of servers without dropping a single request. These are the mechanics that determine whether your system is highly available or fragile — and they are entirely under your control.

Part 1: What a Load Balancer Actually Is

Before the algorithms, the physical reality.

A load balancer is a process that accepts incoming connections on a known address (your domain's IP) and proxies them to one of several backend servers. It sits between the internet and your application servers. Its job sounds simple: forward traffic. Its actual job is more precise: forward traffic intelligently, detect failed backends, and ensure no single backend becomes a bottleneck.

Load balancers operate at different layers of the network stack, and this distinction matters for what they can and cannot do.

Layer 4 load balancing (TCP/UDP): The load balancer sees TCP connections — source IP, source port, destination IP, destination port. It does not see HTTP headers, URL paths, or cookies. It forwards entire TCP connections to backends. Fast because it does not need to parse HTTP. Limited because it cannot make routing decisions based on HTTP content.

Layer 7 load balancing (HTTP/HTTPS): The load balancer terminates the client's TLS connection, reads the HTTP request fully, and makes routing decisions based on URL path, HTTP headers, cookies, or request body content. Then it opens a new connection (or reuses a pooled one) to the chosen backend and forwards the request. Slower than L4 (it must parse HTTP) but capable of content-based routing: send /api/* to API servers, /static/* to a CDN, send requests with a specific X-User-Plan: enterprise header to dedicated enterprise servers.

For most web applications, you want L7 load balancing. Nginx, HAProxy, Caddy, and all cloud load balancers (AWS ALB, GCP HTTPS Load Balancer) operate at L7. The extra parsing overhead is negligible compared to the flexibility it provides.

Part 2: Load Balancing Algorithms

Given a pool of healthy backends, which one should handle the next request? There are six algorithms in common use. Each optimises for different workload characteristics.

Round robin

Distribute requests sequentially across backends in rotation.

Request 1  → Server A
Request 2  → Server B
Request 3  → Server C
Request 4  → Server A  (wrap around)
Request 5  → Server B
...

Implementation: maintain a counter, increment on each request, take modulo by number of servers.

Round robin assumes all requests have equal cost and all servers have equal capacity. When both assumptions hold, it distributes load perfectly evenly. When they do not hold, it distributes requests evenly but not load evenly — a server handling ten expensive requests has more load than a server handling ten cheap ones.

Round robin is the correct default for stateless services where requests have similar cost.

Weighted round robin: Assign each server a weight proportional to its capacity. A server with weight 3 receives three times as many requests as a server with weight 1. Use this when your backend servers have different hardware specs or when you are gradually shifting traffic to a new server during a canary deployment.

Least connections

Route each new request to the backend with the fewest active connections.

Server A: 45 active connections
Server B: 23 active connections  ← next request goes here
Server C: 31 active connections

Least connections adapts to varying request costs automatically. A server processing many slow requests accumulates connections and receives fewer new ones. Servers that finish requests quickly receive more traffic. Load is distributed by actual capacity utilisation, not by request count.

Least connections is the correct algorithm for workloads where requests have highly variable duration — long-running API calls, streaming responses, WebSocket connections, file uploads. For your Samiksha evaluation endpoint (which calls Gemini and can take 3-8 seconds), least connections is strictly better than round robin.

Least response time: A variant that considers both active connection count and average response time. Routes to the server with the best combination of low connections and low latency. More accurate than least connections but requires the load balancer to track response times, which adds overhead.

IP hash (sticky sessions by IP)

Hash the client's IP address to determine the backend. The same client IP always routes to the same backend.

function getBackend(clientIp, backends) {
  const hash = murmurhash(clientIp);
  return backends[hash % backends.length];
}

This is the naive solution to the stateful session problem: if a user's session state is stored in memory on a specific server, they must be routed to that server on every request. IP hash provides this guarantee without requiring coordination between load balancers.

The problems: clients behind NAT share one IP (an entire office building might hash to the same backend, overloading it). Mobile users change IPs frequently (each IP change sends them to a different backend, losing their session). Backend failures cause all clients assigned to that backend to be remapped.

IP hash is a workaround for poor session design. The better solution is externalised session storage (Redis with a session middleware) — then any backend can serve any client because session state is in Redis, not in server memory. Once you externalise sessions, you can use any stateless algorithm.

Random

Select a backend at random. Statistically, at high request volumes, random selection converges to equal distribution (law of large numbers). Requires no state on the load balancer — no counter, no connection tracking.

Random is surprisingly effective for high-throughput stateless services. At 50,000 req/s across 10 backends, random selection distributes each server within ±0.5% of the mean within seconds. Netflix uses a variant of random selection ("power of two random choices") in their load balancing at extreme scale.

Power of two random choices: Pick two backends at random, send the request to the less loaded one. This eliminates the worst case of pure random (occasionally sending multiple consecutive requests to the same overloaded server) while retaining most of the simplicity. Proven to achieve near-optimal load distribution with only two comparisons per request.

Consistent hashing (for caches and shards)

When your backend servers are themselves caches (Redis instances, Memcached nodes) or shards, you need requests for the same data to go to the same backend — for cache locality. Round robin and random fail here: if a request for user:123's data can go to any of eight backends, each backend needs to store all user data, or every request misses the cache.

Consistent hashing maps each key (URL, user ID, cache key) to a backend using a hash ring (Phase 3 Chapter 2). The same key always maps to the same backend unless a backend is added or removed, in which case only 1/N of keys remap. This preserves cache locality while handling backend changes gracefully.

Layer 7 load balancers can implement consistent hashing on any HTTP header, URL parameter, or cookie value. Nginx: upstream { consistent_hash $request_uri; }.

Part 3: Health Checks — How the Load Balancer Knows a Server Is Dead

A backend can fail in many ways:

• Process crashes entirely (port unreachable)
• Process is running but hanging (requests queue up, never respond)
• Process is running and responding but returning errors (application is broken)
• Process is healthy but its dependencies are broken (database is down, it cannot serve correctly)

A naive load balancer detects only the first failure — it notices a connection refused and stops routing to that backend. The other three failures are invisible until requests start failing.

Health checks are how the load balancer continuously probes backends to detect failures and remove unhealthy backends before clients experience errors.

Passive health checks

Monitor the outcomes of real requests. If a backend returns enough errors (5xx responses, TCP connection failures, timeout) within a time window, mark it as unhealthy and stop routing to it.

upstream api_servers {
  server api1:3000;
  server api2:3000;
  server api3:3000;

  # Remove a backend after 3 failures in 30 seconds
  # Try to restore it after 10 seconds
}

Passive health checks are zero overhead — they use real traffic to detect failures. The downside: some real requests fail before the backend is marked unhealthy. If a backend fails on request 1 and the policy is "remove after 3 failures," requests 1, 2, and 3 fail before the backend is removed.

Active health checks

The load balancer proactively sends synthetic requests to each backend on a schedule. If the synthetic request fails or times out, the backend is marked unhealthy. No real client requests are needed to detect failures.

upstream api_servers {
  server api1:3000;
  server api2:3000;
  server api3:3000;
}

server {
  location /health {
    proxy_pass http://api_servers;
    health_check interval=5s fails=3 passes=2 uri=/health;
  }
}

The load balancer sends GET /health to each backend every 5 seconds. After 3 consecutive failures, the backend is removed. After 2 consecutive successes, it is restored.

Active health checks require implementing a /health endpoint in your application. This endpoint must do more than return 200 OK — a trivial health endpoint that always returns 200 is useless. A good health check verifies the application's critical dependencies:

app.get('/health', async (req, res) => {
  const checks = {};
  let healthy = true;

  // Check database connectivity
  try {
    await prisma.$queryRaw`SELECT 1`;
    checks.database = 'ok';
  } catch (err) {
    checks.database = 'error';
    healthy = false;
  }

  // Check Redis connectivity
  try {
    await redis.ping();
    checks.redis = 'ok';
  } catch (err) {
    checks.redis = 'error';
    healthy = false;
  }

  // Check queue depth (warn if too backed up)
  const queueDepth = await evaluationQueue.getWaiting();
  checks.queueDepth = queueDepth;
  if (queueDepth > 10000) healthy = false;

  res.status(healthy ? 200 : 503).json({
    status: healthy ? 'healthy' : 'unhealthy',
    checks,
    timestamp: new Date().toISOString(),
  });
});

When this backend's database connection fails, the health check returns 503. The load balancer stops routing traffic to it within 15 seconds (3 failed checks × 5 second interval). Other backends continue serving. No client requests fail due to a backend with a broken database connection.

Health check failure modes to know

Thundering herd on recovery: A backend goes unhealthy and is removed from rotation. The other backends absorb its traffic. After recovery, the backend is re-added. The load balancer now has an extra backend with no active connections — and routes a burst of new requests to it simultaneously. If the recovery was due to a cascading failure (e.g., the backend was overwhelmed), this burst re-overwhelms it immediately.

Fix: implement slow start — when a backend is restored, add it gradually. Nginx Plus supports slow_start=30s which linearly increases the backend's weight from 0 to full over 30 seconds. HAProxy supports similar functionality. This prevents re-overwhelming a recovering backend.

Health check passes, application fails: The health check endpoint returns 200 but application endpoints are returning 500. This happens when the health check is too shallow — it checks that the process is alive but not that it is functioning. Fix: make health checks verify real application dependencies as shown above.

Try it: one slow backend under round robin, the same under least connections, then a crash with health checks. Each tab asks you to predict first.

Part 4: Connection Pooling at the Load Balancer Layer

Between the load balancer and your backend servers, there are two connection models.

Per-request connections: For each incoming request, open a new TCP connection to a backend, forward the request, wait for the response, close the connection. Simple. Correct. Extremely slow under load because TCP connection establishment (SYN, SYN-ACK, ACK) takes one round trip, and TLS handshake takes another. At 10,000 req/s, you are opening and closing 10,000 TCP connections per second. Connection overhead dominates your latency.

Connection pooling (keepalive): The load balancer maintains a pool of persistent connections to each backend. Incoming requests are multiplexed over these persistent connections. No new TCP handshake per request. The load balancer reuses an existing connection or opens a new one if the pool is exhausted.

upstream api_servers {
  server api1:3000;
  server api2:3000;
  keepalive 32;   # maintain up to 32 idle connections per worker to each backend
}

server {
  location / {
    proxy_http_version 1.1;
    proxy_set_header Connection "";  # clear Connection header to allow keepalive
    proxy_pass http://api_servers;
  }
}

keepalive 32 tells Nginx to keep up to 32 idle connections open to each backend per worker process. New requests immediately reuse existing connections rather than paying TCP handshake overhead. At 10,000 req/s this eliminates millions of SYN/SYN-ACK/ACK packets per minute.

This is why proxy_http_version 1.1 matters: HTTP/1.0 closes connections after each response by default. HTTP/1.1 keeps them alive unless explicitly closed. Without this setting, Nginx sends HTTP/1.0 to backends and keepalive does nothing.

Part 5: Zero-Downtime Deployments

You have 3 API servers behind a load balancer. You need to deploy new code. The wrong way: stop all three, deploy, restart all three. During the deploy window (even if it is 30 seconds), all requests fail. Your users see errors.

The correct approach uses the load balancer's ability to dynamically remove and add backends.

Rolling deployment

Deploy to one server at a time. During deployment, traffic shifts to the remaining servers.

Zero requests fail. The total deployment time is 3 × (deploy time + health check time). The tradeoff: for a brief window, some servers run v1 and some run v2. Your API must be backward compatible with itself during this window — specifically, any database schema changes must be backward compatible (old code must be able to read tables that new code writes to).

This is why the database migration rule exists: never deploy a schema change and the code that requires it in the same deployment step. First deploy the schema change (backward compatible: add a column, but old code doesn't use it yet). Then deploy the new code. Then optionally clean up old columns in a third deploy.

Blue-green deployment

Run two identical environments: Blue (current v1) and Green (new v2). The load balancer points at Blue. When ready to deploy:

Blue-green gives you instant rollback — switching the load balancer back to Blue takes seconds. The tradeoff: you need double the infrastructure (two full environments running simultaneously), and any stateful data written to Green during the switch cannot be rolled back by reverting the load balancer (writes are in the database, not in the server).

Canary deployment

Route a small percentage of traffic to the new version, verify it is healthy, then gradually increase.

Initial:      100% → v1
After step 1:   5% → v2,  95% → v1  (watch metrics for 15 minutes)
After step 2:  25% → v2,  75% → v1  (watch metrics for 30 minutes)
After step 3:  50% → v2,  50% → v1
After step 4: 100% → v2

Canary deployment limits the blast radius of a bad deployment to the percentage of traffic on the canary. If v2 has a bug that affects 1% of requests, and you are routing 5% of traffic to v2, only 0.05% of total requests fail during the canary phase. You catch the bug before it affects all users.

Implementing canary with Nginx using weighted round robin:

upstream api_servers {
  server api-v1:3000 weight=95;
  server api-v2:3000 weight=5;
}

Adjusting the weights shifts traffic between versions. This is the deployment strategy for any change that could affect user experience — new algorithm, new UI endpoint, new database query.

Part 6: SSL Termination and the Load Balancer's Security Role

HTTPS requires TLS. TLS handshakes are computationally expensive — generating session keys, verifying certificates. If every backend server handles TLS, you multiply this cost by the number of backends. If the load balancer handles TLS, it terminates the encrypted connection from the client, then forwards plain HTTP to backends over your internal network.

Client ──(HTTPS/TLS)──► Load Balancer ──(HTTP)──► Backend Server

This is called SSL termination and it concentrates certificate management in one place. You store your TLS certificate and private key only on the load balancer. Backends communicate over unencrypted HTTP on your internal network. This is acceptable because:

1. Internal traffic never leaves your infrastructure
2. Your VPC/private network is not accessible to the internet
3. Centralised certificate management is operationally simpler (renew one cert, not N certs)

If regulatory requirements mandate encryption in transit even on internal networks (certain fintech compliance frameworks), use SSL passthrough (L4 load balancing) or terminate TLS at the load balancer and re-encrypt to backends. This doubles TLS overhead.

The load balancer also handles TLS certificate auto-renewal. Caddy and Nginx with Certbot can automatically renew Let's Encrypt certificates and reload without downtime. This is one of the non-obvious operational tasks the load balancer absorbs for you.

Part 7: Nginx as a Production Load Balancer

Nginx is the standard load balancer for small-to-medium production systems (including your Hetzner VPS setup). A complete production Nginx configuration for a Node.js API with three backends:

upstream api_backend {
  least_conn;                     # use least connections algorithm

  server 10.0.0.1:3000;
  server 10.0.0.2:3000;
  server 10.0.0.3:3000;

  keepalive 32;                   # connection pool
}

server {
  listen 80;
  server_name api.samiksha.app;
  return 301 https://$server_name$request_uri;  # redirect HTTP to HTTPS
}

server {
  listen 443 ssl;
  server_name api.samiksha.app;

  ssl_certificate     /etc/letsencrypt/live/api.samiksha.app/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/api.samiksha.app/privkey.pem;
  ssl_protocols       TLSv1.2 TLSv1.3;

  # Real IP passthrough (so your app logs client IPs, not load balancer IPs)
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header Host $host;

  # Keepalive to backends
  proxy_http_version 1.1;
  proxy_set_header Connection "";

  # Timeouts
  proxy_connect_timeout 5s;    # give up connecting to backend after 5s
  proxy_send_timeout 60s;      # wait up to 60s for backend to accept data
  proxy_read_timeout 60s;      # wait up to 60s for backend response

  location / {
    proxy_pass http://api_backend;
  }

  location /health {
    access_log off;             # don't log health check requests
    proxy_pass http://api_backend;
  }
}

proxy_set_header X-Real-IP $remote_addr is critical: without it, your application sees the load balancer's IP (10.0.0.x) as the client IP for every request. Rate limiting by IP, analytics, and audit logs all become useless. Always pass the real client IP through.

Vocabulary Map

Practice Problems

Problem 1 — Algorithm Selection: Samiksha has two types of API endpoints with very different characteristics:

• GET /api/questions — returns a list of questions, typically 5-20ms, high volume
• POST /api/evaluate — calls Gemini, writes to Postgres, typically 3-8 seconds, lower volume

Both endpoint types go to the same pool of 4 API servers. You are choosing a load balancing algorithm. Which algorithm is correct for this mixed workload and why? What specific failure mode would occur with round robin that your chosen algorithm prevents? Show with concrete numbers what happens to server connection counts under round robin when 100 concurrent evaluate calls are in-flight alongside 2,000 questions requests.

Problem 2 — Health Check Design: AttemptIQ's API server depends on: Postgres (primary + replica), Redis, and BullMQ. Design a health check endpoint that:

1. Distinguishes between "unhealthy, remove from load balancer" and "degraded, but keep routing"
2. Does not add more than 5ms to health check response time
3. Does not itself cause load (e.g., does not run an expensive query)

Write the complete /health endpoint implementation. Define the specific conditions under which it returns 200 (healthy), 207 (degraded), and 503 (unhealthy). Explain why each threshold exists.

Problem 3 — Zero-Downtime Deployment: You are deploying a change to Samiksha that includes both a database migration and application code changes. The migration adds a new column evaluation_version to the evaluations table and the new code writes to this column. You have 3 API servers behind Nginx.

Design the complete deployment sequence. How many separate deployment steps are required? What is deployed in each step? Why can you not deploy the migration and code change together? What happens to requests during each step? Write the Nginx upstream configuration change for each step.

Problem 4 — Go Deeper: A load balancer using least connections routes a new request to the backend with fewest active connections. Consider a backend that is not crashed but is severely degraded — it is accepting connections, not closing them, but taking 30 seconds to respond (instead of the normal 200ms).

What happens to this backend's connection count over time? What does least connections do with this backend? Is this the correct behaviour? What additional mechanism — not covered above — would detect and handle this specific failure mode? What Nginx directive controls it?

Problem 5 — Connect to Previous Chapters: In Chapter 5 (Observability), you learned about distributed tracing and the traceparent header. In this chapter you learned that the load balancer sets headers like X-Real-IP and X-Forwarded-For.

A request arrives at your Nginx load balancer. It already has a traceparent header from a frontend instrumentation SDK. Nginx forwards the request to an API server. The API server calls a BullMQ worker. The worker calls Postgres.

Trace the complete header propagation: which headers must Nginx preserve and which must it set? What happens if Nginx strips the traceparent header? What does the distributed trace look like in Jaeger in that case? Write the Nginx proxy_set_header directives that correctly handle both observability and real-IP concerns simultaneously.

What's Next

You have now completed Phase 4. You understand the full production infrastructure layer: caching strategies, rate limiting algorithms, authentication internals, queue architectures, how to see inside a running system, and how traffic enters and distributes across your application servers.

Phase 5 applies everything you have learned to complete systems — end-to-end designs for real products that combine all the components. The case studies will feel different now than they would have felt if you had read them before Phase 2 and Phase 4. URL shorteners will expose questions of cache invalidation and consistent hashing. Twitter feed will expose questions of fan-out, denormalisation, and read vs write tradeoffs. Google Docs will expose questions of real-time consistency and conflict resolution that connect back to the distributed consensus you studied in Phase 3. Every case study is a synthesis exercise. The goal is not to memorise the designs — it is to understand why each component was chosen and what breaks if you change it.

End of Phase 4, Chapter 6.