Q35 — How does replication work — and what is replication lag?

Why does replication exist?

A single database server is a single point of failure. If it crashes, goes offline for maintenance, or its disk fails, the entire application loses its data store. For any system where availability matters, this is unacceptable.

Replication solves this by maintaining multiple copies of the same data on multiple machines. If one fails, another takes over. This is the fault tolerance reason.

The second reason is read scaling. A single database can only process so many reads per second. If your application is read-heavy (most web applications are — reads outnumber writes by 10:1 or 100:1), you can distribute read traffic across multiple replicas. Each replica can serve reads independently, multiplying your read throughput linearly with the number of replicas.

These two motivations — fault tolerance and read scaling — are the foundation of nearly every production database deployment.

How does single-leader replication work?

The most common replication model: one server is designated the primary (also called leader or master). All write operations go to the primary. The primary records every write in its WAL (Write-Ahead Log — covered in Q13) and streams those changes to secondary servers (also called replicas, followers, or standbys).

Each secondary receives the WAL stream and applies the changes in order, maintaining an identical copy of the data. Read queries can be directed to any secondary, offloading the primary.

This is the default for PostgreSQL (streaming replication), MySQL (binlog replication), and most managed cloud database services.

The primary handles all writes because having a single authority for writes prevents write conflicts — two servers accepting conflicting concurrent writes for the same row. The primary serializes all writes, and replicas apply them in the same order.

What is synchronous vs asynchronous replication?

The crucial question: does the primary wait for replicas to confirm they've applied a write before acknowledging success to the client?

Synchronous replication: The primary sends the write to the replica and waits for the replica to confirm durability before returning success to the client. The client gets a "success" only when the data is on at least two machines.

Benefit: zero data loss on primary failure. If the primary dies immediately after acknowledging a write, the replica already has that write. Promoting the replica loses nothing.

Cost: write latency is now the max of primary write time and replica write time plus network round trip. If the replica is in a different datacenter (100ms away), every write adds 100ms of latency. If the replica is slow or unreachable, writes stall. Synchronous replication to a single replica means that replica's availability determines your write availability.

Asynchronous replication: The primary writes locally, acknowledges success to the client immediately, and sends the change to replicas in the background. The client gets "success" before the replica has the data.

Benefit: write latency is just the primary's local write time. No dependence on replica availability.

Cost: if the primary crashes before the replica received the last write, that write is lost. The replica is promoted, but it's missing some committed transactions. This is data loss on failover — acknowledged writes vanish. For financial systems, this is unacceptable. For social media likes, it's a fine trade-off.

Most systems use semi-synchronous replication as a compromise: wait for at least one replica to confirm, but not all replicas. You tolerate the loss of some replicas while protecting against total loss.

What is replication lag?

In asynchronous replication, the replica is always slightly behind the primary. This gap — measured in bytes, operations, or time — is replication lag.

Under normal conditions, lag is milliseconds: the primary writes, streams the WAL entry, the replica applies it almost instantly. But under high write load, a slow network, a replica under resource pressure, or a large transaction being applied, the lag grows. Lag can reach seconds, minutes, or in pathological cases, hours.

Replication lag is not just a metric to watch — it causes correctness problems for applications that read from replicas.

Why does replication lag cause application bugs?

The read-your-own-writes problem: A user updates their profile photo. The write goes to the primary. The user's next page load reads from a replica. The replica hasn't applied the write yet — the user sees their old photo. From the user's perspective, their update didn't work. They try again. Now there are two updates in flight.

This is a violation of the read-your-own-writes consistency guarantee — a user should always see their own writes, even if other users might see stale data.

Fix: route read requests to the primary for a brief window after a write (say, 1 second). Or, for writes by a specific user, always read their own data from the primary, while reading other users' data from replicas.

The monotonic reads problem: A user refreshes a page twice in quick succession. The first request goes to Replica A (lag: 0.1s — shows the new comment). The second request goes to Replica B (lag: 5s — doesn't show the new comment). The user sees the comment appear and then disappear. Time appears to move backward.

Fix: sticky routing — route a given user's reads always to the same replica for the duration of a session. Use a hash of the user ID to select the replica consistently.

The causality problem: User A posts "What time is it?" User B sees the post and replies "It's 3pm." User C reads from a lagging replica, sees User B's reply without having seen User A's question. The reply appears without context. Causally related operations are seen out of order.

Fix: causal consistency — track which operations depend on which, and ensure replicas apply causally related operations in order.

What is failover and what can go wrong?

When the primary fails, one of the replicas must be promoted to become the new primary. This process is failover.

Automated failover: a monitoring process (like Patroni for PostgreSQL, Orchestrator for MySQL, or the managed service's control plane) detects the primary is down and promotes the replica with the least lag.

Split-brain: What if the primary didn't actually die — it was just slow to respond (network partition, overloaded, long GC pause)? The monitoring system promotes a replica. Now there are two servers that both believe they are the primary. Both accept writes. The data diverges. This is split-brain — one of the most dangerous failure modes in distributed databases.

Fix: fencing — when a new primary is promoted, actively prevent the old primary from accepting writes. Mechanisms include STONITH ("Shoot The Other Node In The Head" — literally cutting power or network access to the old primary), leases (the old primary's write lease expires and it refuses new writes), or consensus protocols that ensure only one node at a time holds the primary lease.

Data loss on failover: The promoted replica has lag — it's missing the last N writes the primary accepted. Those writes are gone. The old primary dies with them. This is unavoidable with asynchronous replication unless you synchronously replicate at least one replica and always promote that one.

Multi-leader and leaderless replication

Multi-leader replication: Multiple nodes accept writes simultaneously. Each acts as a primary for some writes and replicates to others. Used in multi-datacenter deployments where cross-datacenter write latency to a single primary is prohibitive.

The fundamental problem: two leaders can accept conflicting concurrent writes to the same row. Conflict resolution is required — last-write-wins (by timestamp), application-defined merge, or CRDTs (Conflict-free Replicated Data Types). Conflict resolution is hard to get right and application-specific.

Leaderless replication (Dynamo-style): Any replica can accept any write. Reads and writes use quorums: W + R > N (where N is total replicas, W is replicas that must acknowledge a write, R is replicas that must respond to a read). If W=2, R=2, N=3: any write must reach 2 of 3 replicas, any read must consult 2 of 3. At least one replica in the read quorum will have the latest write, guaranteeing you read the most recent value.

This is how Cassandra and DynamoDB work. It provides high availability (writes succeed if W of N are reachable) and tunable consistency (raise W and R for stronger consistency, lower them for higher availability).

The mental model

Why replicate: fault tolerance (one copy dies → others take over) + read scaling

Single-leader:
  Primary: all writes
  Secondaries: apply WAL from primary → serve reads
  
  Sync replication: primary waits for replica ACK before client ACK
    Pro: zero data loss on failover
    Con: write latency = max(primary, replica) + network

  Async replication: primary ACKs immediately, replicates in background
    Pro: low write latency
    Con: data loss window on failover

Replication lag: replica is behind primary by time T
  Causes:
    Read-your-own-writes: write to primary, read from lagging replica → old data
    Monotonic reads: two reads from two replicas at different lag → time goes backward
    Causality: reply visible before the question it's replying to

  Fixes: route own-user reads to primary, sticky replica routing, causal tracking

Failover risks:
  Split-brain: two nodes think they're primary → data diverges
  Fix: fencing (STONITH, leases, consensus)
  Data loss: promoted replica had lag → missing last N writes

Multi-leader: any node accepts writes, conflict resolution required
Leaderless (Cassandra/Dynamo): quorum reads+writes (W+R > N)

Replication is not "make more copies." It's a continuous stream of decisions about who can write, when writes are durable, how far behind replicas are allowed to fall, and what happens when the primary disappears. Each decision is a trade-off between latency, availability, and consistency.