Replication
Copies keep your data alive and your reads fast. The gap between copies is where the bugs live.
The story
A user changes their display name and hits Save. The app writes to the primary database, then redirects to the profile page, which reads from a replica, because the team moved all reads to replicas last month to take load off the primary.
The profile page shows the old name.
The user saves again. Old name again. They file a bug: "Saving my name doesn't work." The engineer who picks it up can't reproduce it: by the time they look, the replica has caught up and the name is correct. The bug is closed as "cannot reproduce". It comes back every week, always at peak hours.
Nothing is broken. The replica was 300 milliseconds behind the primary at peak, and the redirect took 50. The read arrived before the write did.
That gap, replication lag, is behind most replication bugs. The other big one shows up during failover. In October 2018, a 43-second network disruption at GitHub caused databases to be promoted in a different data centre. Some writes had reached one side and not the other, and the cleanup left the site degraded for about 24 hours.
The idea
Replication keeps a copy of the same data on several machines. You do it for four reasons:
- Durability: a dead disk doesn't lose data.
- Availability: if the primary dies, a replica takes over.
- Read scaling: spread reads across copies.
- Locality: put a copy near users in another region.
Single-leader: the default
One node accepts writes and streams its change log (the WAL in Postgres, the binlog in MySQL) to replicas, which apply it in order. It's simple and there are no write conflicts. It's what Postgres, MySQL, MongoDB and most managed databases do by default.
Sync vs async: the core trade-off
| Mode | Write latency | If the primary dies |
|---|---|---|
| Async | Fastest | Writes not yet shipped are lost on failover |
| Sync to all | Slowest. One slow replica stalls every write. | Nothing lost |
| Semi-sync / quorum (one or a majority must confirm) | Middle | Nothing lost if a confirmed replica is promoted |
The usual production setup: one synchronous standby in another zone (for failover with no data loss) plus async replicas for reads.
Living with lag
Async replicas are always a little behind: milliseconds normally, seconds or minutes under heavy load or a long-running query. Two anomalies follow:
- Read-your-writes: users must see their own changes. Fix: for a few seconds after a user writes, send their reads to the primary, or wait until a replica has caught up to the write's log position.
- Monotonic reads: a user shouldn't see data go backwards. Fix: pin each user's session to one replica.
Everyone else can read slightly stale data, and that's fine for almost everything except money, stock and permissions.
Failover: where the real danger is
When the primary dies, something has to detect it, choose a replica, promote it and redirect writes:
- Timeout too short: you fail over on a GC pause and cause an outage yourself. Too long: minutes of failed writes. Typically 10–30 seconds.
- Split brain: the old primary wasn't dead, just cut off, and keeps accepting writes. Prevent it with fencing: revoke its access, or use a consensus system (etcd, ZooKeeper, Patroni, managed RDS/Cloud SQL) so only one node can hold the leader lease.
Beyond single-leader
- Multi-leader: each region has a writable primary. Low write latency everywhere, but concurrent edits conflict and need resolution (last-write-wins loses data, CRDTs merge).
- Leaderless (Dynamo, Cassandra): write to and read from several replicas using quorums (CAP Theorem).
Cheat sheet
| Question | Answer |
|---|---|
| Default topology | Single leader + replicas |
| Production setup | Sync standby in another zone + async read replicas |
| Async risk | Last writes lost on failover |
| Sync risk | Slow replica stalls every write. Use quorum/semi-sync. |
| User doesn't see own change | Read-your-writes: route their reads to the primary briefly |
| Data "goes backwards" | Monotonic reads: pin session to one replica |
| Never read from replicas for | Balances, stock checks at checkout, permission changes |
| Failover timeout | 10–30 s. Too short causes false failovers. |
| Split brain | Fencing + a consensus-backed leader lease |
| Monitor | Replication lag (seconds and bytes), alert when it grows |
| Replication ≠ backup | DROP TABLE replicates instantly. Keep backups + PITR. |
Say it in the interview
"Single primary with a synchronous standby in a second zone, so failover loses nothing, plus two async replicas for reads. Reads that must reflect the user's own write go to the primary for a few seconds after that write. Failover goes through Patroni with an etcd lease so the old primary is fenced, and we alert when replica lag passes a few seconds."