Phase 2 · Chapter 2

Replication: WAL, Lag & Read Consistency

How Postgres copies every write to other machines — and what breaks when it falls behind

In the last chapter you learned that every committed transaction gets written to the WAL before Postgres acknowledges success. You treated the WAL as a crash-recovery tool. Now you are going to see that the same file is the engine of replication — the mechanism that lets you run multiple Postgres nodes with identical data. After this chapter you will understand exactly how a write on one machine appears on another, what "replication lag" means at the byte level, why reads from replicas can return stale data, and how to design systems that get the performance benefit of multiple nodes without silently serving wrong answers to users.

Part 1: Why One Node Is Never Enough

A single Postgres instance has three hard limits that replication exists to address.

Failure. One machine fails and your entire product goes down. There is no redundancy. For any product with paying users, this is unacceptable.

Read throughput. A single Postgres node can handle a finite number of concurrent queries. Once you saturate CPU or I/O on reads, you cannot scale by adding RAM or CPU indefinitely. At some point you need to spread reads across multiple machines.

Backup and analytics. Running heavy analytics queries on your production database competes with your application queries for CPU, I/O, and connection slots. A separate node that receives all the same writes but handles only analytical reads lets you run expensive queries without touching production performance.

All three problems share the same solution: run multiple Postgres nodes where every write goes to one designated node and is automatically copied to the others. The node that accepts writes is called the primary. The nodes that receive copies are called replicas (also: standbys, secondaries, read replicas).

The mechanism that connects them is the WAL.

Part 2: The WAL — From Crash Recovery Tool to Replication Engine

In Chapter 1 you learned that Postgres writes every change to the WAL before writing to data pages. The WAL is an append-only, sequential file stored in $PGDATA/pg_wal/. It is structured as a sequence of records, each describing a physical change to a specific page at a specific location.

A WAL record for an UPDATE does not say "set name = 'Trilochan' where id = 1." It says something closer to: "on page 47 of relation 16384, at offset 312, replace bytes 0x... with 0x...". This is below the SQL level — it is a physical description of the change to a data page.

This matters for understanding what replication actually ships between machines.

The WAL as a stream

The WAL is not a single file — it is a continuous stream of records divided into 16MB segment files (by default). As Postgres processes transactions, it appends records to the current WAL segment. When a segment fills, a new one starts. Old segments are retained for a configurable period and then recycled or archived.

Every record in the WAL has a position called the LSN — Log Sequence Number. An LSN is a 64-bit integer that encodes a byte offset into the WAL stream. You can think of it as a bookmark: "this is how far into the stream we have written." LSNs are monotonically increasing and never repeat for a given Postgres instance.

WAL Stream (conceptually):

LSN 0/1000000  ← transaction 100 begins
LSN 0/1001200  ← row insert (page 12, offset 88)
LSN 0/1002800  ← transaction 100 commits
LSN 0/1003000  ← transaction 101 begins
LSN 0/1005000  ← row update (page 47, offset 312)
LSN 0/1007200  ← transaction 101 commits
...

The primary's job is to generate this stream. The replica's job is to consume it and replay each record against its own data pages. When a replica has replayed every record up to LSN X, its data files are physically identical to the primary's data files at LSN X.

This is the entirety of how Postgres replication works at the core. Everything else — streaming replication, logical replication, synchronous standby — is a variation on this single idea: ship the WAL from primary to replica, replay it on the replica.

Part 3: Streaming Replication — Step by Step

Postgres's standard replication mechanism is called streaming replication, introduced in version 9.0. Here is the complete sequence of what happens when you run a primary with one replica.

Setup

The replica starts by copying the primary's data directory completely — a process called a base backup. This gives the replica a consistent snapshot of all data files at a specific LSN, call it LSN X. From this point forward, the replica needs only the WAL records that come after LSN X.

The WAL sender and receiver

On the primary, a dedicated process called the WAL sender handles replication connections. On the replica, a process called the WAL receiver maintains a TCP connection to the primary's WAL sender.

The WAL receiver sends its current LSN to the WAL sender: "I have replayed everything up to LSN X, send me everything after that." The WAL sender reads from pg_wal/ and streams records to the replica in real time. As new WAL records are written on the primary (new transactions committing), they are immediately streamed to connected replicas.

The startup recovery and catch-up

After the base backup, the replica enters recovery mode. It replays the WAL records it receives, applying each physical page change to its own data files. As it replays records, its LSN advances. Once it catches up to the primary's current LSN, it enters hot standby mode — it is still receiving and replaying WAL in the background, but it can now accept read-only queries.

Normal operation

From this point, the flow is continuous:

1. A transaction commits on the primary
2. Its WAL records are written to pg_wal/
3. The WAL sender detects new records and streams them to the replica
4. The WAL receiver on the replica writes them to its own pg_wal/
5. A process called the startup process (or wal receiver process in newer versions) replays those records against the replica's data pages
6. The replica's data pages now reflect that transaction

In favorable network and load conditions, this entire cycle takes milliseconds.

Part 4: Replication Lag — What It Actually Is

Replication lag is the gap between the primary's current LSN and the replica's replayed LSN. It is measured in bytes (WAL bytes not yet replayed) or in time (how old is the most recently replayed transaction). Both measurements are useful; neither tells the complete story alone.

The three lag components

Lag is not a single thing. It accumulates through three stages, each with its own cause:

Write lag: Time between the WAL record being written on the primary and the WAL sender shipping it to the replica. On a healthy network, this is microseconds. Under network congestion or a slow network link between data centers, this grows.

Flush lag: Time between the WAL receiver receiving a record and writing it to its own pg_wal/. This is disk I/O on the replica side. On a replica under heavy I/O pressure, writes to pg_wal/ can queue up.

Replay lag: Time between the WAL being written to the replica's pg_wal/ and being replayed (applied to data pages). This is the most common source of significant lag. Replaying WAL requires I/O — reading the target pages, applying changes, writing back. If the replica is serving many read queries simultaneously, its I/O is shared between serving reads and replaying WAL. The replay queue grows.

You can observe all three components in Postgres:

-- Run on the primary
SELECT
  client_addr,
  write_lag,
  flush_lag,
  replay_lag,
  pg_wal_lsn_diff(sent_lsn, replay_lsn) AS total_lag_bytes
FROM pg_stat_replication;

Understanding that replay lag comes from I/O contention on the replica is what lets you diagnose and fix it — either by giving the replica more I/O capacity, or by limiting the concurrency of read queries on the replica, or by using a dedicated replica for analytics that runs at a deliberately relaxed replication schedule.

Before reading on, see it happen. Each tab asks you to predict first — the next sections explain the stale read, the three fixes, and what failover costs under async replication.

Part 5: The Read Consistency Problem

Here is the problem that replication introduces which most engineers underestimate until they hit it in production.

Your application writes data on the primary and immediately reads it back — but the read goes to a replica. The replica is 200 milliseconds behind. Your read returns the old value. From the user's perspective, they just saved something and it disappeared.

This is called reading your own writes — or more precisely, the failure to read your own writes. It is not a bug in your code. It is the correct behavior of an eventually consistent read replica.

A concrete scenario

User updates their profile on Samiksha. The PUT request writes to the primary. The response redirects them to their profile page. The GET request for the profile page reads from the replica. The replica has not yet replayed the update. The profile page shows the old data. The user thinks the save failed. They save again. Now you have two write operations for one intended change.

This is not theoretical. It happens on every read-replica setup that does not account for it.

Three strategies to handle it

Strategy 1: Route all reads for a user session to the primary.

Simplest approach. After any write, read from the primary for that user's session for a short window (e.g., 5 seconds). Implement this as a session flag: readsFromPrimary: true, until: timestamp. Every query checks this flag before choosing which connection to use.

Cost: you lose the read-scaling benefit for users who recently wrote something. For most SaaS workloads, this is acceptable — the write rate is much lower than the read rate, and the window is short.

Strategy 2: Synchronous replication — wait for replica confirmation.

Configure Postgres so the primary does not acknowledge a commit until at least one replica has confirmed receipt and replay of the WAL records. The write latency increases by the network round-trip to the replica plus the replica's replay time, but reads from that replica are guaranteed to reflect every acknowledged write.

# postgresql.conf on primary
synchronous_standby_names = 'replica1'
synchronous_commit = on

With this configuration, COMMIT on the primary does not return until replica1 has confirmed flush (or replay, depending on synchronous_commit level). Reads from replica1 will always see data that the primary has confirmed.

Cost: every write takes longer. If the replica is unavailable, the primary waits indefinitely (or until timeout). This trades write availability for read consistency.

Strategy 3: LSN-based read routing.

After a write, record the LSN of that commit. Before reading from a replica, check whether the replica's current replay LSN is at least as large as the write's LSN. If not, wait briefly or fall back to the primary.

// After write
const writeLsn = await primaryDb.query('SELECT pg_current_wal_lsn()');

// Before reading from replica
const replicaLsn = await replicaDb.query('SELECT pg_last_wal_replay_lsn()');
if (replicaLsn < writeLsn) {
  // Fall back to primary or wait
}

This is the most precise approach but requires LSN tracking in your application layer. PgBouncer and some connection poolers support this natively. Libraries like Makara (Ruby) implement it automatically.

Part 6: Synchronous vs Asynchronous Replication — The Real Tradeoff

These are not just configuration options. They represent fundamentally different guarantees and different failure modes.

Asynchronous replication (default)

The primary commits and acknowledges to the client without waiting for any replica. WAL records are shipped to replicas in the background.

Properties:

• Write latency: unaffected by replica health or network to replica
• Durability on primary failure: if primary crashes immediately after acknowledging a commit and before the WAL reaches the replica, that transaction is lost. The replica becomes primary but does not have the last N transactions. This is called replication data loss window.
• Availability: if the replica goes down, the primary continues operating normally.

For most web applications, asynchronous replication is the correct choice. The data loss window is typically less than one second, and the alternative (synchronous replication) imposes write latency penalties that hurt application responsiveness.

Synchronous replication

The primary waits for acknowledgment from one or more replicas before confirming a commit to the client. The synchronous_commit parameter controls exactly what "acknowledgment" means:

synchronous_commit = remote_write
  → primary waits until replica has received and written WAL to OS buffer
  → WAL is not yet on replica's disk, but it left the primary's network stack

synchronous_commit = on (default when sync standby is configured)
  → primary waits until replica has flushed WAL to disk (fsync'd)

synchronous_commit = remote_apply
  → primary waits until replica has replayed WAL and applied changes to data pages
  → the strongest guarantee: reads from replica immediately reflect this commit

The stronger the guarantee, the higher the write latency, because you are adding network round-trips and I/O time on the replica into your commit path.

remote_write latency  ≈ network RTT to replica
on latency            ≈ network RTT + replica disk fsync time
remote_apply latency  ≈ network RTT + replica replay time (slowest, most correct)

The failure mode to understand: With synchronous_standby_names set, if the designated synchronous replica goes down and cannot acknowledge, the primary will queue commits and wait. Your application appears to hang. Writes stop. This is not a bug — it is the correct behavior for a system that has promised not to acknowledge commits until a replica confirms them.

Production configuration usually hedges: use ANY 1 (replica1, replica2) — the primary waits for confirmation from any one of the listed replicas, not all of them. If one replica goes down, the other can still acknowledge.

Part 7: Physical vs Logical Replication

Everything above is physical replication — shipping raw WAL records that describe byte-level changes to data pages. The replica ends up as a byte-for-byte copy of the primary.

Postgres also has logical replication, which ships higher-level change events: "row with id=1 in table users was updated, new values are (name='Trilochan', updated_at='...')".

The differences:

For high-availability and read scaling, physical replication is what you use. Logical replication is for shipping data to analytics systems, migrating between Postgres versions with zero downtime, or syncing specific tables to a different database.

Your Hetzner VPS setup with a primary and replica uses physical replication. When you eventually want to ship events to a data warehouse or run a Postgres major version upgrade with zero downtime, logical replication becomes relevant.

Part 8: Failover — When the Primary Dies

Primary node fails. The replica has replayed WAL up to LSN Y. You need to promote the replica to be the new primary so writes can resume.

Manual promotion

# On the replica server
pg_ctl promote -D /var/lib/postgresql/data
# Or: touch /var/lib/postgresql/data/failover.signal

The replica exits recovery mode, stops accepting WAL from the (now dead) primary, and begins accepting write connections. It generates a new timeline identifier — a monotonically increasing integer that marks "this is where this node became a new primary" — to distinguish its future WAL from the original primary's WAL.

The timeline matters: if the original primary comes back online after failover and tries to rejoin, it may have WAL records that the promoted replica never received (the data loss window from async replication). Postgres uses the timeline to detect this and refuse to let a stale old primary silently reconnect as a replica with diverged data.

Automated failover

Manual promotion requires a human to notice the failure and run a command. Production systems use automated failover managed by tools like:

Patroni: Runs alongside each Postgres instance. Uses etcd or Consul (distributed key-value stores using Raft consensus — you will understand exactly how these work in Phase 3 Chapter 3) to elect a leader. When Patroni on the primary stops reporting, Patroni on a replica detects the absence, runs a leader election, and promotes the winning replica. Application traffic is rerouted via a VIP (virtual IP) or via HAProxy updating its backend list.

Repmgr: Lighter weight. Monitors replication and can trigger promotion scripts. Less robust under network partition scenarios than Patroni.

For your Hetzner VPS setup with a single primary and one replica, Patroni with etcd is the production-grade path when you need high availability. The cost is operational complexity: you now have etcd to manage alongside Postgres.

Part 9: What This Means For Your Application Code

Three concrete rules that apply to every application that reads from replicas:

Rule 1: Never read your own writes from a replica without lag handling. Any flow where a write is immediately followed by a read of the same data must either use the primary for the read, use synchronous replication, or implement LSN-based routing. Violation of this produces silent staleness bugs that are hard to reproduce because they are timing-dependent.

Rule 2: Replica reads are eventually consistent, not strongly consistent. Treat replica reads like cache reads: they are fast, they scale, and they can be slightly stale. Design your UI accordingly. Show "data may be up to X seconds delayed" for analytics views. Never show a user's own recent actions from a replica without lag handling.

Rule 3: Long-running transactions on the replica block vacuum on the primary. This is a subtle production failure mode. A long-running read query on a replica holds an LSN snapshot. The primary cannot vacuum dead tuples that are newer than that snapshot — vacuuming them would break the replica's snapshot. If a query on the replica runs for hours, the primary's tables bloat (table bloat from Chapter 1). Monitor pg_stat_replication's sent_lsn minus replay_lsn, and set max_standby_streaming_delay and hot_standby_feedback appropriately to balance replica query cancellations against primary bloat.

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis: AttemptIQ has a primary Postgres node and one async replica. You route all read queries to the replica to reduce load on the primary. A user submits a test and their answers are written to the primary. The response redirects them to a results page which reads from the replica.

Under what conditions does the results page show no answers submitted? Describe the exact sequence of events. How would you fix this without abandoning replica reads entirely? Write the pseudocode for the routing logic.

Problem 2 — Design From Scratch: Samiksha is growing. You have one primary Postgres on Hetzner. You need to add replication for two reasons: high availability (automated failover under 30 seconds) and analytics (you want to run heavy aggregation queries without impacting production). Design the topology:

How many replicas? What role does each serve? What replication mode (sync/async) for each and why? Where do analytics queries go and what lag is acceptable? What tool manages failover and what does it use for consensus? Draw the topology as ASCII.

Problem 3 — Lag Diagnosis: You run SELECT * FROM pg_stat_replication on your primary and see:

write_lag  = 0ms
flush_lag  = 0ms
replay_lag = 4200ms
total_lag_bytes = 847MB

Write lag and flush lag are near zero. Replay lag is 4.2 seconds. What is the most likely cause? What is happening physically on the replica at this moment? What are two different interventions you could apply and what is the tradeoff of each?

Problem 4 — Go Deeper: Synchronous replication with synchronous_commit = remote_apply guarantees that a read from the replica immediately after a confirmed write will return the written data. But this guarantee has an edge case involving network partitions.

Describe a scenario where: (a) the primary acknowledges a commit with remote_apply, (b) the user reads from the replica, and (c) the read still returns stale data. What network or infrastructure condition makes this possible? How would a system that absolutely cannot tolerate this solve it?

Problem 5 — Connect to Previous Chapters: In Chapter 1 you learned that MVCC keeps dead tuples until VACUUM removes them, and that a long-running transaction on the primary prevents VACUUM from collecting dead tuples newer than that transaction's snapshot.

Replication introduces a related problem: a long-running read query on a replica can also prevent VACUUM on the primary. Explain the physical mechanism by which a query on the replica causes table bloat on the primary. Which Postgres configuration parameter controls this behavior? What are the two sides of the tradeoff when tuning it?

What's Next

You now have two nodes — a primary that accepts writes and a replica that mirrors them. This solves availability and read scaling. But there is a ceiling: every write still goes to one machine. When your write volume exceeds what one node can handle, you need to split the data itself across multiple nodes — each node owning a partition of the rows and serving reads and writes for that partition only. That is sharding. Chapter 3 is about when sharding becomes necessary, how to partition data so queries still work, and what operations become impossible or expensive once your data is split across machines. The consistent hashing you will see in Phase 3 Chapter 2 is one implementation of the same partitioning idea — after Chapter 3 you will understand exactly why it was invented.

End of Phase 2, Chapter 2.