Chapter 5: Leader-Follower Replication

Curriculum: Distributed Systems from First Principles Chapter: 5 of 24 Part: 2 — Replication Prerequisites: Chapter 2 (pages, heap files, WAL introduced), Chapter 4 (replication motivation, sync vs async, split-brain preview) Estimated reading time: 80–90 minutes

Opening

Chapter 4 established that replication is necessary and that the WAL is its mechanism. "The standby replays the WAL" is the right mental model at 10,000 feet. This chapter takes you to ground level. You'll see exactly how a WAL record travels from a client's INSERT on the primary to a matching page write on the standby — every process involved, every disk write, every network message. You'll learn what an LSN is and how it measures position in a stream of changes. You'll understand replication slots — the feature that prevents standbys from falling permanently behind, and the feature that has brought down production databases at dozens of companies because no one was watching the disk fill up. You'll trace what happens during a promotion: the exact moment the standby stops being a standby, what a "timeline" is and why it exists, and why the old primary cannot simply rejoin as a standby after a failover. By the end, when your managed database's monitoring page shows "replica lag: 42 ms," you'll know exactly which of the four lag stages that number represents, and why it matters which one.

Part 1: The WAL as a Physical Stream

In Chapter 2, you learned that the Write-Ahead Log is a sequential file where Postgres records every change before applying it. The crash recovery use was clear: replay the WAL after a crash to reconstruct the committed state. Now look at the same WAL from the replication angle.

The WAL is a complete, ordered, append-only record of every state change in the database. If you gave someone a copy of the database at time T, and then gave them every WAL record generated after T, they could replay those records and arrive at the exact current state of the database. This is precisely what a standby does — except it doesn't wait until a crash; it replays WAL continuously, in real time, as each record is produced.

The WAL is stored as a sequence of segment files in the pg_wal directory. Each segment is 16MB by default. The files are named with a 24-character hexadecimal number representing their position in the global WAL sequence. They look like this:

000000010000000000000001
000000010000000000000002
000000010000000000000003
...

The first 8 characters are the timeline ID (we'll return to timelines later). The rest encodes the position. Postgres writes to the current segment, fills it, closes it, opens the next one, and continues. Old segments are recycled or deleted once no process needs them anymore.

Each WAL record inside a segment has this structure:

[Header: xl_tot_len, xl_xid, xl_prev, xl_info, xl_rmid, xl_crc]
[Body: the actual change data]

The header contains: the total length of the record, the transaction ID that generated it, a pointer to the previous WAL record (for traversal), type flags, which resource manager handles this record type, and a CRC checksum for integrity verification.

The body is resource-manager-specific. The "Heap" resource manager handles row inserts, updates, and deletes. A WAL record for an INSERT says, in binary: "On relation OID 16384 (the users table), on page 312, at offset 3, write these bytes: [the binary tuple data]." An UPDATE record says: "Mark old tuple at (page X, offset Y) as dead, and write new tuple at (page A, offset B) with these bytes." There is no SQL in the WAL. There are no column names. Just page numbers, offsets, and binary data — the same language as the storage engine.

This matters for understanding what the standby actually does. It doesn't re-execute SQL. It applies binary page modifications. The standby's heap files end up byte-for-byte identical to the primary's heap files, applied through the same page-level operations that the primary used internally. This is why it's called physical replication — it replicates the physical page state, not the logical operation that produced it.

Part 2: The Two Processes on Each Side

Replication involves four processes total — two on the primary, two on the standby. Understanding which process does what prevents the confusion that arises when something goes wrong.

On the primary:

The WAL writer process runs continuously, flushing WAL records from the WAL buffer (in-memory) to disk at regular intervals and on every transaction commit. This is what makes writes durable — the WAL writer ensures that committed WAL is on disk before the client's acknowledgment is sent.

The WAL sender process is spawned specifically for each connected standby. When a standby connects, the primary spawns a WAL sender to serve it. The WAL sender reads WAL records from disk (or from the WAL buffer if they haven't been written to disk yet) and streams them over the network connection to the standby. Each standby has its own dedicated WAL sender. You can see them in pg_stat_replication.

On the standby:

The WAL receiver process maintains the network connection to the primary's WAL sender. It receives WAL records as they stream in and writes them to the standby's own pg_wal directory. The WAL receiver is the network half — it deals with the connection, handles reconnections, sends acknowledgment messages back to the primary.

The startup process (sometimes called the recovery process) reads WAL records from the standby's pg_wal directory and applies them to the standby's heap files. This is the same code path used for crash recovery — the startup process was originally designed to replay WAL after a crash; in standby mode it just never stops. It reads records, applies them to pages, reads more records, applies them, continuously.

Notice the separation on the standby: the WAL receiver writes WAL to disk, and the startup process reads it and applies it. These are independent. This separation matters: the WAL receiver can acknowledge to the primary "I have written and fsynced this WAL" before the startup process has applied it to the heap. The heap might lag slightly behind the WAL. For synchronous replication, you can configure exactly which acknowledgment point the primary waits for, which we'll see in Part 5.

Part 3: LSN — The Address of Every Change

To track position in the WAL stream, Postgres uses the Log Sequence Number — LSN. An LSN is a 64-bit unsigned integer representing a byte offset within the global WAL sequence. It looks like 0/3E8A0E8 in Postgres's display format (the two hex numbers are the high and low 32 bits).

The LSN is monotonically increasing. Every WAL record has an LSN. As you move forward in the WAL, LSNs increase. Every standby tracks its current LSN — how far through the WAL stream it has progressed.

LSNs are the common language for measuring replication state:

-- On the primary: what is the current end of the WAL?
SELECT pg_current_wal_lsn();
-- Returns: 0/5F2A310

-- On the standby: what is the last WAL position received?
SELECT pg_last_wal_receive_lsn();
-- Returns: 0/5F2A280

-- On the standby: what is the last WAL position applied to heap?
SELECT pg_last_wal_replay_lsn();
-- Returns: 0/5F2A100

The difference between pg_current_wal_lsn() on the primary and pg_last_wal_replay_lsn() on the standby — in bytes — is the total lag. It combines all the stages between a write on the primary and its effect on the standby's heap.

When the standby connects and initiates streaming replication, it sends its current LSN to the primary: "I have everything up to position X; please start sending from X+1." The primary's WAL sender finds WAL segment X+1 and begins streaming. This handshake is why replication is robust to brief disconnections: as long as the standby's last LSN still exists in the primary's WAL (hasn't been deleted), the standby can reconnect and resume from where it stopped.

Part 4: Replication Slots — The Safety Net with a Sharp Edge

Here is a scenario that, without replication slots, causes a common and painful production failure.

Your standby falls behind due to a network hiccup. The primary keeps generating WAL. At some point the primary's WAL directory fills up with old segments. Postgres cleans up old WAL segments automatically — once a segment is older than the oldest still-needed checkpoint, it's safe to delete (or recycle). The primary deletes the segments the standby needs. The standby reconnects. It asks for WAL starting at its last LSN. The primary's WAL sender looks for that position and discovers the segment has been deleted. The replication connection fails with an error like "requested WAL segment has already been removed." You cannot resume streaming replication. You must re-clone the entire standby from scratch — a process that could take hours for a large database.

Replication slots solve this. A replication slot is a persistent record on the primary that says: "This slot needs WAL up to at least LSN X. Do not delete WAL segments that this slot still needs."

-- Create a replication slot on the primary
SELECT pg_create_physical_replication_slot('standby1_slot');

-- See all slots and their lag
SELECT slot_name, active, restart_lsn, pg_current_wal_lsn() - restart_lsn AS lag_bytes
FROM pg_replication_slots;

The standby connects and identifies itself using this slot name. The primary's WAL cleanup process checks all active slots and retains any WAL segment that any slot still needs. Even if the standby is disconnected for hours, when it reconnects it will find its WAL still intact.

The Danger of Replication Slots

This safety net has a sharp edge. If a standby with a replication slot goes down permanently — the machine dies, the standby is decommissioned, someone forgets to drop the slot — the primary accumulates WAL indefinitely. Every WAL segment generated since the slot's last known position is retained. Since WAL is generated continuously, the primary's pg_wal directory grows without bound. When the disk fills, Postgres cannot write new WAL. When Postgres cannot write WAL, it cannot commit any transaction. The primary crashes. Your entire database goes down because of a forgotten replication slot.

This is not a theoretical risk. It has happened in production at enough companies that it has become a well-known operational hazard. The fix is monitoring:

-- Alert if any slot has accumulated more than X bytes of unreplicated WAL
SELECT slot_name, pg_current_wal_lsn() - restart_lsn AS lag_bytes
FROM pg_replication_slots
WHERE pg_current_wal_lsn() - restart_lsn > 10737418240; -- 10 GB threshold

If a slot's lag exceeds your disk capacity threshold, you must either investigate why the standby is behind, or drop the slot if the standby is gone:

SELECT pg_drop_replication_slot('standby1_slot');

Dropping the slot means the standby cannot resume streaming from that point — it needs a fresh base backup. But a fresh base backup is recoverable. A full disk crashing your primary is not.

Managed database services (RDS, Supabase, etc.) handle slot cleanup automatically. If you're managing Postgres yourself — on a VM, in Kubernetes, with Patroni — replication slot monitoring must be part of your alerting setup from day one.

Part 5: The Four Stages of Lag

Chapter 4 introduced replication lag as a single number. In reality it has four distinct stages, and they mean different things. Postgres exposes all four in pg_stat_replication:

SELECT
  client_addr,
  pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn)   AS send_lag_bytes,
  pg_wal_lsn_diff(sent_lsn, write_lsn)               AS write_lag_bytes,
  pg_wal_lsn_diff(write_lsn, flush_lsn)              AS flush_lag_bytes,
  pg_wal_lsn_diff(flush_lsn, replay_lsn)             AS replay_lag_bytes
FROM pg_stat_replication;

Stage 1 — Send lag (pg_current_wal_lsn() minus sent_lsn): WAL has been generated on the primary but not yet transmitted to the standby. This lag exists when the network pipe is too narrow for the write rate, or when the WAL sender is briefly catching up after a reconnection.

Stage 2 — Write lag (sent_lsn minus write_lsn): WAL has been received by the standby's WAL receiver but not yet written to the standby's pg_wal directory on disk. This lag exists only briefly — the WAL receiver writes to disk almost immediately after receiving.

Stage 3 — Flush lag (write_lsn minus flush_lsn): WAL has been written to the standby's disk but not yet fsynced (durably committed to persistent storage). An fsync call ensures the OS has flushed its write buffers to the actual storage device. Without fsync, a power failure could lose the written-but-not-fsynced WAL.

Stage 4 — Replay lag (flush_lsn minus replay_lsn): WAL has been fsynced on the standby's disk but the startup process has not yet applied it to the standby's heap files. The heap pages don't yet reflect the changes. This is the lag that matters for reads — a read from the standby sees the heap state as of replay_lsn, not flush_lsn.

Primary               WAL record journey                 Standby
generates ──→ sent ──→ written to disk ──→ fsynced ──→ applied to heap
             ←——————— send_lag ———————→
                       ←——— write_lag ——→
                                   ←—— flush_lag ——→
                                             ←— replay_lag →

For synchronous replication, the synchronous_commit parameter controls at which stage the primary waits before acknowledging to the client:

• synchronous_commit = remote_write: primary waits for Stage 2 (written to standby disk, not fsynced). Fast, but a simultaneous power failure on primary and standby could still lose the write.
• synchronous_commit = on (default for synchronous): primary waits for Stage 3 (fsynced on standby). Safe against a single-node power failure.
• synchronous_commit = remote_apply: primary waits for Stage 4 (applied to standby heap). Strongest guarantee — a read from the standby immediately after the primary acknowledges will always see the written value. Slowest, because it waits for the startup process.

For read-your-own-writes guarantees (Chapter 4's problem): remote_apply is the only synchronous level that truly solves it. With on, the WAL is fsynced on the standby but might not be applied to heap yet. A read from the standby could still miss the write for a brief period.

Before reading about promotion, try it. The first two tabs show what replay lag does to reads; the third kills the primary while WAL is still in flight — predict each one first.

Part 6: Promotion — When the Standby Becomes Primary

The primary fails. Monitoring detects it. Someone or something decides to promote the standby. What happens physically?

Step 1: Signal the standby to promote.

In Postgres 12+, you call SELECT pg_promote() on the standby. In older versions you touch a trigger file. Either way, the standby receives a "promote" signal.

Step 2: The standby finishes applying buffered WAL.

Before accepting writes, the standby's startup process applies all WAL records that are already in its pg_wal directory — everything it has received but not yet replayed. This is the "recovery to consistent state" phase. It ensures the standby's heap is as up-to-date as possible before taking writes.

Step 3: The standby creates a new timeline.

This is the critical step that prevents confusion. The standby increments the timeline ID — the first 8 characters of the WAL segment filename — and begins writing new WAL with this new timeline. Timeline 1 becomes Timeline 2. The new primary's WAL segments look like 000000020000000000000001.

The purpose of the timeline: it distinguishes "WAL written by the original primary" from "WAL written by the promoted standby." After a failover, the old primary might come back online with WAL that diverges from the new primary's WAL. Without timelines, there's no way to tell which WAL records belong to which history. With timelines, every WAL record is labelled with the history it belongs to.

Step 4: The standby exits recovery mode and starts accepting writes.

The new primary is live. Clients can now write to it. The standby's process tree restructures — the startup/recovery process exits, and normal backend processes (ready to serve write connections) start.

Part 7: The Old Primary Returns — Why It Can't Just Rejoin

The old primary comes back online. What happens?

It's still running Timeline 1 WAL. The new primary is running Timeline 2. At LSN 0/480, the histories diverged — the old primary continued writing Timeline 1 WAL after the failure (if it came back before anyone stopped it), while the new primary started writing Timeline 2 WAL.

The old primary cannot simply reconnect as a standby to the new primary. Its WAL is on a different timeline. The new primary's WAL sender, seeing a connection from the old primary on Timeline 1, will reject it — the timelines diverged, and accepting the old primary as a replica would mean the old primary tries to apply Timeline 2 WAL records to a heap that diverged at Timeline 1. The page modifications would be wrong.

To rejoin as a standby, the old primary must be re-cloned from the new primary: a fresh pg_basebackup that copies the entire database state from the new primary. The old primary's data directory is discarded. It starts over as a clean replica of the new primary, on Timeline 2.

In production, this is handled by tools like Patroni, Repmgr, or the managed service. But the mechanism is the same: the old primary is fenced (prevented from accepting writes, often by blocking its network or revoking credentials), then re-cloned.

The data that was on the old primary but not on the new primary — the WAL between LSN 0/480 and wherever the old primary got to — is lost. Those were writes that the primary had committed but hadn't yet shipped to the standby. This is the data loss window of asynchronous replication materialising in practice.

Part 8: Hot Standby — Reads During Recovery

By default, a standby in recovery mode rejects all client connections — it's busy replaying WAL and not available for queries. This is safe but wasteful: the standby is a fully functional machine with a copy of all the data.

With hot_standby = on in postgresql.conf (enabled by default in modern Postgres), the standby accepts read-only queries while continuously applying WAL. This is the foundation of read replicas: a standby that serves reads while staying synchronised with the primary.

The mechanism: when a client query starts on the hot standby, Postgres takes a snapshot of the current replay_lsn — the point in the WAL up to which the heap has been applied. The query sees the database state as of that snapshot. Since the heap is being continuously updated by the startup process in the background, queries see a consistent point-in-time view rather than a mutating state.

The complication: query conflicts. The startup process needs to apply WAL records that might conflict with running queries on the standby.

Consider: a query on the standby is doing a long sequential scan of the users table. Simultaneously, the primary runs VACUUM on the users table, which generates WAL records that tell the standby to mark certain page items as dead. The startup process tries to apply these WAL records. But the running query is still reading those pages. Postgres has a choice: wait for the query to finish, or cancel it.

The max_standby_streaming_delay parameter controls how long the startup process waits before canceling a conflicting query. Default: 30 seconds. If a query on the standby runs longer than 30 seconds and a conflicting WAL record arrives, the query is killed with: ERROR: canceling statement due to conflict with recovery.

This is one reason long-running analytical queries don't belong on a hot standby that's also serving a replication stream. Either use a dedicated replica for analytics (with a longer max_standby_streaming_delay or even paused replication during large queries), or use a separate read-only database that isn't a replication target.

Part 9: WAL Archiving and Point-in-Time Recovery

Replication slots protect against the standby falling behind and losing access to recent WAL. But what about older WAL? Once a WAL segment is no longer needed by any replication slot or checkpoint, Postgres deletes it. You lose the ability to replay those changes. If you need to roll back to yesterday's state — say, someone ran DELETE FROM orders WHERE user_id = 42 accidentally and deleted 3 years of order history — you cannot replay WAL back to yesterday because yesterday's WAL is gone.

WAL archiving solves this. You configure Postgres with an archive_command — a shell command that Postgres runs every time a WAL segment is completed. Typically this copies the segment to S3 or Google Cloud Storage:

archive_command = 'aws s3 cp %p s3://your-bucket/wal-archive/%f'

With archiving enabled, every WAL segment is preserved externally before Postgres deletes it locally. The archive contains the complete history of every change to the database since archiving began.

Combined with periodic base backups (snapshots of the entire database directory, typically taken with pg_basebackup), WAL archiving enables Point-in-Time Recovery (PITR):

1. Restore the most recent base backup to a new machine.
2. Point Postgres at the archive: set restore_command (for example restore_command = 'cp /archive/%f %p') so it can fetch archived WAL segments as it needs them.
3. Set recovery_target_time = '2024-06-07 14:30:00' in postgresql.conf and create an empty recovery.signal file in the data directory. (Before PostgreSQL 12 these settings lived in recovery.conf; that file no longer exists.)
4. Start Postgres. The startup process replays WAL up to the target time and stops. With recovery_target_action = 'pause' it waits there, so you can check the data before promoting.
5. You now have the database as it existed at exactly 14:30:00 — before the accidental delete.

This is the most important disaster recovery mechanism for a SaaS product. Without PITR, "user accidentally deleted all their data" means restoring from the most recent nightly backup — losing potentially hours of data. With PITR, you can restore to 30 seconds before the disaster, losing almost nothing.

The operational cost: the archive accumulates continuously. At a write rate of 100 MB/s of WAL, you generate 8.64 GB of WAL per day. Storing 30 days of archives costs 259 GB of S3 storage — at $0.023/GB, about $6/month. The cost of not having it is a customer support nightmare and potential SLA breach. The math is obvious.

Managed database services (RDS, Google Cloud SQL, Supabase) implement PITR automatically. They take base backups, archive WAL to durable storage, and let you restore to any point within a retention window (typically 7–35 days depending on tier). Understanding the mechanism tells you what their restore promise actually means: they're replaying your WAL back to the requested timestamp, applying every change sequentially. A restore to "30 seconds ago" on a database with 5 GB of WAL/hour means replaying ~41 MB of WAL — fast. A restore to "3 hours ago" means replaying ~15 GB of WAL — this will take minutes.

Part 10: A Realistic Production Setup

Let's put this together into what a real production Postgres setup looks like, and map each component to the mechanisms you've learned.

Primary: accepts all writes. Ships WAL to all connected standbys. Archives WAL segments to object storage.

Sync standby (same datacenter, ~500 µs RTT): one synchronous replica, configured via synchronous_standby_names. The primary uses synchronous_commit = on — writes are only acknowledged after the sync standby has fsynced the WAL. RPO = 0 for acknowledged writes. This standby is the automatic failover target.

Async read replicas (same datacenter, potentially multiple): configured with hot_standby = on. Application read traffic is load-balanced across these replicas. They may lag by up to a few seconds. The application routes writes (and consistency-sensitive reads) to the primary; background-tolerant reads (analytics, search, reporting) to replicas.

WAL archive: all WAL segments ship to S3 via archive_command. Combined with nightly base backups, this enables PITR for the last 30 days. This is the safety net against human error — the category of failure that the synchronous standby doesn't protect against (a mistake on the primary is immediately replicated to all standbys).

Replication slots: each standby has a physical replication slot on the primary. A monitoring alert fires if any slot accumulates more than 10 GB of unshipped WAL. Slots for decommissioned standbys are dropped within 1 hour.

Patroni or cloud HA: a coordinator process (using etcd or cloud-native tooling) monitors the primary's health, manages replication slot cleanup, and orchestrates failover: fencing the old primary, promoting the sync standby, updating DNS records so the application's connection string points to the new primary. The application reconnects automatically after the ~30-second failover window.

This is not a theoretical design. It is the architecture — with minor variations — used by most serious production Postgres deployments.

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

You manage a Postgres primary with three standbys: one synchronous standby (same DC), one async read replica (same DC), and one async read replica (different region, for DR). Each has a replication slot.

At 14:00, the different-region DR standby loses network connectivity due to a cloud provider outage. The network outage lasts 72 hours.

Trace what happens to: (a) replication slot lag on the primary, (b) primary's pg_wal directory size (the primary writes 500 MB of WAL per hour), (c) primary write performance, (d) primary disk capacity (the pg_wal directory has 100 GB of space allocated).

At what point does this situation become a disaster? What is the precise sequence of events that crashes the primary? What should the on-call engineer do at hour 1? At hour 6? At hour 48?

Problem 2 — Design from Scratch

Your SaaS product has 100,000 paying customers. During a sales demo, a salesperson accidentally runs the following on production: DELETE FROM customers WHERE plan = 'trial'. This deletes 3,400 trial customers. More critically, it also deleted 12 paying customers who were miscategorised as 'trial' due to a data bug. You have: nightly base backups (at 02:00 UTC), WAL archiving enabled, and PITR configured with a 30-day retention window. The accidental delete occurred at 14:37 UTC.

Design the full recovery procedure. Be specific: which backup do you restore from, what recovery_target_time do you set, what do you do with the restored database, and how do you merge the 12 recovered paying customers back into production without losing the 3,400 legitimately-deleted trial customers or the 20 hours of normal production data generated since 02:00 UTC?

Problem 3 — Synthesis

Connect these three concepts: the four lag stages, synchronous_commit = remote_apply, and the read-your-own-writes problem from Chapter 4.

Your application's checkout flow does this: (1) INSERT the order to the primary; (2) immediately redirect the user to /orders/{id} which fetches the order for display. The fetch goes to a read replica to reduce primary load.

With default async replication, the user sometimes sees a 404 ("order not found") on the redirect. The team's proposed fix is synchronous_commit = remote_apply. Explain precisely why this fix works — which stage of lag it eliminates and why that stage is what causes the 404. Then: what does it cost? Calculate the added latency assuming the replica is in the same datacenter (500 µs network RTT, replica applies WAL within 2 ms of receiving it). Is there a cheaper fix that doesn't require sync replication?

Problem 4 — Go Deeper

Postgres supports a second replication mode called logical replication, which is not covered in this chapter. From the following clues, reason through what it does and when you'd use it over physical replication:

• Physical replication ships binary WAL records that describe page-level changes.
• Logical replication ships row-level changes described as logical operations: "row with id=42 was updated; new values are: name='Bob', email='bob@example.com'".
• A physical replica must have identical Postgres version and platform as the primary. A logical replica does not.
• Physical replication replicates the entire database cluster. Logical replication can replicate a single table or a subset of tables.
• You cannot write to a physical replica (hot standby = read only). You can write to a logical replica's non-replicated tables.

From these clues: what use cases does logical replication enable that physical replication cannot? What would be the trade-offs — what does physical replication give you that logical cannot? Think of at least two concrete scenarios from a SaaS product context where you'd choose each.

Problem 5 — Connect to Previous Chapters

Chapter 2 introduced the WAL as a crash recovery mechanism: Postgres writes changes to the WAL before applying them to the heap, so after a crash it can replay the WAL to reconstruct committed state.

Chapter 5 has shown that the same WAL is used for replication: the primary streams WAL records to standbys that replay them.

Now connect to Chapter 3's indexes: when the primary performs an INSERT that updates both the heap and a B-Tree index (creating new index leaf entries), what WAL records are generated? Are both the heap page modification and the B-Tree page modification in the WAL?

If yes: when the standby replays this WAL, it reconstructs both the heap change and the index change from binary page records. What does this tell you about the standby's indexes — are they identical to the primary's? What happens if you add an index to the primary: does the standby automatically get it?

If you ran CREATE INDEX CONCURRENTLY on the primary, which generates a lot of WAL (it scans the entire table and builds the index while the table is live), what does this do to the standby's replication lag during the index build?

What's Coming Next

You now understand how replication works mechanically, end to end. But a mechanism without a consistency model is incomplete. Chapter 5 showed you how data moves from primary to replica. Chapter 6 asks: what guarantees does this give us about what clients observe? When a client reads from the primary and another from the replica, what can they see? What is "consistent" in a replicated system, and what is the weakest consistency that's safe enough to build on? CAP theorem — the most misunderstood concept in distributed systems — is the answer to these questions, and it will make every subsequent architecture decision legible.

Chapter 5 complete. Problem 3 is the one to do first — it synthesises the four lag stages into a concrete application bug and its fix. Problem 2 is the most practically valuable: walk through PITR as if you were the on-call engineer at 14:38. Problem 5 connects all three chapters of Part 1 to replication — work through it carefully.