Chapter 4: Why Machines Die and What You Do About It

Curriculum: Distributed Systems from First Principles Chapter: 4 of 24 Part: 2 — Replication Prerequisites: Chapter 1 (latency numbers), Chapter 2 (pages, heap files, WAL), Chapter 3 (indexes and write costs) Estimated reading time: 75–90 minutes

Opening

Every architecture decision in this chapter and the nine that follow is a response to one uncomfortable fact: the hardware your software runs on will fail. Not might fail — will fail, on a schedule you cannot predict, at a time you cannot choose. If you've only ever run code on a single machine, this sounds like an abstract risk you manage with backups and prayers. At scale, it is a daily operational reality. Google loses hundreds of servers a week. Amazon's storage systems are designed assuming that any individual disk will fail within 3 years. Netflix deliberately injects random failures into their production systems to test their resilience. These aren't organisations with bad hardware — they're organisations that have done the math on reliability and accepted what it says. This chapter is that math. By the end of it, you will understand why no serious system runs on a single machine, why redundancy is not a luxury but a mathematical necessity, and why copying data across multiple machines — replication — is the foundational primitive of every distributed system that follows.

Part 1: The Unreliability of Hardware — By the Numbers

Let's start with disks, because disk failures are the most studied and the most common.

A modern hard disk drive has an advertised MTBF — Mean Time Between Failures — of 1 to 2 million hours. That sounds like you'd expect the drive to last 114 to 228 years. This number is essentially fiction. It's a statistical calculation based on short-duration tests of many drives simultaneously, extrapolated to a single drive's expected lifetime under ideal conditions. Real-world failure rates look nothing like this.

Google published a study in 2007 analysing over 100,000 hard drives in their datacenters. The annual failure rate in the first year was 1.7%. By year three it climbed to 8.6%. For a drive that cost $100, you'd expect it to fail within 3 years with roughly 1-in-12 odds — worse than a coin flip over a 5-year period.

Backblaze, a cloud storage company, has published annual disk reliability reports since 2013, covering hundreds of thousands of consumer and enterprise drives. Their numbers are similar: 1–3% annual failure rates for most models, with some models spiking to 5–6% in their third year.

Now scale this up. A modest storage cluster — say, 10,000 drives — at a 2% annual failure rate loses 200 drives per year. That's almost 4 per week, or roughly one every two days. The drive failures never stop. They are as predictable and routine as anything in engineering. The question is never "will drives fail?" The question is "when a drive fails, what happens to the data on it?"

Disks are not the only failure mode. A machine has many ways to die:

A CPU can overheat if the cooling system degrades. RAM can flip bits — cosmic rays, which pass through the atmosphere constantly, occasionally flip a bit in a DRAM cell. Error-correcting memory (ECC RAM) detects and corrects single-bit flips, but multi-bit flips still cause crashes. Power supply units fail. Network interface cards fail. The motherboard itself can fail. The operating system kernel can crash due to a bug. A process can exhaust available memory and get killed by the OOM (Out-Of-Memory) killer. A runaway query can fill the disk. A bad configuration push can bring down a service. An engineer can type rm -rf /data in the wrong terminal.

That last category — operator error — is, at mature organisations, the leading cause of outages. Not hardware failure. Not software bugs. Humans making mistakes under pressure, usually during a deployment or incident response. The 2021 Facebook outage that took their entire global infrastructure offline for six hours was caused by a configuration change that, due to a bug in the tool used to apply it, accidentally disabled all of Facebook's border routers. No hardware failed. No software had a bug. One bad command, propagated everywhere simultaneously.

The point is not to catalogue every failure type. The point is to internalise that a machine is a system of components, each of which can fail independently, and the failure of any one of them can make the machine's data inaccessible or permanently lost. This is the physical reality you are building on.

Part 2: The Reliability Paradox at Scale

Here is the counterintuitive result that defines distributed systems engineering.

A single server with well-maintained hardware achieves perhaps 99.9% uptime — the "three nines" standard. That means it's unavailable for 8.76 hours per year. That sounds fine. Most applications could tolerate a few hours of downtime per year.

Now suppose your application runs on 10 such servers, and all 10 must be available simultaneously for the application to function (imagine a system where each server handles a different critical component). What is the probability that all 10 are up at any given moment?

P(all 10 up) = 0.999^10 = 0.990 = 99.0%

Expected downtime: 1% of 8,760 hours/year = 87.6 hours/year

You went from 8.76 hours of expected downtime per year to 87.6 hours — 10× worse — just by adding more machines. This is the reliability paradox: more components means more failure, not less, if all components must work simultaneously.

Push it further:

100 servers all needed:  0.999^100 = 0.905 = 90.5% uptime → 835 hours down/year (34 days)
1000 servers all needed: 0.999^1000 = 0.368 = 36.8% uptime → down more than you're up

A system that requires 1000 components to all be healthy simultaneously — even at 99.9% per component — is effectively always broken. This is not a hypothetical. A large web application with a complex deployment pipeline, dozens of microservices, databases, caches, queues, CDN, DNS, and monitoring systems has thousands of components. If the system required all of them to be healthy to serve any request, it would serve essentially no requests.

The engineering response to this is a fundamental shift in assumption: design systems that continue working when individual components fail, rather than designing systems that assume components won't fail. This shift has two parts.

First: replicate everything that can fail. Don't have one database — have multiple. Don't have one server — have multiple. Design so that losing any single component is a routine, recoverable event.

Second: accept that failures will always be happening somewhere in your system, and build so that a failure in component X does not cascade into failure in components Y and Z. Isolation and fault tolerance. We'll encounter these ideas repeatedly across the remaining chapters.

For now, focus on the first part: replication. If we're going to have multiple copies of our data, how do we do it, and what trade-offs does it introduce?

Part 3: What Replication Actually Is

The word "replication" sounds like a simple thing. Copy the data to another machine. But the mechanism matters enormously, because databases are not static snapshots — they are constantly changing. Every second, rows are being inserted, updated, and deleted. A replica must not just receive a copy of the data at one point in time; it must continuously receive and apply every change that happens on the primary.

The mechanism Postgres uses for this — and the mechanism we'll examine in detail in Chapter 5 — is the Write-Ahead Log, which you first encountered in Chapter 2. Recall that the WAL is a sequential file on disk where Postgres records every change before applying it. If the database crashes mid-write, it replays the WAL to recover. The key insight for replication is: the WAL is a complete, ordered record of everything that happened to the database. If you send that WAL to another machine and replay it there, you get an exact copy of the database.

This is physical replication in Postgres: the primary sends a stream of WAL records to the standby, and the standby continuously applies them. The standby's heap files, B-Tree pages — everything — end up in the same state as the primary, page by page.

The WAL stream can be continuous (streaming replication, milliseconds of lag) or batch-based (WAL shipping, minutes of lag). For any serious production system, streaming replication is the standard.

Now the critical question: when does the primary acknowledge the write to the client? Before or after the standby confirms it received and applied the WAL record? This single decision — the timing of the acknowledgment — is where the entire synchronous vs asynchronous trade-off lives.

Part 4: Synchronous Replication — Durability at a Cost

In synchronous replication, the primary writes the WAL record to its own disk AND waits for at least one standby to confirm it has received (and optionally applied) the record before sending the acknowledgment to the client.

The guarantee: if the primary crashes immediately after sending "success" to the client, at least one standby has the data. The write is never lost if any acknowledged write is considered durable. The data survived on the standby even if the primary's disk is now smoking rubble.

The cost: every write now pays the round-trip time to the standby, plus the standby's own write time. For standbys in the same datacenter (same-DC network round trip ~500 µs), this adds roughly 1–2 ms to every write. For standbys in a different region (cross-continental network ~150 ms), this adds 150 ms to every write — making synchronous cross-region replication effectively unusable for any latency-sensitive application.

But the more dangerous cost is availability coupling. If the standby is slow — say it's running a heavy compaction job — or the network between primary and standby degrades — say a switch is flapping — the primary must wait. Every write queues up behind the standby's acknowledgment. If the standby becomes completely unreachable, the primary either blocks all writes indefinitely (prioritising durability) or gives up on synchronous replication and switches to asynchronous temporarily (prioritising availability). In Postgres, synchronous_commit = on means the primary will block waiting for the standby. synchronous_commit = remote_apply waits for the standby to apply (not just receive) the WAL. Each option has different failure behaviour.

The failure mode to understand here is not the happy path. It's: what happens if the standby goes down? Your primary is now waiting for an acknowledgment that will never come. Every client waiting on a write is now blocked. This is availability coupled to replica health — one of the most important reasons synchronous replication is used sparingly, only for the most durability-critical writes.

Stripe, for example, uses synchronous replication for payment records — a charge being lost is unacceptable. The added latency per write is worth the durability guarantee. For less critical data — user preferences, view counts, analytics events — the trade-off tips the other way.

Part 5: Asynchronous Replication — Speed at a Cost

In asynchronous replication, the primary writes to its own WAL and immediately acknowledges success to the client. The WAL record is then sent to the standby in the background, independently of the client's write.

The client sees 1 ms write latency — just the local write, no network wait. The standby catches up almost immediately under normal conditions.

The guarantee — and this is crucial to state precisely: the write is durable on the primary. If the primary's disk survives, the write is safe. But if the primary crashes between t=1ms (client acknowledged) and t=2ms (standby receives WAL), the write exists on neither machine. The client was told "success." The data is gone.

This scenario — data loss on primary failure — is called replication lag-induced data loss, and it is the fundamental trade-off of asynchronous replication. Under normal operations, the standby is only milliseconds behind the primary, and the window of potential data loss is tiny. But it exists, and it is non-zero.

The question "how much data can I lose?" is answered by the RPO — Recovery Point Objective. For a system with asynchronous replication and a standby that is 100 ms behind, the RPO is 100 ms. If the primary dies, you lose at most 100 ms of writes. For most SaaS applications, losing 100 ms of writes is acceptable — annoying, but survivable. For a payment system where every write is a financial transaction, it is not.

Most managed database products use asynchronous replication by default (AWS RDS's read replicas, most single-region Postgres setups) and offer synchronous options at higher cost or lower throughput.

Part 6: Replication Lag — The Follower Is Always Behind

Even in asynchronous replication under normal conditions, the standby is never perfectly synchronised with the primary. It is always slightly behind. This gap is replication lag — the time between when a write happens on the primary and when it appears on the standby.

Under low write load, replication lag is typically 10–100 milliseconds. The primary sends WAL records, the standby applies them, and the gap stays small. But under high write load — say, a bulk import of 10 million rows — the primary generates WAL records faster than the standby can apply them, and lag grows. If the standby falls significantly behind and then the primary fails, the data loss window is not milliseconds but minutes or hours.

Replication lag has a subtler consequence that you will encounter building real SaaS products: reading from a replica means reading potentially stale data.

Suppose your application has one primary (for writes) and two replicas (for read traffic — "read replicas," a common pattern for scaling read-heavy applications). A user updates their subscription plan. That write goes to the primary. The primary acknowledges success. Now the user's browser immediately fetches their current subscription status — but the load balancer routes that read to a replica that hasn't yet received the update. The user sees their old plan. This is called the read-your-own-writes problem, and it's one of the most common consistency bugs in applications that use read replicas carelessly.

There's a second related problem: monotonic reads. Suppose the user makes two consecutive reads. The first read hits replica 1, which is 50 ms behind the primary and shows balance = 100. The second read hits replica 2, which is 200 ms behind and shows balance = 150 — an older value. The user just saw the balance go backwards. This is a violation of monotonic reads: once you've seen a value, you should never see an older one.

Both problems have practical solutions:

• Route reads that need freshness (the user's own data, immediately after a write) to the primary. Route reads that can tolerate staleness (public content, analytics, anyone else's data) to replicas.
• Or, use sticky sessions: route all reads from a given user to the same replica, so at least the same user sees consistent progress.
• Or, track replication lag and refuse to serve a read from a replica that is lagging more than your application's tolerance.

None of these solutions is perfect. All of them require the application to be aware that it's talking to a replicated system, not a single database. This is a theme that will repeat through the next several chapters: distributing data across machines forces consistency questions into your application layer. The database can no longer make all consistency decisions for you.

Part 7: Partial Failures — The Worst Kind

A machine that is completely dead is, paradoxically, easier to handle than one that is partially dead. A dead machine doesn't respond. Your monitoring detects the silence immediately. You promote a standby, redirect traffic, done.

The dangerous case is a machine that is alive but behaving incorrectly — a partial failure. Examples:

A machine whose disk is so full that new writes fail, but old reads still succeed. The machine responds to health checks (which only check network connectivity) but silently drops all writes.

A machine experiencing intermittent network issues — it can reach some machines but not others. From its perspective it's healthy. From the perspective of the machines it can't reach, it's dead. From the perspective of machines it can reach, it's fine.

A machine that is alive but so overloaded that it responds to requests after a 10-second delay. Is it up or down? Technically up. Functionally useless. Should you promote a standby? If you do, you now have two machines both believing they're the primary.

A machine that has a subtle RAM fault causing it to occasionally return incorrect data. No crash, no error — just wrong answers some of the time.

This last category — a machine that is alive, responds normally, but returns incorrect data — is called a Byzantine fault, named after the Byzantine Generals Problem in distributed systems theory. Byzantine faults are the hardest to handle. Most real-world systems (including Postgres replication) assume non-Byzantine failures: machines either work correctly or crash. They don't assume that a machine might actively lie. This assumption is reasonable for a closed datacenter where you control all the hardware. In adversarial environments (blockchain, some consensus protocols), Byzantine fault tolerance is required — but that's a different domain.

For your purposes: the failure modes that matter are crash-stop failures (machine crashes, stops responding) and network failures (machines can't reach each other even though both are running). Partial failures of the intermittent-network variety are what make distributed systems hard to debug and what motivate the complexity of the failure detection mechanisms in Chapter 5.

Part 8: Detecting Failure — The Timeout Problem

Before you can respond to a failure, you must detect it. And detecting failure turns out to be a deep problem with no perfect solution.

The naive approach is a heartbeat. Every N milliseconds, the primary sends a "I'm alive" message to a monitoring process. If the monitoring process doesn't receive a heartbeat for M milliseconds, it declares the primary dead and promotes a standby.

The problem: the heartbeat is a network message. The monitoring process not receiving the heartbeat could mean any of:

1. The primary has crashed. (The intended interpretation.)
2. The network between primary and monitoring process is down, but the primary is healthy.
3. The primary is alive but overloaded and couldn't send the heartbeat in time.
4. The monitoring process itself crashed.

You cannot distinguish between these cases from the monitoring process's perspective alone. A missing heartbeat is evidence of a problem, not proof of a specific problem. And the consequences of misidentifying the situation are severe.

If the primary is actually alive and you promote a standby, you now have two primaries: the old one (which is still serving writes from its perspective) and the new one (which just started serving writes). Both accept writes from clients. Their data diverges. When the network issue resolves, you have two conflicting histories. This is split-brain, and it is one of the most feared failure modes in database replication. Recovering from split-brain requires manually merging the divergent histories — an expensive, error-prone, often lossy operation.

The standard mitigation is to use conservative timeouts — wait longer before declaring a primary dead, to reduce false positives. But longer timeouts mean longer unavailability during real failures. This is an irresolvable tension: you cannot distinguish a dead primary from a temporarily-unreachable one without the ability to communicate with it, and if you could communicate with it, you wouldn't need to declare it dead.

More sophisticated approaches use consensus protocols — systems where a majority of nodes must agree before any significant decision is made. These are the subject of Chapters 11 and 12. For now, the key intuition is: split-brain happens when two parts of a system make independent decisions about who is in charge. Preventing it requires a mechanism that ensures only one part can make that decision at a time.

Practical tools like Patroni (for Postgres) and Galera (for MySQL) implement watchdog processes, fencing (actively preventing the old primary from accepting writes, using techniques like cutting off its network or STONITH — "Shoot The Other Node In The Head"), and consensus-based leader election to handle failover more robustly. Managed databases (RDS Multi-AZ, Google Cloud SQL with high availability) handle all of this for you — but you need to understand the mechanism to reason about the guarantees they provide and the failure modes they don't eliminate.

Part 9: The Durability Spectrum — Placing Your Data on It

Every piece of data in your system sits somewhere on a spectrum between "if the primary crashes, this data is definitely lost" and "this data cannot be lost even if we lose an entire datacenter." Your job as an engineer is to consciously place each piece of data where it belongs.

In-memory only: fastest writes, zero durability. Data disappears on crash. Use for session caches, rate limit counters, feature flag states — anything you can reconstruct.

Async single replica: fast writes, small durability window (milliseconds of potential loss). Use for most application data where losing the last few milliseconds of writes is survivable.

Sync single replica: slightly slower writes, guaranteed no acknowledged write is lost on primary failure. Use for financial transactions, audit logs, anything where data loss is unacceptable.

Sync multi-replica (quorum): writes must be acknowledged by a majority of replicas. Survives failure of a minority of nodes. Use for the most critical data in high-availability systems.

Multi-region sync: writes propagate across geographic regions before acknowledging. Survives entire datacenter failure. Extremely high latency (cross-region round trips). Use for truly critical data where geographic failure is a concern.

Most SaaS applications live primarily in the async single replica tier, with specific components (payment records, audit trails) in sync single replica. Understanding the spectrum prevents over-engineering (sync multi-region replication for a user's theme preference) and under-engineering (async replication for payment records).

Part 10: What This Means for Your Application

You're building a SaaS product. You're probably using a managed database — RDS, Supabase, PlanetScale, Neon, or similar. These services handle replication for you. What do you need to know?

Connection failover. When a primary fails and a standby is promoted, there is a brief period — typically 30–60 seconds for managed services — where no primary is available. Your application's database connection pool needs to handle this gracefully: retry failed connections with exponential backoff, surface appropriate errors to users rather than hanging indefinitely. If your connection pool just fails fast on a connection error and you have no retry logic, a 60-second failover becomes a 60-second outage for users — all of them.

Read replicas are eventually consistent. If you add read replicas (to scale read traffic), never route writes to them and never assume they're up-to-date. Route the user's own data reads to the primary after a write. Only route clearly-stale-tolerant reads (public content, other users' data) to replicas.

RPO vs RTO. These are two metrics that define your recovery posture.

RPO (Recovery Point Objective): how much data can you afford to lose? If you use async replication with typical 100 ms lag, your RPO is ~100 ms. If you use synchronous replication, your RPO is zero (for acknowledged writes).

RTO (Recovery Time Objective): how quickly must the system be available after a failure? An automatic failover to a hot standby has RTO of ~30–60 seconds. A restore from backup has RTO of hours. For a SaaS product where downtime directly costs revenue, you need both to be clearly defined and you need your infrastructure to meet them.

Managed services publish their RTO and RPO guarantees (e.g., "RDS Multi-AZ provides RPO of ~1 second and RTO of ~60 seconds for most failure scenarios"). Read those guarantees. Know whether they match what your customers expect from you.

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

Your SaaS application's database is a single Postgres instance with no replication. The database runs on a server with the following characteristics: NVMe SSD with 1% annual failure rate, 32 GB ECC RAM, redundant power supply, single network interface card (NIC) with 0.5% annual failure rate.

Calculate the annual probability that the server is unavailable due to hardware failure (assume failures are independent). Then: your application has 50,000 paying customers and earns $20 per customer per month. If a disk failure causes 12 hours of downtime while you restore from backup, what is the estimated revenue impact? At what monthly revenue does it become economically rational to invest in a hot standby with automatic failover (which reduces downtime from 12 hours to 60 seconds)?

Problem 2 — Design from Scratch

You are designing the replication strategy for a fintech SaaS product with three types of data:

• Payment transactions (user charges $X, ledger updated)
• User profile data (name, email, preferences)
• Analytics events (page views, button clicks, feature usage)

For each data type: choose synchronous or asynchronous replication and justify the choice in terms of RPO, write latency impact, and the consequences of data loss. Then: design where each type's reads should be served from (primary or replica) and why. Calculate the write latency added by your synchronous choice, assuming same-datacenter replicas (500 µs round trip).

Problem 3 — Synthesis

Connect replication lag, the read-your-own-writes problem, and your SaaS application's user experience.

A user upgrades from the Free to the Pro plan. Your application writes the new subscription status to the primary. The primary acknowledges. Your application immediately redirects the user to the dashboard, which makes an API call to fetch their subscription status — but this API call hits a read replica that is 800 ms behind the primary.

Trace exactly what the user sees. Then design a solution. Your solution must work across a stateless API layer (no sticky sessions available — each API call can hit any server). You cannot route all reads to the primary (too much load). What is the minimum change to your application that guarantees the user sees their Pro status immediately after upgrading, without overloading the primary?

Problem 4 — Go Deeper

The split-brain problem is described as occurring when "two machines both believe they are the primary." But consider this more carefully: what is it that makes one machine the primary and another a standby?

In a single-machine database, "primary" is trivial — there's only one machine. In a replicated system, "primary" is a role that must be assigned, tracked, and enforced across the network. The assignment of this role is itself a distributed systems problem.

Reason through this: if you have a primary and a standby, and the network between them fails (but both are still running and reachable by some clients), what should each machine do? Build the argument for why neither machine can unilaterally decide whether the other is dead. What information would you need to make a safe decision? What would a system that makes the safe decision look like?

(This problem is foreshadowing Chapters 11 and 12 on consensus. Don't look those up yet — reason from first principles. The goal is to discover the problem, not solve it.)

Problem 5 — Connect to Previous Chapters

In Chapter 2, you learned that Postgres uses a Write-Ahead Log (WAL) for crash recovery: changes are written to the WAL before being applied to the heap, so the database can replay the WAL after a crash.

In Chapter 4, you've learned that replication works by shipping the WAL to standbys.

Connect these two: if a primary crashes and its heap files are corrupted, but its WAL file is intact, can you recover? What if the WAL file is also corrupted, but a standby has been receiving the WAL stream and is only 200 ms behind? Walk through the physical recovery sequence — what does the standby have, what does it need to become the primary, and is there anything a 200 ms lag means you might not be able to recover?

What's Coming Next

You now understand why replication exists and the fundamental trade-off at its core. Chapter 5 goes inside the mechanism. You'll see exactly how Postgres streaming replication works: how WAL records flow from primary to standby, what a "replication slot" is and why it exists, what happens when you promote a standby to primary, and what the specific failure modes look like at the WAL level — including the exact moment at which a write is "safe" and what makes it unsafe. The abstract concept of "the standby replays the WAL" will become a concrete picture you can hold in your head during a 3am incident.

Chapter 4 complete. The most important problem here is Problem 4 — sit with it before looking at anything about Raft or Paxos. The reasoning you do in isolation will make Chapters 11 and 12 click immediately instead of feeling like new information.