Chapter 11: The Consensus Problem — Why Agreement Is Hard

Curriculum: Distributed Systems from First Principles Chapter: 11 of 24 Part: 5 — Consensus Prerequisites: Chapter 4 (failure taxonomy), Chapter 5 (leader election, split-brain), Chapter 6 (consistency models, CAP theorem) Estimated reading time: 75–85 minutes

Opening

Every hard problem in distributed systems eventually reduces to the same question: how do multiple independent machines agree on a single fact? In Chapter 5, you saw that when a primary database fails, some process must elect a new leader — but who decides? If two nodes simultaneously believe they are the leader (split-brain), data gets corrupted. In Chapter 6, you saw that strong consistency (linearizability) requires reads to see the latest write — but if nodes can't agree on which write was latest, there is no "latest." In Chapter 7, you saw that distributed transactions require all shards to atomically commit or abort — but if one shard crashes mid-transaction, the others don't know whether to commit. All three problems — leader election, linearizable reads, distributed transactions — are instances of the same fundamental problem: distributed consensus. Getting N machines to agree on one value.

This chapter is about why that problem is hard. Not "hard to implement correctly" in the way that any complex code is hard. Hard in a mathematical, provably-unavoidable sense. Certain things are impossible, and understanding why they're impossible tells you exactly what assumptions you must make to get past them — assumptions that every real consensus algorithm (Paxos, Raft, Zab) makes, which is why they look the way they do. Chapter 12 then builds Raft from scratch on top of exactly those assumptions.

Part 1: What Consensus Actually Is — The Precise Definition

Before understanding why consensus is hard, you need a precise definition of what it requires. "Agreement" is vague. Consensus has three specific properties that any correct algorithm must satisfy.

A consensus algorithm operates over a set of N nodes, each of which proposes a value. The algorithm must ensure:

Agreement (Safety): No two nodes decide on different values. If Node A decides "value=42," then every other node that decides must also decide "value=42." No divergence.

Validity: The decided value must be a value that was actually proposed by some node. The algorithm cannot invent values out of thin air. If the only proposed values are 42 and 77, the decided value must be either 42 or 77.

Termination (Liveness): Every non-faulty node eventually decides on a value. The algorithm doesn't run forever without reaching a conclusion.

These three properties seem completely reasonable. Of course you want agreement. Of course the value should be something someone proposed. Of course you want the algorithm to finish. The remarkable result is that satisfying all three simultaneously is impossible under certain conditions — and understanding those conditions is the entire point of this chapter.

The two properties that create the fundamental tension are agreement (safety) and termination (liveness). It turns out that in an asynchronous network where nodes can fail, you cannot guarantee both simultaneously. You must sacrifice one. Which one you sacrifice, and under what conditions, defines the character of every real distributed consensus algorithm.

Part 2: The Two Generals Problem — Why Unreliable Channels Break Everything

Start with the simplest possible version of the agreement problem. Two armies, separated by a valley, need to coordinate an attack. They can only communicate by sending messengers through the valley. A messenger might be captured (message lost). They need to agree: attack at dawn, or don't attack. If only one army attacks, it loses.

Can they guarantee agreement with unreliable message delivery?

General A sends: "Attack at dawn. Confirm." General B receives it and wants to confirm. General B sends: "Confirmed." But General B doesn't know if the confirmation arrived. If it didn't, General A won't attack. So General B sends again: "Confirmed, please acknowledge." But now General A doesn't know if that message arrived. And so on.

No matter how many messages they exchange, neither general can ever be certain the last message was received. There is always one final unconfirmed message. The protocol can never terminate with guaranteed certainty on both sides.

This is not a solvable problem with unreliable communication. It's been proven formally. The Two Generals Problem demonstrates that consensus over a lossy channel is impossible — not just hard to implement, but mathematically provably impossible.

Why does this matter for distributed systems? TCP solves reliable message delivery between two hosts — but TCP itself relies on the same two-generals problem at the IP layer, which is why TCP connections can fail silently (a half-open connection). More importantly: the internet is not a reliable channel at the application level. Packets are dropped, servers crash mid-send, processes are killed. Your distributed database nodes communicate exactly like the two generals.

The practical escape hatch that real systems use: timeouts plus retries plus idempotency. TCP doesn't guarantee delivery but it retries until acknowledged or the connection is declared dead. Distributed systems similarly use timeouts to detect likely failure and retries with idempotent operations to recover. The insight is that you don't need mathematical certainty — you need sufficient confidence that the system can make forward progress most of the time.

Part 3: FLP Impossibility — The Most Important Theorem You'll Never Implement

In 1985, Fischer, Lynch, and Paterson published a result that shook the distributed systems community: it is impossible to design a deterministic consensus algorithm that is both safe (always reaches the right decision) and live (always terminates) in an asynchronous system where even one node can crash.

This is the FLP impossibility result. It's worth understanding precisely, because it's often misunderstood.

The setting: N nodes communicate by message passing. There are no clocks — you can't use timeouts, because in an asynchronous system, a slow node is indistinguishable from a crashed node. Messages are eventually delivered but can be delayed arbitrarily. One node can fail by crashing (stopping forever).

The result: in this model, no deterministic algorithm can guarantee that all non-faulty nodes always decide, while also guaranteeing that they never disagree. At any point during the algorithm's execution, there exists some sequence of events (delays, reorderings) that will prevent the algorithm from ever reaching a decision.

The proof sketch: define a "bivalent" configuration as one where the outcome is still undecided — the algorithm could still decide either 0 or 1. The proof shows that from any bivalent configuration, there exists a way to keep the algorithm bivalent forever — by carefully delaying messages so that the deciding step is perpetually postponed. The adversary (the network) can always do this in a fully asynchronous model.

What FLP does NOT say: it does not say consensus is impossible in practice. It says it's impossible in a fully asynchronous model. Real systems are not fully asynchronous — they have approximate clocks, they make timing assumptions, they use randomness. FLP is a bound on what is achievable in the worst-case theoretical model.

The practical implication is the key insight: every real consensus algorithm escapes FLP by adding an assumption that the theoretical model doesn't allow. Specifically:

Paxos and Raft escape FLP by adding timeouts — they assume the network is eventually synchronous (messages eventually arrive within some time bound, even if that bound is unknown). With timeouts, you can distinguish "slow node" from "crashed node" with sufficient probability.

Randomized consensus algorithms (like Ben-Or's algorithm) escape FLP by adding randomness — a node that's stuck between choices flips a coin. Random decisions break the adversary's ability to perpetually maintain a bivalent state.

The assumption that Paxos, Raft, and Zab make is partial synchrony: the network is sometimes asynchronous (periods of high latency, packet loss), but eventually becomes synchronous (eventually, messages are delivered within some bounded time). During asynchronous periods, the algorithm may pause. During synchronous periods, it makes progress. This is the realistic model for the internet: sometimes things are slow or broken, but not forever.

Part 4: Safety vs Liveness — The Fundamental Trade-off

FLP forces every consensus algorithm to make a choice: when things go wrong (network partitions, node failures, message delays), do you prioritise safety or liveness?

Safety: Never make a wrong decision. If you can't guarantee correctness, do nothing — wait, block, refuse to proceed. A safe system may halt, but it never corrupts.

Liveness: Always make progress. Keep deciding, keep moving forward, even if you might occasionally be wrong. A live system may make mistakes, but it never stalls forever.

These are not just abstract properties — they map directly to the observable behaviour of your system under failure.

A safety-prioritising consensus algorithm (Raft, Paxos) will stop accepting new writes during a network partition where it can't achieve quorum. Your application gets errors. Users can't complete actions. Dashboards show failure. But the data already committed is guaranteed correct — no corruption, no split-brain. When the partition heals, the algorithm resumes. You had an availability outage, not a correctness outage.

A liveness-prioritising system (some eventually-consistent databases in certain configurations) will keep accepting writes on both sides of a partition. Users can complete actions. Dashboards look normal. But when the partition heals, the two sides have conflicting histories that must be merged — potentially with lost updates, conflicting values, or application-level inconsistencies. You avoided an availability outage but paid with a correctness problem.

This maps directly to Chapter 6's CAP theorem: during a partition, choose consistency (safety, stop accepting writes) or availability (liveness, accept writes and reconcile later). CAP is not a theorem about distributed systems in general — it is specifically about the safety/liveness trade-off under network partitions in the consensus context.

Network partition occurs:
          
Side A (3 nodes)    │    Side B (2 nodes)
                    │
Safety system:      │
  → Side A: quorum, │    → Side B: no quorum,
    accept writes   │      REFUSE writes
  (Consistency ✓,   │      Availability ✗)
                    │
Liveness system:    │
  → Side A: accept  │    → Side B: also accept
    writes          │      writes
  (Availability ✓,  │      Consistency ✗ — 
    diverged!)      │      data will conflict)

Raft makes the safety choice. Cassandra (at consistency level ONE) makes the liveness choice. Neither is wrong — the correct choice depends on whether your application tolerates availability outages or data inconsistencies.

Part 5: The Quorum — Why Majority Is the Magic Number

Every fault-tolerant consensus algorithm uses quorums. A quorum is the minimum number of nodes that must participate in an operation for it to be considered valid. For a cluster of N nodes, the standard quorum is floor(N/2) + 1 — a strict majority.

Why majority? The answer is in the intersection property: any two majorities of N nodes must share at least one node in common.

N = 5 nodes: {A, B, C, D, E}
Majority quorum = 3

Write quorum: {A, B, C}
Read quorum:  {C, D, E}
Intersection: {C}   ← at least one node always in both

This intersection is what guarantees consistency. When a write is acknowledged by a quorum, at least one node in any subsequent read quorum has seen that write. The read is guaranteed to return the latest write, because the latest write's quorum and the read's quorum overlap.

If you used a quorum smaller than majority, you could have two write quorums with no overlap — two writes both "succeeded" on completely disjoint sets of nodes, with no way to reconcile which was later or authoritative. That's the split-brain.

BAD: quorum = 2 (less than majority for N=5)
Write 1 quorum: {A, B}
Write 2 quorum: {C, D}
No intersection! Both writes "succeeded" on different nodes.
Nobody knows which is authoritative. Split-brain.

GOOD: quorum = 3 (majority for N=5)
Write 1 quorum: {A, B, C}
Write 2 quorum: ??? — any set of 3 must share at least one with {A, B, C}
Write 2 cannot succeed without touching a node that knows about Write 1.

The quorum size is also the fault tolerance: a cluster of N nodes with quorum floor(N/2) + 1 can tolerate floor(N/2) failures. A 5-node cluster tolerates 2 failures (quorum of 3). A 3-node cluster tolerates 1 failure (quorum of 2). A 7-node cluster tolerates 3 failures (quorum of 4).

This is why consensus clusters are always deployed with odd numbers of nodes. An even number wastes fault tolerance: a 4-node cluster has quorum 3 and tolerates only 1 failure — the same as a 3-node cluster but with one more machine to manage and pay for.

N=3: quorum=2, tolerates 1 failure. Cost: 3 machines.
N=4: quorum=3, tolerates 1 failure. Cost: 4 machines. ← WASTEFUL
N=5: quorum=3, tolerates 2 failures. Cost: 5 machines.
N=6: quorum=4, tolerates 2 failures. Cost: 6 machines. ← WASTEFUL
N=7: quorum=4, tolerates 3 failures. Cost: 7 machines.

Odd numbers always give you more fault tolerance per machine. This is why etcd clusters are 3 or 5 nodes, almost never 4 or 6.

Here is the quorum rule at work in a 5-node cluster. Crash two followers and writes still commit; crash a third and the cluster chooses safety and stops committing.

Part 6: Paxos — The Original Algorithm and Its Infamous Complexity

In 1989 (published 1998), Leslie Lamport described Paxos — the first correct, practical consensus algorithm. It's worth understanding Paxos at a conceptual level because it reveals why consensus requires the structure it does, even if you'll implement Raft (which Lamport himself calls "easier to understand").

Paxos operates in two phases.

Phase 1 (Prepare/Promise): A node that wants to become the leader (called a "proposer") sends a Prepare(n) message to a quorum of nodes, where n is a proposal number higher than any it has used before. Each node that receives this message (an "acceptor") promises not to accept any proposal numbered less than n, and reports back the highest-numbered proposal it has already accepted (if any).

Phase 2 (Accept/Accepted): If the proposer receives promises from a quorum, it sends an Accept(n, v) message with its proposal number n and value v. The value v must be the value from the highest-numbered accepted proposal reported in Phase 1 (if any acceptor already accepted a value); otherwise the proposer can choose any value. Each acceptor accepts the proposal unless it has promised a higher number.

Why does the proposer have to use the previously accepted value rather than its own? This is the safety mechanism. If some earlier round had already achieved consensus on 42 (at least a quorum accepted it), the new proposer must preserve that decision. By requiring the new proposer to continue with any previously-accepted value, Paxos ensures that a value once decided is never changed.

This is the core insight of Paxos: the second phase's value constraint is what preserves safety. Any concurrent proposer who might have achieved consensus in an earlier round is "absorbed" into the new round via Phase 1's promise mechanism.

What makes Paxos notoriously difficult: the paper describes single-value consensus (agreeing on one value once). Real systems need multi-Paxos — reaching consensus on a sequence of values (a log of operations). The paper doesn't describe Multi-Paxos completely. Different teams implemented it differently. Subtle corner cases around leader changes, message reordering, and log gaps make correct implementation extremely difficult. Chubby (Google's distributed lock service, the predecessor to Zookeeper) is built on Paxos, and the team wrote a famous paper, "Paxos Made Live," describing all the engineering challenges they encountered implementing it in practice. The paper is basically a catalogue of all the ways Paxos is underspecified.

This incompleteness of the Paxos specification is why Diego Ongaro and John Ousterhout designed Raft in 2013, explicitly optimised for understandability. Chapter 12 is entirely Raft.

Part 7: What a Consensus Algorithm Must Actually Do — Decomposing the Problem

Paxos and Raft are both solving the same problem, but it's worth being precise about what "consensus in a real system" means, because it's more than just agreeing on one value.

Real systems need replicated state machine consensus: all N nodes must execute the same sequence of operations in the same order, so that their state (database contents, lock state, configuration) remains identical. This is called the replicated log — a sequence of entries that all nodes apply in order.

Replicated log (3-node cluster):

Node A: [op1: SET x=1] [op2: SET y=2] [op3: DEL z] ...
Node B: [op1: SET x=1] [op2: SET y=2] [op3: DEL z] ...
Node C: [op1: SET x=1] [op2: SET y=2] [op3: DEL z] ...

All nodes apply same operations in same order → same state

For a replicated log, consensus requires solving several sub-problems simultaneously:

Leader election: One node must be designated as the coordinator for a given term — the one that proposes new log entries. If the leader crashes, a new election must occur and a new leader must be chosen. The new leader must not lose any entries that the old leader had committed.

Log replication: The leader receives client requests, appends them to its log, and replicates them to followers. An entry is "committed" once a quorum of nodes has it in their log — at that point, it can be applied to the state machine.

Safety under leader change: When a new leader is elected, it must not overwrite committed entries. Its log must be at least as up-to-date as a quorum of other nodes — ensuring it has all committed entries.

Log consistency: At any given log index, all committed entries must be identical across all nodes. You can never have a situation where Node A's entry at index 17 is "SET x=1" and Node B's entry at index 17 is "SET x=2."

Membership changes: Adding or removing nodes from the cluster while it's running, without a window where two independent quorums can simultaneously make decisions (the dreaded "split quorum" problem during transition).

These sub-problems are what Raft fully specifies. Paxos partially specifies them, leaving the rest to implementers. The resulting Paxos implementations at Google, Apache, etc. all look different from each other in their details — a sign that the specification was incomplete.

Part 8: The Leader Bottleneck and the Performance Cost of Consensus

Consensus has a real performance cost. Every operation that needs consensus (a write to a linearizable database, a leader election, a lock acquisition) must:

1. Go to the leader node
2. Be appended to the leader's log
3. Be replicated to a quorum of followers (one network round trip to each)
4. Wait for quorum acknowledgement
5. Be committed and applied
6. Response returned to client

Steps 3-4 involve waiting for a network round trip. From Chapter 1: a network round trip within a datacenter takes about 0.5 ms. A network round trip to another datacenter takes 10–100 ms. Consensus throughput is therefore bounded by: throughput = quorum_latency / per_operation_overhead. If each consensus round trip takes 1 ms and operations can be pipelined, a well-implemented leader can commit 10,000–100,000 operations per second. If the quorum spans datacenters (geo-replication), that drops to hundreds per second.

This is the consensus tax: you pay a minimum of one network round trip per committed write, plus the coordination overhead. You cannot make this faster by adding more nodes — adding nodes makes it slightly slower (larger quorums have higher latency to reach) and doesn't improve the fundamental single-leader bottleneck.

The leader bottleneck is inherent. All writes go to one node — the leader — so write throughput is bounded by one machine's capacity. This is why consensus-based systems (etcd, ZooKeeper, Consul) are not designed for high write throughput. They're designed for correctness and coordination: storing configuration, holding locks, electing leaders, maintaining small amounts of critical state. The metadata of distributed systems, not the data itself.

The practical implication: do not use etcd or ZooKeeper as your primary database. Use them to store configuration, cluster membership, leader identity, and distributed locks — things that change infrequently and require absolute correctness. Store your actual data in a scalable database (sharded Postgres, Cassandra, etc.) that doesn't pay the consensus tax on every write.

Use consensus (etcd, ZooKeeper) for:
  - Which node is the current database leader?
  - Is the distributed lock held, and by whom?
  - What is the current cluster membership?
  - What is the current configuration version?

Use regular databases for:
  - User records (billions of rows)
  - Events and logs (high write throughput)
  - Product catalog, orders, sessions

Part 9: Practical Consensus — The Assumptions Real Systems Make

You've seen that FLP makes deterministic consensus impossible in a fully asynchronous system. Real systems escape FLP with one or more of these assumptions:

Partial synchrony: The network is eventually synchronous — there exists a time after which all messages are delivered within some time bound Δ. We don't know what Δ is or when the synchronous period begins, but it eventually exists. In practice: the internet is usually within 50 ms latency; outliers exist but are not permanent. Raft and Paxos both require partial synchrony to elect leaders reliably.

Failure detector: A component that provides hints about which nodes are suspected to have failed. An eventually-perfect failure detector is one that eventually stops suspecting correct nodes and eventually suspects all failed nodes. In practice: timeout-based heartbeats. If Node B doesn't respond to Node A's heartbeat within 150 ms, Node A suspects Node B has failed. This might be wrong (Node B is just slow) — hence "suspected" rather than "known failed."

Leader stability: When the leader is healthy and the network is stable, progress is fast. Consensus slows down only during elections (leader failures). In practice: leaders stay elected for minutes or hours. The election overhead is amortised over thousands of committed entries.

Randomised timeouts: To prevent multiple nodes from simultaneously trying to become leader (a "split vote" where no candidate achieves quorum), nodes wait a random amount of time before starting an election. Randomness breaks the symmetry that would otherwise cause repeated failed elections. Raft uses this explicitly.

These assumptions are realistic for production systems in a single datacenter or well-connected datacenters. They break down in: extreme network partitions lasting hours, Byzantine failures (nodes behaving arbitrarily, not just crashing), or adversarial network conditions. For those scenarios, more complex protocols (PBFT, Tendermint for blockchain) are needed.

Part 10: The Role of Consensus in Real Architectures — Where It Lives

Now that you understand consensus and its cost, you can see the architectural pattern used by almost all serious distributed systems: use consensus for coordination, not for data.

The pattern: one small cluster (3–5 nodes) running a consensus protocol manages the metadata and coordination state. A larger cluster of data nodes manages the actual data, coordinating via the consensus cluster.

Kubernetes uses etcd as its consensus cluster — all cluster state (Pod definitions, Service configs, Secrets) lives in etcd. The actual workloads (containers) run on data plane nodes that are not using consensus for every operation. Kafka uses ZooKeeper (historically) or KRaft (its own embedded Raft, since Kafka 2.8) for broker metadata, topic partition leadership, and controller election — not for the actual message storage, which is a replicated log on each broker.

This separation means: the consensus cluster needs to be correct and available, but not high-throughput. The data layer needs to be high-throughput, but can use lighter-weight consistency mechanisms (replication with a clear leader, rather than full Paxos/Raft on every write).

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

You are running a 5-node etcd cluster across two datacenters: 3 nodes in DC-West and 2 nodes in DC-East. The WAN link between datacenters goes down for 30 seconds (network partition).

Trace exactly what happens: which partition can achieve quorum? Which cannot? What does the partition that can't achieve quorum do when a write request arrives? When the link comes back up, what does the cluster do to reconcile? Now consider: a human operator, seeing that DC-East is "unavailable," manually forces DC-East to form its own cluster (a common mistake under pressure). What has just been created? What is the data state when the WAN link is restored and both sides try to rejoin? What is the correct operational procedure instead of forcing DC-East to form its own cluster?

Problem 2 — Design from Scratch

Design the consensus layer for a distributed database primary election system. Requirements: a Postgres primary + 2 replicas. If the primary fails, a new primary must be elected within 10 seconds. The new primary must have all committed transactions from the old primary. Once elected, the new primary must prevent the old (possibly recovered) primary from accepting writes (fencing).

Specify: how many nodes in your consensus cluster? Where does it run (same machines as Postgres, or separate)? What information does the consensus cluster store? How does a replica contact the consensus cluster to propose itself as leader? How does the consensus cluster ensure the most up-to-date replica (highest LSN from Chapter 5) wins the election rather than just the first to ask? How does the consensus cluster revoke the old leader's ability to write (STONITH/fencing from Chapter 4)?

Problem 3 — Synthesis

Connect the quorum intersection property to Chapter 6's consistency models.

A system uses read quorum = write quorum = majority (standard Raft/Paxos). Argue from the intersection property why this configuration provides linearizability: every read must see the most recent write. Now weaken the quorums: write quorum = N (all nodes must acknowledge), read quorum = 1 (any one node). Does the intersection property still hold? What consistency level does this provide? Now: write quorum = 1 (one node acknowledges), read quorum = N (all nodes must respond). Does the intersection property hold? What is the practical problem with this configuration under node failure?

Finally: Cassandra allows W + R > N configurations (e.g., W=2, R=2, N=3). Does this guarantee linearizability? Explain why or why not by tracing a concurrent write and read through the intersection property — specifically, what happens if the write completes to nodes {A, B} and the read goes to nodes {B, C}?

Problem 4 — Go Deeper

The FLP impossibility result says consensus is impossible in a fully asynchronous system. Randomized algorithms escape FLP — how?

Ben-Or's randomized consensus algorithm works roughly as follows: in each round, a node either reports its proposed value (if it has seen a quorum agree) or flips a coin to choose a value if undecided. The coin flip has probability 1/2 of choosing 0 or 1.

From first principles: why does randomness break the FLP impossibility argument? (Hint: the FLP proof relies on the adversary being able to arrange message delivery to keep the system in a bivalent state. What property of the adversary does randomness defeat?) What is the expected number of rounds before consensus is reached? What happens to this algorithm under Byzantine failures (nodes that lie or behave arbitrarily, not just crash)? How does Raft's randomized election timeout use the same insight as Ben-Or's coin flip, even though Raft is not a fully randomized algorithm?

Problem 5 — Connect to Previous Chapters

Connect consensus to Chapter 5's WAL replication and Chapter 7's distributed transactions.

In Chapter 5, Postgres streaming replication uses a primary that sends WAL records to replicas. This is not consensus — there is no quorum, no leader election protocol, no guarantee that the primary has the latest data after a failover. It relies on external tooling (Patroni, which uses etcd) to manage elections.

Part A: Trace exactly what Patroni does during a primary failure that uses consensus. What information does Patroni store in etcd? How does Patroni use etcd's consensus to ensure only one replica is promoted? Why can't the replicas elect a new leader among themselves without etcd?

Part B: Chapter 7 mentioned that distributed transactions across shards require all shards to atomically commit or abort — two-phase commit (2PC). 2PC is not consensus — why not? (It does not satisfy the termination property: if the coordinator crashes after sending PREPARE but before sending COMMIT, participants block forever.) How would you modify 2PC to be fault-tolerant by replacing the coordinator with a consensus cluster? What is the resulting protocol called (research: Paxos Commit, or 3PC)?

What's Coming Next

You now understand why consensus is hard, what properties it must satisfy, how quorums provide safety, and why every real consensus algorithm is an escape from FLP via specific assumptions. Chapter 12 builds Raft from scratch — the algorithm that provides fully-specified, understandable replicated log consensus. After reading the academic description of Paxos (and its notorious incompleteness), Raft will feel like a relief: every design decision is explained, every edge case is specified, and the algorithm can be implemented correctly from the paper alone. Raft is used in etcd (Kubernetes' brain), CockroachDB, TiKV, Consul, and dozens of other production systems. After Chapter 12, you'll be able to read their source code and recognise exactly what each piece is doing.

Chapter 11 complete. This chapter has no code and no implementation — it is pure reasoning. The most important things to internalise before Chapter 12: the safety/liveness trade-off (Raft sacrifices liveness during partitions to preserve safety), the quorum intersection property (why majority is magic), and FLP's practical implication (timeouts + partial synchrony are not bugs in Raft's design, they're the intentional escape hatch from impossibility). Problem 3 (working through quorum math for different W/R configurations) and Problem 5B (why 2PC is not consensus) are the conceptual anchors — they appear verbatim in distributed systems interviews.