Leader Election
Exactly one node in charge, even when nodes pause, crash and lose touch. "Exactly" is the hard part.
The story
A nightly job emails each customer their monthly statement. It runs from cron on the app servers. When the team scales from one app server to five, every customer gets five statements.
The quick fix is a lock: before running, each server tries to SET statement-job-lock in Redis with a 60-second expiry, and only the one that gets it does the work. It works for months.
Then one night the server holding the lock has a long garbage-collection pause, about 70 seconds. Its lock expires during the pause. A second server takes the lock and starts sending. The first server wakes up, has no idea it was paused, believes it still holds the lock, and carries on sending too.
Martin Kleppmann described exactly this failure in his well-known 2016 critique of Redis-based locking: a lock with an expiry can't stop a paused process from acting after its lock has expired. The holder can't tell it was paused. The fix has to be on the receiving side.
The idea
Some work must be done by exactly one node at a time:
- Accepting writes for a database (the primary)
- Running scheduled jobs
- Assigning partitions to consumers (Kafka)
- Holding a lock on a resource (a migration, a file)
Leader election chooses that node, notices when it dies, and chooses a new one. The hard part is that from the outside, a dead leader and a slow or cut-off leader look the same.
Leases, not permanent locks
The leader holds a lease: leadership for a limited time, renewed by heartbeats. If the leader stops renewing, the lease expires and someone else can take over.
The lease must live in something that is itself consistent and fault-tolerant: etcd, ZooKeeper, Consul (all built on consensus), or your database's own locking. Kubernetes uses Lease objects in etcd for exactly this. A single Redis node isn't enough when correctness matters.
Fencing tokens: the fix for the story
Every time the lease is granted, it comes with a number that only goes up. The leader sends it with every action, and the resource being acted on rejects any number lower than the highest it has already seen.
That's the key point: the leader can't be trusted to know it's still the leader, so the system it acts on checks for it. In a database this can be as simple as UPDATE … WHERE fencing_token < $new_token.
How consensus does it: Raft
Databases and coordination services elect leaders among themselves with a consensus algorithm, usually Raft:
- Majority vote: a node needs votes from more than half the cluster, so two leaders can't be elected in the same term. With 5 nodes, 3 are needed, and the cluster survives 2 failures.
- Terms act as fencing tokens: a message from an old term is ignored.
- Random timeouts make it unlikely that several nodes stand for election at once and split the vote.
- A minority side of a partition can't elect anyone, so it can't accept writes. That's the CP choice from CAP Theorem.
Cheat sheet
| Question | Answer |
|---|---|
| Need one cron runner? | Lease in your DB (Postgres advisory lock) or etcd/Consul. Or a scheduler that runs it once. |
| Where the lease lives | etcd, ZooKeeper, Consul, or the DB, not a single Redis for correctness-critical work |
| Lease length | Several heartbeat intervals (e.g. 10s with 3s renewals) |
| Paused / partitioned old leader | Fencing token checked by the resource |
| Cluster size | 3 or 5 (odd). 5 survives 2 failures. |
| Minority partition | Can't elect, can't write. By design. |
| Split vote | Random election timeouts |
| Do you need it at all? | Not if the work is idempotent and can safely run twice, or can be partitioned across all nodes |
| Kafka partitions | The consumer group coordinator assigns them. You don't elect. |
Say it in the interview
"The statement job takes a lease in etcd with a 10-second TTL, renewed every 3 seconds, so if the holder dies another node takes over within about 10 seconds. Each lease carries a fencing token, and the job writes it with every batch, so the database rejects work from a holder that paused and lost its lease. The job is also idempotent per customer and month, so even a slip doesn't send twice."
Go deeper
- SD 1 · Leader Election and Raft.
- SD 2 · Chapter 11 — The Consensus Problem and Chapter 12 — Raft.
- SD 3 · Q41 — Consensus and Raft.
- Replication: failover is leader election for your database.