Q28 — Why is distributed locking hard, and how do you do it safely?

Why can't you just use a database lock across multiple machines?

On a single machine, locking is solved. The operating system provides mutexes. The CPU provides atomic instructions. Memory is shared, so a lock in memory is visible to all threads on the machine simultaneously. Locking is safe because there's one clock, one memory space, and the OS guarantees what "locked" and "unlocked" mean.

Distributed systems have none of these guarantees. You have multiple machines, each with its own memory, each running on its own clock, communicating over a network that can delay, drop, or reorder messages. "Locking" across these machines requires coordination — and coordination over an unreliable network is where the fundamental difficulty lives.

The need arises constantly: only one worker should process a given job, only one service instance should run a scheduled cron task, only one server should write to a shared external resource at a time. On a single machine, a mutex solves this trivially. On ten machines, you need distributed locking.

Why is the naive Redis approach almost right but dangerously wrong?

The most common approach: use Redis as a coordinator. To acquire a lock, a service sets a key in Redis with SET lock_key unique_value NX PX 30000 — set the key only if it doesn't exist (NX), with a 30-second expiry (PX 30000). Only one caller can successfully set the key. The caller that gets OK holds the lock. Others get nil and must wait or fail.

To release the lock, the holder deletes the key (with a Lua script to ensure only the holder can delete it, by checking the unique value matches).

This works in the happy path. But distributed systems are full of unhappy paths.

Problem 1: The process pauses.

Imagine the timeline:

Service A acquires the lock at time T=0. The lock has a 30-second TTL.
At T=29s, Service A's process pauses — a garbage collection stop-the-world, a VM being live-migrated, a page fault causing a slow disk read. The pause lasts 2 seconds.
At T=30s, while Service A is paused and unaware, Redis expires the lock. The TTL ran out.
At T=30.5s, Service B acquires the lock. Service B now holds the lock and begins modifying the shared resource.
At T=31s, Service A's pause ends. It resumes execution, believes it still holds the lock (it has no way to know the TTL expired during the pause), and begins modifying the shared resource.

Two processes now simultaneously believe they hold the exclusive lock. The entire point of the lock is violated.

This is not a hypothetical — GC pauses of seconds are real in JVM applications. VM live migrations cause pauses. The assumption "I checked the lock recently, I still hold it" breaks under process pauses.

Problem 2: The response is lost.

Service A sends SET lock_key value NX PX 30000 to Redis. Redis successfully sets the key. The response travels back to Service A but the network drops it. Service A receives no confirmation.

Service A doesn't know whether the lock was acquired or not. If it retries, it gets nil (the key exists), which looks like "lock already held by someone else." The service may conclude it lost the lock and give up — even though it actually holds it. The lock is now held by nobody (no service thinks it holds it) but the key exists in Redis, so nobody can acquire it until TTL expires.

Problem 3: Redis restarts or becomes unavailable.

If Redis crashes and restarts without persistence, all keys are gone. Every held lock is silently released. Multiple services that were waiting for locks now all acquire them simultaneously.

If Redis is unreachable due to a network partition, nobody can acquire or release locks. Services block or fail, depending on how you handle the error case.

What is Redlock and why is it still not completely safe?

Antirez (Redis's creator) proposed Redlock: instead of one Redis instance, run 5 independent Redis instances (with no replication between them). To acquire the lock, contact all 5 and attempt to set the key. If a majority (3 of 5) succeed within a time bound, the lock is acquired. The lock's TTL minus the time spent acquiring it is the safe hold time.

Redlock improves fault tolerance — a single Redis node failure doesn't break the lock. But it doesn't solve the process pause problem. Martin Kleppmann (author of "Designing Data-Intensive Applications") wrote a detailed critique: even with Redlock, a process pause between acquiring the lock and using it can lead to two processes simultaneously believing they hold the lock.

Redlock is better than single-node Redis locking, but it is not safe for use cases where correctness is critical (financial transactions, exactly-once processing).

What actually makes distributed locking safe? Fencing tokens.

The key insight from Kleppmann's critique: the lock alone is not enough. The lock tells processes "you have permission to act." But if that permission is revoked (TTL expires) without the process knowing, the process acts without permission.

The solution is a fencing token: every time a lock is acquired, the lock service returns a monotonically increasing integer (the token). The holder includes this token in every operation on the shared resource. The shared resource rejects any operation whose token is lower than the highest token it has already seen.

Timeline with fencing tokens:

Service A acquires lock, gets token 33. Begins operating on the shared resource.
Service A pauses (GC). Lock TTL expires.
Service B acquires lock, gets token 34. Begins operating on the shared resource with token 34.
Service A resumes. Tries to operate with token 33.
Shared resource sees token 33 < 34 (already seen). Rejects Service A's operation.

Now the resource itself enforces safety, not just the lock. Even if the lock coordination fails, operations with stale tokens cannot cause damage. The lock is the performance optimization (avoid contention); the fencing token is the correctness mechanism.

This requires the shared resource (your database, your storage service, your API endpoint) to understand and enforce fencing tokens — which most systems don't do out of the box. Implementing it requires custom code. But for genuinely critical use cases, it's the only approach that is provably safe.

What about ZooKeeper and etcd?

For distributed locking with stronger guarantees, use a system built on consensus — ZooKeeper or etcd. These systems use Raft (etcd) or ZAB (ZooKeeper) to maintain a strongly consistent, fault-tolerant key-value store. A lock in etcd is a lease with a TTL maintained by a heartbeat; the consensus protocol ensures the lease state is consistent across the etcd cluster even through node failures.

The lock primitives in these systems are designed from the ground up for distributed coordination, not bolted on top of a cache. etcd's lease mechanism naturally produces monotonically increasing revision numbers — a built-in fencing token equivalent.

The cost: ZooKeeper and etcd are heavier infrastructure than Redis, with more operational complexity. For most applications that need distributed locking, Redis with awareness of its limitations (short lock holds, idempotent operations, retry logic) is sufficient. For exactly-once processing or financial-grade correctness, use etcd/ZooKeeper with fencing tokens.

Practical guidance

For most distributed locking needs:

Use Redis with SET NX PX for simple coordination (job deduplication, cron exclusivity)
Keep lock TTL short — seconds, not minutes
Make the operations the lock protects idempotent — if the lock fails and two instances run, idempotency catches it
Never hold the lock across network calls or user-facing waits

For critical correctness:

Use etcd or ZooKeeper
Implement fencing tokens on the resource being protected
Treat the lock as a performance hint, not a correctness guarantee

The mental model

code

Single-machine lock: shared memory + OS → trivially safe
Distributed lock: need coordination across machines over unreliable network

Redis NX + TTL: simple, works in happy path
  Breaks:
    Process pause → lock expires mid-hold, two holders simultaneously
    Lost response → holder doesn't know it holds the lock
    Redis restart → all locks silently released

Redlock (5 Redis nodes): better fault tolerance, same process-pause vulnerability

Fencing tokens: every lock acquisition returns a monotonically increasing token
  Resource enforces: reject operations with token < max-seen
  Process pause cannot corrupt: stale token gets rejected at the resource
  Requires: resource to understand and enforce tokens

ZooKeeper / etcd: consensus-based, designed for coordination
  Built-in leases + revision numbers = fencing tokens
  Heavier infrastructure, stronger guarantees

Choose by stakes:
  Low stakes (cron deduplication) → Redis NX, idempotent ops, short TTL
  High stakes (financial, exactly-once) → etcd + fencing tokens

Distributed locking is hard because the distributed systems problems — process pauses, network drops, partial failures — attack the very assumption locking relies on: that a lock holder is alive and exclusive. Making that assumption safe requires either accepting that it can break (and designing idempotent operations to survive the break) or implementing fencing to make the resource protect itself.