Q40 — What is a distributed transaction — and why is two-phase commit (2PC) dangerous?

What problem does a distributed transaction solve?

A transaction in a single database is atomic: all operations succeed or all are rolled back. The database's WAL and recovery mechanism guarantee this (Q13, Q15). If the machine crashes mid-transaction, recovery rolls back the partial work. The application always sees either the full result or nothing.

Now suppose the same logical operation spans multiple systems: debit an account in Database A, credit an account in Database B, send a confirmation event to a message queue. All three must succeed together or fail together. A partial success — the debit succeeds but the credit fails — leaves the data in an inconsistent state: money gone from one account, not arrived in the other.

A distributed transaction is a transaction that spans multiple independent systems (different database instances, different services, different datastores), providing atomicity across all of them. It must guarantee: either all participants commit, or all participants abort. No partial commits.

This is the hardest problem in distributed systems to solve correctly, and two-phase commit (2PC) is the classic protocol for it — with well-known, serious failure modes.

How does two-phase commit work?

2PC introduces a coordinator — a process that orchestrates the transaction across all participants (the individual systems involved).

Phase 1 — Prepare: The coordinator sends a PREPARE message to all participants. Each participant executes the transaction locally, writes the changes to its WAL, but does not commit. It holds all locks. It responds to the coordinator with either VOTE-COMMIT (I'm ready and have durably prepared my changes) or VOTE-ABORT (I encountered an error, abort).

A VOTE-COMMIT is a promise: the participant is telling the coordinator "even if I crash and restart, I have recorded what I need to do. I will commit if you tell me to."

Phase 2 — Commit or Abort: If the coordinator receives VOTE-COMMIT from all participants, it sends COMMIT to all of them. Each participant commits its local changes and releases locks.

If any participant votes ABORT (or doesn't respond in time), the coordinator sends ABORT to all participants. Each rolls back its local changes and releases locks.

The coordinator writes its decision to its own durable log before sending Phase 2 messages, so it can re-send the decision if participants don't acknowledge.

Why is 2PC dangerous — what are the failure modes?

Problem 1: The coordinator is a single point of failure.

In the window between Phase 1 (participants have voted COMMIT and prepared) and Phase 2 (coordinator has sent COMMIT), the coordinator crashes.

The participants are in limbo. They have prepared their changes — locks are held, resources are reserved, they've voted to commit. But they cannot commit unilaterally (what if the coordinator was going to send ABORT because another participant voted ABORT?). They cannot rollback unilaterally (what if the coordinator was going to send COMMIT?). They must wait for the coordinator to recover.

This is the blocking problem — 2PC can block indefinitely if the coordinator fails at the wrong moment. Every resource locked by the prepared participants stays locked. Other transactions that need those rows are blocked. In a busy OLTP system, a coordinator crash during the commit phase can freeze significant portions of the database for however long recovery takes.

Recovery: when the coordinator restarts, it reads its log to find in-progress transactions and re-sends the Phase 2 decisions. But if the coordinator's log is also lost (disk failure), it may not know what it decided. Manual intervention may be required.

Problem 2: Locks held across network round trips.

2PC requires holding row-level locks from the start of Phase 1 through the end of Phase 2. This is two network round trips worth of lock-hold time — far longer than a local transaction. During those round trips, contending transactions are blocked.

In a multi-datacenter setup where Phase 1 involves cross-datacenter communication (100ms+), locks are held for hundreds of milliseconds. Under high concurrency, this lock contention severely degrades throughput.

Problem 3: Participant failure after PREPARE.

A participant votes COMMIT and then crashes before receiving the Phase 2 COMMIT message. The coordinator sends COMMIT, other participants commit, but this participant is down. When it recovers, it reads its log, finds the prepared-but-not-committed transaction, and asks the coordinator: "what did you decide?" The coordinator (if still running) sends COMMIT and the participant completes. But if the coordinator has also crashed and lost its log, the participant is stuck.

Problem 4: Network partitions.

If the coordinator and a participant are partitioned after Phase 1, the participant holds its prepared state indefinitely — it can't hear the coordinator's Phase 2 message. The partition heals eventually, the Phase 2 message arrives, and the participant acts. But during the partition window, resources are locked and unavailable.

Why does 2PC's safety guarantee exist alongside its danger?

It's important to understand: 2PC is safe — if implemented correctly and assuming coordinator recovery, it provides exactly-once semantics and never produces inconsistent results. The coordinator's log ensures the decision is durable before Phase 2 messages are sent. The participants' prepared state ensures they can always complete the committed transaction after recovery.

The danger is not in correctness (2PC is correct) — it's in availability and performance. A correct 2PC system that blocks for 5 minutes while the coordinator recovers is correct but practically unusable. The tradeoff is: 2PC provides distributed atomicity at the cost of availability (blocking during coordinator failure) and performance (locks held across network round trips).

What are the alternatives to 2PC?

Saga pattern (covered in Q50): Break the distributed transaction into a sequence of local transactions, each with a compensating transaction (undo). If step 3 fails, run compensating transactions for steps 1 and 2 to undo their effects. No distributed locks, no coordinator single point of failure. The trade-off: intermediate states are visible (between step 1 succeeding and step 2 succeeding, the system is temporarily inconsistent). Sagas are eventually consistent, not atomically consistent.

Outbox pattern (covered in Q49): For the specific case of "write to database AND send a message," write the message to an outbox table in the same local transaction as the database write. A separate process reads the outbox and publishes the messages. The local transaction is atomic (both the business write and the outbox write are in one ACID transaction). The message delivery is at-least-once (with idempotent consumers). No distributed transaction needed.

Avoid distributed transactions by design: The best solution is often to structure the system so that atomicity is needed only within a single service and database. Co-locate data that must be updated atomically in the same database. Use event-driven architecture where services react to events rather than being called synchronously in a shared transaction. This is the fundamental insight behind domain-driven design and microservices: service boundaries should align with consistency boundaries.

The mental model

Distributed transaction: atomic commit across multiple independent systems
  Debit DB-A + Credit DB-B + Publish event → all or nothing

2PC protocol:
  Phase 1 (Prepare):
    Coordinator → all participants: PREPARE
    Each participant: execute locally, write to WAL, hold locks, respond VOTE-COMMIT/ABORT
  Phase 2 (Commit/Abort):
    All VOTE-COMMIT → coordinator sends COMMIT → all commit, release locks
    Any VOTE-ABORT → coordinator sends ABORT → all rollback, release locks

2PC is correct — never produces inconsistent results
2PC is dangerous:
  Coordinator failure between phase 1 and 2 → participants block indefinitely (locks held)
  Cross-network lock hold → contention, throughput degradation
  Coordinator log loss → manual intervention required

Alternatives:
  Saga: sequence of local transactions + compensating transactions
    Pro: no distributed locks, high availability
    Con: intermediate states visible (eventual consistency)

  Outbox pattern: write message to DB in same local transaction as business write
    Pro: exactly-once message delivery without 2PC
    Con: requires polling/CDC infrastructure to publish outbox

  Design around it: co-locate atomic data, service boundaries = consistency boundaries
    Pro: eliminates the problem entirely
    Con: requires careful domain modeling upfront

Real-world usage: 2PC is used within distributed databases (CockroachDB, Spanner) where
the coordinator is highly available by design. Avoided in microservices architectures.

2PC is not wrong — it's a powerful, correct protocol. Its danger is that distributed transactions become a dependency on coordinator availability and a performance bottleneck from cross-network lock holds. The better answer, most of the time, is designing the system so that atomicity is a local concern — not distributed at all.