Distributed Transactions
Once each service owns its database, "all or nothing" stops being free. Sagas and the outbox put it back, step by step.
The story
In the monolith, placing an order was one database transaction: reserve stock, charge the card, create the order. If anything failed, ROLLBACK undid all of it.
Then the system was split into an Order service, a Payment service and an Inventory service, each with its own database. Placing an order became three calls:
The support queue fills with "charged but no order" tickets. The fix that gets suggested first, "just wrap it in a transaction", doesn't work: there's no transaction that spans three databases owned by three services.
There are two real options: a protocol that makes several databases commit together (two-phase commit), or a sequence of local transactions where each step knows how to undo itself (a saga).
Option 1: Two-phase commit (2PC)
A coordinator asks every participant to prepare (do the work, lock the rows, promise it can commit), then tells them all to commit, or all to abort.
It gives true atomicity, but:
- It blocks. Participants hold locks between prepare and commit. A coordinator crash at the wrong moment leaves them stuck.
- It's slow. Several round trips, with locks held throughout.
- Every participant must support it (XA transactions). Most queues, third-party APIs (like a payment gateway) and many NoSQL stores don't.
2PC fits inside one database system that spans machines (Spanner, CockroachDB and similar handle it internally). It rarely fits between microservices.
Option 2: Sagas
A saga is a chain of local transactions, one per service. If a step fails, the saga runs compensating actions for the steps that already succeeded, in reverse order.
Notice the order: reserve stock before charging. Put the step that's hardest to undo, or most likely to fail, where its failure triggers the least compensation. A refund is a real money movement with fees and customer emails. Releasing a reservation is a row update.
Two ways to run one:
| Orchestration | Choreography | |
|---|---|---|
| Flow is visible in | One place (code or Temporal / Step Functions workflow) | Spread across services' event handlers |
| Good for | 4+ steps, branching, timeouts | 2–3 simple steps |
| Risk | Orchestrator becomes a "god service" | Nobody can see the whole flow, and cycles creep in |
What sagas don't give you: isolation. Between steps, other requests can see the in-between state (an order that's PENDING, stock that's reserved but not paid for). Design for it: explicit states, "pending" shown in the UI, and reservations that expire.
The glue: the transactional outbox
Every saga step has to update its database and publish an event. Do both directly and you've got two writes that can each fail:
- DB commit succeeds, publish fails → the next step never happens.
- Publish succeeds, DB commit fails → the next step runs for something that didn't happen.
The outbox pattern turns it back into one local transaction:
The relay may publish an event more than once, so every consumer must be idempotent (Fault Tolerance and Idempotency).
Cheat sheet
| Question | Answer |
|---|---|
| Can I use one DB transaction? | If the data can live in one database, do that. It's the best fix. |
| 2PC | Atomic, but blocking and slow. Use within one distributed DB, not across services. |
| Saga | Local transactions + compensations. The default across services. |
| Orchestration vs choreography | Orchestrate for 4+ steps or branching. Choreograph for 2–3 simple ones. |
| Step order | Hard-to-undo steps (charging money) last |
| Compensation ≠ rollback | It's a new action (refund, release), visible and possibly failing too: retry it |
| Isolation | None between steps. Use explicit PENDING states and expiring reservations. |
| DB write + publish event | Transactional outbox |
| Duplicates | Every step and compensation idempotent |
| Tooling | Temporal, AWS Step Functions, or a state table + outbox |
Say it in the interview
"Order, inventory and payment each own their data, so there's no shared transaction. I'd run an orchestrated saga: create the order as PENDING, reserve stock with a 10-minute expiry, charge the card last, then confirm. If the charge fails we release the stock and cancel the order. Each step writes its event through an outbox, and every handler is idempotent on the order ID."
Go deeper
- SD 2 · Chapter 13 — Distributed Transactions: 2PC, sagas and their failure modes.
- SD 3 · Q40 — Distributed Transactions and 2PC, Q50 — Saga Pattern, Q49 — Outbox Pattern.
- Monolith vs Microservices: why the split created this problem.