Q50 — What is the saga pattern — how do you handle failures across services without 2PC?

The problem: business transactions that span service boundaries

In a monolith, "place an order" is a single database transaction: deduct inventory, charge the customer, create the order record. If any step fails, roll back. The database handles it atomically — ACID guarantees that either all steps commit or none do.

In a microservices architecture, those three operations live in three different services with three different databases: the inventory service, the payment service, and the order service. There is no single transaction that spans all three.

You might reach for 2PC (see Q40) — the distributed protocol that tries to atomically commit across multiple participants. But 2PC has critical problems: it blocks if the coordinator crashes mid-protocol, holds locks across the network during the vote phase, and is often not supported by modern databases and message queues. In practice, 2PC is too fragile and slow for most microservices workflows.

The saga pattern is the alternative: break the business transaction into a sequence of local transactions, each within one service, where each step publishes an event that triggers the next. If any step fails, execute compensating transactions to undo the steps that already succeeded.

What is a saga?

A saga is a sequence of local transactions. Each local transaction:

1. Updates its own service's database
2. Publishes an event (or sends a command) to trigger the next step

If step N succeeds, it triggers step N+1. If step N fails, it triggers compensating transactions for steps 1 through N-1 — effectively rolling back the work that already committed.

The critical difference from 2PC: saga steps commit immediately and independently. There is no "prepare" phase where all participants hold locks waiting for a global commit decision. Each step commits as soon as it completes. This means failures after a step has committed require compensation (running a corrective action), not rollback.

Example — placing an order:

1. Order service: Create order with status PENDING → publish "OrderCreated"
2. Inventory service: Reserve items → publish "InventoryReserved"
3. Payment service: Charge customer → publish "PaymentProcessed"
4. Order service: Update order status to CONFIRMED → publish "OrderConfirmed"

If payment fails (card declined):

• Compensate inventory: release the reserved items → publish "InventoryReleased"
• Compensate order: mark order as FAILED → publish "OrderFailed"

The inventory reservation and the pending order record were committed to their respective databases. The compensation steps explicitly undo them — they're new transactions that reverse the effect, not rollbacks.

Two coordination models: choreography vs orchestration

There are two ways to coordinate the steps in a saga.

Choreography: No central coordinator. Each service listens for events and reacts. The order service publishes "OrderCreated." The inventory service listens, reserves items, publishes "InventoryReserved." The payment service listens, charges the customer, publishes "PaymentProcessed." Services know what events to react to and what events to publish.

Pros: fully decentralized, no single point of failure, services are loosely coupled. Cons: the overall flow is implicit — it's distributed across multiple service codebases. Tracing a failed saga requires correlating events across multiple logs. Compensating logic is similarly scattered. As the number of steps grows, choreography becomes hard to reason about.

Orchestration: A central coordinator service (the "orchestrator" or "saga manager") explicitly commands each step. The orchestrator calls the inventory service: "reserve these items." When it gets a response, it calls the payment service: "charge this card." On failure, it calls the inventory service: "release the reservation."

Pros: the overall flow is in one place — easy to read, debug, and change. Compensation logic is centralized in the orchestrator. Cons: the orchestrator is a new service to build and operate; it can become a bottleneck; it introduces coupling between the orchestrator and each participant service.

For complex workflows with many steps and complex compensation paths, orchestration (via a dedicated workflow engine — Temporal, AWS Step Functions, or a custom saga orchestrator) is generally preferred. For simple 2-3 step workflows, choreography is simpler to implement.

What makes compensation hard?

Saga compensation is not free rollback — it's real work that has real side effects, and some side effects can't be undone.

Non-reversible actions: Suppose step 3 in your saga sends a confirmation email. If step 4 fails and you need to compensate, you can't un-send the email. The user got a "your order is confirmed!" message for an order that subsequently failed. This is a real user-facing problem. The standard approach: delay irreversible actions to the last step of the saga, so they only execute after all reversible steps have succeeded.

Partial completion race conditions: What if a compensating transaction fails? You tried to release the inventory reservation but the inventory service was down. Now you have a permanently reserved item. Saga orchestrators must retry compensation until it succeeds (with retries and backoff), and compensating transactions must themselves be idempotent. This requires careful design — compensation is not "just undo the thing," it's a full operation with its own failure handling.

Isolation violations: In a database, a transaction's intermediate state is invisible to other transactions (MVCC, isolation levels). In a saga, each step commits immediately and is visible. Between step 1 (order created) and step 4 (order confirmed), other services can observe the order in PENDING state. Other sagas or queries might read this intermediate state and act on it. This is called "dirty reads" at the saga level — sagas are not isolated from each other the way database transactions are. Strategies to mitigate: use semantic locking (the PENDING status communicates "don't act on this yet"), or design around the intermediate state being visible.

When does a saga fail vs when does it succeed?

A saga is complete when all steps succeed (happy path). It is "rolled back" when a failure is detected and all compensation steps complete. But there's a third state: stuck — a step failed and compensation is also failing.

Stuck sagas are a serious operational problem. They require a workflow engine that:

• Persists saga state durably (so crashes don't lose progress)
• Retries failed steps and compensations with backoff
• Escalates to a dead-letter/human review queue if compensation repeatedly fails
• Provides visibility into the current state of every in-flight saga

This is why production saga implementations use a dedicated workflow engine (Temporal, AWS Step Functions, Conductor) rather than homegrown coordination code. A workflow engine handles state persistence, retries, timeouts, and observability as infrastructure.

The mental model

Problem:
  Business transaction spans multiple services and databases
  2PC is fragile, slow, locks across the network → not viable in practice

Saga = sequence of local transactions, each in one service
  Each step: write to local DB → publish event/command → trigger next step
  If step N fails: execute compensating transactions for steps 1..N-1
  Compensation = new forward transactions that undo prior effects
                 NOT database rollback (prior steps already committed)

Coordination:
  Choreography: services react to events → decentralized, hard to trace
  Orchestration: coordinator commands each step → centralized, easier to reason about

What makes compensation hard:
  Non-reversible side effects (emails, SMS) → defer to last step
  Compensation can itself fail → retry with backoff, idempotent
  Intermediate state is visible to others → no saga-level isolation
    → use semantic locks (PENDING status), design around visibility

Stuck sagas (step failed + compensation failing):
  Require persistent state + retries + escalation to human review
  → Use a workflow engine (Temporal, Step Functions, Conductor)

Saga vs 2PC:
  2PC: all participants lock + vote → atomic commit or abort → synchronous
       Blocks on coordinator crash, holds cross-network locks, slow
  Saga: each step commits immediately → compensation on failure → async
        No global locks, no coordinator needed for commit
        Cost: intermediate state visible, compensation is real work

Use saga when:
  Business transaction spans multiple services
  2PC is not viable (almost always)
  Steps can be modeled as local transactions with compensating actions

Sagas are the standard pattern for distributed business transactions in microservices. They accept that distributed systems cannot provide the same atomicity as a local database transaction, and instead provide correctness through explicit compensation — rolling forward (via undo operations) rather than rolling back (via lock release).