Q49 — What is the outbox pattern — why does it exist?
What problem does it solve?
Consider a common operation in a microservices system: a user places an order. The order service must do two things atomically:
- Write the new order to its database (so the order exists)
- Publish an "OrderPlaced" event to a message queue (so the inventory service, notification service, and billing service can react)
The obvious code writes to the database, then publishes to the queue. What can go wrong?
Scenario A: The database write succeeds, but before the queue publish happens, the service crashes. The order exists in the database, but no event was ever published. The downstream services never heard about this order. Inventory was never decremented. The notification was never sent. The order is in a ghost state — it exists but the system doesn't know about it.
Scenario B (the reverse attempt): You publish to the queue first, then write to the database. The publish succeeds, but the database write fails (constraint violation, timeout, deadlock). Downstream services received an event for an order that doesn't exist. They try to process a phantom order, fail, or create inconsistent state.
Both orderings have a window where one side happened and the other didn't. These two operations — database write and message publish — are in two different systems, and there's no distributed transaction spanning them (publishing to Kafka cannot participate in a Postgres transaction). There's no atomic "do both or neither."
The outbox pattern is the solution.
What is the outbox table?
The outbox pattern uses a simple insight: your database is a durable, transactional store. Use it to durably record the intent to publish, within the same transaction as your business write.
The implementation: add an outbox table to your service's database. It stores pending events — the messages that need to be published but haven't been yet.
CREATE TABLE outbox (
id UUID PRIMARY KEY,
event_type TEXT NOT NULL, -- e.g., 'OrderPlaced'
payload JSONB NOT NULL, -- event payload
created_at TIMESTAMPTZ NOT NULL,
published BOOLEAN DEFAULT FALSE
);
Now, in your order service, the operation changes:
BEGIN;
INSERT INTO orders (id, user_id, total, ...) VALUES (...);
INSERT INTO outbox (id, event_type, payload)
VALUES (gen_random_uuid(), 'OrderPlaced', '{"order_id": ..., "user_id": ...}');
COMMIT;
Both writes happen in a single database transaction. Either both commit, or neither does. The outbox row is the durable record of the intent to publish. If the service crashes immediately after the commit, the outbox row survives — it's in the database, with published = false.
How do events get from the outbox to the message queue?
A separate process — the outbox relay (sometimes called the outbox publisher or CDC processor) — is responsible for picking up unpublished events from the outbox table and publishing them to the message queue.
Polling approach: The relay runs a query every few seconds: SELECT * FROM outbox WHERE published = false ORDER BY created_at LIMIT 100. For each row, it publishes the event to Kafka (or whichever queue), then updates the row: UPDATE outbox SET published = true WHERE id = ?. If the relay crashes after publishing but before marking published = true, it will republish on restart — this is at-least-once delivery. The downstream consumer must be idempotent (Q26, Q43).
CDC (Change Data Capture) approach: Instead of polling a flag, the relay uses the database's replication log (Postgres WAL, MySQL binlog) to detect new rows in the outbox table. Tools like Debezium consume the WAL and forward inserts to the outbox table directly to Kafka. This is more efficient than polling — zero delay, no load from repeated queries, no published flag needed — and is the production-grade approach for high-throughput systems. The relay reads from the WAL offset it last processed, so events are never missed (the WAL is the reliable source of truth about what happened).
What guarantees does the outbox pattern provide?
Atomicity between DB write and event publication: The business data and the event intent are written in one transaction. You can never have an order without a corresponding outbox row, or an outbox row without a corresponding order. The two are coupled at the database level.
Durability: If the service crashes, the outbox rows survive in the database. The relay will pick them up and publish when it next runs (or when it reconnects).
At-least-once delivery: The relay may publish an event and then crash before marking it processed. On restart, it publishes again. Downstream consumers see a duplicate. This is inherent to the pattern and by design — the alternative (at-most-once) means accepting lost events. Consumers must handle idempotent processing.
Decoupling from the queue's availability: If Kafka is temporarily down, the service can still accept orders — they're written to the database + outbox. When Kafka recovers, the relay drains the backlog. The service's write path is not dependent on the queue's availability.
What is dual write and why does the outbox replace it?
"Dual write" is the naive alternative: write to the database, then write to the queue, in sequence, without a transaction. As described in the problem statement, this creates a failure window between the two writes where you can end up in an inconsistent state.
The fundamental reason dual write fails: two separate systems cannot be updated atomically without a coordinator. You'd need a 2PC coordinator (see Q40) to guarantee atomicity across the database and the queue — and as explained there, 2PC is dangerous, slow, and often not supported.
The outbox pattern avoids the need for cross-system atomicity by collapsing the queue write into the database write. The event is not in Kafka — it's in your database, in the same transaction. Kafka only gets it later, via the relay, as an eventual propagation step. The atomicity is local, which is safe and fast.
When do you use the outbox pattern?
Use it any time you need to publish an event to a message queue as part of a business transaction, and you cannot accept the possibility of: (a) the event not being published when the DB write succeeded, or (b) the event being published when the DB write failed.
Common scenarios: order placement, payment processing, user registration, status updates — any operation where downstream services need to be informed reliably of something that just happened.
You don't need the outbox pattern if your operation is read-only (no event to publish), if you're fine with at-most-once event delivery (loss is acceptable), or if your system uses an event-sourcing model where the event log is the primary store (Q51).
The mental model
The problem:
Write to DB + publish to queue = two separate systems
No atomic transaction spans them
DB write succeeds → crash → event never published (downstream services miss it)
Event published → DB write fails → downstream processes phantom event
Dual write fails because:
Two separate systems cannot be updated atomically without a coordinator
2PC is dangerous and often unsupported by message queues
Outbox pattern:
Add outbox table to service's own DB
In one DB transaction: write business data + write event to outbox
→ Atomicity is local (same DB transaction) → safe, fast
Separate relay process reads outbox and publishes to queue:
Polling: SELECT unpublished rows → publish → mark published
CDC (Debezium): watch WAL/binlog → forward new outbox rows to Kafka
Guarantees:
✅ DB write and event intent are always in sync (same transaction)
✅ Events survive crashes (in DB until relay publishes)
✅ Queue availability doesn't block service's write path
⚠️ At-least-once delivery (relay may republish after crash)
→ downstream consumers must be idempotent
Use when:
Any business operation that must reliably trigger downstream events
Order placement, payment, registration, status changes
Don't need it when:
Read-only operations (no events)
At-most-once delivery is acceptable
Using event sourcing (event log is primary store)
The outbox pattern is the canonical solution to the "I need to write to a database and publish a message atomically" problem. It works by recognizing that you don't need to atomically write to two systems — you just need to atomically record the intent to publish in the same system as the business data, and let a relay handle the eventual propagation.