Chapter 14: Message Queues — Decoupling Time and Space

Curriculum: Distributed Systems from First Principles Chapter: 14 of 24 Part: 6 — Communication Prerequisites: Chapter 1 (latency numbers), Chapter 2 (WAL, durability), Chapter 4 (failure modes), Chapter 13 (Saga choreography) Estimated reading time: 80–90 minutes

Opening

Every interaction so far in this curriculum has been synchronous: a client sends a request, waits for the server to process it, and gets a response. The client and server are coupled in time — the client is blocked until the server finishes. They're also coupled in space — the client must know the server's address, the server must be running when the client needs it, and if the server is slow or crashed the client feels it immediately.

This coupling causes three classes of problems in real systems. First, traffic spikes: your order service receives 10,000 checkout requests per second during a flash sale, but your inventory service can only process 1,000/second. The excess 9,000 requests either fail immediately or queue inside the order service's connection pool until they time out. Second, cascading failures: the inventory service gets slow (GC pause, slow disk). Every order service request that calls inventory now waits. Order service's thread pool fills up. Order service becomes slow. API gateway requests start timing out. The entire system is down because one service was slow for 30 seconds. Third, tight coupling: the order service must know inventory service's address, retry logic, circuit breaker behaviour, and timeout policy. Adding a new consumer of order events — a fraud detection service, an analytics pipeline — requires changing the order service.

A message queue solves all three by introducing a buffer between producer and consumer. The producer writes a message to the queue and returns immediately — it does not wait for the consumer to process it. The consumer reads from the queue at its own pace. They are decoupled in time (producer and consumer don't run simultaneously) and space (producer doesn't know who consumes, consumer doesn't know who produced). The queue absorbs spikes, isolates slow consumers, and lets new consumers be added without touching the producer. This chapter builds message queues from the physical level up — what they actually are on disk, what "delivery guarantee" means in terms of real failure scenarios, and where they break.

Part 1: What a Message Queue Is Physically

Strip away all abstraction. A message queue is, at its core, a durable ordered buffer stored on disk. Let's build it from the simplest possible design.

A producer writes a message. The queue appends it to a log file on disk — the same sequential append from Chapter 10's LSM Trees. The message gets an offset (its position in the file, starting at 0). Consumers read messages by specifying an offset and receiving the message at that position plus subsequent messages.

Queue file on disk (append-only log):

Offset 0:  {id: m1, payload: "order #123 created", ts: 10:00:00}
Offset 1:  {id: m2, payload: "order #124 created", ts: 10:00:01}
Offset 2:  {id: m3, payload: "order #125 created", ts: 10:00:01}
Offset 3:  {id: m4, payload: "order #126 created", ts: 10:00:02}
...

Consumer A is at offset 2: next read returns m3, then m4, etc.
Consumer B is at offset 0: next read returns m1, m2, m3, m4, etc.

Each message is written once (sequential append — fast, from Chapter 1) and can be read many times by different consumers at different offsets. The queue doesn't delete messages when a consumer reads them — deletion happens by advancing an offset pointer or by time/size-based retention.

This is the physical reality behind every message queue system. RabbitMQ, Amazon SQS, Redis Streams, and Kafka all store messages as durable log files with offsets. The differences are in: how they track which consumer has read which message, how many times a message can be delivered, what happens when a consumer crashes mid-processing, and how much of the log is retained.

Part 2: Delivery Semantics — The Most Important Property to Get Right

The delivery guarantee is what a queue promises about whether and how many times a consumer will receive each message. There are three models, and choosing the wrong one causes data corruption or data loss. This is the single most important thing to understand about message queues.

At-Most-Once

The queue delivers each message zero or one times. If the consumer crashes after receiving the message but before processing it, the message is never redelivered. The consumer might have processed it (and crashed after), or might not have processed it at all. The queue has no way to know — it delivered once and moved on.

At-most-once is appropriate when: message loss is acceptable (metrics sampling, non-critical analytics, telemetry where you just need trends not exact counts). It is inappropriate when: each message represents a financial transaction, a user action, or anything where loss causes business harm. In practice, at-most-once is rarely the right choice for production systems. When in doubt, do not use it.

At-Least-Once

The queue delivers each message one or more times. A message is considered delivered only after the consumer explicitly acknowledges it. If the consumer crashes before acknowledging, the queue redelivers the message — possibly to the same consumer after recovery, or to a different consumer. The consumer might process the same message twice.

At-least-once is the default for most message queue systems (RabbitMQ, SQS, Kafka with manual offset commits). It guarantees no message is lost — at the cost of possible duplicates. The consumer must be designed to handle duplicate delivery gracefully. The mechanism for handling duplicates is idempotency: the consumer's processing of a message must produce the same result whether it runs once or ten times.

"Create order #123 if it doesn't exist already" is idempotent. "Add $10 to account balance" is not idempotent — running it twice adds $20. The idempotent version: "Set account balance to $110 if current balance is $100 and this transaction_id hasn't been applied." Idempotency often requires storing a record of which message IDs have been processed.

Exactly-Once

The queue delivers each message exactly one time, and the consumer processes it exactly one time, even under crashes and retries. This sounds like the obvious correct choice, but it's the hardest to implement and has important caveats.

True exactly-once delivery across a network is impossible (Chapter 11's Two Generals Problem — you can never be certain the consumer received and processed a message without a round trip confirmation, which can itself fail). What systems call "exactly-once" is actually "effectively exactly-once" or "idempotent at-least-once" — the system ensures duplicates are detected and suppressed.

Kafka's exactly-once semantics (EOS, introduced in Kafka 0.11) works like this: the producer writes messages with an idempotent producer ID and sequence number. The broker deduplicates writes with the same (producer_id, sequence_number). On the consumer side, offsets and processing results are committed atomically in a transaction — if the consumer crashes, it rolls back both the processing result and the offset, so the message is reprocessed from the last committed offset. This gives exactly-once semantics within the Kafka ecosystem (producer to broker to consumer to output), but requires the output sink (database, downstream service) to also support transactional writes.

The practical guidance: design for at-least-once with idempotent consumers. True exactly-once either isn't available (most systems), is expensive (Kafka EOS adds 50-100% latency overhead), or requires the entire pipeline to be within one system (can't span a Kafka consumer writing to a non-transactional API). Idempotent at-least-once gives you the same effective guarantee at lower complexity.

Part 3: How Acknowledgement Works Physically

The acknowledgement mechanism is what separates at-least-once from at-most-once. Understanding it physically removes the magic.

When a consumer fetches a message from the queue, the queue notes that the message is "in flight" — delivered to a consumer but not yet acknowledged. The queue starts a visibility timeout (SQS terminology) or ack timeout (RabbitMQ terminology): a timer. If the consumer doesn't acknowledge within the timeout, the queue assumes the consumer crashed and redelivers the message.

In-flight tracking:

Queue state after delivering m3 to Consumer A:

Offset 0: m1  [ACKed by Consumer A at 10:00:05]
Offset 1: m2  [ACKed by Consumer A at 10:00:06]
Offset 2: m3  [IN FLIGHT → Consumer A, timeout at 10:00:40]
Offset 3: m4  [pending]

At 10:00:40, if no ACK received:
  m3 → back to pending, re-delivered to Consumer A or Consumer B

The consumer's processing pipeline:

The ordering rule: always ACK after processing, never before. This seems obvious but is a common implementation bug — some frameworks auto-ACK on receipt rather than on processing completion, giving at-most-once semantics when at-least-once is expected.

The visibility timeout must be longer than your longest processing time. If processing a message takes up to 30 seconds, set visibility timeout to 60+ seconds. If the timeout is too short, the queue redelivers messages to other consumers while the original consumer is still processing — causing duplicate processing even without crashes.

Part 4: Queue vs Pub-Sub — Two Delivery Patterns

The same physical log can be read in two fundamentally different ways, each enabling different architectural patterns.

Queue (point-to-point)

A queue is consumed by competing consumers: multiple consumer instances all read from the same queue, but each message is delivered to exactly one consumer. Messages are distributed across consumers — Consumer A gets some, Consumer B gets others. This is horizontal scaling: add more consumer instances to process more messages in parallel.

(Each message processed by exactly one consumer.)

The queue model is appropriate when: you have work items that should be processed once (send one email, create one record, run one job). Adding consumers scales throughput. RabbitMQ and SQS primarily implement the queue model.

Pub-Sub (publish-subscribe)

A topic is consumed by multiple independent consumer groups: each group gets a complete copy of every message. Within a group, messages may be distributed among instances (like a queue), but each group sees all messages independently of other groups.

(Each group independently processes all messages.)

The pub-sub model is appropriate when: multiple downstream systems need to react to the same event independently. An order event might need to update inventory, trigger fraud detection, send a confirmation email, and update analytics — all independently, all seeing the same order event. Adding a new consumer group doesn't require changing the producer. Chapter 13's Saga choreography pattern relies on pub-sub: each service subscribes to events it cares about.

Kafka implements pub-sub natively. Consumer groups in Kafka are independent — each group maintains its own offset into the topic. RabbitMQ implements pub-sub through "fanout exchanges" — a message is copied to multiple queues, each read by a different consumer.

Part 5: Message Ordering — What You Can and Cannot Guarantee

"Messages are processed in order" sounds like an obvious requirement. It's harder to achieve than it sounds, and the cost of achieving it is reduced parallelism.

Global ordering

All messages from all producers arrive at all consumers in the exact order they were produced. This is only achievable with a single producer and a single consumer with no parallelism. As soon as you have multiple consumers processing in parallel, consumer A might finish processing message 3 after consumer B finishes message 4 — the effective processing order is not the production order.

Global ordering is rarely necessary in practice. It's appropriate for: a single command sequence where each command depends on the previous (a database WAL, a game state machine). For most business events ("order created", "payment charged"), ordering within a single entity's events matters, but global ordering across all entities does not.

Partition-level ordering (the practical answer)

Kafka and other modern queue systems solve the ordering problem by partitioning: a topic is divided into N partitions. Messages are assigned to partitions by a key (e.g., order_id % N determines the partition). All messages for the same key go to the same partition. Within a partition, messages are strictly ordered. Across partitions, there is no ordering guarantee.

Topic order_events, 4 partitions. Within each partition, events for that order are in order. Across partitions, no ordering guarantee.

This gives you ordering where it matters — all events for a given order are in order — while allowing massive parallelism across partitions. A topic with 100 partitions can be consumed by up to 100 consumers in parallel, each processing one partition's worth of messages in strict order.

The consequence for producers: if ordering within an entity matters, all events for that entity must use the same partition key. If order #100's events go to different partitions (e.g., because you're using a random partition key), they may be processed out of order.

Out-of-order delivery under failure

Even with a single partition, at-least-once delivery can violate ordering. If message m5 fails processing and is retried while m6 and m7 have already been processed successfully, the effective processing order is m1, m2, m3, m4, m6, m7, m5. The retry of m5 arrives "out of order" relative to subsequent messages.

This is why strict ordering + at-least-once delivery is fundamentally in tension. Solving it requires either: (a) processing retries in order, blocking m6 and m7 until m5 succeeds (reduces throughput), or (b) designing consumers to be order-independent (events carry enough state to be applied regardless of order), or (c) using exactly-once semantics where retries don't skip ahead.

Before reading the four options, try it. The first two tabs are this section; the third is the poison message in Part 7.

Part 6: Backpressure — When Consumers Are Slow

Backpressure is what happens when producers write faster than consumers can process. The queue fills up. Messages accumulate. The queue grows without bound — until it runs out of disk space and crashes, or until processing falls so far behind that the messages are no longer relevant.

Backpressure scenario:

Producer rate:   10,000 messages/second
Consumer rate:    1,000 messages/second
Queue growth:     9,000 messages/second

After 1 hour:   9,000 × 3,600 = 32.4 million messages accumulated
At 1 KB/message: 32.4 GB of unprocessed messages

There are four ways to handle backpressure:

Accept and accumulate: Let the queue grow. This is the default behavior. It works if the backlog is temporary (traffic spike that will subside) and you have enough disk. Kafka is designed for this — it retains messages for days or weeks by default, and consumers catch up when they can. The risk: if the backlog grows faster than available disk, the queue crashes.

Scale consumers: Add more consumer instances to process the backlog. With a partitioned topic, add consumers up to the partition count. Beyond the partition count, adding consumers doesn't help (each consumer must have at least one partition, and partitions can't be shared). Scale consumers first — this is the correct response to sustained backpressure.

Drop or shed: Configure the queue with a maximum size. When full, reject new messages (producer gets an error) or drop the oldest messages to make room. This is appropriate for low-priority real-time data (live sensor readings where a 10-second-old reading is worthless) but disastrous for critical data (dropped order events = lost revenue).

Apply backpressure to the producer: Signal the producer that the consumer is overwhelmed. The producer slows down or blocks until consumers catch up. This is reactive flow control — used in reactive programming frameworks and streaming systems. The challenge: in distributed systems, communicating backpressure across a queue requires a feedback channel from consumer to producer, which many queue systems don't provide natively.

The operational monitoring metric: consumer lag — the difference in offset between the latest produced message and the consumer's current position. A growing consumer lag indicates backpressure. Decreasing lag means consumers are catching up. Flat lag at zero means the system is in steady state. Alert on consumer lag thresholds, not just error rates.

Part 7: Dead Letter Queues — Handling Poison Messages

A poison message is a message that consistently causes the consumer to crash or error. The consumer receives it, tries to process it, fails, NACKs it (negative acknowledgement), the queue redelivers it, the consumer fails again, and this repeats indefinitely. The consumer is stuck on one message and never makes progress on subsequent messages.

The solution: a dead letter queue (DLQ) — a separate queue where messages go after failing delivery a configured number of times (e.g., 5 attempts). After the 5th failed attempt, the message is moved to the DLQ instead of being retried. The consumer continues processing m5 and m6. The DLQ holds m4 for manual inspection and remediation.

DLQs are not just error handling — they're an observability tool. The DLQ tells you which messages your system couldn't process. A DLQ filling up is an alert that something is systematically wrong: a new message format your consumer doesn't understand, a dependency that's down, a bug introduced in a recent deploy. Monitor DLQ depth as a first-class operational metric.

Every message queue in production should have DLQs configured. Not having DLQs means poison messages either block the queue forever or are silently dropped — both outcomes are worse than inspecting failures.

Part 8: Push vs Pull — Two Delivery Models

In the push model, the queue proactively sends messages to consumers. The queue controls the delivery rate. SNS (Simple Notification Service), webhook delivery, and some RabbitMQ configurations push messages to consumers.

In the pull model, consumers request messages from the queue on their own schedule. The consumer controls the fetch rate. Kafka and SQS both use the pull model.

The pull model is almost always superior for distributed systems, for a fundamental reason connected to Chapter 9's backpressure discussion. In the push model, if consumers are slow, the queue pushes faster than consumers can handle — the consumer's in-memory buffer fills up, it falls behind, and either messages are dropped or the consumer crashes. The queue can't easily know when to slow down.

In the pull model, consumers naturally slow down when they're busy — they simply don't fetch the next batch until they've finished the current one. The queue doesn't need to know the consumer's processing rate; the consumer self-regulates. Backpressure is automatic.

The pull model also enables consumers to process messages in batches, which dramatically improves throughput. Instead of processing one message at a time (one DB write per message), a consumer can fetch 1,000 messages, insert them in a batch, and commit once. This reduces I/O overhead by orders of magnitude for I/O-bound consumers.

The only advantage of push: lower latency. A pushed message arrives at the consumer as soon as it's available — no polling delay. Pull consumers have a polling interval — if they poll every 100ms, they have up to 100ms of additional delivery latency. Kafka mitigates this with long polling — a pull request blocks on the server until a message is available or a timeout expires, giving push-like latency with pull's flow control.

Part 9: Message Queue Internals — How Durability Is Achieved

From Chapter 2: durability means a write survives crashes. For a database, the WAL provides this. For a message queue, the same principle applies: a message is durable once it's written to the queue's on-disk log and that write has been fsynced.

The fsync is the durability guarantee. Without fsync, the OS may buffer the write in its page cache (Chapter 2's kernel buffer), and a crash before the buffer is flushed means the message is lost even though the broker acknowledged it. This is the same trade-off as synchronous_commit = off in Postgres — you get lower latency but risk losing recent writes on crash.

Queue systems typically offer a durability knob: synchronous fsync per message (durable, slow), periodic fsync (configurable trade-off), or async write (fastest, not durable). For financial messages, use synchronous durability. For metrics or logs where occasional loss is acceptable, async is fine.

Replication adds another layer: a message is considered durable once written to the broker's disk AND replicated to N-1 other brokers. This mirrors Chapter 5's synchronous replication — the producer waits for the replica acknowledgement before receiving confirmation. In Kafka: acks=all means the leader waits for all in-sync replicas to acknowledge before returning success to the producer.

Part 10: RabbitMQ, SQS, and Redis Streams — Three Real Systems

Understanding three production queue systems reveals the design trade-offs in practice.

RabbitMQ

RabbitMQ is a traditional message broker implementing the AMQP (Advanced Message Queuing Protocol) standard. Its model: producers send messages to exchanges, exchanges route messages to queues based on routing rules, consumers read from queues.

The routing layer is RabbitMQ's distinguishing feature. Exchange types: direct (route by exact key), topic (route by pattern — order.*.created), fanout (send to all bound queues — pub-sub), headers (route by message headers). This routing flexibility lets you express complex message distribution patterns that Kafka (which has simpler routing) cannot express directly.

RabbitMQ stores messages in RAM by default (for speed) and persists to disk only when explicitly marked "persistent" or when memory pressure forces it. This makes RabbitMQ faster than Kafka for small workloads but less durable by default. In production: always enable message persistence (delivery_mode=2) and set queue durability.

RabbitMQ deletes messages after they're consumed and ACKed — queues don't retain history. This is the point-to-point model: messages are work items, consumed once. There is no concept of "replay from offset 0" — once consumed, messages are gone.

Weakness: RabbitMQ struggles with very high throughput (>100k messages/second) and very large message backlogs. When queues get large, performance degrades significantly as RabbitMQ pages messages to disk and back. For high-throughput event streaming, Kafka is the better choice.

Amazon SQS

SQS (Simple Queue Service) is Amazon's fully-managed message queue. It's the simplest queue to operate — no servers to manage, automatic scaling, pay per message. Standard SQS provides at-least-once delivery with best-effort ordering (not guaranteed). FIFO SQS provides exactly-once processing and strict ordering, at lower throughput (3,000 messages/second vs. nearly unlimited for Standard).

SQS's visibility timeout model: when a consumer reads a message, it becomes invisible to all other consumers for the visibility timeout period (default 30 seconds, max 12 hours). If the consumer doesn't delete the message within the timeout, it reappears in the queue. The consumer deletes the message after successful processing — this is the ACK. SQS does not use NACK — to immediately make a message visible again (force redelivery), the consumer changes the visibility timeout to 0.

SQS integrates deeply with AWS Lambda — Lambda functions can be triggered automatically when messages appear in an SQS queue, scaling concurrency with queue depth. For serverless architectures, SQS + Lambda is the standard event-driven pattern.

Limitation: SQS messages are delivered to one consumer at a time and deleted after consumption. No pub-sub (use SNS + SQS fan-out for pub-sub). No replay. No consumer groups in the Kafka sense.

Redis Streams

Redis Streams (Redis 5.0+) is a Redis data structure implementing a persistent, append-only log with consumer groups. It combines Redis's in-memory speed with the durability and consumer group semantics of a proper message queue.

A stream is an ordered sequence of entries, each with an auto-generated ID (timestamp + sequence number). Consumer groups track which entries each group has consumed, including which are in-flight (delivered but not ACKed). Entries that aren't acknowledged can be reclaimed by other consumers in the group using XAUTOCLAIM.

Redis Streams is excellent for: moderate throughput use cases (up to a few hundred thousand messages/second) where you want Redis's operational simplicity and already use Redis for caching. It's not appropriate for: very high throughput (Kafka handles tens of millions/second), very large message retention (Redis stores data in RAM; streams on disk require AOF/RDB persistence), or complex routing (Redis Streams has no exchange/routing layer).

The fan-out pattern with Redis: publish to multiple streams (one per consumer group), or use Redis Pub/Sub for fire-and-forget fan-out (no persistence, at-most-once). Redis Streams with multiple consumer groups gives at-least-once fan-out with persistence.

Part 11: Idempotent Consumers — The Critical Design Pattern

Because at-least-once delivery is the practical standard and exactly-once is expensive or unavailable, every consumer you write should be idempotent by design. This is the most important implementation discipline in message-driven systems.

An idempotent consumer processes a message once and produces the same result if it processes the same message again. The two techniques:

Natural idempotency: The operation itself is idempotent. SET user:42.email = 'alice@example.com' is idempotent — running it twice produces the same final state. INSERT INTO users (id, email) VALUES (42, 'alice@example.com') ON CONFLICT DO NOTHING is idempotent. DELETE FROM orders WHERE order_id = 123 is idempotent (deleting an already-deleted row is a no-op). Design operations to be naturally idempotent where possible.

Deduplication with message ID: For operations that are not naturally idempotent (e.g., INSERT INTO payments (amount) VALUES (99.99) — each call creates a new payment), track which message IDs have been processed. Before processing, check if this message ID was already processed. If yes, skip. If no, process and record the message ID atomically.

-- Idempotent payment processing
BEGIN;
  -- Check if already processed
  INSERT INTO processed_messages (message_id, processed_at)
  VALUES ('msg-abc-123', NOW())
  ON CONFLICT (message_id) DO NOTHING;
  
  IF rows_affected = 0 THEN
    ROLLBACK;  -- Already processed, skip
    RETURN;
  END IF;
  
  -- Process the payment
  INSERT INTO payments (amount, user_id, order_id)
  VALUES (99.99, 42, 456);
COMMIT;

The processed_messages table is the deduplication store. Its key is the message ID. The atomic INSERT + ON CONFLICT ensures that even if two consumer instances simultaneously receive the same message (possible under at-least-once delivery), only one will successfully insert the message ID and proceed with processing.

The deduplication store must be durable — if you store it in Redis with no persistence and Redis crashes, your deduplication history is lost and past messages would be reprocessed. Use a durable store (Postgres table, or Redis with AOF persistence) for deduplication records.

Message IDs must be assigned by the producer, not the consumer. If the producer sends the same message twice (after a network failure where it didn't receive the queue's ACK), both copies have the same ID and the consumer can deduplicate them. If the queue assigns IDs, two copies of the same message would get different IDs and both would be processed.

Part 12: When Not to Use a Message Queue

Message queues add operational complexity: another system to run, monitor, and debug. They introduce eventual consistency (the consumer processes the message after the producer has moved on). They make distributed tracing harder (a request spans multiple services and a queue, breaking the synchronous call chain). Use them deliberately, not by default.

Don't use a queue when you need a synchronous response. If the client must know the result before proceeding — a login check, a balance query, a product search — a queue introduces latency and complexity for no benefit. Use synchronous RPC (REST, gRPC).

Don't use a queue when the downstream is always fast and available. If your inventory service responds in 5ms and has 99.99% uptime, adding a queue between the order service and inventory service adds latency and complexity. The buffering and decoupling aren't needed. Direct RPC is simpler and faster.

Don't use a queue when exactly-once is truly required and unavailable. Some financial operations must be executed exactly once with no possibility of retry. If the queue system doesn't support transactions with your downstream database (making exactly-once impossible without idempotency design), direct database writes with distributed transactions (Chapter 13) may be more appropriate.

Don't use a queue to avoid thinking about distributed transactions. A common mistake: using a queue to "decouple" two operations that actually need to be atomic. If creating an order and reserving inventory must both succeed or both fail, putting a message in a queue between them doesn't solve the atomicity problem — it defers it to the consumer side, where it becomes a saga. If a saga isn't appropriate (you need strict atomicity), use a distributed transaction. A queue without saga semantics doesn't give you atomicity.

The correct use cases for message queues: absorbing traffic spikes, fan-out to multiple independent consumers, decoupling services where eventual consistency is acceptable, work queues for background processing, and event-driven Saga choreography (Chapter 13).

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

Your order processing consumer fetches a message from SQS with visibility timeout = 30 seconds. Processing involves: (a) validating the order, (b) calling the Stripe API to charge the card (takes up to 25 seconds), (c) inserting the order into the database, (d) deleting the message from SQS.

Trace exactly what happens in each failure scenario: (1) Consumer crashes between step (a) and (b). (2) Consumer crashes between (b) and (c) — card has been charged, order not created. (3) Consumer crashes between (c) and (d) — order created, message not deleted. (4) Stripe API takes 28 seconds, consumer finishes all steps at second 33.

For each scenario: what does the customer see? What is the queue state? What happens when SQS redelivers? Which scenarios require idempotency to be correct? Design the idempotency key and deduplication logic to handle scenario (2) without double-charging the customer.

Problem 2 — Design from Scratch

Design the message queue architecture for a real-time ride-hailing system (Uber/Ola). Events in the system: RideRequested, DriverAccepted, DriverArrived, RideStarted, RideCompleted, PaymentProcessed, RatingSubmitted. Consumers: pricing service, notification service (SMS/push), analytics pipeline, fraud detection, driver earnings service, support ticket service.

Specify: how many topics? What partition key for each? What consumer groups per topic? What delivery guarantee is required for each consumer group (at-least-once vs exactly-once, and why)? What is the maximum acceptable consumer lag for the notification service (SLA: driver acceptance notification within 5 seconds of ride request)? What ordering guarantees do you need — does it matter if RideStarted is processed before DriverArrived by the analytics pipeline? What happens if the fraud detection consumer falls 10 minutes behind — what is the business impact?

Problem 3 — Synthesis

Connect message queues to Chapter 13's Saga pattern and Chapter 6's consistency models.

The e-commerce order saga from Chapter 13 used choreography: each step publishes an event, the next service reacts. Now implement this in terms of specific queue operations.

The saga: OrderCreated → Inventory reserves → InventoryReserved → Payment charges → PaymentCharged → Shipping schedules → ShipmentScheduled. Each arrow is a message queue publish.

For each step: which queue/topic does the event go to? What is the consumer's idempotency key? What delivery guarantee is needed? Now consider: the payment service consumes InventoryReserved and publishes PaymentCharged. What consistency model (Chapter 6) describes the state visible to a concurrent read between these two publishes — inventory is reserved but payment not yet charged? Is this an acceptable intermediate state? Under what business rule does it become unacceptable? How does a DLQ change the saga behaviour if the payment service fails 5 times — what happens to the reserved inventory?

Problem 4 — Go Deeper

Pull-based consumers with long polling have a fundamental latency floor: messages are only fetched when the consumer polls, and batch size determines throughput vs latency trade-off.

A consumer fetches messages in batches of 100. Processing each batch takes 200ms. The topic receives 500 messages/second. Calculate: what is the consumer lag growth rate? At what batch size does the consumer keep up with the producer (zero lag growth)? What is the end-to-end latency (time from message produced to message processed) at that batch size? Now introduce 3 consumer instances each reading from a separate partition: how does this change the lag growth rate and end-to-end latency?

A second scenario: instead of batching, the consumer processes one message at a time. Processing takes 5ms per message. What is the maximum throughput (messages/second) for a single consumer? For 10 consumers across 10 partitions? At what processing latency per message does single-consumer throughput drop below the producer rate?

Problem 5 — Connect to Previous Chapters

Connect message queue durability to Chapter 2's WAL mechanics and Chapter 5's replication.

A Kafka broker receives a message from a producer. The producer uses acks=all (wait for all in-sync replicas to acknowledge). There are 3 brokers: a leader and 2 followers. Trace the full write path: where is the message written first, what is the role of the Kafka leader's WAL (segment files on disk), how does the leader replicate to followers (compare to Chapter 5's WAL sender/receiver), and what acknowledgement does the producer receive and when?

Now connect to Chapter 2's fsync discussion: Kafka's log.flush.interval.messages controls how often the leader fsyncs its segment files. If set to 1 (fsync every message), what is the performance impact per message using Chapter 1's latency numbers? If set to 10,000 (fsync every 10,000 messages), what is the durability risk — specifically, under what failure scenario is data loss possible? How does Kafka's replication (all ISR acknowledge before producer ACK) interact with fsync — if replication happens before fsync, is the message durable on followers even if the leader's OS buffer hasn't been flushed to disk?

What's Coming Next

This chapter built message queues from first principles: the append-only log, delivery semantics, acknowledgement mechanics, ordering constraints, and backpressure. Chapter 15 takes these foundations and builds Kafka — the distributed log that runs at internet scale. Kafka is not simply a "fast message queue." It's a rethinking of what a message queue is: an immutable, replayable, distributed log that serves simultaneously as a queue, a pub-sub system, a database change feed, a stream processing backbone, and an event store. Understanding Kafka means understanding why LinkedIn, Airbnb, Netflix, and Uber run their entire real-time data infrastructure through it. After Chapter 15, you'll understand why "just use a message queue" is like saying "just use a database" — the details of which one and why define your system's capabilities.

Chapter 14 complete. Problem 1 (SQS visibility timeout failure scenarios with Stripe) is the most practically grounding — the double-charge scenario happens in production more often than teams admit. Work through the idempotency key design for scenario 2 carefully: it requires an idempotency key that is known before the Stripe API call, survives a crash, and is checked before charging. Problem 2 (ride-hailing queue architecture) is a full system design sub-problem that appears in Uber and similar company interviews. The consumer lag SLA for notifications (5 seconds end-to-end) is the kind of constraint that drives partition count and consumer scaling decisions.