Q46 — What is the difference between a queue and a pub/sub system?

What are the two fundamental questions messaging systems answer?

Every messaging system answers two questions: who receives the message, and what happens to it after delivery?

The answer to "who receives" defines the model: one receiver (queue) or many receivers (pub/sub). The answer to "what happens after delivery" defines the retention model: delete the message (traditional queue) or keep it (log-based systems like Kafka).

Understanding the difference between queue and pub/sub is understanding which message distribution model fits your problem — and why many modern systems like Kafka collapse the distinction by supporting both.

What is a queue?

A queue implements point-to-point messaging: one message, one consumer. Messages enter at one end, consumers pull from the other. When a consumer successfully processes a message and acknowledges it, the message is gone — removed from the queue, not visible to anyone else.

If you have multiple consumers reading the same queue, the queue distributes work: consumer A gets message 1, consumer B gets message 2, consumer A gets message 3. Each message is processed by exactly one consumer. This is load balancing — the queue is a work distributor.

The semantics: a message in a queue represents a unit of work to be done. It's claimed by one worker, executed, and acknowledged. Nobody else needs to know it existed.

Classic examples: A job queue for processing uploaded videos — each video needs to be transcoded once, by one worker. An email sending queue — each email should be sent once. A payment processing queue — each transaction handled by one worker. In all of these, processing twice would be a bug.

Traditional implementations: AWS SQS, RabbitMQ (default mode), ActiveMQ, Redis with LPUSH/BRPOP. These are designed around work distribution — configurable visibility timeouts, dead-letter queues for failed messages, and deletion on acknowledgment.

What is pub/sub?

Pub/sub (publish-subscribe) implements a broadcast model: one message, many receivers. A publisher sends a message to a topic (or channel). All subscribers to that topic receive a copy of the message independently.

The semantics: a message in pub/sub represents an event that occurred. Multiple interested parties can react to it. Publisher and subscribers are fully decoupled — the publisher doesn't know or care who the subscribers are. New subscribers can be added without modifying the publisher.

Classic examples: A "user signed up" event published to a topic — the email service subscribes and sends a welcome email, the analytics service subscribes and records the signup, the onboarding service subscribes and creates the user's initial profile. All three react to the same event independently. Adding a new subscriber (say, a fraud detection service) requires no changes to the publisher.

Traditional implementations: AWS SNS, Google Cloud Pub/Sub, Redis Pub/Sub. These deliver to all current subscribers; subscribers that aren't connected when the message is published typically miss it (though Pub/Sub systems with durability can retain messages).

What each model gets wrong for certain use cases

Queue's limitation: A queue naturally distributes work among consumers, but it cannot broadcast. If you need the same message processed by three different services, you'd need to publish three copies to three different queues, or use a fan-out mechanism in front. The queue has no concept of "multiple independent consumers with different purposes."

Pub/sub's limitation: Pub/sub broadcasts to all subscribers, but it doesn't distribute work among instances of the same subscriber. If you have 5 instances of the notification service all subscribed to "user-signed-up" events, all 5 receive every event — you'd send 5 notifications per signup. Pub/sub has no concept of "among these 5 instances, only one should handle each message."

In practice, real systems need both behaviors simultaneously:

The notification service (pub/sub): the event must reach the notification service, the analytics service, and the audit service independently
Within the notification service (queue): one of the 5 notification service instances handles each event

How Kafka unifies both

Kafka's consumer group model (Q45) solves both problems with one abstraction:

Multiple consumer groups = pub/sub between services: Each service (notification, analytics, audit) creates its own consumer group. Each group reads every message on the topic independently. Adding a new service means creating a new consumer group — zero producer changes.

Multiple consumers within a group = queue within a service: The notification service's 5 instances form one consumer group. Each partition is assigned to one instance. Work is distributed. No notification is sent twice.

The same Kafka topic delivers pub/sub semantics across groups and queue semantics within groups — simultaneously, automatically, with no additional configuration.

This is why Kafka has largely replaced both traditional message queues and pub/sub systems in modern architectures. For simple, low-volume use cases, the operational complexity of Kafka isn't justified. But for any system at meaningful scale that needs durability, replayability, and both distribution patterns, Kafka's unified model is the right choice.

Message retention — the critical difference in practice

Beyond distribution semantics, there's a deeper operational difference in how these systems handle messages after delivery.

Traditional queues: Messages are deleted after acknowledgment. Once consumed, they're gone. There's no way to replay, no way to add a new consumer and have it process historical messages, no audit trail. The queue is ephemeral work storage.

Pub/sub systems: Messages are typically delivered to current subscribers and then discarded (or held briefly). Historical messages are usually inaccessible to new subscribers.

Kafka: Messages are retained on disk for a configurable period (7 days, 30 days, or indefinitely based on your retention policy). A new consumer group can start from offset 0 and read the entire history of the topic. A bug fix can reprocess all messages since the bug was introduced. A new service can bootstrap its state by replaying historical events.

This retention model is the foundation of event sourcing — treating all application state as derived from a durable, replayable log of events. It's only possible with a log-based system like Kafka, not with a traditional delete-on-consume queue.

Choosing between them in practice

Use a traditional queue (SQS, RabbitMQ) when:

Simple job/task processing — one consumer processes each task
Low volume — Kafka's operational complexity isn't justified
No need for message replay or historical consumers
Short message retention is acceptable

Use pub/sub (SNS, Google Pub/Sub) when:

Broadcasting events to multiple independent subscribers
No need for ordered processing or consumer group coordination
Fire-and-forget fan-out (coupled with SQS for durable delivery per subscriber: SNS → SQS)

Use Kafka when:

Both fan-out across services AND load balancing within a service
Message retention for replay (new consumers, debugging, reprocessing)
High throughput and ordering within partitions
Event sourcing or audit log use cases
Stream processing pipelines

The mental model

code

Queue (point-to-point):
  1 message → 1 consumer (claimed and processed by one worker)
  Distribution model: work queue / load balancing
  After delivery: message deleted
  Limitation: no fan-out to multiple independent consumers
  Use for: task queues, job processing, payment handling

Pub/sub:
  1 message → N subscribers (each gets a copy independently)
  Distribution model: broadcast / event notification
  After delivery: message discarded (or briefly retained)
  Limitation: all instances of same subscriber each get message (no work distribution)
  Use for: event notifications, fan-out to different services

Kafka: both simultaneously via consumer groups
  Within group: queue behavior (partitions distributed to consumers)
  Across groups: pub/sub behavior (each group reads all messages)
  + Retention: messages kept on disk (7 days, 30 days, forever)
  + Replay: any new consumer can start from offset 0
  
  Queue → ephemeral (delete on consume)
  Kafka → durable log (messages persist regardless of consumer state)

Rule of thumb:
  Simple job distribution, no history needed → SQS / RabbitMQ
  Pure fan-out, no consumer coordination → SNS / Pub/Sub
  Both patterns + high throughput + replay → Kafka

Queue and pub/sub are not competing technologies — they're different answers to different questions. The question is: what should happen to this message, and who needs to see it? The answer tells you which model you need.