Q45 — What are consumer groups in Kafka — why do they exist?

What happens without consumer groups?

Without any grouping concept, every consumer reading a Kafka topic reads the entire topic independently from its own offset. If you have 5 consumers reading the "orders" topic, each consumer reads every message — you get 5 copies of every order processed. This is fan-out: every subscriber gets every message.

For some use cases this is exactly what you want. An analytics system and a notification system both want to process every order independently. Neither cares what the other is doing. They maintain their own offsets and read at their own pace.

But for other use cases, this is wrong. Suppose you have a payment processing service, and you want to process orders faster by running 5 instances of it. You don't want each instance processing every order — you want each order processed by exactly one instance. The work should be divided, not duplicated.

Consumer groups solve both use cases simultaneously.

What is a consumer group?

A consumer group is a named group of consumers that cooperate to consume a topic. The key rule: each partition in the topic is assigned to exactly one consumer within the group at any given time.

If a topic has 12 partitions and a consumer group has 3 consumers:

• Consumer 1 gets partitions 0, 1, 2, 3
• Consumer 2 gets partitions 4, 5, 6, 7
• Consumer 3 gets partitions 8, 9, 10, 11

Each message (from a given partition) is delivered to exactly one consumer in the group at a time. No consumer is idle — they share the work. But "one consumer at a time" is not "processed once": if a consumer crashes or a rebalance moves its partition before it commits its offset, the new owner resumes from the last committed offset and processes those messages again. Consumers in a group must still be idempotent (Q43). This is queue-like behavior: work is distributed across workers.

From the perspective of the consumer group as a whole, every message is seen once. The group processes the topic collectively, with the workload parallelized across its members.

How does this give you both queue and pub/sub simultaneously?

The elegance of consumer groups: the behavior depends on how many groups you create.

One group → queue behavior: All consumers share the work. Each message goes to one consumer. This is a distributed work queue. Adding more consumers to the group parallelizes processing.

Multiple groups → pub/sub behavior: Each group independently reads every message. Group A (analytics service) sees all messages. Group B (notification service) sees all messages. Group C (audit service) sees all messages. Each group maintains its own offset, progresses at its own pace, and is completely independent of the others.

You get pub/sub between groups and queue within each group — both from the same Kafka topic, with no configuration changes to the producers.

This is why Kafka can replace both traditional message queues (RabbitMQ) and pub/sub systems (SNS) with a single abstraction. The consumer group is the mechanism that unifies them.

How does Kafka assign partitions to consumers?

Partition assignment within a group is managed by one of the brokers, called the group coordinator. When a consumer joins a group, it sends a JoinGroup request to the coordinator. The coordinator collects all group members and assigns partitions.

One consumer in the group is designated the group leader (not a broker — a consumer). The group leader receives the full membership list and executes the partition assignment algorithm, sending the result back to the coordinator, which distributes assignments to each consumer.

The default assignment strategy is range (assign consecutive partitions) or round-robin (alternate across consumers). Kafka 2.4+ added the sticky assignor, which tries to minimize partition movement during rebalances — keeping existing assignments stable and only reassigning what's necessary.

What is rebalancing — and why is it dangerous?

When the membership of a consumer group changes — a consumer joins, crashes, or leaves — Kafka must redistribute partitions. This is a rebalance.

During a rebalance:

1. The coordinator sends a signal to all consumers: "stop consuming and rejoin"
2. All consumers pause — they stop reading and processing messages
3. The group leader recomputes the partition assignment
4. Each consumer receives its new assignment
5. Consumers resume from their committed offsets on the new partitions

During this pause — the stop-the-world phase — no messages are being consumed. For a busy topic, this means lag accumulates. A rebalance that takes 10 seconds means 10 seconds of unprocessed messages, which may then need to catch up quickly.

Frequent rebalances are harmful. Common causes:

• Consumer heartbeat timeout (the consumer is alive but didn't send a heartbeat to the coordinator within session.timeout.ms) — coordinator thinks it's dead, triggers rebalance
• Consumer takes too long to process a batch (exceeds max.poll.interval.ms) — coordinator assumes it's stuck, removes it from the group, triggers rebalance
• Rolling deployment — consumers restart one at a time, each restart triggers a rebalance

Cooperative rebalancing (Kafka 2.4+): The incremental cooperative rebalancer avoids stop-the-world. Instead of revoking all assignments at once, it identifies only the partitions that must move and revokes those specifically. Consumers keep their existing partitions while the reassignment happens in two rounds. The pause is much shorter and affects only the moving partitions, not the entire group.

What happens when consumers exceed partition count?

There's a hard limit: the maximum parallelism within a consumer group is bounded by the number of partitions. If a topic has 12 partitions and a group has 15 consumers, 3 consumers will be idle — no partitions to assign to them.

This is why partition count is a critical topic configuration decision made at creation time. Adding partitions is possible but operationally complex (it changes the partition that keys hash to, breaking ordering guarantees for key-based messages). Most teams set partition count generously at topic creation (e.g., 50–100 for expected high-throughput topics) to allow future scaling of consumer groups without repartitioning.

What is committed offset and why does it matter for consumer groups?

Each consumer group maintains its own committed offset per partition — stored in the __consumer_offsets internal Kafka topic. This offset is the group's bookmark: "the next message we need to process is at this offset."

When a consumer crashes and a replacement starts (or an existing consumer takes over a partition during rebalance), it reads the committed offset and resumes from there. This provides at-least-once delivery within the group — messages after the last committed offset will be reprocessed.

The offset commit strategy matters:

Auto-commit (default): Kafka commits the offset periodically in a background thread (every 5 seconds by default). If the consumer crashes between commits, messages processed since the last auto-commit are reprocessed on restart. Simple but can cause duplicates.

Manual commit: The consumer explicitly calls consumer.commitSync() or commitAsync() after processing each batch. More control — you commit only after you've successfully processed, so you know exactly what's been done. Used for at-least-once guarantees with clear control over the commit point.

Transactional commit: Combined with idempotent producers or downstream transactional systems, the offset commit and the downstream write are made atomically — for exactly-once semantics within Kafka.

The mental model

Without groups: every consumer reads every message independently → pure pub/sub fan-out

Consumer group = set of consumers sharing partition assignments
  Rule: each partition assigned to exactly one consumer in the group at any time
  Effect within group: queue behavior — each message processed by one consumer
  Effect across groups: pub/sub behavior — each group sees all messages independently

Parallelism:
  12 partitions → up to 12 consumers in a group can work in parallel
  > 12 consumers → extras sit idle (no partitions to assign)
  → Set partition count generously at topic creation

Rebalance: triggered when group membership changes
  Classic: stop-the-world → all consumers pause → reassign → resume
  Cooperative (2.4+): only move necessary partitions → shorter pause
  Minimize rebalances: tune heartbeat/poll timeouts, use sticky assignor

Committed offset per group per partition:
  Group's bookmark for where it left off
  Auto-commit: every N seconds (simple, possible duplicates)
  Manual commit: after successful processing (explicit control, at-least-once)
  Transactional commit: offset + downstream write atomic (exactly-once)

One topic, multiple behaviors:
  1 group → distributed work queue (order processing, payment processing)
  N groups → each group independently processes all messages
            (analytics + notifications + audit, all from same topic)

Consumer groups are Kafka's mechanism for solving the "how do I scale consumers and support multiple independent subscribers" problem with a single, elegant abstraction. The partition-to-consumer assignment is simple, deterministic, and rebalanceable — making Kafka's consumer model both powerful and operationally predictable.