Phase 3 · Chapter 4

Kafka — Event Streaming at Scale

The append-only log that changed how large systems are built

What this chapter does to you: Kafka is not a message queue. That distinction matters. A message queue deletes a message after it's consumed. Kafka retains every event as a durable, ordered, replayable log. This one difference — the log — changes what you can build with it. After this chapter, you will understand Kafka's physical architecture, why it achieves millions of writes per second, how consumer groups work, and when to use Kafka instead of BullMQ. You will also understand the event sourcing pattern — the architectural consequence of treating your system's history as the source of truth.

Part 1: The Problem Kafka Solves

BullMQ Is Not Enough

Reading order note: this chapter compares Kafka to BullMQ, which is taught in Phase 4, Chapter 4 (Queues). If you haven't used BullMQ, read that chapter first and come back — or read on and treat BullMQ as "a Redis-backed job queue."

You've been using BullMQ (backed by Redis) for async jobs. It's great for:

• "Grade this test submission"
• "Send this email"
• "Update this leaderboard"

One producer adds a job. One worker consumes it. Job is deleted after processing.

Now consider a different problem. You have a financial transaction system. Transaction occurs. Multiple things must happen:

1. Update user balance (accounting service)
2. Send transaction email (notification service)
3. Update fraud detection model (ML service)
4. Write to audit log (compliance service)
5. Update real-time dashboard (analytics service)

With BullMQ: you'd need 5 separate queues. Each producer pushes to all 5. When you add a 6th consumer (say, a new reporting service), you modify the producer. The producer must know about every consumer. Tight coupling.

Also: if the ML service is down for an hour, you lose that hour's data forever — the queue messages were consumed and deleted by the other services, the ML service got nothing.

Kafka solves both problems:

• One event written once. Any number of consumers read it independently.
• Events are retained for days or weeks. A consumer can restart and replay.

Part 2: The Append-Only Log — Kafka's Core Idea

Kafka's fundamental data structure is a log — an append-only, ordered sequence of records.

Think of it as a file that only grows. Records are appended to the end. Each record gets an offset — its sequential position in the log.

offset:  0          1          2          3          4
         ┌──────────┬──────────┬──────────┬──────────┬──────────
         │ event A  │ event B  │ event C  │ event D  │ event E  →
         └──────────┴──────────┴──────────┴──────────┴──────────
         oldest                                              newest

Properties of this log:

• Writes are sequential (always append to end) — fastest possible disk I/O
• Each record is immutable — never updated, never deleted (until retention expires)
• Any consumer can read from any offset — multiple consumers don't interfere
• Consumers track their own position (offset) — Kafka doesn't track it per-consumer

The key difference from a queue: In BullMQ, when a worker processes a job, it's gone. The next worker gets the next job. In Kafka, reading a record doesn't remove it. Consumer A reads offset 5. Consumer B can also read offset 5. They're both reading the same log.

This is why Kafka can fan-out to arbitrary numbers of consumers without the producer knowing about any of them.

Part 3: Topics, Partitions, Offsets

Topic

A topic is a named log. It's the unit of data organisation in Kafka.

topic: "transactions"    → all financial events
topic: "user-events"     → all user actions (login, click, view)
topic: "test-submissions" → all test submission events from AttemptIQ

Producers write to topics. Consumers read from topics.

Partitions — The Unit of Parallelism

A topic is divided into partitions. Each partition is its own independent log.

topic: "test-submissions" with 6 partitions:

Partition 0: [sub:100] [sub:106] [sub:112] ...
Partition 1: [sub:101] [sub:107] [sub:113] ...
Partition 2: [sub:102] [sub:108] [sub:114] ...
Partition 3: [sub:103] [sub:109] [sub:115] ...
Partition 4: [sub:104] [sub:110] [sub:116] ...
Partition 5: [sub:105] [sub:111] [sub:117] ...

Why partitions? A single log can only be written to and read from at one speed — limited by one machine's disk I/O. Partitions let you spread the log across multiple machines (Kafka brokers), each handling a subset of the load.

6 partitions across 3 brokers = 2 partitions per broker. Write throughput scales horizontally with the number of partitions.

How records are assigned to partitions:

// With a key: hash(key) % numPartitions
producer.send({ topic: 'test-submissions', key: userId, value: submissionData })
// Same userId always goes to same partition → ordering guaranteed per user

// Without a key: round-robin across partitions
producer.send({ topic: 'test-submissions', value: submissionData })
// Maximises throughput, but ordering not guaranteed

The ordering guarantee: Kafka guarantees ordering within a partition, not across partitions. If you need "all events for user:123 in order," use userId as the key. All of user:123's events land in the same partition, in order.

Offsets — Your Position in the Log

Every record in a partition has an offset — a sequential integer starting at 0.

Partition 0:
offset 0: {sub:100, userId:A, score:85, timestamp:...}
offset 1: {sub:106, userId:B, score:92, timestamp:...}
offset 2: {sub:112, userId:A, score:78, timestamp:...}

Consumers track their current offset. "I've read up to offset 42." This offset is stored in a special Kafka topic: __consumer_offsets.

After processing, the consumer commits its offset: "I've processed everything up to offset 42. Next time I start, begin at 43."

If a consumer crashes at offset 38 (mid-processing), it restarts from 38. It reprocesses 38 and onward. This is why your Kafka consumers must be idempotent — exactly as we discussed for BullMQ jobs.

Part 4: Brokers, Replication, Leaders

Brokers

A Kafka cluster consists of multiple brokers — individual Kafka server processes. Each broker stores some partitions. Partitions are distributed across brokers.

Kafka Cluster (3 brokers):

Broker 1: Partition 0 (leader), Partition 3 (replica), Partition 5 (replica)
Broker 2: Partition 1 (leader), Partition 0 (replica), Partition 4 (replica)
Broker 3: Partition 2 (leader), Partition 1 (replica), Partition 3 (replica)

Each partition has one leader broker (handles all reads/writes for that partition) and N-1 replica brokers (maintain a copy of the partition log).

This is the leader election problem from Chapter 3 — applied to each partition. Kafka uses ZooKeeper (classic) or KRaft (modern, Kafka's own Raft implementation) to elect partition leaders.

Replication — Kafka's Durability

Every write to a partition leader is replicated to its replica brokers.

acks=all producer setting: wait for all in-sync replicas (ISR) to confirm. acks=1: only wait for leader. Faster but loses data if leader crashes before replication. acks=0: fire and forget. Maximum throughput. No durability guarantee.

ISR (In-Sync Replicas): The set of replicas that are caught up with the leader. A replica falls out of ISR if it's too far behind (configurable lag threshold). When leader fails, only ISR members are candidates for new leader — ensures no committed data is lost.

Why Kafka Is Fast — The Physics

Kafka achieves millions of writes per second per broker. Two physical reasons:

Sequential disk writes: Kafka only appends to the end of the log. Sequential writes on an SSD or even a spinning disk are dramatically faster than random writes. An SSD can do ~500MB/s sequential vs ~50MB/s random. 10× difference. Kafka is designed to saturate your disk's sequential write throughput.

Zero-copy transfers for reads: When a consumer reads from Kafka, the broker uses sendfile() — an OS syscall that transfers data from disk to network socket without copying it through user-space memory.

Normal read: disk → kernel buffer → user buffer → socket buffer → network
Zero-copy:   disk → kernel buffer ──────────────► socket buffer → network

Two fewer memory copies per read. At millions of reads per second, this is the difference between a Kafka broker using 30% CPU vs 80% CPU.

Part 5: Consumer Groups — The Scaling Mechanism

A consumer group is a set of consumers that collectively consume a topic.

Kafka assigns partitions to consumers within a group:

• Each partition is consumed by exactly one consumer in the group
• If consumers < partitions: some consumers handle multiple partitions
• If consumers > partitions: some consumers are idle (no partition assigned)

topic: "test-submissions" with 6 partitions
consumer group: "grading-workers" with 3 consumers

Consumer 1: Partition 0, Partition 1
Consumer 2: Partition 2, Partition 3
Consumer 3: Partition 4, Partition 5

The scaling property: To process faster, add more consumers (up to the number of partitions). 6 partitions → maximum 6 consumers processing in parallel. Want more parallelism? Increase partition count when creating the topic. (Changing partition count after topic creation is possible but complex — design partition count upfront based on expected throughput.)

Multiple consumer groups — independent consumption:

All three groups read all events independently. One group being slow or failing doesn't affect others. Each group maintains its own offset. Kafka just keeps the log.

Rebalancing — What Happens When Consumers Come and Go

The partition-to-consumer assignment is not fixed. A rebalance reassigns partitions whenever:

• a consumer joins the group (you scaled up, or deployed a new version)
• a consumer leaves cleanly (shutdown)
• a consumer is presumed dead: it stopped sending heartbeats (session.timeout.ms), or it took longer than max.poll.interval.ms between poll() calls — i.e. one message took too long to process

Before: C1 → P0,P1   C2 → P2,P3   C3 → P4,P5
C3 crashes ─▶ rebalance
After:  C1 → P0,P1,P4   C2 → P2,P3,P5
        C1 and C2 resume P4/P5 from the LAST COMMITTED offset
        → anything C3 processed but hadn't committed is processed AGAIN

Three things to know for interviews:

1. Rebalances cause duplicates. Work done after the last commit is redone by the new owner. Another reason consumers must be idempotent.
2. Rebalances cause pauses. With the classic "eager" protocol, every consumer in the group stops while partitions are reassigned. Frequent deploys or a slow handler that keeps tripping max.poll.interval.ms cause rebalance storms — throughput drops to near zero. Mitigations: the cooperative (incremental) rebalance protocol, static group membership (group.instance.id) so a quick restart doesn't trigger a rebalance, and keeping per-message processing time well under the poll interval.
3. A "zombie" consumer that was presumed dead but is actually just slow may still try to commit offsets or write output. Kafka rejects commits from a consumer that is no longer in the current group generation — the same fencing idea as in the Leader Election chapter.

Delivery Semantics — At-Most-Once, At-Least-Once, Exactly-Once

When you commit the offset decides what a crash does to you:

At-most-once:   read → COMMIT → process
                crash after commit, before processing → message LOST, never retried

At-least-once:  read → process → COMMIT          ← the default you should choose
                crash after processing, before commit → message processed TWICE

Exactly-once:   not achievable by commit order alone

At-least-once + an idempotent consumer (dedup by event ID, or upsert instead of insert) is what almost every production system uses. It is effectively-once: no loss, and duplicates are harmless.

The producer side has the same problem: if the broker's ACK is lost, the producer retries and the message is written twice. Fixes:

• Idempotent producer (enable.idempotence=true, the default in modern clients): each producer has an ID and per-partition sequence numbers, so the broker drops retried duplicates.
• acks=all + min.insync.replicas=2 (with replication factor 3): a write is only confirmed once at least two replicas have it. acks=all alone is weaker than it sounds — if the ISR has shrunk to just the leader, "all" means one copy.

Kafka "exactly-once" (EOS) = idempotent producer + transactions: a consumer that reads from topic A and writes to topic B can commit its output and its input offset atomically. It is real, but it only covers Kafka → Kafka. The moment your consumer writes to Postgres, sends an email, or calls an API, you are back to at-least-once + idempotency on that side effect.

Interview phrasing: "At-least-once delivery, idempotent consumers keyed on event ID. I'd use Kafka transactions only for a Kafka-to-Kafka stream processing stage."

Part 6: Kafka vs BullMQ — When to Use Which

This is the practical decision you'll make repeatedly.

Use BullMQ when:

• You have discrete tasks with clear completion (grade test, send email)
• You need retry with backoff and dead-letter queue
• Scale < 100k msgs/sec
• You don't need to replay events

Use Kafka when:

• Multiple independent consumers need the same events
• You need event replay (ML model retraining, audit, debugging)
• Scale > 100k msgs/sec
• You're building event sourcing or CQRS architecture
• Events must be retained for compliance or analysis

For AttemptIQ specifically:

• Test grading: BullMQ (one consumer type, clear job lifecycle)
• Analytics pipeline: Kafka (multiple consumers: dashboards, ML, reporting)
• Audit log: Kafka (compliance requirement to retain all events)

Part 7: Event Sourcing — The Architectural Pattern

Kafka enables a pattern called event sourcing:

Traditional approach: Store current state. Update it when things change.

UPDATE users SET score = 1850 WHERE id = 123

Current state is all you have. History is gone.

Event sourcing approach: Store every event that led to the current state. Derive state from events.

event: { type: "TEST_COMPLETED", userId: 123, score: 85, at: t1 }
event: { type: "TEST_COMPLETED", userId: 123, score: 92, at: t2 }
event: { type: "BONUS_AWARDED",  userId: 123, points: 50, at: t3 }

Current score = sum of all events = 85 + 92 + 50 = 227

Why event sourcing matters:

• Complete audit history: you know exactly what happened and when
• Replay: reprocess all events to recompute state (for bugs, new features)
• New consumers: add a new service that reads all historical events and builds its own view
• Debug: find the exact event that caused a bug

CQRS (Command Query Responsibility Segregation): Events are written to Kafka (the write side — commands). Multiple consumers read events and build their own read-optimised views (the read side — queries).

Each read model is optimised for its query pattern. No one DB must serve all reads. This is the architectural realisation of the read/write separation you've been building implicitly in the caching and queue patterns (Phase 4).

Part 8: Kafka in Production — The Operational Reality

Retention

Kafka retains messages based on:

• Time: log.retention.hours=168 (7 days default)
• Size: log.retention.bytes=1073741824 (1GB per partition)

Whichever limit is hit first triggers log segment deletion (oldest segments first).

Retention is per-topic. Financial events: 90 days. Clickstream: 7 days. Audit: 1 year.

Compaction — Infinite Retention for State

For some topics, you want to retain only the latest value per key — not the entire history.

log.cleanup.policy=compact

Event stream for user profiles:
offset 0: {key:"user:123", value:{name:"Trilok"}}
offset 5: {key:"user:123", value:{name:"Trilok S"}}
offset 9: {key:"user:123", value:{name:"Trilok Kumar"}}

After compaction:
offset 9: {key:"user:123", value:{name:"Trilok Kumar"}}  ← only latest retained

Compacted topics: effectively infinite retention but storage-efficient. Used for: configuration, state snapshots, anything where "latest value per key" is enough.

Consumer Lag — Your Most Important Metric

Consumer lag = the difference between the latest offset produced and the latest offset consumed by a consumer group.

Partition 0:
  Latest produced offset: 10,000
  Consumer group "grading" latest committed: 9,850
  Lag: 150 messages behind

Zero lag: consumers keeping up in real time. Growing lag: consumers falling behind — scale up workers or optimise processing.

Monitor consumer lag. Alert when it exceeds your latency SLA. Kafka exposes this via JMX metrics and the kafka-consumer-groups.sh tool.

Dead Letter Topic

Kafka has no built-in retry/DLQ like BullMQ. You implement it yourself:

consumer.run({
  eachMessage: async ({ message, topic, partition }) => {
    try {
      await processMessage(message)
    } catch (error) {
      // Failed — check retry count
      const retries = parseInt(message.headers['retry-count'] || '0')

      if (retries < 3) {
        // Republish to retry topic with incremented count
        await producer.send({
          topic: `${topic}.retry`,
          messages: [{
            ...message,
            headers: { 'retry-count': String(retries + 1) }
          }]
        })
      } else {
        // Exhausted retries — send to dead letter topic
        await producer.send({
          topic: `${topic}.dead`,
          messages: [message]
        })
      }
    }
  }
})

Part 9: The Vocabulary Map

Part 10: Practice Problems

Problem 1 (Partition Design): You're adding Kafka to AttemptIQ. You create a topic test-submissions.

Expected throughput: 10,000 submissions/day peak. You want to process submissions in order per student (student's submissions must be graded sequentially). You expect to run 5 grading workers at peak.

How many partitions do you create? What do you use as the message key and why? What is the maximum parallelism you could achieve with this topic? If you later need 10 workers, what do you change?

Problem 2 (Consumer Groups — Fan-out): Your test-submissions topic is consumed by 3 consumer groups:

• grading-workers (5 consumers)
• analytics-workers (2 consumers)
• notification-workers (1 consumer)

The topic has 10 partitions.

Draw the partition assignment for each consumer in each group. If one grading-worker crashes, what happens to its partitions? If you add a 6th grading-worker, what happens?

Problem 3 (Ordering Guarantee): User A submits test X (offset 42, partition 3). User A immediately submits test Y (offset 43, partition 3). Your grading consumer crashes after processing offset 42 but before committing.

When the consumer restarts, from which offset does it begin? Does it reprocess offset 42? If reprocessing offset 42 triggers a re-grade, what could go wrong? How do you make your grading consumer idempotent?

Problem 4 (Kafka vs BullMQ): AttemptIQ is adding a new "learning recommendations" feature. When a student completes a test, the recommendation engine must: a) Analyse the student's weak topics b) Select 5 relevant practice questions c) Send them a personalised push notification

Additionally:

• A new "parent dashboard" feature needs to read the same test completion events to show parents their child's progress in real time

Should you use Kafka or BullMQ for test-completion events? If Kafka: design the topic, partition key, and consumer groups. If BullMQ: explain how you handle the parent dashboard requirement.

Problem 5 (The Final Synthesis): You are the backend architect of AttemptIQ. The company has grown. You now have:

• 500,000 students taking tests simultaneously at peak
• 10 microservices consuming test submission events
• A compliance requirement to retain all events for 7 years
• A data science team that needs to retrain their ML models monthly using 6 months of historical submission data

The current BullMQ setup cannot handle this. Design the migration from BullMQ to Kafka:

1. Which topics do you create? (name them, give partition count and reason)
2. Which consumer groups exist for each topic?
3. How do you handle the transition — new events go to Kafka, old BullMQ jobs must still complete?
4. How do you ensure no events are lost during migration?
5. What retention policy for compliance (7 years) vs hot analytics (30 days)?

Use vocabulary from Phases 1–3 — this is the capstone exercise for distributed systems.

Where You Are

Phase 1 — Foundation
  ✅ Chapter 1: Processes, Threads, Memory, Event Loop
  ✅ Chapter 2: DNS, TCP, TLS, HTTP
  ✅ Chapter 3: Database Internals (Postgres)

Phase 2 — Databases in Depth
  ✅ Chapter 1: Transactions, ACID & Isolation
  ✅ Chapter 2: Replication, WAL & Lag
  ✅ Chapter 3: Sharding
  ✅ Chapter 4: Query Optimisation & EXPLAIN

Phase 3 — Distributed Systems
  ✅ Chapter 1: CAP Theorem & Consistency Models
  ✅ Chapter 2: Consistent Hashing
  ✅ Chapter 3: Leader Election & Raft
  ✅ Chapter 4: Kafka — Event Streaming

Phase 4 — Backend Patterns & Production Infrastructure
  ⬜ Chapter 1: Caching (Redis)
  ⬜ Chapter 2: Rate Limiting
  ⬜ Chapter 3: Authentication
  ⬜ Chapter 4: Queues (BullMQ)
  ⬜ Chapter 5: Observability
  ⬜ Chapter 6: Load Balancing

Phase 5 — System Design Case Studies
  ⬜ Chapter 1: URL Shortener
  ⬜ Chapter 2: Twitter Feed
  ⬜ Chapter 3: YouTube
  ⬜ Chapter 4: WhatsApp
  ⬜ Chapter 5: Uber
  ⬜ Chapter 6: Google Docs (real-time collaboration)
  ⬜ Chapter 7: Notification System

You now have the theory: why distributed data disagrees, how it is partitioned, how nodes agree on a leader, and how an append-only log decouples whole systems.

Phase 4 is next: the patterns you'll reach for every day — caching, rate limiting, auth, queues, observability, load balancing. Each one will feel more solid because you already know what happens underneath (Redis is single-threaded, replicas lag, locks need fencing, consumers must be idempotent).

The work that remains, for every chapter: reading is not owning. Work every practice problem and speak your answers out loud — the inability to explain something clearly is the signal that you don't yet own it.

End of Phase 3, Chapter 4. Do not proceed to Phase 4 until you can explain, out loud, what happens to an in-flight message when a consumer crashes before committing its offset.