Chapter 15: Kafka — The Distributed Log

Curriculum: Distributed Systems from First Principles Chapter: 15 of 24 Part: 6 — Communication Prerequisites: Chapter 1 (sequential vs random I/O), Chapter 2 (WAL, sequential append), Chapter 5 (replication, ISR), Chapter 10 (LSM Trees, sequential writes), Chapter 14 (message queues, delivery semantics, consumer groups) Estimated reading time: 90–100 minutes

Opening

LinkedIn had a problem in 2010. Their activity data — page views, clicks, searches, profile updates — was flowing through dozens of pipelines: one for real-time recommendations, one for analytics, one for A/B testing, one for the activity feed, one for search indexing. Each pipeline had its own data ingestion system. Each one was slightly different. Adding a new data consumer meant building yet another pipeline. The pipelines were fragile, not replayable, and couldn't be replayed if something went wrong downstream. Jay Kreps, Neha Narkhede, and Jun Rao decided to solve the root problem: they needed a single, durable, replayable log that any number of consumers could read independently, at their own pace, from any point in time.

The result was Kafka, open-sourced in 2011. But Kafka is widely misunderstood as "a fast message queue." It is not primarily a message queue. It is a distributed commit log — the same log abstraction you met in Chapter 2 (database WAL) and Chapter 5 (replication log), but made the primary product rather than an implementation detail. The key insight that Kreps later wrote about in "The Log: What every software engineer should know about real-time data" (2013): the log is not just a mechanism inside a database — it is the correct abstraction for integrating all the data systems in a company. Every state change, every event, every write in your system can be represented as an entry in a log. That log becomes the single source of truth that all other systems derive their state from. This chapter builds Kafka from the physical storage level up to that architectural vision.

Part 1: Kafka's Physical Storage — What's Actually on Disk

Kafka stores data in topic partitions. A topic is divided into N partitions (configured at creation, changeable with caveats). Each partition is an ordered, immutable sequence of messages. Each broker in the cluster stores some partitions.

On disk, each partition is stored as a series of segment files in a directory. A segment file is just an append-only log file — exactly the same sequential append from Chapter 10. When a segment reaches its configured size limit (default: 1 GB) or age limit (default: 7 days), it is rolled — the current segment is sealed (no more writes) and a new segment is created.

Directory for partition: orders-topic-partition-3/

Segment files:
  00000000000000000000.log       ← segment starting at offset 0
  00000000000000000000.index     ← sparse offset index for this segment
  00000000000000000000.timeindex ← sparse time index for this segment
  00000000001073741824.log       ← segment starting at offset 1,073,741,824
  00000000001073741824.index
  00000000001073741824.timeindex
  00000000002147483648.log       ← current (active) segment
  00000000002147483648.index
  00000000002147483648.timeindex

The filename is the base offset of the first message in that segment — a 20-digit zero-padded number. This naming scheme enables binary search: to find offset 1,500,000,000, Kafka looks for the segment file with the largest base offset ≤ 1,500,000,000, finds 00000000001073741824.log, then uses its index to seek to the right position within the file.

The .index file is a sparse offset index — analogous to the sparse SSTable index from Chapter 10. It stores (relative_offset, file_position) pairs every N bytes (default every 4 KB). To find a specific offset: binary search the index file to find the nearest entry, then scan the log file from that position. This turns a full-file scan into two small seeks — one to the index, one to the approximate log position.

The .timeindex file is a sparse index by timestamp — allows fetching messages by time (consumer.seekToTimestamp(timestamp)) rather than offset. Critical for: replaying from a specific point in time after a disaster, or implementing time-windowed stream processing.

This storage design — segment files + sparse indexes — is why Kafka performs well even with terabytes of data per broker. No random writes ever touch message data. Reads are sequential reads through segment files. The OS page cache (Chapter 2's kernel buffer) caches recently-read segments in RAM automatically. Kafka is deliberately designed to let the OS handle caching rather than building its own buffer pool.

Part 2: Why Kafka Is Fast — The I/O Performance Story

Kafka's throughput numbers are remarkable: a single broker can sustain 100–500 MB/s of throughput on commodity hardware. Understanding why requires Chapter 1's I/O latency numbers and two OS-level mechanisms.

Sequential I/O exploited maximally

From Chapter 1: sequential writes to an SSD: ~500 MB/s. Random writes: ~0.1 MB/s. Kafka's entire write path is sequential. Producers write to the active segment's end. No reads. No random seeks. No in-place updates. Pure append. This is the same insight as LSM Trees (Chapter 10), applied to message storage.

From Chapter 1: reading sequentially from RAM (page cache): ~10 GB/s. Reading from SSD: ~500 MB/s. A broker that has recently-written data in the OS page cache serves reads at RAM speed — effectively 10 GB/s — without any custom caching logic. Kafka explicitly designs for this: it writes messages in the same binary format they'll be read in, allowing the OS to serve reads directly from the page cache without deserialization.

Zero-copy transfer

The typical data path for serving a file over a network: read file from disk → kernel buffer (page cache) → user space buffer (application memory) → kernel socket buffer → network. That's 4 copies and 2 system calls.

Kafka uses the sendfile system call (Linux) or transferTo (Java NIO), which copies data directly from the OS page cache to the network socket: page cache → network. Two copies eliminated. This is called zero-copy because no user-space copies occur. For a broker serving 500 MB/s of read traffic, zero-copy reduces CPU usage by 50–70% compared to a standard file-serving approach.

Batching and compression

Producers don't send one message at a time. They batch messages into a ProducerBatch — a buffer of up to batch.size bytes (default 16 KB) or linger.ms milliseconds (default 0 ms — flush immediately, but can be set to 5ms to allow batching). The entire batch is sent in one network request, compressed together, and written to the broker in one sequential append.

Compression of a batch of related messages (e.g., 100 JSON order events) is far more effective than compressing each message individually — the repeated structure across messages (field names, schema) is amortised across the batch. Kafka supports LZ4 (fast, moderate compression), Snappy (fast, Google's codec), zstd (better compression ratio, slower), and GZIP (best compression, slowest). LZ4 is the standard choice for throughput-sensitive workloads.

Without batching: 100 messages × 1 KB = 100 network requests, 100 KB written
With batching + compression:
  100 messages × 1 KB = 100 KB → compressed to ~20 KB → 1 network request
  Throughput: 5× higher, network bandwidth: 5× lower

The combination of sequential writes, zero-copy reads, and batch compression is why Kafka achieves throughput an order of magnitude higher than traditional message brokers.

Part 3: Replication — ISR and What It Means for Durability

Each partition has one leader and N-1 followers (replicas). The leader handles all reads and writes. Followers replicate from the leader — pulling new messages at their own pace, similar to Chapter 5's streaming replication.

The set of replicas that are "caught up" with the leader is called the In-Sync Replica set (ISR). A replica is in-sync if it has replicated all messages up to a lag threshold (configurable: default replica.lag.time.max.ms = 30 seconds). A replica that falls too far behind is removed from the ISR — it's a lagging replica that can't be trusted to take over if the leader fails.

ISR example (3 brokers, replication-factor=3):

Leader (Broker 1):   [m1][m2][m3][m4][m5]  ← latest offset: 5
Follower (Broker 2): [m1][m2][m3][m4][m5]  ← caught up: in ISR
Follower (Broker 3): [m1][m2][m3]           ← lagging: removed from ISR

ISR = {Broker 1, Broker 2}

Producer acknowledgement and min.insync.replicas

The acks producer configuration determines when the broker acknowledges a write:

acks=0: The producer sends the message and doesn't wait for any acknowledgement. Fire and forget. No durability guarantee. Highest throughput. Used for metrics/logs where loss is acceptable.

acks=1: The broker acknowledges after the leader writes to its local log (before replicating). If the leader crashes before replication, the message is lost. Moderate durability, good throughput.

acks=all (or -1): The broker acknowledges only after all ISR members have written the message. Combined with min.insync.replicas, this is Kafka's strongest durability guarantee.

min.insync.replicas is a broker-level setting (e.g., set to 2). When a producer sends with acks=all, the leader waits until at least min.insync.replicas have acknowledged before confirming. If the ISR has fallen below min.insync.replicas (e.g., two brokers are down and ISR has only 1 member), the leader rejects writes with a NotEnoughReplicasException. The partition is unavailable for writes rather than accepting writes that might be lost.

Safe configuration for no data loss:
  replication-factor = 3
  min.insync.replicas = 2
  producer acks = all

This means:
  - A message is acknowledged only after 2 of 3 brokers have it
  - If only 1 broker is available (ISR size < 2), writes are rejected
  - Can tolerate 1 broker failure without data loss (2 copies remain)
  - Tolerates 0 broker failures for availability (if 2 go down, writes stop)

This is the same safety/liveness trade-off from Chapter 11: you can always add more replicas (more safety) but availability decreases as min.insync.replicas increases. The correct configuration depends on your durability vs availability requirement.

Leader election and unclean leader election

When a partition leader fails, Kafka elects a new leader from the ISR. An ISR member is guaranteed to have all committed messages (same logic as Chapter 12's Raft election restriction — only up-to-date nodes can lead).

If the entire ISR is down and only out-of-sync replicas are available, Kafka faces a choice: wait for an ISR member to recover (availability sacrifice), or elect an out-of-sync replica (data loss risk). The unclean.leader.election.enable setting controls this. Default: false (don't elect a lagging replica). Set to true only when availability is more important than data loss — for example, log aggregation pipelines where losing a few messages is acceptable.

Part 4: Consumer Groups and Partition Assignment

Chapter 14 introduced consumer groups conceptually. Here's how they work mechanically.

A consumer group is identified by a group.id string. When consumers join a group, one broker is designated the group coordinator for that group. The group coordinator manages group membership and triggers rebalancing — reassigning partitions to consumers when membership changes.

Partition assignment is simple: each partition is assigned to exactly one consumer in the group. If there are more consumers than partitions, some consumers are idle. If there are more partitions than consumers, some consumers handle multiple partitions.

Topic: orders (8 partitions: P0–P7)
Consumer group: order-processors (3 consumers: C1, C2, C3)

Assignment after rebalancing:
  C1: P0, P1, P2
  C2: P3, P4, P5
  C3: P6, P7

When C3 crashes, a rebalance is triggered:

  C1: P0, P1, P2, P6
  C2: P3, P4, P5, P7

The rebalancing problem

Rebalancing is disruptive. During a rebalance, all consumers in the group stop consuming (the "stop the world" rebalance in older Kafka versions). They revoke their partitions, the coordinator reassigns, and consumers resume. For the duration of the rebalance (typically 1–10 seconds), no messages are processed and consumer lag grows.

Frequent rebalances happen when: consumers crash and rejoin repeatedly, new consumer instances are deployed (rolling deploys), or the session timeout fires due to slow processing (consumer takes too long to heartbeat the coordinator).

Kafka 2.4+ introduced incremental cooperative rebalancing (replacing the stop-the-world rebalance). Consumers only revoke the specific partitions being reassigned — not all partitions. Consumers that keep their partition assignments continue processing during the rebalance. Only the moving partitions experience a pause. This dramatically reduces the impact of frequent rebalances.

Offset management

The consumer's position in each partition is stored as an offset — the index of the next message to consume. Offsets are committed to a special Kafka topic: __consumer_offsets. This topic is replicated like any other Kafka topic.

Consumer commits:
  { group: "order-processors", topic: "orders", partition: 0, offset: 1247 }
  { group: "order-processors", topic: "orders", partition: 1, offset: 983 }
  ...

If C1 crashes and restarts, it reads its committed offsets from __consumer_offsets
and resumes from offset 1247 on partition 0 and 983 on partition 1.

The commit is what determines delivery semantics:

Commit before processing (auto-commit or premature manual commit): Consumer reads message at offset 1247, commits offset 1248 (next to read), then crashes mid-processing. On restart, resumes from 1248 — message 1247 is never reprocessed. At-most-once semantics.

Commit after processing (manual commit post-processing): Consumer reads 1247, processes it, then commits 1248. If it crashes after processing but before committing, it resumes from 1247 and reprocesses. At-least-once semantics.

Commit atomically with processing result: Consumer reads 1247, writes the processing result and the offset 1248 in a single database transaction. If the database commits, both the result and the offset are persisted atomically. If the database aborts, neither is saved — the consumer retries from 1247. Exactly-once semantics (within the boundary of Kafka + this database). This is the Kafka Streams transactional model.

In code, always use enable.auto.commit=false and commit manually after processing. Auto-commit (the default) commits on a timer regardless of processing state — giving at-most-once semantics that most teams don't intend.

Part 5: Log Retention vs Log Compaction — Two Ways to Manage Storage

Kafka partitions grow indefinitely unless managed. Two strategies control partition size.

Time/size-based retention (the default)

Segments older than log.retention.hours (default 168 hours = 7 days) or larger than log.retention.bytes are deleted. The broker deletes entire segment files — since segments are sealed and never modified, deletion is just unlink() on the file. O(1) deletion, no compaction needed.

This is appropriate for: event streams where old events have no ongoing value, pipelines where consumers are expected to process within the retention window, and high-volume streams where storing all history is prohibitively expensive.

The retention window must be long enough that: the slowest consumer can process through a full retention period's worth of data without the start of the window being deleted. If your consumer is 12 hours behind and retention is 24 hours, you have 12 hours before the consumer's position falls off the retention cliff — the messages it needs to read have been deleted. Alert on consumer lag approaching the retention window.

Log compaction

Log compaction is Kafka's mechanism for keeping only the latest value for each key — turning the partition into a key-value store where each key's most recent value is always readable.

Before compaction:
Offset 0:  {key: "user:42", value: {name: "Alice", email: "old@example.com"}}
Offset 1:  {key: "user:17", value: {name: "Bob"}}
Offset 2:  {key: "user:42", value: {name: "Alice", email: "new@example.com"}}
Offset 3:  {key: "user:42", value: {name: "Alice", email: "newest@example.com"}}
Offset 4:  {key: "user:17", value: null}  ← tombstone (delete marker)

After compaction (old user:42 entries removed, user:17 tombstone retained briefly):
Offset 3:  {key: "user:42", value: {name: "Alice", email: "newest@example.com"}}
Offset 4:  {key: "user:17", value: null}  ← tombstone (kept until configurable delay)

After compaction, a consumer reading from the beginning of the partition sees only the latest value for each key. This enables log compaction as a persistent key-value store — a consumer that starts fresh doesn't need to replay all history; it reads the compacted partition to get current state.

Log compaction is what makes Kafka useful as a changelog — a database change feed that can be replayed to reconstruct the current state. Change Data Capture (CDC, discussed in Part 8) uses log-compacted topics to represent the current state of database tables.

Tombstones (null value messages) are retained for a configurable period (delete.retention.ms, default 24 hours) to allow consumers to see the deletion before the tombstone itself is compacted away. After the delay, the tombstone is removed in the next compaction — equivalent to Chapter 10's LSM Tree tombstone removal during compaction.

Part 6: Exactly-Once Semantics — Idempotent Producer and Transactions

Chapter 14 explained that exactly-once is "effectively exactly-once via idempotent at-least-once." Kafka's implementation is worth understanding precisely.

Idempotent producer

Each producer is assigned a Producer ID (PID) by the broker. Each message in a batch is given a sequence number that monotonically increases per partition. The broker tracks the highest sequence number received from each (PID, partition) pair. If a message arrives with a sequence number ≤ the tracked maximum (a duplicate from a retry), it's silently discarded. If a message arrives with a sequence number that skips (gap indicates lost messages), the broker returns an error.

This makes the producer idempotent: retrying a failed batch (the producer doesn't know if the broker received it) won't cause duplicate messages. The idempotent producer setting: enable.idempotence=true (default since Kafka 3.0).

Transactional producer

For exactly-once across multiple partitions (producer writes to partition A and partition B atomically), Kafka uses a transactional API. The producer begins a transaction, writes to multiple partitions, and commits the transaction. Consumers configured with isolation.level=read_committed only see messages from committed transactions.

Producer:
  producer.beginTransaction()
  producer.send(topicA, "message for partition A")
  producer.send(topicB, "message for partition B")
  producer.commitTransaction()  ← both writes become visible atomically
  
Consumer (isolation.level=read_committed):
  reads topicA → sees "message for partition A" only after commitTransaction()
  reads topicB → sees "message for partition B" only after commitTransaction()

Internally, transactions work by writing a transaction marker (commit or abort) to all affected partitions after the commit. The broker holds transactional messages invisible to read_committed consumers until the marker arrives. This is a two-phase commit where the transaction coordinator is a special broker role backed by Kafka's internal __transaction_state topic.

Exactly-once end-to-end (read-process-write) requires: idempotent producer + transactional API + committing the input offset atomically with the output write. Kafka Streams implements this transparently. Manual implementations require explicitly including the consumer offset commit in the producer transaction:

producer.beginTransaction()
producer.send(outputTopic, processedResult)
producer.sendOffsetsToTransaction(inputOffsets, consumerGroupId)
producer.commitTransaction()

The performance cost: read_committed isolation adds read latency (consumers must wait for transaction markers) and producer commit adds a round trip. For exactly-once pipelines, expect 2–5× higher latency compared to at-least-once.

Part 7: Kafka Streams and the Stream-Table Duality

Kafka's most intellectually interesting contribution beyond being a fast queue is the concept of stream-table duality — the observation that a stream of events and a table of state are two representations of the same information.

A stream is a sequence of events ordered by time: "at 10:00, user 42 clicked product 99. At 10:01, user 42 added product 99 to cart. At 10:02, user 42 checked out."

A table is the current state: "user 42 has product 99 in their cart."

The duality: a table is a stream folded over time — each table entry is the result of applying all stream events for that key up to now. A stream is a table unfolded over time — each event is a change to the table state. Converting between them: aggregate a stream to get a table (apply all events, keep latest state). Differentiate a table to get a stream (emit an event for each change — this is CDC).

Stream → Table (aggregation):
Event: user:42 cart=[]            → Table: {user:42: cart=[]}
Event: user:42 cart=[product:99]  → Table: {user:42: cart=[product:99]}
Event: user:42 cart=[p:99, p:12]  → Table: {user:42: cart=[product:99, product:12]}

Table → Stream (change log):
Table changes:
  user:42 email updated → emit {key: user:42, value: {email: new@example.com}}
  user:17 deleted       → emit {key: user:17, value: null}  (tombstone)

Log compaction is the mechanism that makes a Kafka topic behave like a table — the compacted topic always holds the latest value for each key. This is why log-compacted topics serve as the persistent representation of a KTable in Kafka Streams.

Kafka Streams is a Java library (not a separate cluster) for stream processing on top of Kafka. It lets you write processing logic that reads from Kafka topics, transforms/aggregates/joins, and writes results back to Kafka topics — all with exactly-once guarantees and with local state stores backed by RocksDB (Chapter 10).

Stream processing example: order revenue per user

KStream<String, Order> orders = builder.stream("orders");
KTable<String, Double> revenuePerUser = orders
  .groupByKey()
  .aggregate(
    () -> 0.0,
    (userId, order, currentRevenue) -> currentRevenue + order.amount(),
    Materialized.as("revenue-store")  ← stored in local RocksDB
  );
revenuePerUser.toStream().to("revenue-per-user");

The local RocksDB state store is backed by a Kafka changelog topic (log-compacted). If the stream processor restarts, it restores state from the changelog rather than reprocessing the entire input. This is the Kafka Streams recovery model: fail fast, restore from log.

Part 8: Kafka Connect and Change Data Capture

Kafka Connect is a framework for moving data between Kafka and external systems. It manages: source connectors (pulling data from databases, APIs, files into Kafka) and sink connectors (pushing data from Kafka into databases, data warehouses, search indexes).

The most powerful source connector pattern is Change Data Capture (CDC) — reading a database's change log (the WAL from Chapter 5) and converting every database change (INSERT, UPDATE, DELETE) into a Kafka message.

Debezium is the dominant open-source CDC framework. It connects to Postgres, MySQL, MongoDB, Oracle, and others via their replication protocols (Postgres logical decoding, MySQL binlog). Every committed change in the database produces a Kafka message containing the before-and-after state of the row:

Postgres row update: UPDATE users SET email='new@example.com' WHERE id=42

Debezium Kafka message (topic: postgres.public.users):
{
  "op": "u",  // update
  "before": {"id": 42, "name": "Alice", "email": "old@example.com"},
  "after":  {"id": 42, "name": "Alice", "email": "new@example.com"},
  "ts_ms": 1700000000000,
  "source": {"db": "production", "table": "users", "lsn": 12345678}
}

This message carries enough information to reconstruct the full change — useful for: keeping a search index (Elasticsearch) in sync with the database, maintaining a cache (Redis) invalidation stream, building read models in event-sourced systems, feeding an analytics data warehouse, auditing who changed what and when.

The beauty of CDC via Kafka: the database doesn't know it's being watched. Debezium reads the WAL (a standard replication slot in Postgres) without any application-level changes. You can add new consumers of database changes without touching the application or the database schema. The decoupling is complete.

The risk: the Postgres replication slot used by Debezium holds WAL segments until Debezium has processed them (Chapter 5's replication slot disk-filling danger). If Debezium falls behind or stops, the WAL accumulates on disk without bound. Monitor the replication slot lag and alert aggressively.

Part 9: KRaft — Kafka Without ZooKeeper

Kafka historically used ZooKeeper (a ZAB-based distributed coordination service, similar to etcd) for: storing cluster metadata, tracking broker membership, electing partition leaders, storing consumer group offsets (before __consumer_offsets topic).

ZooKeeper as a separate system added operational complexity: two systems to manage, two failure domains to monitor, a separate consensus cluster that Kafka operators often misconfigured. Kafka 2.8 introduced KRaft (Kafka Raft) — an embedded Raft implementation within Kafka itself, making ZooKeeper optional. Kafka 3.3 declared KRaft production-ready. Kafka 4.0 removes ZooKeeper support entirely.

In KRaft mode, a subset of Kafka brokers are designated as controllers (typically 3 or 5). The controllers run Raft consensus among themselves to maintain cluster metadata: topic configurations, partition assignments, ISR membership, and broker registration. The metadata is stored in a special internal topic: __cluster_metadata, which is itself a Raft-replicated log.

The benefit: significantly faster metadata propagation. In ZooKeeper-based Kafka, a partition leader change required a ZooKeeper write + notification + broker read — several round trips across two systems. In KRaft, the controller cluster can propagate a partition leader change to all brokers in a single batch metadata push.

Practically, new Kafka deployments should use KRaft. Existing ZooKeeper deployments can migrate using Kafka's migration tooling.

Part 10: Kafka in Architecture — The Centralised Log Pattern

The original insight that motivated Kafka: instead of point-to-point pipelines between every pair of systems, centralise all data flow through a single log. Every system writes its events to Kafka. Every system that needs data reads from Kafka. The pipelines become: N systems × 1 Kafka integration each, rather than N × (N-1) point-to-point integrations.

Each arrow is a custom integration. N=5 systems = 10 integrations. Any system added = N new integrations.

N systems write to Kafka. M systems read from Kafka. Adding a new consumer: 1 new Kafka consumer. Done.

This pattern is called the data lake or event-driven architecture (EDA) at the organisation level. Kafka becomes the central nervous system of the company's data. Every state change is an event in Kafka. Every downstream system derives its state from those events.

The practical consequence: downstream systems can be rebuilt from scratch by replaying Kafka history. Deleted a table by accident? The events are in Kafka — replay and rebuild. Want to add a new ML model trained on historical user behaviour? The events are in Kafka — read from the beginning. Want to test a new recommendation algorithm on production traffic without risk? Shadow-consume Kafka events in a test environment. This replayability is what distinguishes Kafka's log model from a traditional message queue.

Part 11: When Kafka Is the Wrong Choice

Kafka is operationally complex. Before choosing it, understand when it's overkill.

Kafka is wrong for request-response patterns. Kafka is a fire-and-forget log. If your producer needs to know the result of processing (did the order succeed? what's the user's balance?), Kafka adds a round trip: write event, wait for response event. Synchronous RPC is simpler for request-response. Use Kafka for events that don't need immediate acknowledgement.

Kafka is wrong when message volumes are low. If your system generates 100 events/minute, a Kafka cluster (minimum 3 brokers for production, plus ZooKeeper or KRaft controllers, plus monitoring) is enormous overhead for the benefit. RabbitMQ, SQS, or even a Postgres table with LISTEN/NOTIFY is appropriate. Kafka's operational benefit (centralised log, replay, high throughput) only justifies its complexity at meaningful scale — roughly above 10,000 events/second or when replayability is a hard requirement.

Kafka is wrong when message ordering across all keys is required. Within a partition, Kafka guarantees order. Across partitions, there is no ordering guarantee. If your use case requires total global ordering of all events, you need a single partition — which means no parallelism and a throughput ceiling of one consumer. Consider whether true global ordering is actually necessary or whether partition-level ordering (all events for a given entity in order) suffices.

Kafka is wrong when you need complex routing. Routing messages to different destinations based on message content requires reading, inspecting, and re-routing — effectively writing a stream processor. RabbitMQ's topic exchanges handle content-based routing natively. If your routing logic is complex and doesn't require Kafka's throughput or replayability, RabbitMQ is simpler.

Kafka is wrong when exactly-once to external systems is required. Kafka EOS works within the Kafka ecosystem (Kafka to Kafka, or Kafka to a database that supports transactions). If your consumer must write to an API that has no transaction support, exactly-once is impossible to guarantee — you must design for idempotent at-least-once regardless of what Kafka provides.

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

A Kafka cluster has 3 brokers, replication-factor=3, min.insync.replicas=2. A topic has 6 partitions. Current ISR for partition P0: {Broker1 (leader), Broker2, Broker3}.

Trace each failure scenario: (a) Broker2 crashes. What is the new ISR for P0? Can producers still write to P0 with acks=all? (b) Then Broker3 also crashes. What is the new ISR? Can producers write? What error do they receive? (c) Broker1 now also crashes. Is there a new leader for P0? What happens to the partition? (d) Broker2 recovers but its log for P0 is at offset 9,500, while Broker1 (still crashed) was at 10,000. If unclean.leader.election.enable=true, can Broker2 become leader? What messages are lost? How does this connect to Chapter 5's recovery after async replication lag?

Problem 2 — Design from Scratch

Design the complete Kafka architecture for a payment processing system. Events: PaymentInitiated, FraudCheckPassed, FraudCheckFailed, PaymentAuthorised, PaymentCaptured, PaymentFailed, RefundInitiated, RefundCompleted. Consumers: fraud detection service, payment gateway integration, ledger service, notification service (SMS/email), analytics pipeline, compliance audit log.

For each topic: how many partitions and why, what partition key, what retention policy (time-based or compacted — and why), what min.insync.replicas, what acks setting for the producer. For each consumer: what consumer group name, what isolation level (read_uncommitted vs read_committed), what is the idempotency key for its processing, what is the DLQ strategy. How does the compliance audit log consumer differ from all others — what special retention requirement does it have?

Problem 3 — Synthesis

Connect Kafka's ISR mechanism to Chapter 5's replication and Chapter 6's consistency models.

In Chapter 5, Postgres synchronous replication (synchronous_commit=on) waits for the standby to flush WAL before acknowledging the write. In Kafka, acks=all with min.insync.replicas=2 waits for 2 ISR members to write before acknowledging.

First: what consistency level (Chapter 6) does acks=all, min.insync.replicas=2 provide? Trace a read from any ISR member after a committed write — can you get linearizable reads from Kafka? (Hint: can consumers read from followers?) Second: Kafka consumers typically read from the partition leader only. How does this compare to Chapter 5's hot standby reads (replica reads)? What isolation level does the Kafka leader-only read provide? Third: a consumer commits its offset to __consumer_offsets after processing. This topic is itself replicated. What happens if the offset commit fails (the __consumer_offsets partition leader crashes between processing and committing)? What delivery semantics result and what does the consumer do on restart?

Problem 4 — Go Deeper

Kafka's zero-copy uses sendfile to transfer data from the OS page cache to the network socket. But this only works when the data is uncompressed on disk (or when the network and disk compression formats match). If you enable SSL/TLS encryption of network traffic, sendfile cannot be used — the data must pass through user space for encryption. If you enable compression on the broker (recompressing messages between producer and consumer), zero-copy is also lost.

From first principles: in a high-throughput Kafka cluster (500 MB/s write, 2 GB/s read), estimate the CPU overhead difference between zero-copy reads and standard copy reads. (Hint: use Chapter 1's memory bandwidth numbers and assume a 2-4 copy overhead). At what throughput level does this CPU difference become significant — at 10 MB/s? 100 MB/s? 1 GB/s?

Now: a company enables end-to-end SSL for all Kafka traffic (required by their security policy). What other architectural changes should they make to compensate for the lost zero-copy benefit? (Consider: producer-side compression, smaller clusters with more CPU, larger fetch sizes.) What is the residual throughput after SSL overhead at 1 GB/s of read traffic on a 4-core broker?

Problem 5 — Connect to Previous Chapters

Connect Kafka's log compaction to Chapter 10's LSM Tree compaction and Chapter 13's Saga pattern.

Part A: Kafka log compaction and LSM Tree compaction (Chapter 10) both keep only the latest value for each key and use tombstones to represent deletions. List three structural similarities and one fundamental difference. (Hint: what does Kafka compaction guarantee about read-after-compaction vs what LSM compaction guarantees? Consider the ordering of offsets after compaction.)

Part B: A Saga implemented via Kafka choreography (Chapter 13) has this problem: the compensation events (refund, release inventory) must be processed by the correct consumer in the correct order relative to the original events. If the payment service's Kafka consumer is behind by 2 minutes, and the inventory service processes InventoryReleased (compensation) before InventoryReserved (the original event) due to consumer lag differences across services, what happens?

Trace this failure through the stream-table duality: the inventory service maintains a KTable of current reservations. What is the state of this KTable after receiving InventoryReleased before InventoryReserved? How does partition-level ordering (all events for order #123 on the same partition) prevent this specific failure? What is the partition key for the saga topic and why?

What's Coming Next

You now have the full communication layer: synchronous RPC for request-response, message queues for decoupling (Chapter 14), and Kafka's distributed log for high-throughput event streaming and system integration. The next two chapters shift from data movement to system infrastructure: how traffic reaches your services in the first place. Chapter 16 covers load balancing — the algorithms and architectures that distribute incoming requests across multiple server instances. Chapter 17 covers rate limiting — how systems protect themselves from traffic spikes, abusive clients, and resource exhaustion. Together, these form the edge infrastructure that every large-scale system needs before requests even touch your application logic.

Chapter 15 complete. Problem 2 (payment system Kafka design) is the most complete system design exercise so far — it requires integrating topic design, partition key selection, delivery guarantees, idempotency, and DLQ strategy for a real business domain. Work through it in writing, not just in your head. Problem 5B (Saga event ordering across slow consumers) is the subtlest failure mode in event-driven systems and is almost never addressed correctly without explicit partition key design. If you can explain why partition-level ordering prevents Saga compensation event reordering, you understand Kafka better than most engineers who use it daily.