Q44 — How does Kafka work internally — why is it fast?

What is Kafka fundamentally?

Kafka describes itself as a "distributed event streaming platform," but at its core it is a distributed, partitioned, replicated commit log. Every topic is a log. Messages are appended. Nothing is deleted on consumption. Consumers read by tracking a position in the log. This append-only log model is the key to almost everything that makes Kafka fast, scalable, and replayable.

To understand why Kafka is fast, you need to understand what it's doing at the hardware and OS level — because Kafka's design is explicitly built around making efficient use of modern hardware.

Why is Kafka fast — sequential I/O

The most fundamental reason: Kafka writes are sequential appends to disk.

Random I/O — jumping around on disk to find and update specific records — is slow. On an HDD, a random seek takes 5–10ms. Even on an SSD, random writes cause write amplification (as covered in Q12). Sequential writes bypass this entirely: data goes to the end of the file, the disk head doesn't move, and the OS can use write-ahead buffering efficiently.

Kafka's brokers don't maintain complex in-memory data structures. They write messages to a log file sequentially. The OS page cache handles the caching — frequently-read messages are in RAM already from the page cache, while the durable copy is on disk. Kafka effectively uses the OS as a large RAM cache, avoiding Java heap management and GC pressure.

This is why Kafka can handle high throughput on commodity hardware. A single broker can sustain hundreds of MB/s of writes because those writes are sequential.

Why is Kafka fast — zero-copy data transfer

When a consumer reads messages from Kafka, the data must travel from disk to the consumer's network socket. In a naive implementation:

Kernel reads data from disk into kernel buffer
Data copied from kernel buffer to user-space application buffer
Application copies data back to kernel socket buffer
Kernel sends data to network

That's 4 steps and 2 extra copies through user space. Kafka sidesteps all of this using the sendfile() system call (or transferTo() in Java NIO), which tells the kernel: "copy data from this file descriptor directly to this network socket descriptor." The kernel does it entirely in kernel space:

Kernel reads data from disk into kernel buffer
Kernel DMA-copies from kernel buffer directly to network card buffer
Network card sends to consumer

No data touches user space. No extra copies. This is called zero-copy — user-space copies are eliminated. For read-heavy consumers (Kafka replication, consumer catchup), this dramatically reduces CPU usage and increases throughput.

Why is Kafka fast — batching

Kafka producers don't send one message at a time. They accumulate messages in a batch (configurable by size and time — e.g., "batch up to 16KB or 5ms, whichever comes first") and send the batch as a single network request.

This batching reduces the per-message overhead of network round trips, TCP packet headers, and broker processing. A single batch of 1,000 messages costs roughly the same as a single message in terms of network overhead — 1,000x more efficient.

Brokers also compress batches (lz4, zstd, snappy) before writing them to disk, further reducing I/O. Consumers receive compressed batches and decompress locally — the heavy I/O path stays compressed.

What is a partition, and why does it matter?

A Kafka topic is a logical category for messages ("user-events", "order-updates"). A topic is divided into partitions — numbered, ordered, immutable logs.

Each partition is an independent, ordered sequence of messages. Messages within a partition are assigned monotonically increasing offsets starting at 0. The offset is the message's permanent address in the partition. Consumer reads look like: "give me messages starting at offset 1,547,823 in partition 3."

Why partition? Three reasons:

Parallelism: Each partition can be on a different broker. A topic with 12 partitions spread across 3 brokers means 3 parallel write streams, each going to a different machine. Read parallelism is similarly multiplied. Throughput scales linearly with partition count (up to hardware limits).

Ordering: Kafka guarantees order within a partition. Messages from the same producer to the same partition arrive in the order they were sent. This is the ordering guarantee — not across partitions, only within. A topic with 12 partitions has 12 independent orderings.

Replication: Each partition is replicated independently across brokers. One broker is the leader for each partition, handling all reads and writes. The other brokers with copies are followers. If the leader crashes, a follower takes over. Replication is at the partition level — you can have a 3-way replicated partition without replicating the whole topic uniformly.

How does partition assignment work?

When a producer sends a message, it must decide which partition to write to. Two modes:

With a partition key: The producer specifies a key (e.g., user_id). Kafka hashes the key and sends it to a consistent partition: partition = hash(key) % num_partitions. All messages with the same key go to the same partition — guaranteeing ordering for that key. All events for user 42 are in the same partition, in order.

Without a key (round-robin): Messages are distributed across partitions in a round-robin fashion. Maximum throughput, but no ordering guarantee across messages — each message might go to a different partition.

What are offsets and how do consumers use them?

Each message in a partition has an offset — a sequential integer assigned by the broker. Offset 0 is the first message, offset 1 the second, and so on. Offsets are immutable and forever — a message at offset 5,421 will always be at offset 5,421.

Consumers track their position using offsets. After processing a message at offset N, the consumer commits offset N (or N+1 as the "next offset to read"). The committed offset is stored in a special Kafka topic called __consumer_offsets.

On restart, the consumer reads its last committed offset and resumes from there. If the consumer committed too early (before processing), a crash causes message loss. If the consumer committed too late (after processing but before the commit), a crash causes the message to be reprocessed — at-least-once behavior.

Crucially: consumers don't delete messages. They just advance their offset pointer. The messages remain in the log until Kafka's retention policy removes them (by time — e.g., 7 days — or by size). This means multiple consumer groups can independently read the same topic at different positions, and a new consumer can start from the beginning and replay the entire history.

How does Kafka replication work?

Each partition has a leader broker and one or more follower brokers (configured by the replication factor). Producers write to the leader. The leader appends to its local log and broadcasts to followers. Followers replicate the data and acknowledge.

The ISR (In-Sync Replica) set is the set of brokers that are caught up with the leader (within a configurable lag threshold). A message is considered committed once every replica in the ISR has it — the leader then advances the partition's high watermark, and consumers only read up to the high watermark. That rule is independent of the producer's acks setting: acks only decides when the producer is told "success" — before or after the message is committed.

acks=0: producer doesn't wait — at-most-once, maximum throughput acks=1: producer waits for leader to write — fast, but an acknowledged message can be lost if the leader crashes before followers replicate it acks=all: producer waits for all ISR — maximum durability, slowest. Pair it with min.insync.replicas=2 (replication factor 3): if the ISR shrinks to just the leader, "all" would otherwise mean one copy.

The mental model

code

Kafka = distributed, partitioned, replicated append-only commit log

Why fast:
  Sequential writes → OS page cache + no seek penalty (like LSM, like WAL)
  Zero-copy reads → sendfile() syscall → kernel copies file→socket directly, no user-space
  Batching → 1000 messages → 1 network round trip, compressed on disk

Topic → Partitions:
  Each partition = independent ordered log with per-message offsets
  Partition = unit of parallelism + ordering guarantee
  Partition key → consistent hash → same partition → ordered for that key

Consumer position = offset (integer, committed to __consumer_offsets)
  Messages NOT deleted on consume
  Multiple consumer groups read independently at different offsets
  New consumer can replay from offset 0

Replication:
  Each partition: 1 leader (reads/writes) + N followers
  ISR = followers caught up with leader
  Committed = written by all ISR (high watermark) — regardless of acks
  acks = when the producer hears "success": 0 never, 1 leader wrote, all = committed

acks=0: at-most-once, max throughput
acks=1: fast, small loss window
acks=all: durable, slower

Kafka is fast because it does one thing well: append bytes to a file, and read them back. Every design decision — sequential I/O, zero-copy, batching, the immutable log model — is optimized for that single operation. The result is a system that can handle millions of messages per second on commodity hardware, while providing durability, replayability, and multi-consumer fan-out that traditional message queues cannot.