Kafka End-to-End, Past the Point Where Tutorials Stop

Part I - The physical model

1. The basic shape: broker, topic, partition

Simply put: "Within Broker you have topics. A topic is a stream of messages, and it's split into partitions for scalability." True, but which of these three words refers to an actual machine, because the rest of this post hinges on it:

• Broker = a physical or virtual server. It owns CPU, RAM, disk, and NIC - this is the actual hardware in the cluster.
• Topic = a logical name, like user-signups. It is not a file anywhere, and it doesn't live on any single broker.
• Partition = the real thing. A topic is split into some number of partitions, and each partition is an ordered, append-only sequence of records living as an actual directory on one specific broker's disk, replicated to a handful of other brokers for safety.

Everything about how Kafka scales falls out of that partition being the true unit of physical storage. A topic with one partition can only ever live on one broker, no matter how big your cluster is; give it ten partitions and Kafka can spread those ten physical directories across ten different brokers, so ten machines share the read/write load instead of one.

Also, leadership is assigned per-partition, not per-topic or per-broker.

Topic: orders (3 partitions, replication factor 2)

Every partition has one broker acting as its leader (handling all reads and writes for it) and some number of followers replicating it. Notice Broker 2 above is a leader for partition 1 and a follower for partition 0 at the same time - a busy topic's partitions get scattered across brokers on purpose, so no single server becomes a hotspot just because it hosts something popular.

This also dictates how producers actually route messages: a producer doesn't write to "a topic." It fetches cluster metadata (which broker leads which partition), hashes the record's key to pick a partition, and sends the record directly to whichever broker currently leads that partition.

With that physical model in place, everything downstream in this post (segment files, replication, rebalancing) is really about managing that one unit: the partition. Next: what's actually inside one.

2. The storage engine: segments, not trees

The tutorial version: "a partition is an append-only log." Also true, and also where most explanations stop - without ever opening the directory to see what's in it.

Not one giant file but a set of segments. At any time there's exactly one active segment being appended to, plus a set of older, closed segments. Each segment is really three files sharing a base offset as their filename:

• .log - the actual records, appended sequentially, never rewritten in place.
• .index - a sparse mapping of offset → byte position in the .log file. Sparse means it doesn't index every message, only one entry every few KB (log.index.interval.bytes). Finding a specific offset is a binary search over this sparse array to locate the right block, followed by a short linear scan through that block to the exact record. That's not the O(log n) exact-key lookup a B-Tree gives you - it's "get close, then walk a few KB" - and that's fine, because Kafka only ever needs a starting point for sequential reads, never an arbitrary record.
• .timeindex - the same trick, but mapping timestamp → offset, which is what powers offsetsForTimes (seeking by wall-clock time instead of a raw offset).

New segments roll over when the active one hits log.segment.bytes or log.roll.ms, whichever comes first. This matters operationally: segment size directly controls how much data can be deleted or compacted in one unit, and how much a broker has to scan when it restarts and rebuilds its index files.

Retention works at the segment level, not the message level - a whole segment is deleted once every record in it is older than retention.ms (or the partition exceeds retention.bytes). That's deliberate: deleting whole files is cheap; deleting individual records from the middle of an immutable append-only file is not.

Log compaction (used for topics like __consumer_offsets and for building changelog/KTable-style state) is a different, optional cleanup mode: instead of deleting by age, a background cleaner thread rewrites closed segments to keep only the latest record per key, dropping everything superseded and eventually removing keys marked with a null-value "tombstone" once it's safe to do so. It only touches closed segments - never the active one - and it triggers based on a dirty ratio (min.cleanable.dirty.ratio), not on a schedule. This is the one place Kafka does something LSM-flavored, but notice it's opt-in and narrow in scope, not the default write path.

Record batch format: producers don't send one record at a time - they send record batches, and the batch itself (not each record) carries a header with a base offset, a CRC checksum, an attributes bitmap (compression codec, timestamp type, whether it's part of a transaction, whether it's a control batch), a producer ID, a producer epoch, and a base sequence number. Individual records inside the batch store only small deltas (relative offset, relative timestamp) against that header, which is why compression works so well here - repetitive structure compresses cheaply, and Kafka compresses the whole batch as one unit rather than record-by-record.

Hold onto that producer ID and sequence number - they resurface in Part IV when we get to idempotence and transactions.

We now know what's on disk. Part II is about what happens when that disk gets read from and written to under load.

Part II - Speed and safety

3. The read path: page cache, zero-copy, and its asterisk

The tutorial soundbite: "Kafka is fast because of zero-copy." True, but usually left as a slogan rather than a mechanism.

Here's the actual path: Kafka doesn't manage its own read cache. It deliberately leaves that job to the Linux page cache, and uses the sendfile() syscall to move bytes straight from page cache to network socket, bypassing the JVM heap and user space entirely. A conventional "read file, write to socket" path involves up to four context switches and four data copies; sendfile collapses that to two of each. If a consumer is reading data that was just produced, it's often still sitting in page cache, so the broker never touches disk for that read at all.

The asterisk that a lot of "Kafka is zero-copy" claims skip: this optimization disappears the moment TLS is enabled. Encrypting the bytes in flight requires pulling them into user space to run through the SSL engine before they hit the socket - there's no such thing as kernel-space encryption via sendfile. Since most production clusters run with TLS for good reasons, a lot of real-world deployments aren't actually getting the zero-copy benefit people assume they are. Kafka still performs fine here - the network, not the CPU, is usually the real ceiling - but it's worth knowing which optimization you traded away for encryption.

4. Replication and durability: LEO, high watermark, and what acks=all really promises

The tutorial version: "set acks=all and Kafka is durable." True, but this is where the mechanism sounds backwards at first if you stop there: Kafka gets its durability guarantees from replication, not from forcing disk flushes (fsync) on every write. Disk flushing is deliberately left to the OS's background threads, because blocking on physical disk for every message would tank throughput and can make a broker miss heartbeats and get evicted from the cluster - trading a marginal durability gain for a much bigger availability loss.

To see how replication actually provides that safety, it helps to be precise about the offsets involved, because "offset" gets used loosely:

• Log End Offset (LEO) - every replica (leader and followers) tracks its own LEO: the offset of the next record it will write. This is just "how far this specific copy has gotten."
• High Watermark (HW) - the leader tracks the LEO of every follower via their fetch requests, and sets the HW to the minimum LEO among all replicas currently in the ISR (in-sync replica set). The HW is the offset up to which a record is guaranteed to exist on every in-sync copy. Consumers can only ever read up to the HW - never past it - so they never observe a record that could vanish in a leader failover.

With acks=all (the default since Kafka 3.0), a producer's write is acknowledged only once it's been replicated into the page cache of every broker in the required ISR set (min.insync.replicas), advancing the HW past that record - not once anything has hit a physical platter. The bet is that "every in-sync replica loses power at the same instant before any of them flushes" is astronomically rarer than "one disk hiccups," and in practice that bet holds.

A few honest caveats that make this less of a blanket guarantee than it sounds:

• If the ISR shrinks below min.insync.replicas, producers using acks=all simply start getting errors and you get backpressure, not silent data loss. That's the correct trade-off, but it surprises people who expect the cluster to keep accepting writes no matter what.
• Unclean leader election (unclean.leader.election.enable) allows an out-of-sync replica to become leader to preserve availability, at the cost of silently truncating any records that were never replicated to it. It defaults to disabled for exactly this reason - enabling it is explicitly choosing availability over durability during a bad outage.
• Eligible Leader Replicas (KIP-966), generally available since Kafka 4.0, tightens this further by tracking exactly which replicas are provably caught up to the HW and therefore safe to promote, closing some of the historical gap between "technically in the ISR" and "actually safe to elect as leader."

Reads are fast, writes are durable - but both of those depended on brokers agreeing on facts like "who's the leader" and "what's in the ISR." Part III is about how that agreement actually happens.

Part III - Coordination

5. Cluster metadata: how brokers agree on all of this

Tutorials tend to mention this only in passing - "ZooKeeper coordinates the cluster" - as background trivia rather than something with mechanics worth understanding. Everything in Part II - who leads which partition, which replicas are in the ISR, which brokers are even alive - is itself a piece of state that the whole cluster has to agree on, and that agreement has to survive brokers crashing and restarting.

For most of Kafka's life, an external ZooKeeper ensemble stored this metadata and ran controller elections. As of Kafka 4.0 (March 2025), ZooKeeper mode has been removed entirely - not deprecated, gone. The only supported mode now is KRaft, Kafka's own Raft-based consensus protocol, where a set of dedicated controller nodes replicate cluster metadata (broker liveness, partition leadership, configs) through an internal Raft-replicated topic called __cluster_metadata, instead of relying on an external system. Controller failover is now sub-second instead of the several seconds typical of ZooKeeper, and there's one less distributed system to operate.

If you're reading (or drawing) an architecture diagram with ZooKeeper still in the write path, it's describing pre-4.0 Kafka.

That's the server side settled - brokers agreeing on brokers. Part IV moves to the client side: what a producer's flags and a consumer group's membership actually do against all of this machinery.

Part IV - The client contracts

6. The producer path, end to end

The tutorial version covers .send(), maybe batch.size and linger.ms as performance knobs, and acks as a durability knob. What it usually doesn't cover is what's actually happening underneath those flags - and that's where "exactly-once" claims get made or broken.

Batching risk window. Producers buffer records client-side before sending, controlled by batch.size and linger.ms.