Why Kafka is fast and when to use it

"Fast" is not a property of the code — it's a property of the access pattern. Kafka is fast because it refuses to do the slow things: it never seeks randomly, never serializes per message, and never copies bytes it doesn't have to.

Mechanism companion to C23 Sequential IO Zero-copy

This is the mechanism page, not the design page

C23 is the design anchor: it derives the partitioned append-log queue, replication, consumer groups, and delivery semantics. This page owns the low-level why is it fast — the physics and the syscalls underneath that design. If you're asked to design the queue, go to C23; if you're asked why a log broker out-throughputs a "normal" message queue by 10–100×, you're in the right place.

The hinge: throughput is a fight against the disk head

First principle

A spinning disk (and to a lesser but real degree an SSD) is fast at one thing: reading or writing bytes that are physically adjacent. The instant you ask it to jump to a new location, you pay a seek. So the entire performance question collapses to: can I express my workload as long sequential runs, or am I forcing the device to chase pointers? Kafka's design is a single-minded answer — make every write an append to the end of a file, and every read a forward scan — so the disk does the only thing it's good at.

Put numbers on the gap, because the whole architecture is a reaction to it. A commodity disk streams sequential data at ~100s of MB/s (call it 200–500 MB/s for a modern SATA SSD, north of a GB/s for NVMe). But random access is capped not by bandwidth but by IOPS — a few hundred IOPS on spinning rust, tens of thousands on a good SSD. Take a workload of small 1 KB records:

sequential: 250 MB/s ÷ 1 KB ≈ 250,000 records/s | random (HDD): 300 IOPS × 1 KB ≈ 300 records/s

That is roughly a 100×–1000× gap, and it does not come from a faster CPU or a cleverer language — it comes from refusing to seek. A traditional message broker that keeps a per-message index, deletes messages as they're consumed, and updates state in place is, structurally, a random-IO workload. Kafka deletes nothing on consume, indexes almost nothing, and only ever writes to the tail. This is exactly DDIA Ch. 3's log-structured storage argument: an append-only log turns random writes into sequential ones, and that single decision is the dominant lever.

Clarify the contract — what "fast" must and must not mean

Before optimizing, pin down what guarantees Kafka is allowed to keep cheap, because "fast" is bought by narrowing the contract:

Must: sustain very high aggregate write and read throughput for an append/replay workload, with durable, ordered-per-partition storage and many independent consumers reading the same data.
Must NOT (deliberately): support per-message random update or delete, in-place mutation, low-latency request/response, or a global total order across partitions. Kafka is a log, not a database and not an RPC layer. The moment your problem needs "fetch this one record and mutate it," you've left Kafka's sweet spot — that's the Kafka vs RPC trade below.

Lever 1 — batching: amortize the fixed cost per message to near zero

Sequential IO removes the seek; batching removes the per-message overhead. Sending one message end-to-end is never just "write 1 KB." It is: a system call to hand bytes to the kernel, a TCP segment (with its headers and round-trip accounting), broker-side handling, optionally an fsync, and a replication acknowledgment. Those fixed costs are paid per network/syscall event, not per byte. So the question is how many messages you fold into one event.

Worked example — amortizing fixed cost

Suppose each produce request carries a fixed overhead of O = 200 µs (syscall + TCP + broker dispatch + ack accounting) and each 1 KB message costs m = 1 µs of actual data work.

1 message/request: cost per message = O + m = 201 µs → ≈ 4,975 msg/s. The overhead is the workload.
1,000 messages/request: cost per message = O/1000 + m = 0.2 + 1 = 1.2 µs → ≈ 833,000 msg/s. The fixed cost is amortized ~1000× and you're now bound by data work, not overhead.

Same code, same hardware, ~167× the throughput — purely from grouping. This is why a "Kafka is slow!" report almost always turns out to be a producer sending one tiny message per request (linger.ms=0, batch.size tiny): you're paying the 200 µs tax on every record.

Batching also unlocks compression (DDIA Ch. 4, encoding): compressors find redundancy across records, so a batch of similar JSON events compresses far better than each event alone — and Kafka stores and ships the batch compressed, so compression buys network and disk and page-cache savings in one move. But batching is not free: holding messages to fill a batch adds latency. That is the central tunable, and it deserves a static table.

Batch / linger setting	Throughput	Added p99 latency	Compression ratio	Fits
1 msg, linger 0	~5K msg/s (overhead-bound)	~0 ms (no wait)	poor (per-msg)	low-latency, low-volume signals
~64 KB, linger 5 ms	~10–20× higher	+~5 ms	good	general-purpose default
~1 MB, linger 100 ms	~100× higher, near disk/NIC bound	+~100 ms	best (large window)	analytics / ingest pipelines

The trade is monotone and unavoidable: throughput is bought with latency. linger.ms is literally "how long will I wait, accumulating messages, before flushing?" Set it to 0 and you optimize latency at the cost of overhead; raise it and you trade tail latency for amortized efficiency. There is no setting that wins both — you choose where on this curve your workload lives.

The physical layout — segments, the offset index, and the zero-copy path

A partition is not a fancy structure; it is a directory of segment files — append-only byte logs that roll over at a size/time limit. Alongside each segment is a sparse offset index mapping a logical offset to a byte position, so a consumer's "give me from offset N" becomes a binary search in a tiny index plus a sequential read. No B-tree, no per-message lock, no delete-on-consume.

PARTITION = ordered sequence of append-only segment files ┌──────────────────────────────────────────────────────────────┐ │ 00000.log 00000.index 00512.log 00512.index ... │ │ ┌────────────────┐ ┌──────┐ ┌────────────────┐ ┌──────┐ │ │ │ rec rec rec ...│ │off→pos│ │ rec rec rec ...│ │off→pos│ │ │ └────────────────┘ └──────┘ └────────────────┘ └──────┘ │ │ ▲ append only ▲ sparse: every ~4KB, not per-msg │ └──────┼─────────────────────────────────────────────────────────┘ │ producers only ever write here (the tail) → sequential fetch(offset N): binary-search index → byte pos → SEQUENTIAL read forward THE ZERO-COPY READ PATH (consumer fetch served via sendfile) ┌────────┐ DMA ┌────────────┐ sendfile ┌─────┐ DMA ┌──────┐ │ DISK │ ───────▶ │ PAGE CACHE │ ───────────▶ │ NIC │ ──────▶ │socket│ └────────┘ (kernel) └────────────┘ (kernel→kernel)└─────┘ └──────┘ ▲ └─ NO copy into user-space broker heap broker never touches the message bytes contrast — the naive read (4 copies, 2 context switches): disk → page cache → USER buffer → SOCKET buffer → NIC (read()) (broker) (write())

Deep dive 1 — the append-only shape avoids locks and indexes entirely

A mutable store needs an index to find a record and a lock to mutate it safely under concurrency; both are random-access, contention-prone structures. An append-only log sidesteps all of it. Writes go to one place — the tail — so the only synchronization is appending in order (and Kafka serializes per partition, which is also why a partition is the unit of ordering). There is nothing to lock for readers because readers never mutate: a consumer just remembers its own offset and reads forward. Critically, the consumer owns its offset, not the broker. The broker doesn't track "who has read what" or delete on consume — it retains by time/size and hands out byte ranges. This is the inversion that makes fan-out cheap: ten consumer groups reading the same partition are ten independent forward scans over the same page-cached bytes, costing the broker almost nothing extra. This is DDIA Ch. 11's partitioned-log abstraction: the log is the source of truth, consumption is a cursor over it, and replay is just resetting the cursor.

Deep dive 2 — page cache + sendfile: the bytes the broker never touches

Here is the part that surprises people: Kafka does not maintain its own message cache. It writes to the file and lets the OS page cache hold hot pages. Because producers append and consumers read the tail, the data a consumer wants was written moments ago and is almost certainly still in page cache — so a "read" is frequently served entirely from RAM with zero disk IO, and Kafka gets a giant cache for free, shared across processes, that survives a broker JVM restart and never doubles memory by caching the same bytes twice (once in JVM heap, once in OS cache).

On top of that sits zero-copy via sendfile. The naive way to serve a file to a socket copies bytes four times and crosses the user/kernel boundary twice: disk → page cache → user-space buffer → socket buffer → NIC. sendfile(2) tells the kernel "ship these page-cache bytes straight to this socket," so the data goes page cache → NIC without ever being copied into the broker's user-space heap. The broker becomes a coordinator of byte movement, not a transformer of bytes — it decides which range of which segment to send and lets the kernel DMA it out. This is also why broker-side transformations (decompress, re-encode, filter) are expensive: they force the bytes through user space and forfeit zero-copy. Kafka keeps the batch opaque on the hot path precisely to preserve it.

The sharpest senior question: "Walk me through every copy a message avoids from disk to socket."

Answer it as a subtraction from the naive 4-copy path. (1) Disk → page cache: one DMA copy by the kernel — unavoidable, but often skipped entirely because the tail is already cached from the recent write. (2) Page cache → user-space buffer: avoided — sendfile never reads the bytes into the broker process. (3) User buffer → socket buffer: avoided — there is no user buffer. (4) Then DMA page-cache → NIC. So the message that would cross the kernel/user boundary twice and be copied four times instead crosses zero user boundaries and (on modern NICs with scatter-gather DMA) is copied effectively once. The broker CPU never touches the payload. That is the whole magic trick.

Deep dive 3 — partition parallelism is the scaling unit

Everything above makes a single partition fast. Partitions are how you make the cluster fast: a topic is split into P partitions spread across brokers, and within a consumer group each partition is consumed by exactly one consumer. So P is your maximum parallelism — for reads (at most P consumers do useful work in a group) and for writes (each partition is an independent sequential append on possibly different disks/brokers). Want 2× the throughput? Add partitions and consumers. This is DDIA Ch. 6/Ch. 11 partitioning applied to a log, and it carries the same tension we cover in lesson 07: the partition key decides distribution, so a skewed key (everything keyed by one tenant) creates a hot partition that no amount of total partitions will relieve — see the failure modes below. The cost of more partitions isn't throughput, it's metadata: more files, more open handles, longer leader-election/rebalance times, and more controller pressure.

Trade-offs

Choice	Buys	Costs	Choose when
Large batches vs small	throughput + compression (amortize fixed cost ~1000×)	added per-message latency (linger)	ingest/analytics; not latency-sensitive signals
More partitions vs fewer	read/write parallelism = P	weaker ordering, more files, slower rebalance	high-volume topics with a good key
Long retention vs short	replay, new consumers, event sourcing	disk, longer recovery scans	event-sourced / reprocessable pipelines
Kafka (log) vs RPC	decoupling, buffering, fan-out, replay	no synchronous response; eventual	async workflows; never for request/response

The sharpest of these is the last one, and it's where candidates most often misuse Kafka. Kafka is fast precisely because it gave up request/response: there is no "process this and tell me the answer." A producer fires and the data lands in a log; a consumer reads it whenever it gets there. If your call site needs an answer before it can proceed — a synchronous read, a "did this succeed?" — Kafka is the wrong tool, and bolting a reply-topic on top reinvents RPC badly. Use it for the workloads its physics rewards: high-volume, append-shaped, replayable streams where decoupling and buffering are features. See lesson 11 for where a log broker sits versus a traditional task queue, and lesson 02 for why batching trades latency for throughput on the same utilisation curve.

Failure modes

Failure	Why it happens	Mitigation
"Kafka is slow" with tiny messages	linger 0 / tiny batch → paying the fixed per-request overhead on every 1 KB record (the 201 µs case above).	Raise `batch.size` / `linger.ms` and enable compression before blaming hardware.
Too many partitions	Open file handles, controller metadata, and rebalance time all scale with partition count.	Right-size P to required parallelism; consolidate low-volume topics; monitor controller pressure.
Slow consumer / lag	Consumer falls behind the tail; reads now miss page cache and hit disk, and may fall off the retention window entirely.	Scale the group up to P, isolate the slow consumer, or lengthen retention so it can catch up.
Hot partition	Skewed partition key sends most traffic to one partition, defeating parallelism (lesson 07's hot shard).	Pick a higher-cardinality key, or split/salt the hot key.
Broker transformation kills zero-copy	Server-side decompress/filter/re-encode forces bytes through user space.	Keep batches opaque on the hot path; do transformation in consumers, not the broker.

Interview prompts you should be ready for

Why is sequential IO so much faster than random, with numbers? (senior answer) Random access is bounded by IOPS (a few hundred on HDD), sequential by bandwidth (~100s of MB/s). For 1 KB records that's ~300 records/s random vs ~250K/s sequential — a 100×–1000× gap. Kafka makes every write a tail append and every read a forward scan, so the disk does the only thing it's good at. DDIA Ch. 3's log-structured argument.
Walk me through every copy a message avoids from disk to socket. (senior answer) Naive path copies 4 times across 2 kernel/user boundaries: disk→page cache→user buffer→socket buffer→NIC. sendfile drops the two middle copies — page cache goes straight to the NIC, never into the broker's heap. The broker CPU never touches the payload; with scatter-gather DMA it's effectively one copy. And the first copy is often skipped because the tail is already in page cache from the recent write.
How much does batching actually buy, and what does it cost? (senior answer) If fixed per-request overhead is ~200µs and data work ~1µs/msg, 1 msg/request ≈ 5K msg/s; 1000 msgs/request ≈ 833K msg/s — ~167×, purely from amortizing the fixed cost. The cost is latency: linger.ms is how long you wait to fill a batch. Throughput is bought with tail latency; there's no setting that wins both.
Why doesn't Kafka keep its own cache? (senior answer) It leans on the OS page cache. Producers append and consumers read the tail, so hot data is in cache for free, shared across processes, surviving a JVM restart, with no double-buffering. A JVM heap cache would just duplicate what the kernel already holds and add GC pressure.
Why is the partition the unit of both ordering and parallelism? (senior answer) Within a partition, appends are serialized so order is well-defined; within a consumer group, each partition is owned by one consumer, so P caps useful parallel consumers. Ordering and parallelism are the same knob from opposite ends — more partitions buys throughput but only per-partition ordering, never a global total order.
When should you NOT use Kafka? (senior answer) When you need request/response or a synchronous answer — Kafka gave up exactly that to be fast. Also when you need per-record random read/update/delete (use a database), or a global total order across keys. A reply-topic to fake RPC reinvents RPC badly.
A consumer is lagging and reads got slow — what changed? (senior answer) It fell off the tail, so its reads now miss page cache and hit disk (back to random-ish, slow IO), and risk falling off the retention window. Scale the group toward P, isolate it, or extend retention. The page-cache assumption only holds while you read near the tail.
How does this page relate to the queue design in C23? (senior answer) C23 designs the queue — partitions, replication, consumer groups, delivery semantics. This page is the mechanism underneath: sequential IO, batching, page cache + sendfile, partition parallelism. C23 tells you what the log is; this tells you why it out-throughputs a conventional broker.

Related lessons

Read C23 first for the queue design this page explains the speed of. Lesson 11 places a log broker against task queues and brokers generally. Lesson 02 is the throughput-vs-latency curve that batching rides. Lesson 04 covers the encoding/compression that batching unlocks, and lesson 07 is the partitioning (and hot-shard) story applied here to partitions.

C23 · queue design anchor Async messaging Partitioning Latency & throughput Data modeling & storage

Takeaway

Kafka's speed is not magic; it is the disciplined avoidance of slow operations. Make writes sequential appends so the disk never seeks (~100× over random IO). Batch so the fixed per-request cost is amortized ~1000× — bought with latency, the one unavoidable trade. Lean on the OS page cache and sendfile so hot bytes flow disk→cache→NIC without ever entering the broker's user space. And use partitions as the unit that turns one fast log into a fast cluster. Every one of those is a refusal to do random, per-message, or extra-copy work. When the workload is append-shaped and replayable, that refusal is a superpower; when you need a synchronous answer, it's the wrong tool — that's the line C23 and this page draw together.