Design a distributed message queue
The whole design turns on one fork: do you delete a message when it's consumed, or keep it and let consumers replay? Pick "keep," commit to a partitioned append-only log, and almost everything else follows.
0. The hinge: queue vs. log
Before any boxes, force the one decision that everything hangs off. Two designs share the word "queue" but are architecturally opposite:
Log (retain + replay): a message is appended and kept for a retention window regardless of who read it. Consumers track their own position (an offset) and advance independently; a new consumer can start from offset 0 and replay history. This is the right shape for event streams, change-data-capture, and anything where multiple downstreams need the same data — and it is what Kafka, Pulsar, and Kinesis are. We design the log, because it is the strict superset: a log can emulate a queue (one consumer group, short retention), but a queue cannot emulate a log.
This is exactly DDIA Ch. 11's distinction between messaging systems (broker holds messages transiently, deletes on ack) and log-based message brokers (broker is a durable, partitioned, replayable log). The chapter's punchline is ours: making the broker a log is what unlocks replay, multiple independent consumers, and reprocessing after a bug fix — at the cost of storing data you've "already delivered."
1. Clarify the contract
Treat the prompt as a product contract before a component diagram. The system must:
- Let producers publish messages to a named topic, optionally with a partition key.
- Let consumers read at their own pace and resume exactly where they left off after a crash.
- Retain messages for a configured window so retry, replay, and new consumers all work.
- Scale throughput horizontally by partitioning a topic across brokers.
- Expose ordering and delivery guarantees explicitly, not by accident.
And, just as important, what it need not do: it need not provide total order across the whole topic (we'll show that kills throughput), it need not do per-message random-access reads (it's a log, not a database — see C26 for why sequential access is the whole performance story), and it need not guarantee true exactly-once on its own (we'll be honest about that in §6).
2. Put numbers on the shape — throughput is bytes, not just messages
The single most common mistake in this case is to size by message rate and forget that the disk, network, and replication links all move bytes. Size everything in bytes/sec. Take a target of 1,000,000 msg/s at an average 1 KB per message:
ingress = 1{,}000{,}000 \text{ msg/s} \times 1\text{ KB} = 1 \text{ GB/s}
That 1 GB/s is the number that governs the design. Now retention. At 7-day retention before replication:
1 \text{ GB/s} \times 86{,}400 \text{ s/day} \times 7 = 604{,}800 \text{ GB} \approx 600 \text{ TB}
With a replication factor of 3 (so a broker can die without data loss — DDIA Ch. 5):
600 \text{ TB} \times 3 = 1.8 \text{ PB on disk}
Partition count falls straight out of the per-partition ceiling. A single partition is an ordered log served by one leader broker; a realistic sustained ceiling is ~10 MB/s per partition once you account for replication and consumer fan-out. So:
N_{\text{partitions}} = \frac{\text{target throughput}}{\text{per-partition ceiling}} = \frac{1\,\text{GB/s}}{10\,\text{MB/s}} = 100 \text{ partitions (floor)}
Round up for headroom and to allow more consumer parallelism — partition count is the cap on how many consumers in one group can read concurrently. Finally, the operational signal that matters most at runtime, consumer lag, is also just arithmetic — the backlog grows whenever consumption falls behind production:
\text{lag(messages)} = (\text{produce rate} - \text{consume rate}) \times \text{time}
If producers push 1M msg/s and consumers drain 900k msg/s, lag grows at 100k msg/s — 6M messages in a minute. Lag measured in time ("the consumer is 4 minutes behind head") is the alert you page on (see lesson 16).
3. Define the surface area
Keep the API small. It should express publish / consume / commit, and nothing about the internal log layout.
| API / operation | Why it exists |
|---|---|
produce(topic, key, payload) | Append a message. key selects the partition (hash(key) % N), which is how same-key messages stay ordered. |
poll(topic, partition, offset) → [messages] | Read a batch starting at offset. Pull-based: the consumer controls its own rate (this is the backpressure mechanism). |
commit_offset(group, partition, offset) | Persist "this group has processed up to here." The commit, not the read, is what makes progress durable across crashes. |
Note the API is pull, not push. A consumer that asks for the next batch only when it's ready can never be overwhelmed — slow consumers simply poll less often and their lag grows. That single choice is most of the backpressure story (§6).
4. Model the data
The data model is the architecture. A topic is a logical stream; it is split into N partitions, each an independent append-only log of (offset, key, payload, timestamp) records. Offsets are monotonically increasing integers within a partition — there is no global offset. Each consumer group stores one committed offset per partition; groups are independent, which is exactly what lets the same topic feed analytics, billing, and search without them interfering.
5. Linearized design
Walk an event from publish to processed; the bottlenecks appear in order.
- 1. Route. The producer hashes the key to pick a partition (
hash(key) % N); no key ⇒ round-robin for pure throughput. This is lesson 07's partitioning applied to a stream — the key choice decides both load balance and ordering scope. - 2. Append. The partition's leader broker appends the record to the end of its log and assigns the next offset. Append-only means sequential writes; the why this is fast (sequential I/O, batching, page cache, zero-copy send) is the companion mechanism lesson — see C26 — and we deliberately don't re-derive it here.
- 3. Replicate. Followers pull the new records; once enough replicas are caught up (the in-sync replica set, DDIA Ch. 5), the write is acknowledged to the producer.
acks=allwaits for the ISR;acks=1waits only for the leader (faster, can lose the tail on leader failure). - 4. Consume. Each partition is assigned to exactly one consumer in a group. The consumer polls from its committed offset forward and advances at its own pace.
- 5. Commit. After processing a batch, the consumer commits its offset. Crash-and-restart resumes from the last commit — this is where at-least-once vs. at-most-once is decided (§6).
- 6. Retain / reclaim. Old segments are deleted by time or size policy, independent of whether anyone read them. (Or compacted: keep only the latest record per key — a changelog.)
The diagram makes the central property visible: groups A and B read the same physical log at different positions. Deleting on consume would make that impossible — which is why the log model, not the queue model, is the anchor design.
6. Deep dives
6.1 Queue (delete-on-consume) vs. log (retain + replay) — the central fork
We already named this in §0; here is why it is load-bearing in the design and not just terminology. In a queue, the broker must track per-message state (delivered? acked? redelivered?) and delete on ack — that's random-access mutation, the enemy of sequential I/O, and it makes "two independent readers" structurally impossible because consuming is destructive. In a log, the broker tracks nothing per consumer; the consumer owns its offset. The broker's only job is append + serve-by-offset + age-out. That asymmetry is why log brokers hit GB/s on commodity disks while classic brokers (RabbitMQ-style) top out far lower under fan-out. The cost you accept: you store messages you've already delivered (the 1.8 PB above), and you must reason about retention as a correctness property — if a consumer is down longer than retention, it loses data permanently. DDIA Ch. 11 frames this precisely as "logs vs. messaging," and the partitioned log is the design pattern that makes replay cheap.
6.2 Per-partition ordering vs. total ordering
The honest contract is ordering within a partition, none across partitions. Why not total order? Total order means every message gets a single global sequence number, which means a single serialization point — one leader, one log, no horizontal scaling. You'd be back to a 10 MB/s ceiling for the whole topic instead of per partition. Total order and throughput are in direct tension: the moment you shard for throughput, you give up a global order.
The practical move is to make ordering scope match the business requirement via the partition key. You almost never need "all orders globally in order"; you need "all events for a given account in order." So key by account_id: every event for that account lands in the same partition and is therefore totally ordered relative to each other, while different accounts parallelize freely. This is DDIA Ch. 11's "partitioned logs" insight — you buy as much ordering as you need and not a drop more.
account_id means a single hot account (a celebrity, a bot, a misconfigured integration) sends all its traffic to one partition — a hot partition (lesson 07). One partition can't be split without breaking the ordering guarantee for that key. Mitigations: composite keys (account_id:subkey) when per-subkey order is enough, or accept that the hottest key sets your per-partition ceiling.6.3 Backpressure and consumer-group rebalance
Because consumption is pull-based, backpressure is built in at the read path: a slow consumer simply polls less, its lag grows, and the producer is never blocked by it. The producer side gets backpressure differently — through quotas (the broker throttles a producer exceeding its byte budget) and ultimately through retention: the log is a bounded buffer, and if consumers fall far enough behind, the oldest unread data ages out. That makes "consumer lag in time" the metric you alert on long before data loss.
The subtle failure is rebalance. Partitions are assigned to consumers so each partition has exactly one active reader per group. When a consumer joins, leaves, or is presumed dead (missed heartbeat), the group rebalances — partitions are reassigned. During the classic "stop-the-world" rebalance, all consumers in the group pause, which is a latency spike and a lag jump precisely when you're least able to afford it (e.g. a deploy that restarts every consumer triggers a cascade of rebalances). Senior mitigations: sticky / cooperative-incremental rebalancing (only the moved partitions pause), generous session timeouts, and committing offsets frequently so a reassigned partition replays as little as possible.
6.4 Exactly-once across produce → process → publish
The sharpest senior question is: "Give me exactly-once from produce, through processing, to a downstream publish — or tell me why you can't." The honest answer is that end-to-end exactly-once delivery is impossible in the general case (the consumer can always crash in the gap between "did the side effect" and "committed the offset"), so what we actually engineer is effectively-once:
- Idempotent producer: each producer batch carries a sequence number; the broker dedups retried appends, so a producer retry after a network blip doesn't write the message twice.
- Transactional offset commit: commit the output (the downstream write or the next-topic publish) and the input offset atomically — either both happen or neither — so reprocessing after a crash is a no-op. This is the read-process-write transaction (DDIA Ch. 11).
- Idempotent consumer: when the side effect leaves the system (an external API, a non-transactional store), make the operation idempotent via a dedup key so a redelivery is harmless.
This case only summarizes the mechanism — the full treatment (at-least-once vs. at-most-once vs. effectively-once, the outbox pattern, dedup-key TTLs) is taught in C27, the delivery-semantics anchor. The interview move is to name the impossibility, then describe idempotent producer + transactional offsets as the effectively-once construction, and point at C27 for depth rather than hand-wave "Kafka does exactly-once."
7. Trade-offs
| Choice | Buys | Costs | Choose when |
|---|---|---|---|
| Log (retain) vs. queue (delete) | replay, multiple independent consumers, recovery | storage for already-delivered data; retention is now a correctness property | event streams, CDC, anything with >1 downstream — the default here |
| More partitions vs. fewer | parallelism, higher throughput ceiling | more metadata, more open files, slower & more frequent rebalances | high-throughput topics needing many concurrent consumers |
| acks=all (sync ISR) vs. acks=1 | durability — no tail loss on leader failure | producer latency (wait for replicas) | critical events (payments, audit); use acks=1 for lossy telemetry |
| Long retention vs. short | replay window, late-joining consumers, recovery | storage cost (scales linearly — the 1.8 PB) | audit/analytics streams; shorten for high-volume ephemeral data |
The sharpest one to narrate is acks vs. latency under leader failure. With acks=1 the producer is acked as soon as the leader writes; if that leader dies before followers pull the record, the message is silently lost — fine for metrics, fatal for a payment event. With acks=all you wait for the in-sync replica set, paying replication latency on every write, but a leader can die and a follower takes over with no data loss (DDIA Ch. 5's leader election over the replicated log). You don't pick one globally — you pick per topic, by how much each message is worth.
8. Failure modes
| Failure | Mitigation |
|---|---|
| Consumer lag grows unbounded | Scale out the group (add consumers up to partition count), fix the downstream bottleneck, or shed; alert on lag age well before it approaches retention, or you lose data. |
| Poison message (always fails) | Bounded retries, then route to a dead-letter queue with the original payload + error for inspection; never let one message block the partition forever. |
| Broker / leader failure | Elect a new leader from the in-sync replica set and resume from the replicated log (DDIA Ch. 5). Data acked under acks=all survives; acks=1 may lose the tail. |
| Duplicate processing | At-least-once is the default; make consumers idempotent and commit offsets after the side effect succeeds. Full treatment in C27. |
| Hot partition | Composite keys when sub-ordering suffices; otherwise the hottest key bounds your per-partition throughput — size for it. |
| Rebalance storm on deploy | Cooperative/sticky rebalancing, rolling restarts with pauses, generous session timeouts, frequent offset commits. |
9. What to say in the interview
- Queue or log — and why does it matter? (senior answer) A queue deletes on consume; a log retains and lets consumers replay via independent offsets. I'd build the log because it's a strict superset — it can emulate a queue, supports multiple independent consumers, and enables replay/reprocessing, at the cost of storing already-delivered data. That single choice drives the whole append-only, offset-based design.
- How do you size this? (senior answer) In bytes, not messages. 1M msg/s × 1 KB = 1 GB/s ingress; ×7 days ≈ 600 TB; ×3 for replication ≈ 1.8 PB on disk. Partition count = target throughput ÷ per-partition ceiling ≈ 1 GB/s ÷ 10 MB/s = 100+ partitions. Lag = (produce − consume) × time is the runtime signal.
- What ordering do you guarantee? (senior answer) Per-partition ordering, none across partitions. Total order needs a single serialization point and kills throughput. I match ordering scope to the business by keying — e.g. key by account_id so each account's events are ordered while accounts parallelize.
- Give me exactly-once produce → process → publish, or tell me why you can't. (senior answer) True end-to-end exactly-once is impossible — the consumer can crash between the side effect and the offset commit. What I build is effectively-once: idempotent producer (broker dedups retried appends) plus transactional offset commit (output + input offset atomic) plus idempotent consumers for external side effects. Full treatment is the delivery-semantics anchor, C27.
- What's a consumer-group rebalance and why is it dangerous? (senior answer) Partition-to-consumer reassignment when membership changes. A stop-the-world rebalance pauses the whole group — a lag spike exactly when you can't afford it, and deploys can cascade them. Mitigate with cooperative/sticky rebalancing, frequent commits, and tuned session timeouts.
- acks=1 vs. acks=all? (senior answer) acks=1 returns when the leader writes — fast, but loses the tail if the leader dies before replicas catch up. acks=all waits for the in-sync replica set — durable across leader failure, at replication latency. Choose per topic by message value: acks=all for payments, acks=1 for telemetry.
- What happens when retention expires before a consumer catches up? (senior answer) The data is gone — retention is a correctness property in a log, not just a cost knob. So I alert on lag age relative to retention and size retention with recovery time in mind. This is why "log" buys replay only within the window.
- Why is the log fast enough to hit GB/s? (senior answer) Because every write is a sequential append and every read is a forward cursor — no random-access mutation, batching amortizes syscalls, the page cache serves hot reads, and zero-copy ships bytes from disk to socket. That's the mechanism lesson, C26 — I'd defer the internals there and keep the design here at the architecture level.
Related foundation lessons
This case is the partitioned-log anchor. The mechanism of why an append-log is fast (sequential I/O, batching, zero-copy) is the companion lesson C26 — deferred there on purpose. Delivery semantics / effectively-once live in the anchor C27, which this case links to rather than re-deriving. For the broader queue-vs-broker landscape see lesson 11 (async messaging); for the key-hashing and hot-partition mechanics see lesson 07 (partitioning); and for ISR replication and leader election see lesson 08 (replication).