Asynchronous messaging & streaming
Taking work off the synchronous critical path. A queue is a shock absorber, a buffer, and a decoupler — and it costs you eventual consistency, an extra moving part, and a lifetime of reasoning about duplicates.
A synchronous call couples caller and callee in time. The caller blocks until the callee answers, so a slow or dead callee directly degrades the caller — the tail-latency coupling from lessons 02 and 07. If A calls B calls C synchronously, A's p99 is the sum of three tails and A's availability is the product of three availabilities.
A message broker breaks that coupling. The producer hands off a message and returns immediately; the consumer processes it whenever it can. The two sides no longer have to be up at the same time, fast at the same time, or scaled to the same size. That is the entire value proposition — and everything else in this lesson is a consequence of that one move.
Why decouple: what a queue buys and what it costs
Putting a buffer between producer and consumer is a classic engineering trade. Be able to recite both columns:
| You buy | You pay |
|---|---|
| Spike absorption — a 10× burst fills the buffer instead of overwhelming the consumer; the queue smooths bursty arrivals into steady drain. | Eventual consistency (09) — the result isn't ready when the API returns. "Your video is processing." |
| Independent scaling — add consumers without touching producers; scale the slow side only. | Extra component — the broker is now a dependency that can fail, lag, fill its disk, or lose data. |
| Resilience — consumer down ≠ producer down. Work waits in the queue and drains on recovery. | End-to-end latency now includes queue wait — under load, time-in-queue can dominate (lesson 02's queueing term). |
| Fan-out — one event, many independent consumers (search index, email, analytics) with no producer changes. | Ordering & duplicates get subtle — global order is gone; you must design for redelivery. |
The decision rule. Keep the user-blocking path short: validate the request, persist the minimum, enqueue, respond. Do everything derived asynchronously — resize the image, transcode the video, send the email, update the search index, fan out the post to followers, recompute the recommendation. If the answer is "the user does not need to see the result of this work to consider their action done," it belongs off the critical path.
Two broker shapes: queue vs. log
The single most important distinction in this space. They look similar (producers, consumers, messages) but the data model is opposite, and that drives every other property.
| Queue / message broker | Log / stream | |
|---|---|---|
| Examples | RabbitMQ, ActiveMQ, AWS SQS | Apache Kafka, AWS Kinesis, Pulsar, Redpanda |
| Delivery model | A message goes to one of N competing consumers, then is removed | An append-only ordered log; messages are retained, consumers read by offset (the consumer's bookmark — how far into the log it has read) |
| After consumption | Gone (acked & deleted) | Still there until retention expires — replayable |
| Multiple subscribers | Hard — each message is consumed once; need separate queues / exchanges to copy | Native — each consumer group (a set of consumers that split a topic's partitions so each message is handled once by the group) reads the whole stream independently at its own offset |
| Scaling parallelism | Add consumers; broker load-balances messages across them | Add consumers up to partition count (the hard ceiling) |
| Ordering | Best-effort / per-queue; weak once you parallelise | Strict within a partition; none across partitions |
| Best for | Distributing tasks/work — job queues, RPC offload, "do this once" | Event streaming, replay, reprocessing, multiple subscribers, audit log |
Mental model: a queue is a to-do list — items are pulled off and disappear. A log is a ledger / journal — entries are appended and stay, and any reader can scan from any point. Pub/sub (one publish → many subscribers) is the fan-out pattern; a log gives it to you for free via consumer groups, while a classic broker implements it with a topic-exchange that copies the message into each subscriber's queue.
Why a log is so fast: a log broker like Kafka stores each partition as an append-only commit log on disk — the same write-optimised, sequential-append shape that makes an LSM-tree's (04) writes cheap. Because every write goes to the end of a file rather than seeking to a random spot, the disk does sequential I/O — and sequential writes are an order of magnitude faster than random ones, even on SSDs. That single physical property is why a log sustains such high throughput while still being durable on disk.
Choosing between them
- Need to replay a week of events because you found a bug in the consumer? You need a log. A queue already deleted them.
- Need three different teams to each react to "order placed" without coordinating? Log + three consumer groups. With a queue you'd fan into three queues and own the duplication.
- Pure "run this background job once" with no replay and no multiple readers? A task queue (SQS/RabbitMQ) is simpler, cheaper, and has built-in visibility timeouts and DLQs.
- Need per-key ordering at scale (all events for one user in order)? Log with key-based partitioning.
The scaling & ordering model (log)
This is partitioning (lesson 07) wearing a different hat. A topic is split into partitions. A partition is simultaneously:
- The unit of parallelism — one partition is consumed by at most one member of a consumer group at a time. So max consumer parallelism = partition count. Ten partitions cap you at ten working consumers; the eleventh sits idle.
- The unit of ordering — messages are ordered within a partition only, never globally. There is no "the order of the topic"; there are only N independent ordered sub-streams.
Key-based partitioning routes all messages with the same key (e.g. user_id) to the same partition via hash(key) % partitions, so per-key order is preserved even though global order is not. This is usually exactly what you want: you don't care that user A's events interleave with user B's, only that each user's own events stay in order.
The trade: more partitions = more parallelism and throughput, but weaker effective ordering (more independent streams), more open files / replication overhead, slower rebalances, and — note — you generally can't reduce partition count later without breaking key→partition stability. Size for ~1–2× your projected peak consumer count, not 1000× "just in case."
Delivery semantics — the classic interview trap
Ask any candidate "how do you get exactly-once?" and watch what happens. Here is the honest table:
| Semantic | How | Failure behaviour | Use when |
|---|---|---|---|
| At-most-once | Ack before processing (fire-and-forget) | May lose messages on crash; never duplicates | Metrics, telemetry, logs — a dropped data point is fine |
| At-least-once | Ack after processing; redeliver on timeout/failure | Never loses; produces duplicates (crash between side-effect and ack → reprocess) | The practical default. Almost everything that matters |
| "Exactly-once" | Not deliverable across a network. Achieved as at-least-once + idempotent consumer / dedup key, or transactionally within one system | Effect happens once even though delivery may repeat | Payments, inventory, anything where a duplicate is harmful |
The line to deliver: "Exactly-once delivery is a myth — two generals; you can't atomically commit a network send and a remote side-effect. Exactly-once processing is the real goal, and you get it with at-least-once delivery plus an idempotent consumer." Idempotency means: include a stable message_id / dedup key (the same idea as the Idempotency-Key request header in 03), and on the consumer either (a) check-and-set against a dedup store before applying, (b) use an idempotent operation (SET not INCR, upsert by key), or (c) bind the side-effect and the offset commit in one transaction. Kafka's "exactly-once" is precisely this — atomic produce + offset-commit within Kafka; it does not extend to your external database or email provider. The dedup-store mechanics and the inbox/outbox patterns that make this crash-safe are the whole subject of 12.
A consumer pulls a message, charges a card (the side-effect), then sends the ack. Suppose processing takes 200 ms and the consumer crashes uniformly at random during processing with probability 0.1% per message (deploys, OOMs, node loss).
The dangerous window is "side-effect done, ack not yet sent." If the side-effect lands at, say, 150 ms of the 200 ms and the crash falls in the final 50 ms, the broker never sees an ack, its visibility timeout fires, and it redelivers — a second charge.
P(dup per message) ≈ P(crash) × (vulnerable fraction) = 0.001 × (50ms / 200ms) = 0.00025
At 2,000 msg/s that is 0.00025 × 2000 = 0.5 double-charges per second — ~43,000 per day. "Rare" crashes produce a steady stream of duplicates at scale. Conclusion: you do not hope duplicates won't happen; you make the charge idempotent (dedup on payment_id) so a redelivery is a no-op. The math is why "we'll just be careful" is not an answer.
Operational realities: backpressure, lag, and poison messages
A queue is a buffer, and buffers have a failure mode: they hide a permanently-too-slow consumer until they explode.
- Consumer lag is THE health metric (forward-ref lesson 16): lag = latest produced offset − latest committed offset, i.e. how many messages are waiting. Flat lag = keeping up. Steadily rising lag = capacity < arrival rate, and end-to-end latency is heading to ∞.
- Backpressure — what you do when consumers can't keep up. Options: a bounded queue that pushes back / sheds (drop or reject producers, return 429), slow the producer (flow control), or let an unbounded buffer grow and watch lag. The unbounded buffer is seductive and wrong: it converts an under-capacity error into rising latency that surfaces hours later, far from the cause.
- Dead-letter queue (DLQ) — a "poison" message that always throws (malformed payload, a bug) will be redelivered forever and, in a log, blocks its whole partition (offset can't advance). After N retries, route it to a DLQ so the partition keeps draining; alert on DLQ depth and inspect offline.
- Retention / replay — logs keep data for a window (time or size). Long enough retention lets you fix a consumer bug and reprocess from an old offset; too long and you pay storage and risk replaying stale events into downstreams.
If consumer capacity is even 1 msg/s below arrival rate, the backlog grows without bound and time-in-queue → ∞ — the queue "works" (no errors) while the data gets arbitrarily stale. A queue buys time to absorb spikes; it cannot fix a sustained capacity deficit. The fix is always more consumers (up to partition count), more partitions, a faster consumer, or shedding load — never "make the buffer bigger."
Backpressure simulator
The arithmetic that decides whether a queue is a shock absorber or a time bomb. Capacity is C × perConsumer, capped by partitions; the net drain rate is capacity − λ.
Interview prompts you should be ready for
- When would you put a queue between two services, and when would a synchronous call be better? (senior answer: queue when the work is derived/deferrable and you want spike absorption, independent scaling, or fan-out, and the caller doesn't need the result inline. Stay synchronous when the caller needs the answer to proceed, or when the added eventual-consistency + extra-component cost isn't justified — don't queue a read that must return data.)
- Kafka vs. SQS/RabbitMQ — pick one and defend it. (senior answer: log vs. queue. Kafka if you need replay, multiple independent consumer groups, per-key ordering at scale, or an audit/event-sourcing stream. A task queue if it's "run this job once," no replay, single reader — you get visibility timeouts and DLQs with far less operational weight.)
- How does ordering work in Kafka, and how many consumers can actually run in parallel? (senior answer: ordering is per-partition only, never global; one partition → at most one consumer per group, so max parallelism = partition count. Key-based partitioning gives per-key order. The eleventh consumer on a ten-partition topic is idle.)
- Can you guarantee exactly-once delivery? (senior answer: no — exactly-once delivery across a network is a myth; you can't atomically commit a send and a remote side-effect. Use at-least-once + idempotent consumer (dedup key, upsert, or transactional offset commit) to get exactly-once processing. Kafka's "exactly-once" is internal to Kafka and doesn't cover your external DB.)
- Your consumer just got a 10× duplicate redelivery storm. Why, and how is your system unharmed? (senior answer: at-least-once redelivers when an ack is lost — rebalance, timeout, crash between side-effect and ack. Unharmed because every handler is idempotent: dedup on a stable message_id, or operations are naturally idempotent (set/upsert), so reprocessing is a no-op.)
- Consumer lag is climbing steadily. Walk me through diagnosis and fixes. (senior answer: rising lag = capacity < arrival. Check if all partitions have a consumer (idle members mean you're partition-capped → add partitions); check per-message latency for a slow dependency or a poison message stuck on a partition; then scale consumers up to partition count, speed up the handler, or shed load. Making the buffer bigger fixes nothing.)
- What's a dead-letter queue and why does it matter for a partitioned log? (senior answer: a poison message that always fails gets redelivered forever and blocks its partition's offset, stalling everything behind it. After N retries, route it to a DLQ so the partition advances; alert on DLQ depth and inspect offline.)
- How do you apply backpressure when producers outrun consumers? (senior answer: bounded queue with shed/reject (429) or producer flow control — propagate the pressure to the source. The anti-pattern is an unbounded buffer that converts an over-capacity condition into silently rising latency. A queue absorbs spikes; it cannot fix a sustained deficit.)
A broker decouples producer and callee in time — that's the whole game, and everything else is a consequence. Choose a queue for "do this task once," a log for replayable, multi-subscriber, per-key-ordered streams. Remember the ceilings: in a log, max parallelism = partition count and order is per-partition only. Assume at-least-once and make consumers idempotent — exactly-once delivery is a myth, exactly-once processing is the goal. And watch lag: a queue absorbs spikes but cannot fix a sustained capacity deficit; an unbounded buffer just hides the failure until latency goes to ∞.