At-most, at-least, and effectively-once delivery

"Exactly once" is the most over-promised phrase in distributed systems. The honest question is narrower: what happens in the microsecond between the side effect and the acknowledgement, and how do you make the inevitable duplicate harmless instead of corrupting?

Anchor case Delivery semantics Idempotency & outbox

First principle — the crash window is unavoidable

Doing useful work means two distinct events must happen: the side effect (charge the card, write the row, send the email) and the acknowledgement (commit the consumer offset / ack the broker so the message is not redelivered). These cannot be fused into one atomic event across two systems — a card processor and your message broker do not share a transaction. So there is always an instant between them where a crash leaves the side effect done but the message un-acked. On restart the broker, having seen no ack, redelivers. That redelivery is not a bug you can engineer away; it is a structural property of crossing a system boundary. The whole discipline of delivery semantics is choosing which side of that window you fail on, and how you make the inevitable duplicate a no-op.

This page is the anchor for the concept of delivery semantics, idempotency, dedupe keys, and the transactional outbox across the whole case set. C07 (notifications), C25 (stream processing), C29 (inventory), C30 (admission), and C32–C36 (payments & ledger) all defer here for the general mechanism and show only their case-specific dedupe key. If you internalise one page on "exactly-once," make it this one.

The three semantics, stated precisely

The names sound like a quality ranking — they are not. They are three choices about where the ack sits relative to the work, each correct for a different workload.

Semantic	Mechanism	Failure mode	Use when
At-most-once	*Ack before* the work.** Mark the message consumed, then process. A crash after the ack but before the work loses the message.	Can lose. Never duplicates.	Loss is cheaper than a duplicate: metrics, telemetry, "user is typing" pings, best-effort cache busts.
At-least-once	*Ack after* the work is durable.** Process, confirm durability, then commit the offset. A crash before the ack causes redelivery.	Can duplicate. Never loses.	The default for anything that matters. Loss is unacceptable; you accept duplicates and handle them.
Effectively-once	At-least-once delivery + an idempotent consumer keyed by a dedupe id. Redelivery still happens; the second apply is detected and skipped.	None visible — duplicates become no-ops.	Whenever a duplicated side effect corrupts state: payments, inventory, balances.

The slogan trap

"Exactly-once delivery" is impossible across two independent systems — the Two-Generals / FLP intuition (DDIA Ch. 9): you cannot make a remote side effect and a local ack atomic without a shared transaction you do not have across a boundary. What is achievable is exactly-once processing, i.e. effectively-once: deliver as often as the network forces (at-least-once), then dedupe so the effect happens once. The honest senior answer is never "we use exactly-once delivery" — it is "at-least-once delivery plus an idempotent consumer."

Put a number on it — duplicates are not hypothetical

Engineers wave duplicates away as a rare edge case. At scale they are a daily certainty, and the arithmetic decides whether you need idempotency or can skip it.

Take an events pipeline at 10⁹ messages/day. Suppose each message has a crash-window probability — the chance a consumer dies in the gap between side effect and offset commit — of roughly p ≈ 10^-6 (a few-millisecond window against rare-but-real process crashes, partition rebalances, OOM kills, deploys, and pod evictions). The expected duplicate redeliveries per day are:

10⁹ messages/day × 10^-6 = ~1 000 duplicates/day

That is the load-bearing number. Without an idempotent consumer those are 1 000 corruptions per day — 1 000 double charges, 1 000 double inventory decrements, 1 000 duplicate shipping labels. With a dedupe key they become 1 000 no-ops per day: detected, logged, dropped. Same delivery layer, same crash rate — the only difference is whether the consumer is idempotent. This is why "we'll handle duplicates if they come up" is wrong: at 10⁹/day they come up roughly every 86 seconds.

Throughput

10⁹ msg/day

Crash-window p

~10^-6

Duplicates/day

~1 000

With idempotency

1 000 no-ops

Two more estimates steer the design. Dedupe-store size: for a 7-day replay window you store ~7×10⁹ keys; at ~64 bytes/key that is ~450 GB — bounded and shardable, which is exactly why a finite TTL matters (trade-off table below). Retry amplification: during a downstream outage, naive immediate retries multiply load — a 5× retry policy is 5× the traffic precisely when the downstream is weakest — which is why backoff + dead-lettering (lesson 13) is part of correctness, not polish.

The crash window, drawn

Here is the exact sequence that produces the duplicate: the consumer does the side effect, then dies before the offset commit lands. The broker, never having seen the ack, redelivers on restart.

AT-LEAST-ONCE CONSUMER — the crash window t0 poll message m (offset 42) t1 apply side effect ──▶ [ DB: row written / card charged ] durable t2 ╳╳╳ CRASH ╳╳╳ ← offset 42 NOT yet committed to broker ────────────────────────────────────────────────────────── t3 consumer restarts; broker still shows last-committed offset = 41 t4 broker REDELIVERS m (offset 42) ← the duplicate t5 apply side effect AGAIN ──▶ [ DB: SECOND row / SECOND charge ] corruption THE FIX — idempotent apply keyed by m's dedupe id t1 BEGIN txn INSERT processed_keys(m.id) ── unique PK; fails on duplicate apply side effect ── same txn as the dedupe row COMMIT ── side effect + key land together t4' redelivery: INSERT processed_keys(m.id) → conflict → SKIP work, just ack ↑ the 1000 corruptions become 1000 no-ops

The crucial detail is in the fix: the dedupe-key insert and the side effect are in one local transaction. If the side effect is itself a database write you control, this is genuinely atomic — and that is the only place true exactly-once lives: locally, inside a single store's transaction. The moment the side effect is a call to a foreign system (a card processor, an email gateway), you cannot include it in your DB transaction, and you fall back to making that foreign call itself idempotent (pass it your dedupe id as its idempotency key — see C35 for the key mechanics).

Clarify the contract

What the design must do — and, just as important, what it need not:

Define at-most-once, at-least-once, effectively-once and pick the right one per workload, not globally.
Make consumers tolerate redelivery: idempotent apply keyed by a stable dedupe id.
Coordinate a DB write with a published event so neither can happen without the other (outbox).
Support replay/backfill that re-runs safely against existing processed keys.
Need NOT achieve exactly-once delivery — that is an illusion across systems; aim for exactly-once effect.
Need NOT dedupe forever — a TTL covering the replay/retry window is enough for most workloads.

Linearized design — one message through the system

Walk an event in the order it actually moves; the bottleneck and the ordering constraint fall out naturally.

Stamp identity. Every message carries a stable dedupe id — a natural key (payment_intent_id, order_id+line) where one exists, else a producer-assigned UUID. This id, not the broker offset, is the unit of dedupe.
Apply + record in one txn. In a single local transaction, insert the dedupe id into processed_keys (unique constraint) and apply the side effect. A conflict means "already done" → skip.
Commit the offset only after the side effect is durable. This is the ordering rule that defines at-least-once. Commit-before-work would make it at-most-once (loss); commit-after-work makes it at-least-once (duplicate-but-safe).
For "write DB and publish an event," use the outbox (next section) — never a dual write.
A relay publishes outbox rows and marks them sent idempotently; redelivery of an already-sent row is a no-op.
Replay re-runs against existing keys — the same dedupe table that absorbs crash duplicates absorbs intentional replays, throttled off the live path.

Deep dive 1 — the three semantics are an ack-placement choice

Strip away the names and there is one knob: does the ack precede or follow the work?

At-most-once = ack first. You tell the broker "I've got it" before you've done anything. If you crash, the work never happens — but you never do it twice. This is the right choice when a duplicate is worse than a loss: a duplicated "CPU at 91%" metric skews a dashboard; a lost one is invisible noise. Telemetry, presence pings, fire-and-forget logs.
At-least-once = ack last. You do the work, confirm it's durable, then ack. If you crash in the window, you redo — but you never drop. This is the correct default for anything that matters, because for most business operations a missing event (a lost order, an undelivered payment) is catastrophic while a duplicate is merely recoverable — provided the consumer is idempotent.
Effectively-once = at-least-once + idempotent consumer. You keep the never-lose guarantee of at-least-once and neutralise its duplicate cost with a dedupe key. You are not changing the delivery layer at all — the broker still redelivers — you are changing the consumer so the redelivery has no effect.

The interview tell: a candidate who says "I'll use exactly-once" without naming the dedupe key has not understood that the delivery layer cannot give it to them. The strong answer always factors into delivery (at-least-once) + consumer property (idempotent).

Deep dive 2 — true exactly-once is local-only, an illusion across systems

Kafka markets "exactly-once semantics" (EOS), and within its own boundary it is real: a Kafka transaction can atomically write output records and commit the input offsets, because both live in Kafka — one system, one transaction (DDIA Ch. 11). Flink's checkpoint-and-rollback gives the same within its managed state. The trick is that the side effect and the ack are in the same system, so they share a transaction.

The illusion breaks the instant the side effect leaves that boundary. Charging a card is not a Kafka record; sending an email is not a state-store mutation. No Kafka transaction can roll back a real charge. So for any external effect you are back to: deliver at-least-once, and make the external call idempotent by passing it your dedupe id (the processor stores it and refuses a second charge for the same key). Effectively-once is exactly-once's reach extended past the boundary by pushing the dedupe responsibility onto the receiver. The senior framing: exactly-once is a property of a closed system; effectively-once is how you fake it across an open one.

Deep dive 3 — the transactional outbox & offset-commit ordering

The most common correctness bug in event-driven services is the dual write: a service updates its database and publishes an event as two separate operations. There is no atomicity between them, so a crash in the gap leaves one done and the other not — a lost event (DB updated, event never sent) or a phantom event (event sent, DB write rolled back). At 10⁹/day this is the same ~1 000 corruptions as the consumer crash window, now on the producer side.

The transactional outbox (DDIA Ch. 11) collapses the two writes into one. Instead of publishing directly, the service inserts the intended event as a row in an outbox table in the same DB transaction as the business change. Either both commit or neither does — true local atomicity. A separate relay then reads unsent outbox rows and publishes them to the broker, marking each sent. The relay is at-least-once (it may crash after publish, before marking sent → it republishes), which is exactly why downstream consumers must be idempotent — the outbox guarantees the event is never lost, and the dedupe key guarantees it is never double-applied.

TRANSACTIONAL OUTBOX — one DB transaction, no dual write service handles request ┌──────────────────── ONE DB TRANSACTION ────────────────────┐ │ UPDATE orders SET status='PAID' WHERE id=42 │ │ INSERT INTO outbox(event_id, payload) VALUES(uuid, {...}) │ └────────────────────────── COMMIT ─────────────────────────┘ both land together, or neither does │ ▼ ┌───────── RELAY (poll or CDC tail) ─────────┐ │ read unsent outbox rows │ │ publish to broker ──▶ [ Kafka topic ] │ │ mark row sent (idempotent; may republish) │ └────────────────────────────────────────────┘ │ at-least-once ▼ consumer dedupes on event_id → effectively-once

Note the offset-commit ordering that ties it all together, on both sides. Producer side: business write and outbox row in one txn (atomic). Consumer side: side effect + dedupe-key insert in one txn, then commit the offset. The rule is identical at both ends — commit the "I'm done" marker only after the durable effect, in the right transactional grouping — and getting the order backwards is what silently converts at-least-once into at-most-once and starts losing data. (The outbox key mechanics — insert-if-absent, TTL, request-hash — are anchored in C35; this page owns the pattern and ordering.)

Data model & API

Small and intentional. Two tables carry the whole pattern.

Table / op	Shape	Why it exists
`processed_keys`	`(dedupe_id PK, applied_at, result?)` + TTL	Consumer-side dedupe. Unique PK turns a duplicate apply into a conflict → skip. Optional cached result lets a retry return the original response.
`outbox`	`(event_id PK, payload, created_at, sent_at NULL)`	Producer-side atomic intent. Written in the business txn; drained by the relay.
`process(msg)`	idempotent, keyed by `msg.dedupe_id`	The only consumer entry point; insert-key-then-apply in one txn.
`commit_offset(p, n)`	called after durable apply	The ack. Ordering here is the at-least-once guarantee.

Trade-offs

Choice	Buys	Costs	Choose when
At-most-once vs at-least-once	No duplicates, zero dedupe machinery	Can silently lose messages	Loss is cheaper than a duplicate: telemetry, presence, best-effort caches
Dedupe forever vs TTL	Forever: replay-safe at any horizon	Forever: unbounded storage (the ~450 GB/7-day figure, growing without limit)	Forever only for financial/audit records; TTL covering the retry+replay window otherwise
Outbox vs dual write	Atomic write-and-publish; never a lost/phantom event	Relay component + publish latency; CDC or poll cost	Any service that both mutates its DB and emits an event (almost always)
Retry aggressively vs backoff + DLQ	Aggressive: fast recovery from blips	Aggressive: retry storms amplify a downstream outage	Backoff + dead-letter for anything crossing a fragile downstream; aggressive only for cheap local retries

The sharpest of these is dedupe forever vs TTL, because it is where correctness meets cost. A dedupe key only needs to outlive the longest interval over which a duplicate can arrive — i.e. the broker's redelivery/retention window plus your maximum intentional replay horizon. Keep keys one second past that and you are paying storage for impossible duplicates; expire them one second too early and a late redelivery sails through as a fresh message and corrupts state. Size the TTL to the actual redelivery window (often hours to a few days), not to a comfortable round number — and only the genuinely permanent-audit workloads justify "forever."

Failure modes

Failure	Mitigation
Duplicate charge after consumer crash	Idempotent apply keyed by `payment_intent_id`; insert-key + charge in one txn; pass the same id to the processor as its idempotency key.
Lost event after DB commit (dual write)	Transactional outbox — event row in the same txn as the business change; relay drains it.
Phantom event after rollback	Same — outbox row only exists if the txn committed, so no event without a durable change.
Poison message blocks the partition	Bounded retries, then dead-letter (DLQ); never block the partition behind one bad message (lesson 13).
Replay overwhelms downstream	Throttle replay, run it off the live path; dedupe keys make re-processing safe so replay is idempotent by construction.
Dedupe store TTL too short	Size TTL ≥ broker redelivery window + max replay horizon; late redelivery within the window must still find its key.

Interview prompts you should be ready for

Your consumer wrote to the DB then crashed before committing the offset — walk through restart and how you make it safe. (senior answer) On restart the broker still shows the last-committed offset (the message un-acked), so it redelivers. Naively I re-apply and double the side effect. The fix is an idempotent apply keyed by the message's dedupe id: in one local transaction I insert the dedupe id into processed_keys (unique PK) and perform the side effect, then commit the offset only after that txn is durable. On the redelivery the insert conflicts, I skip the work and just ack. The side effect and the dedupe record share one transaction, so they are atomic — the duplicate becomes a no-op.
Why is exactly-once delivery impossible, and what do you deliver instead? (senior answer) Because the side effect and the ack live in different systems with no shared transaction, there is always a crash window between them — Two-Generals. So I deliver at-least-once and make the consumer idempotent: effectively-once. Exactly-once is real only inside a single system (Kafka EOS, Flink checkpoints), where the effect and the ack share a transaction.
At-most-once vs at-least-once — what's the actual difference? (senior answer) Where the ack sits relative to the work. Ack-before-work loses on crash but never duplicates (at-most-once — telemetry). Ack-after-durable-work duplicates on crash but never loses (at-least-once — anything that matters). The choice is "is a loss or a duplicate worse for this workload?"
What's wrong with updating the DB and publishing an event as two steps? (senior answer) It's a dual write — no atomicity, so a crash in the gap loses the event or emits a phantom. Use a transactional outbox: insert the event as a row in the same DB transaction as the business change, then a relay publishes it at-least-once. The outbox guarantees never-lost; the consumer's dedupe key guarantees never-double-applied.
How long do you keep dedupe keys? (senior answer) Long enough to cover the broker's redelivery window plus the max intentional replay horizon — often hours to days. Too short and a late redelivery sneaks through as new; too long and you pay unbounded storage (~450 GB per 7 days at 10⁹/day). Only audit/financial records justify "forever."
A downstream is flapping and your retries are making it worse — what do you do? (senior answer) Retry storms amplify load exactly when the downstream is weakest. Exponential backoff with jitter, a circuit breaker, and a bounded retry count that dead-letters the message rather than blocking the partition (lesson 13). Replay the DLQ off the live path once the downstream recovers — safe because the consumer is idempotent.
Your side effect is an external API call, not a DB write — does the outbox still help? (senior answer) The outbox guarantees the intent is durably recorded and published, but it can't make the external call atomic with your DB. So I make the external call itself idempotent by passing my dedupe id as its idempotency key; the receiver stores it and refuses a second effect for the same key (the C35 mechanics). Effectively-once = my outbox + their idempotency.

Related lessons

This case is the concept anchor; others link here rather than re-deriving. Build on foundation lesson 12 (distributed transactions) — the hub for idempotency, outbox, and sagas — which this page makes concrete at case scale. The delivery layer itself comes from lesson 11 (async messaging) (brokers, offsets, at-least-once semantics), and the partitioned append-log those offsets index is C23. Backoff, circuit breakers, and dead-lettering — the retry-storm mitigations — are lesson 13 (fault tolerance); lesson 16 is where you watch duplicate-skip and DLQ rates to know the pattern is working. The idempotency-key mechanics (insert-if-absent, TTL, request-hash) are anchored in C35; payment-specific dedupe keys live in C32–C34/C36, inventory's atomic conditional decrement in C29 — each shows only its case-specific key and defers the concept to this page.

Transactions & idempotency Async messaging Fault tolerance Observability C23 · partitioned log C35 · idempotency key

Takeaway

Delivery semantics come down to one unavoidable gap — the crash window between the side effect and the ack. You cannot close it across two systems, so you choose which side to fail on: ack-first loses (at-most-once), ack-last duplicates (at-least-once). For anything that matters, pick at-least-once and neutralise the duplicate with an idempotent consumer keyed by a dedupe id — that is effectively-once, and it is the only honest meaning of "exactly-once" across a boundary. At 10⁹ msg/day a ~10^-6 crash window is ~1 000 duplicates daily; idempotency is the single change that turns 1 000 corruptions into 1 000 no-ops. Pair it with a transactional outbox on the producer so you never dual-write, commit offsets only after durable work, and size dedupe TTLs to the real redelivery window. True exactly-once is local; everywhere else, you build effectively-once.