Part 6 · Invariants under concurrency

Messaging, Idempotency, and the Transaction Boundary

Lesson 13 gave the application a clean fiction: wrap a set of reads and writes in a transaction and a chosen set of concurrency and crash faults simply do not happen — atomicity, isolation, and durability all hold inside one database. But a real operation rarely stays inside one database. "Place the order" writes an orders row and publishes an order_placed event to Kafka so the warehouse, the billing service, and the analytics pipeline can react. "Register the user" writes a users row and calls a remote email service. The instant an operation touches a database and a message broker and a second service, the ACID box no longer wraps the whole thing — there is no transaction that spans them. A crash between the two steps leaves an order with no event, or an event for an order that was rolled back. This lesson is about keeping invariants alive across that boundary, where ACID stops.

DDIA source

Arc drawn from Martin Kleppmann, Designing Data-Intensive Applications, Chapter 7 (Transactions — and specifically why a single-system transaction cannot be stretched across a broker or a remote service) and Chapter 11 (Stream Processing — exactly-once / effectively-once message processing, idempotence, and the read-process-write boundary). Original synthesis; the outbox/inbox framing and the ML-infra examples are ours.

Linear position

Prerequisite: Lesson 13 — a transaction is the unit that makes a bundle of reads and writes atomic, isolated, and durable within one database; you can name the anomalies and know that the C in ACID is the application's invariant. We reuse exactly that and ask the next question: what happens to the invariant when the operation also has to tell a second system?
New capability: Diagnose the dual-write / dual-system boundary problem, design the outbox pattern (one local transaction + an at-least-once relay) so a database write and a published message can never disagree, make a consumer safe to re-deliver with idempotency keys and an inbox, state precisely what "exactly-once" can and cannot mean (it is effectively-once = at-least-once delivery plus idempotent processing), and pick a saga or a durable workflow engine for multi-step cross-system operations that must survive a crash mid-flight.

The plan

Six moves. (1) Name the boundary problem precisely: there is no ACID transaction across a database and a broker, so "write the DB then publish" can fail in the gap and lose or phantom a message. (2) Say why the obvious fix — a distributed transaction / two-phase commit across both systems — is usually avoided (blocking, coupling), pointing forward to lesson 17 for the mechanics. (3) Build the outbox pattern: write the business row and an outbox event row in one local transaction, then a relay tails the outbox and publishes — turning two systems into one atomic local write plus an at-least-once relay. (4) Make the consumer safe with the inbox / dedup pattern and idempotency keys, with a worked duplicate-rate number. (5) Pin down exactly-once as a scoped guarantee — effectively-once — and link it to stream processing (lesson 19). (6) Stretch to multi-step operations: sagas with compensations and durable workflow engines that persist progress so a crash resumes rather than restarts. Then failure modes, the "Where is truth?" artifact, and the hand-off to partial failure and clocks.

1 · The boundary problem: there is no transaction across two systems

What this is: why "write the DB, then publish a message" is not atomic, and the two ways it breaks.

A transaction (lesson 13) is a property of one resource manager — one database, with one commit log, one lock manager, one notion of abort. The moment your operation has to update a database and hand a message to Kafka (or call a second service over HTTP), you have two independent systems that commit independently, and nothing forces their two outcomes to agree. The naive code looks innocent:

the "dual write" — looks atomic, is not def place_order(o): db.begin() db.insert(orders, o) # step A: durable in the DB db.commit() # <-- COMMIT POINT of system 1 broker.publish("order_placed", o) # step B: durable in the broker # <-- COMMIT POINT of system 2 there are TWO commit points and a GAP between them. A crash, a network drop, or a broker timeout in the gap leaves the two systems disagreeing: crash AFTER db.commit, BEFORE publish -> order exists, NO event (LOST message: warehouse never ships) publish succeeds, then db.commit fails -> event exists, NO order (if you reorder the two) (PHANTOM message: ship a rolled-back order) publish times out, you retry, first one -> order exists, TWO events actually went through (DUPLICATE message)

There is no ordering of the two steps that is safe. Publish-then-write risks a phantom message (an event for a row that never committed); write-then-publish risks a lost message (a committed row no one is told about); and any retry of a step whose acknowledgement you did not see risks a duplicate. This is the dual-write problem, and it is the central failure that this lesson exists to fix. The deep reason: the C in ACID — the invariant — was "an order_placed event exists for every committed order, and only for committed orders," and that invariant now spans two systems, so no single transaction can enforce it.

Why you can't just "make publish reliable"

Engineers reach for retries: if publish fails, retry until it succeeds. But the crash can happen between db.commit and the line that even attempts the publish — there is no durable record that a publish is owed, so after a restart the process has no idea it still has to send the event. Reliability of the publish call does not help; the gap is in remembering that a publish is required at all. The fix has to make "a message is owed" part of the same durable transaction as the business write. That is the outbox (§3).

2 · Why not a distributed transaction (2PC) across both systems?

What this is: the textbook fix that exists, and why production usually refuses it.

There is a classical answer to "make two systems commit atomically": a distributed transaction using two-phase commit (2PC). A coordinator asks every participant (the DB, the broker) to prepare — durably promise it can commit — and once all vote yes, tells them all to commit. If any votes no, all abort. Done correctly it gives genuine atomicity across heterogeneous systems. So why is it rare in modern stacks? Three reasons, and we only sketch them here because the mechanism is the subject of lesson 17.

It blocks on the coordinator

Between "prepared" and "commit/abort", every participant holds locks and cannot release them. If the coordinator crashes after collecting votes but before deciding, participants are stuck in doubt — locks held, rows frozen — until it recovers. A single coordinator becomes a correctness-critical single point that can stall many systems at once.

It couples independent systems

2PC requires every participant to speak the same protocol (XA, or equivalent) and to expose a durable prepare phase. Most message brokers and most microservice APIs do not, and you do not want your database's availability tied to a remote email vendor's. Heterogeneous systems with independent operators are exactly where 2PC is most painful.

It scales and fails poorly

Throughput drops with the slowest participant; every distributed transaction is at least one extra round trip of prepare; and the failure modes (orphaned prepared transactions, coordinator log corruption) are operationally nasty. The industry verdict for DB-plus-broker has largely been: avoid it.

So instead of forcing two systems to commit atomically, the dominant pattern collapses the problem to a single system: make the only thing that must be atomic be a write to one database, and turn the second system into a downstream consumer of that database's durable log. That is the outbox.

3 · The outbox pattern: one local transaction, then an at-least-once relay

What this is: the workhorse fix — write the business row and the event in the same transaction, then relay.

The outbox pattern (also "transactional outbox") removes the gap by refusing to have two commit points. In one local database transaction you write both the business row and a row into an outbox table describing the event you owe. Because both writes are in the same ACID transaction, they commit together or not at all — the invariant "an event is owed for exactly the committed orders" is now a single-database constraint, which lesson 13 already taught you how to keep. A separate relay (also called a message relay or dispatcher) then reads unpublished outbox rows and publishes them to the broker, marking each as sent.

OUTBOX + RELAY — two systems become one atomic local write + a relay WRITE PATH (one ACID transaction, one commit point): db.begin() insert(orders, o) # business row insert(outbox, {id, topic:"order_placed", payload:o, sent:false}) db.commit() <-- BOTH rows or NEITHER. No gap, no dual write. RELAY (a separate, restartable process — runs forever): loop: rows = select * from outbox where sent=false order by id limit N for r in rows: broker.publish(r.topic, r.payload, key=r.id) # at-least-once update outbox set sent=true where id=r.id sleep(poll_interval) crash anywhere in the relay -> on restart it re-reads sent=false rows and re-publishes. A row may be published TWICE (publish ok, crash before the update) but NEVER lost. The relay's guarantee is AT-LEAST-ONCE.

The trade the outbox makes is precise and worth saying out loud: it converts an unbounded correctness problem (messages can be lost or phantom, with no bound) into a bounded duplication + latency problem (messages are never lost, may be duplicated, and arrive after a small relay delay). Duplication is recoverable on the consumer side (§4); loss is not. That asymmetry is the whole point.

Polling relay vs CDC relay

The relay can learn about new outbox rows two ways. Polling (shown above) repeatedly queries where sent=false — dead simple, but it adds query load and its latency is the poll interval. Change data capture (CDC) tails the database's own write-ahead log (lesson 19) and streams every insert to the outbox table directly into the broker — no polling, lower latency, no sent column needed because the log is the source of truth. Tools like Debezium do exactly this. CDC is the same move taken to its limit: the database's durable log is the event stream. Lesson 32 builds an entire search-index pipeline on this idea precisely to avoid a dual write into the index.

Worked number: relay lag

Suppose the relay polls every 200 ms, fetches up to 500 rows per poll, and each publish + mark-sent takes 2 ms. At steady state with an arrival rate of 300 events/sec: each 200 ms window accumulates 300 x 0.2 = 60 rows, well under the 500-row batch, so the relay drains each batch in 60 x 2 = 120 ms and keeps up. End-to-end lag from db.commit to broker is then bounded by poll wait + drain ≈ 200 + 120 = 320 ms worst case. Now a burst to 3,000 events/sec: each window accumulates 600 rows but the batch caps at 500, so the outbox backlog grows by 100 rows per window — the relay is behind. The knobs: raise the batch limit, shorten the poll interval, or run multiple relay workers partitioned by id ranges. The lesson: outbox lag is real and measurable; size the relay for peak, not average, and alarm on outbox depth (count where sent=false) the way you alarm on a queue.

4 · The inbox and idempotency keys: making re-delivery safe

What this is: the consumer-side half — because the relay is at-least-once, the consumer must tolerate duplicates.

The outbox guarantees at-least-once delivery, which means duplicates are not an edge case — they are guaranteed to happen eventually (relay restart, broker re-delivery on a missed ack, consumer crash after processing but before committing its offset). So the consumer must be built so that processing the same message twice has the same effect as processing it once. That property is idempotence, and there are two standard ways to get it.

Idempotency key + dedup ledger (inbox)

Every message carries a stable, unique idempotency key (the outbox row id works perfectly — it is stable and unique per business event). The consumer keeps an inbox / processed_keys table. In one local transaction it checks "have I seen this key?", and if not, applies the effect and inserts the key. A duplicate finds the key already present and is dropped. The dedup record and the effect commit together, so the consumer can crash anywhere and stay correct.

Naturally idempotent effects

Make the operation itself insensitive to repetition. An upsert keyed by a business id (insert ... on conflict do update) lands the same row no matter how many times it runs. Setting a value (status = 'shipped') is idempotent; incrementing a counter (count = count + 1) is NOT — that is the lost-update shape from lesson 13 in reverse, and re-delivery double-counts. Prefer set-to-value over add-delta whenever the message could repeat.

INBOX dedup — the consumer makes at-least-once look effectively-once on message m (carries idempotency_key = outbox row id): db.begin() if exists(inbox where key = m.key): # already processed db.commit(); ack(m); return # drop the duplicate apply_business_effect(m) # e.g. mark order paid insert(inbox, {key: m.key, at: now()}) # remember we did it db.commit() <-- effect + dedup record commit TOGETHER ack(m) crash before db.commit -> message redelivered, reprocessed cleanly (no effect yet) crash after db.commit, before ack -> redelivered, key found, dropped. SAFE.

Worked number: the duplicate rate you are signing up for

Say the broker re-delivers on a missed ack and your consumer crashes (or its ack is lost to a network blip) on 0.5% of messages, and the relay independently double-publishes on 0.2% of its rows. Then roughly 0.005 + 0.002 = 0.007, about 1 in 143 messages, arrives at the consumer more than once. At 300 events/sec that is about 2 duplicates every second — 181,000 per day. Without idempotent processing, each one is a double-charge, a double-ship, or a corrupted counter. With the inbox, all 181,000 are silently dropped and nothing downstream notices. This is why "we'll fix duplicates later" is never acceptable: the duplicate rate is not zero, it is a steady fraction of a percent, every day, forever.

5 · "Exactly-once" is effectively-once: a scoped guarantee, stated precisely

What this is: the most over-promised phrase in messaging — what it really means and what it cannot.

Marketing says "exactly-once delivery." That phrase, taken literally, is impossible: a sender that does not receive an acknowledgement cannot know whether the message was lost before arrival or the ack was lost after, so it must either risk losing the message (at-most-once) or risk sending it again (at-least-once) — there is no third delivery option over an unreliable network. You cannot get exactly-once delivery.

What you can get is effectively-once processing: at-least-once delivery plus idempotent processing. The message may arrive several times (delivery is at-least-once), but the observable effect on state happens exactly once (because the consumer dedups by idempotency key, §4). The trick is to be precise about which step is exactly-once:

what is and isn't guaranteed at-most-once delivery : send, never retry. May LOSE. Effect <= 1 time. (rarely ok) at-least-once delivery : retry until acked. May DUPLICATE. Effect >= 1. (the relay) exactly-once DELIVERY : impossible over an unreliable network. (a myth) effectively-once : at-least-once delivery + idempotent processing. Delivery may repeat; STATE CHANGE happens once. (the goal)

This is the same guarantee a stream processor offers when it claims "exactly-once": frameworks like Kafka Streams or Flink achieve it by atomically committing the output write and the input offset together (an inbox by another name), so re-reading an un-acked offset reprocesses without double-applying. Lesson 19 builds that read-process-write loop in detail. The honest summary to give in an interview: "There is no exactly-once delivery. We get effectively-once by combining at-least-once delivery with an idempotent, deduplicating consumer, and the guarantee is scoped to state changes inside systems we control — it does not extend to a third party's side effect we cannot dedup, like an email that was already sent."

Where effectively-once leaks: non-idempotent external side effects

Idempotence is easy when the effect is a row in your database (dedup table, upsert). It is hard or impossible when the effect is external and observable: sending an email, charging a card, calling a partner API that itself is not idempotent. For those, push the idempotency key to the external system if it supports one (Stripe accepts an Idempotency-Key header and dedups on its side), or record "email for key K sent" in your inbox before the send and accept that a crash in the tiny window between send and record-commit can double-send. You cannot make a non-idempotent third party exactly-once; you can only shrink and document the window.

6 · Multi-step operations: sagas and durable workflows

What this is: when the operation is not one write but a chain of cross-system steps that must survive a crash mid-flight.

The outbox handles "one local write, then tell others." But many operations are a sequence of cross-system steps, each of which can fail: reserve inventory → charge the card → allocate a shipping slot → send a confirmation. There is no single transaction over all four (different systems, §1–2), and a crash after step 2 must not leave the card charged with no shipment. Two patterns manage this.

Saga (with compensations)

Model the operation as a sequence of local transactions, each in its own system, linked by events (often via outboxes). If a later step fails, run compensating transactions that semantically undo the earlier ones — refund the charge, release the reservation. A saga gives no isolation (other transactions can observe the half-done intermediate state) and no automatic rollback; you write the compensations and accept that the system is eventually-consistent across steps. It is the distributed analogue of "do, and if it goes wrong, do the opposite."

Durable workflow engine (Temporal-style)

A workflow engine persists the progress of the operation — which steps have completed, with what results — to a durable store as it runs. If the worker crashes, the engine resumes the workflow from the last completed step rather than restarting from the top, replaying recorded results instead of re-executing them. Each step is still required to be idempotent (it may be retried), but the engine removes the burden of hand-rolling progress tracking, timers, and retries. This is "durable execution": the call stack itself becomes crash-resilient state.

The relationship to everything above: a saga is built out of outboxes and idempotent consumers — each step's event is published via an outbox and consumed idempotently. A durable workflow engine is the same guarantees packaged so the application writes ordinary-looking sequential code while the engine supplies the persistence, retries, and at-least-once-plus-idempotent semantics underneath. Both turn "a crash restarts the whole multi-step operation" into "a crash resumes it," which is the cross-system echo of the atomicity lesson 13 gave you inside one database.

ML-infra tie — a training-job launch is a saga

"Launch fine-tuning run R" must: write a runs row (system of record), reserve a GPU pool (scheduler), copy the dataset snapshot (object store), and register the run in the experiment tracker. Four systems, no shared transaction. Do it as a dual write and a crash after the GPU reservation leaks 8 expensive GPUs with no run to bill them to. Do it as a saga driven by an outbox: write the runs row + an outbox event in one transaction; a relay drives the scheduler, store, and tracker, each step idempotent (keyed by run id) so retries are safe; if dataset copy fails, a compensating step releases the GPU reservation. Or run the whole launch inside a durable workflow so a controller restart resumes at "dataset copied" instead of re-reserving GPUs. Either way the invariant — "GPUs are reserved if and only if a run exists to use them" — survives the crash that a dual write would have broken.

Trade-offs

Choice	Buys	Costs / still allows
Dual write (DB then publish)	Nothing — it is the bug. Simple to type.	Lost / phantom / duplicate messages with no bound; invariant broken on any crash in the gap
Distributed txn / 2PC	True atomicity across heterogeneous systems	Blocking on coordinator (in-doubt locks), tight coupling, poor scale; broker/services rarely support it (lesson 17)
Outbox + polling relay	No dual write; at-least-once, never lost; trivial to operate	Poll-interval latency; extra query load; outbox table growth (needs pruning)
Outbox + CDC relay	Lowest latency; the DB log is the stream; no `sent` column	Operate a CDC pipeline (Debezium etc., lesson 19); log-format coupling
Inbox / idempotency key	Re-delivery is safe; effectively-once state change	Dedup table growth; must key every message stably; external side effects still leak (§5)
Saga + compensations	Multi-step cross-system ops with bounded recovery	No isolation (visible intermediate state); you hand-write every compensation; eventually consistent
Durable workflow engine	Crash resumes not restarts; retries/timers/progress handled for you	New infrastructure to run; steps still must be idempotent; lock-in to the engine's model

Failure modes

The dual write. "Write the DB, then publish" shipped as-is. Symptom: occasional orders with no downstream event (or events for rolled-back orders) that no single-service test reproduces, because the gap only opens on a crash between two commits.
"Exactly-once" taken literally. A consumer written assuming each message arrives once. Symptom: double-charges and double-ships at a steady fraction-of-a-percent rate (§4 worked number) under normal operation, not just incidents.
Non-idempotent effect on the consumer. The handler does count = count + 1 or sends an un-keyed email. Symptom: counters drift high, customers get duplicate emails, only under retry/redelivery.
Unbounded outbox / inbox table. Rows are never pruned; the table and its indexes bloat, the relay's where sent=false scan slows. Symptom: rising relay lag and storage with no corresponding traffic increase.
Relay not sized for peak. Batch limit or single worker can't drain bursts; backlog grows (§3 worked number). Symptom: outbox depth climbs during traffic spikes, downstream sees stale events.
Saga with no compensation for a step. A mid-saga failure leaves a side effect with nothing to undo it. Symptom: leaked reservations / charges with no matching final entity.
Idempotency key that isn't stable. Keying on a timestamp or a per-attempt UUID instead of the business event id — every retry looks new, dedup never fires.

Decision checklist

Does any code path write a database and publish / call a second system? If yes, it is a dual write — replace it with an outbox.
Are the business write and the outbox row in the same local transaction (one commit point)?
Is the relay at-least-once, and is outbox depth (count where sent=false) alarmed like a queue? Is the relay sized for peak, not average?
Does every message carry a stable, unique idempotency key (the outbox row id), and does the consumer dedup against an inbox in the same transaction as the effect?
Is every consumer effect idempotent — upsert / set-to-value, never blind increment, never un-keyed external side effect?
For external side effects (email, charge): does the third party accept an idempotency key, and is the unavoidable double-send window documented?
For multi-step operations: is it a saga with a compensation for every step, or a durable workflow that persists progress so a crash resumes? Have you decided you do NOT need 2PC (lesson 17)?
Are outbox and inbox rows pruned once safely past their retention horizon?

Where is truth?

System of record: the business row in the database (the orders row), committed by one local ACID transaction — it is the only thing that is authoritatively true. Copies / derived views: the message in the broker is a derived, at-least-once copy of that fact, and every downstream consumer's state is a further derived view of the stream. Freshness budget: the relay lag (the §3 worst-case ≈ 320 ms, plus broker and consumer lag) — the window during which the broker has not yet been told a committed truth. Owner: the service that owns the database and its outbox owns the truth; consumers own their own derived views. Deletion path: deleting a fact means writing a tombstone/delete event through the same outbox so every derived view converges; you cannot reach into the broker and unsend. Reconciliation / repair: the consumer's idempotency-key dedup (inbox) is the reconciliation that absorbs duplicates and re-deliveries; for divergence, replay the stream and let idempotent processing converge. Evidence it is correct: the idempotency-key ledger — the inbox table of processed keys plus the outbox's sent flags — is the audit trail proving each committed fact was delivered and applied exactly once in effect, and lets you prove no order is missing an event and no event was applied twice.

Checkpoint exercise

Try it

Your service handles "confirm payment": it must write a payments row to its database, publish a payment_confirmed event so the fulfillment service ships, and send the customer a receipt email via a third-party API. (a) Show the naive dual-write code and enumerate every way a crash corrupts the invariant "fulfillment ships iff payment is confirmed." (b) Redesign it with an outbox: give the schema of the two tables, the single transaction, and the relay loop, and state the relay's delivery guarantee. (c) The fulfillment consumer might receive payment_confirmed twice; show the inbox-based handler that makes shipping idempotent, and name the stable idempotency key you'd use. (d) The receipt email is a non-idempotent external side effect — explain why you cannot make it strictly exactly-once and describe the smallest window in which a double-send can still occur and how you'd minimize it. (e) Now suppose confirm-payment also has to reserve a coupon and could need to refund — sketch whether you'd use a saga or a durable workflow and why.

Where this points next

The outbox, the inbox, and the relay all quietly assumed something: that a crashed process stays crashed, that a "missed ack" cleanly distinguishes a dead consumer from a slow one, and that when two relay workers split the outbox by id range, only one of them is really running. None of that is safe in a distributed system. A consumer you declared dead by timeout may be merely slow and is still processing; a relay you failed over may have a zombie predecessor still publishing; the clock you stamped the inbox row with may run backward relative to another node's. Lesson 15 confronts these partial failures directly: why you cannot trust timeouts or wall-clock time to tell live from dead, and how a fencing token — a monotonically increasing number issued with a lease — lets a system reject a write from a process that only thinks it still holds the lock. That is exactly the guarantee that keeps a failed-over relay's zombie from corrupting the very outbox this lesson built.

Takeaway

A transaction protects an invariant inside one database, but real operations cross a broker and a second service, where ACID stops. "Write the DB then publish" is the dual-write bug: a crash in the gap loses or phantoms or duplicates the message, because the invariant now spans two systems with independent commit points. A distributed transaction (2PC, lesson 17) could make them atomic but blocks on its coordinator and couples heterogeneous systems, so it is usually avoided. The dominant fix is the outbox pattern: write the business row and an outbox event row in one local transaction, then a restartable relay (polling or CDC, lesson 19) publishes them — collapsing two systems into one atomic local write plus an at-least-once relay that may duplicate but never loses. The consumer closes the loop with an inbox and a stable idempotency key, deduping in the same transaction as the effect so re-delivery is safe. There is no exactly-once delivery; what you get is effectively-once = at-least-once delivery + idempotent processing, scoped to state changes in systems you control (external side effects still leak a small window). For multi-step cross-system operations, use a saga with compensations or a durable workflow engine that persists progress so a crash resumes rather than restarts. The system of record is the DB row; the broker carries a derived copy; the idempotency-key ledger is the evidence it all reconciles.

Interview prompts

Why is "write the database, then publish to Kafka" not safe, even with retries? (§1 — two independent commit points with a gap; a crash between them loses the message, and reordering risks a phantom; there's no durable record that a publish is owed, so retrying the publish call doesn't help — the gap is in remembering a publish is required.)
You could use a distributed transaction across the DB and broker — why is that usually avoided? (§2 — 2PC blocks on the coordinator with locks held in-doubt, couples heterogeneous systems that must all speak the protocol, and scales/fails poorly; most brokers and service APIs don't support it. Mechanics in lesson 17.)
Explain the outbox pattern and exactly what guarantee the relay provides. (§3 — write the business row and an outbox event row in one local transaction so they commit atomically; a separate relay tails the outbox and publishes, marking rows sent. The relay is at-least-once: never lost, may duplicate on a crash between publish and mark-sent.)
Given at-least-once delivery, how do you make a consumer correct? (§4 — idempotent processing: a stable idempotency key (the outbox row id) plus an inbox/dedup table checked-and-inserted in the same transaction as the effect, or naturally idempotent effects like upsert/set-to-value; never blind increment.)
Is exactly-once delivery achievable? What does "exactly-once" actually mean? (§5 — no exactly-once delivery: a lost ack is indistinguishable from a lost message, so you choose at-most-once or at-least-once. What's real is effectively-once = at-least-once delivery + idempotent processing; the state change happens once even though delivery may repeat. Scoped to systems you control; external side effects leak a small window.)
Why is incrementing a counter in a message handler dangerous, and what do you do instead? (§4 — re-delivery double-counts because add-delta isn't idempotent; use set-to-value or upsert keyed by the business id, or dedup the message by idempotency key before applying.)
A multi-step operation spans four systems and must survive a worker crash — saga or workflow engine? (§6 — a saga chains local transactions linked by events (outboxes) with hand-written compensations and no isolation; a durable workflow engine persists progress and resumes from the last completed step on crash, with retries/timers handled for you but steps still idempotent. Both turn restart into resume.)
Where does the outbox/idempotency design break, and how would you catch it in production? (§3–5 failure modes — unstable idempotency keys, non-idempotent external effects, unbounded outbox/inbox tables, a relay not sized for peak; catch via outbox-depth alarms, dedup-ledger audits proving every committed row was delivered once in effect, and pruning past retention.)