Distributed transactions & idempotency

On one database, atomicity is free: BEGIN…COMMIT and the database guarantees all-or-nothing. The instant a unit of work touches two independent systems, that guarantee evaporates — partial failure becomes the normal case. This lesson is the menu of ways to buy correctness back, and the exact price each one charges.

Why this is hard the moment you have two systems

A single ACID database makes a multi-row change atomic and isolated for you. The engine holds locks, writes a redo/undo log, and on COMMIT flips everything visible at once; on crash it rolls back the half-done work. You never see it.

Now your write must span two independent systems — two shards (lesson 07), a database and a message broker (11), or two microservices each with its own database. There is no shared transaction log, no shared lock manager, no single COMMIT. The defining failure is the partial write:

step 1: debit account A ── COMMIT ✔ (durable) ──────── process crashes / network partition ──────── step 2: credit account B ── never runs ✘ Result: money left the system. No rollback exists. System A is committed; System B never heard about it.

Every technique below is a way to reconstruct atomicity, isolation, or both across that gap. The staff-level skill is choosing the weakest mechanism that still meets the correctness requirement — strong distributed atomicity is expensive and fragile, and most business flows do not need it.

The trap

Engineers reach for "a distributed transaction" reflexively. In practice that means two-phase commit, and 2PC across services is almost always the wrong answer: it is blocking, synchronous, lock-heavy, and its availability is the product of every participant's availability. The right answer is usually saga + outbox + idempotency keys.

Two-phase commit (2PC): atomicity at a brutal price

2PC introduces a coordinator that drives all participants through two phases:

PREPARE — coordinator asks every participant "can you commit?" Each one does the work, writes it durably to a prepared/in-doubt state, holds its locks, and votes YES or NO. A YES is a binding promise: "I can commit if you tell me to, even across a crash."
COMMIT / ABORT — if all voted YES, coordinator tells everyone to COMMIT; any NO (or timeout) → ABORT. Participants apply and release locks.

It delivers genuine atomicity. The costs are severe and worth memorizing:

Property	Cost in 2PC	Why it bites
Latency	2 round-trips × slowest participant	Tail latency dominated by the worst node (06/02 tail effect).
Availability	Product: A = ∏ aᵢ	One participant down → no commit possible. Series availability (07/11).
Locks	Held from PREPARE to COMMIT	Rows locked across network round-trips → contention, reduced throughput.
Blocking	Coordinator dies after PREPARE	Participants stuck in doubt, holding locks, unable to commit or abort. A liveness disaster.

The blocking problem is the killer. Between a YES vote and hearing the verdict, a participant cannot decide on its own (it might have voted YES while a peer voted NO). If the coordinator vanishes, the participant holds locks indefinitely — other transactions queue behind it, and the only fix is human intervention or a timeout that risks splitting the decision. Three-phase commit (3PC) adds a phase to reduce blocking but assumes bounded network delay (a synchronous model), which the real internet violates, so it is rarely used.

Verdict on 2PC

Avoid on the hot path and across microservices. It legitimately appears inside a single database cluster (e.g. distributed SQL engines coordinating their own shards, where the coordinator and participants share fate and fast links). Across independent services owned by different teams, it couples their availability and their on-call pagers — almost never worth it.

The saga pattern: trade isolation for availability

A saga models the work as a sequence of local transactions T₁…Tₙ, each committing in its own service/database. Each Tᵢ has a compensating action Cᵢ that semantically undoes it. If step k fails, you run C_k−1…C₁ in reverse to roll the whole thing back.

HAPPY PATH T1 reserve ──▶ T2 charge ──▶ T3 ship ──▶ T4 notify ✔ done FAILURE AT T3 (ship fails) T1 reserve ──▶ T2 charge ──▶ T3 ✘ │ │ ▼ ▼ C1 release ◀── C2 refund ◀── (compensate in reverse) Net effect: semantically undone, eventually consistent.

One order through a failed saga

T1 reserve inventory — the order row is now PENDING; the customer sees "processing".
T2 charge the card — succeeds. The money has genuinely left the customer's account; they still see "processing".
T3 ship — fails: the warehouse service is down. This is the pivot from forward progress to rollback.
The orchestrator runs C2: refund the card. Note what the ledger holds now — a charge and a refund, not nothing. The money is returned by adding an entry, not by erasing one.
C1: release the reserved inventory — the stock goes back on the shelf.
The order row moves to FAILED and the customer is notified that the order could not be completed.

The order was semantically undone — but the intermediate charged state was briefly visible to the customer between steps 2 and 4. That visible intermediate state is exactly the isolation you gave up by leaving a single ACID transaction; a database would have hidden it behind COMMIT.

The crucial trade-off: a saga is not isolated. Between T₂ (charge) and C₂ (refund), the customer's money is gone and visible to other reads. There is no global rollback that hides intermediate state. You manage this with:

Semantic locks / pending states — mark the order PENDING so other transactions know it is in flight; the UX shows "processing".
Commutative / reorderable updates — design so concurrent operations don't conflict (e.g. balance adjustments via deltas).
Compensations are themselves reliable and idempotent — a refund that runs twice must not double-refund. This is non-negotiable.

Compensations are also semantic, not byte-for-byte rollback. You cannot un-send an email; C might be "send a correction". You cannot un-charge cleanly; C is "issue a refund" — the ledger now has two entries, not zero. Some actions are irreversible (a launched rocket, a wire transfer to an external bank); order your saga so irreversible steps come last, after everything reversible has already succeeded (the "pivot transaction").

Orchestration vs choreography

	Orchestration	Choreography
Control	Central orchestrator drives each step and compensation	Each service reacts to events, emits the next event
Coupling	Orchestrator knows all services	Services know only events, fully decoupled
Observability	Flow is explicit, one place to read, easy to trace	Flow is implicit — emergent from event subscriptions
Failure mode	Orchestrator is a brain (and a SPOF to harden)	Hard to follow / debug cyclic dependencies at scale
Best for	Complex flows, > ~4 steps, strong audit needs	Simple 2–3 step flows, max decoupling

Default to orchestration for anything non-trivial: the saga's state machine lives in one observable place and compensation logic is explicit. Choreography is elegant for small flows but becomes a distributed "what happens next?" mystery as steps multiply.

The dual-write problem and the transactional outbox

A specific and extremely common partial-failure case (it connects directly to 09): a service must "update its database and publish an event." These are two systems. Whichever order you pick, a crash in the middle is corrupting:

Write DB, then publish — crash before publish → event lost, downstream never learns.
Publish, then write DB — crash before write → orphan event for a state that doesn't exist.

The fix is the transactional outbox: in one local DB transaction, write the business row and an outbox-event row. They commit atomically because they're in the same database. A separate relay (a poller, or change-data-capture tailing the DB's replication log — the very write-ahead log from 04 that also drives the replicas in 08) reads unsent outbox rows and publishes them to the broker, marking each sent.

┌──────────── ONE local DB transaction ────────────┐ │ INSERT orders(...) -- business row │ │ INSERT outbox(event, payload) -- event row │ │ COMMIT ── atomic, both or neither │ └────────────────────────────────────────────────────┘ │ outbox table ▼ ┌─────────────┐ reads unsent rows ┌────────┐ │ relay / CDC │ ─────────────────────▶ │ broker │ └─────────────┘ marks sent / deletes └────────┘

Concretely, the outbox table is just an ordinary table living beside your business tables:

TABLE outbox ┌──────────────┬───────────────────────────────────────────────┐ │ id │ BIGINT PK, auto-increment (preserves order) │ │ aggregate_id │ the business entity this event is about │ │ event_type │ e.g. 'OrderPlaced' │ │ payload │ JSON/BLOB — the event body the broker carries │ │ created_at │ TIMESTAMP │ │ published │ BOOLEAN DEFAULT false │ └──────────────┴───────────────────────────────────────────────┘ relay loop: SELECT ... FROM outbox WHERE published = false ORDER BY id → publish each to the broker → UPDATE outbox SET published = true WHERE id = ? (or skip the polling and let a CDC relay tail the WAL instead)

The relay publishes at-least-once — it may publish a row, crash before marking it sent, and republish on restart. That is fine because consumers are idempotent (next section). The dual on the consumer side is the inbox pattern: record each processed message id and skip duplicates, again in one local transaction with the business effect.

Idempotency: the workhorse that makes all of it safe

Retries are not exceptional — they are guaranteed. At-least-once delivery (11), client retries on timeout, relay republishing, and failover (13) all will redeliver. So the foundational rule is: every side-effecting operation must be safe to apply more than once.

Naturally idempotent	Not idempotent (needs wrapping)
`SET balance = 100` (absolute value)	`balance += 10` (increment)
`PUT /user/42 {…}`	`POST /charge` (charge card)
Set-membership: `add x to set`	Append to log / list
Delete by id (idempotent: gone is gone)	"Send email", "ship item"

For non-idempotent operations, wrap them with an idempotency key: the client generates a unique key per logical request (e.g. a UUID), sends it on every retry, and the server records processed keys in a dedup store along with the result. On a duplicate key, the server returns the stored result without re-applying the effect. Over HTTP this has a real, widely-copied convention — the Idempotency-Key request header, popularised by Stripe's payments API — which we cover from the API side in 03; here we are building the server-side dedup machinery that header relies on.

Idempotency key lifecycle

Receive request with key K. Look up K in dedup store.
If present and completed → return stored response, do nothing else.
If present and in-progress → another attempt is running; wait or return 409.
If absent → insert K as in-progress (atomic, e.g. unique constraint), do the effect and store the result in the same local transaction, mark completed.

Step 4's "effect + key in one transaction" is exactly the outbox/inbox idea again — that is what makes the dedup itself crash-safe.

This is the answer to the interview question "how do you make at-least-once delivery correct?" You do not chase exactly-once delivery (impossible in general); you achieve exactly-once effect via at-least-once delivery + idempotent processing. Set the dedup store's TTL to cover the maximum realistic retry window (minutes to a few days), not forever.

Worked numbers: why long synchronous chains are fragile

Suppose each step (or 2PC participant) is independently up with probability p = 0.99. The whole flow succeeds only if all n steps succeed: P(success) = pⁿ (the series-availability law from 07/11).

n = 2 → 0.99² = 0.9801 → ~2.0% fail
n = 5 → 0.99⁵ = 0.9510 → ~4.9% fail
n = 8 → 0.99⁸ = 0.9227 → ~7.7% fail

At p = 0.95 it is far worse: 0.95⁸ ≈ 0.663 — a third of flows fail. Each added participant multiplies failure in. For 2PC every failure means held locks and a blocking window; for a saga every failure means running compensations — on average, if failure strikes uniformly, roughly (n−1)/2 already-completed steps to undo. The lesson: shorten the chain, prefer single-shard transactions by co-locating data (07), and make the steps you keep idempotent so retries are cheap rather than catastrophic.

Reliability calculator: 2PC vs saga

Move the sliders. Watch P(whole flow succeeds) = pⁿ collapse as you add participants, and see why a saga's "retry + compensate" posture survives where a synchronous 2PC chain locks up.

Distributed-flow reliability

P(success) = pⁿ — each of n independent steps must succeed (series law). On a saga failure you must run the steps already completed as compensations; on a 2PC failure those steps sit blocked holding locks. Higher n and lower p both punish synchronous atomicity hardest.

steps / participants n: 5 per-step success p: 0.990

P(flow succeeds) = pⁿ

—

P(some step fails)

—

E[compensations | failure]

—

2PC blocking exposure

—

Read-out

—

Choosing the mechanism

A decision order that survives interview scrutiny:

Can the data be co-located on one shard? Pick the shard key (07) so the unit of work is a single local ACID transaction. Cheapest, strongest, no distributed problem at all.
If it must span systems, can it be eventually consistent? Almost always yes for business flows → saga (orchestrated) + transactional outbox + idempotency keys. The UX shows "pending"; correctness is reached via compensations and retries.
Does money/inventory truly demand synchronous atomicity? Only then consider 2PC, and confine it inside a single DB cluster with fast shared-fate links — never across team-owned services.

Interview prompts you should be ready for

Why can't you "just write to the DB and publish to Kafka" in one transaction? (senior answer) They are two independent systems with no shared commit; a crash between them either loses the event or orphans it. Use the transactional outbox — write the business row and an outbox row in one local DB transaction, then a relay/CDC publishes at-least-once.
What exactly is the blocking problem in 2PC? (senior answer) After voting YES a participant holds locks and cannot unilaterally decide; if the coordinator dies before sending the verdict, the participant is stuck "in doubt" holding locks indefinitely, blocking other work. It's a liveness failure, not just a latency one — and why 2PC is unfit across services.
How is a saga different from a transaction? (senior answer) A saga gives atomicity (via compensations) but sacrifices isolation — intermediate states are visible. There is no global rollback; Cᵢ semantically undoes Tᵢ. You manage exposure with pending states/semantic locks and order irreversible steps last.
Orchestration or choreography, and why? (senior answer) Orchestration for non-trivial flows: the state machine and compensation logic live in one observable place, easy to trace and audit. Choreography maximizes decoupling for 2–3 step flows but the overall flow becomes implicit and hard to debug as it grows.
How do you make at-least-once delivery correct? (senior answer) You don't pursue exactly-once delivery; you get exactly-once effect via idempotent processing. Client-supplied idempotency key + a dedup store; record the key and the result in the same local transaction as the effect, and return the stored result on duplicates.
Which operations are naturally idempotent, and how do you fix the ones that aren't? (senior answer) Absolute SETs, PUTs, set-adds, delete-by-id are naturally idempotent; increments, charges, appends, sends are not. Wrap the latter with an idempotency key + dedup store, or redesign as absolute/commutative updates.
Your saga's compensation itself fails — now what? (senior answer) Compensations must be retriable and idempotent; the orchestrator persists saga state and retries with backoff until success, escalating to a dead-letter/alert after a bound. Compensations should never have a path that requires a further compensation — keep them simple and reversible-by-construction.
When is 2PC actually acceptable? (senior answer) Inside a single database/cluster that owns its participants on fast shared-fate links — e.g. a distributed SQL engine coordinating its own shards. Across independently-owned microservices it couples their availability (A = ∏ aᵢ) and on-call, and almost never earns its cost.

Takeaway

Atomicity is free on one database and expensive across two. Reach for the weakest mechanism that meets the need: co-locate data for a single-shard transaction first; otherwise use an orchestrated saga with a transactional outbox and idempotency keys, accepting eventual consistency as a deliberate product decision ("pending"). Reserve 2PC for inside a single cluster where shared fate makes its blocking, lock-heavy, series-availability cost tolerable. Above all: assume every message is delivered more than once, and make every side effect idempotent — that single discipline is what makes at-least-once delivery, retries, and failover correct instead of corrupting.