Avoid double charges

The network can lose your response but not your side effect. The client retries; the charge already happened. Idempotency is the machinery that makes "do this once" survive a world where you can never be sure whether the last attempt landed.

The retry storm, with numbers

Make the failure concrete before designing anything. The client's HTTP timeout is 3 s. The payment provider (PSP) actually settles the authorization at 4 s — slow, but successful. Here is the window that mints duplicate charges:

The window is open for t_settle − t_timeout = 4 − 3 = 1 s on every request whose true latency exceeds the client's patience. At scale this is not a corner case: if 0.5% of 2,000 payments/s straddle the timeout, that is 2000 · 0.005 = 10 double-charge attempts per second — 864k/day of liability if unguarded. The fix is not "make the PSP faster" (you don't control it) — it is to make the second call recognize itself as a repeat and short-circuit. That recognition needs a stable name for the operation that is identical across retries: the idempotency key.

Clarify the contract

Treat the prompt as a product contract before a box diagram. The system must:

• Make a retry with the same key a no-op that returns the original response — not a second charge.
• Guarantee one business intent → at most one successful charge, even under concurrent retries on different servers.
• Dedup the PSP's webhooks — the provider also retries its callbacks at-least-once.
• Expose a clean state machine: pending → succeeded | failed → refunded.
• Support audit / dispute investigation for the full chargeback window.

And explicitly what it need not do: it need not guarantee the charge happens exactly once at the PSP on the first try — that is impossible across a network (DDIA Ch. 9). It guarantees the observable effect is exactly-once: at-least-once delivery + dedup = effectively-once. We make the operation safe to repeat, then repeat freely.

Sizing: the key TTL is the whole storage problem

The only real sizing question is how long you keep idempotency records, because everything downstream multiplies off it. The naive instinct — "expire keys after an hour" — is a bug. A retry can arrive late; worse, a customer can dispute a charge months later, and support must replay exactly what happened. So:

TTL = max( PSP settlement window , card dispute / chargeback window ) ≈ 90–180 days

Card-network chargeback windows run 120 days (Visa/Mastercard) up to 180. Take 180 days. Now storage is mechanical: at 2,000 payments/s and a record of ~1 KB (key, request hash, status, serialized response, timestamps):

31 TB of dedup metadata is the price of correctness — and the trade-off table's first row exists precisely because someone always proposes a 1-hour TTL to shrink it. That choice trades 31 TB for the inability to honor a 90-day dispute, which costs far more in fraud and support than disk. The TTL is set by the business window, not the engineer's comfort. (Hot path stays in a fast store keyed for lookup; cold records age to cheaper storage but must remain queryable for the dispute window.)

Data model & API

The data model is the architecture here. The key insight: the idempotency record and the payment intent are two different things — the key dedups requests; the intent tracks the money.

Linearized design

Follow one POST /pay through the system, then watch the duplicate try collapse on it.

1. 1. Fingerprint. Compute request_hash = H(method, path, body). The client supplies Idempotency-Key; the hash binds the key to this exact request.
2. 2. Insert-if-absent. Attempt INSERT INTO idempotency_key(key, request_hash, status='in_progress'). The UNIQUE constraint makes exactly one of N concurrent attempts win.
3. 3a. Insert succeeds → first-ever request. Create the payment_intent (pending) and an outbox row in the same transaction. Commit. A separate worker drains the outbox and calls the PSP.
4. 3b. Insert fails (key exists) → a repeat. Read the existing row. If status=completed, return the stored response_blob verbatim. If in_progress, return 409 / "retry shortly" (the first attempt is still running). If the stored request_hash ≠ this one, return 422 — the client reused a key with a different body.
5. 4. PSP result lands (via outbox worker response or webhook). Transition the intent pending → succeeded, store the PSP response in response_blob, flip the key to completed.
6. 5. Webhook dedup. The PSP also calls /provider/webhook, possibly twice. INSERT psp_event_id; on conflict, ack and drop. Same mechanic, different key.

The bottleneck this exposes: step 2's insert is the single point every request funnels through, and it must be atomic. Get that wrong and the whole edifice leaks duplicates — which is the first deep dive.

Deep dives

1. The simultaneous-miss race — why read-then-write is wrong

The trap implementation: read the key, if absent then write it. Now the sharpest senior scenario: two requests with the same key arrive simultaneously on two different servers, and both miss the cache. Server A reads "absent," Server B reads "absent," both proceed to charge the PSP, both write the key. Two charges. This is DDIA Ch. 7's classic write-skew / lost-update — a read-modify-write with a gap between read and write under concurrency.

The fix is to never read-then-write. Make the existence check and the claim a single atomic operation at the place the data lives:

• Relational: a UNIQUE constraint on key plus INSERT … ON CONFLICT DO NOTHING. The database serializes the two inserts; exactly one returns "inserted," the loser sees the conflict and falls into the read-existing branch.
• Redis: SET key value NX EX ttl — set-if-not-exists in one round trip. NX is the atomicity.

The loser must not charge — it polls the winner's row (the in_progress state) until it flips to completed, then returns that response. The principle generalizes well beyond payments and is the same one C35's neighbors lean on: a distributed dedup is only correct if the check-and-claim is atomic where the data lives. This is also why the idempotency key is not "client politeness" — it is the only stable name two independent servers can agree on without talking to each other.

2. Client-generated key vs. server-issued intent — and the mismatch trap

Who mints the key? Two models:

• Client-generated (Stripe-style): client sends a UUID in Idempotency-Key. Simple, one round trip — but the client can misuse it.
• Server-issued intent: client first POST /payment_intent to get an intent_id, then confirms it. Two round trips, but the server controls the namespace.

The client-generated model has a dangerous failure: a buggy client reuses one key for two genuinely different payments — same key, $10 then $1000. A naive server sees the key, returns the first result, and silently swallows the $1000 charge (or, worse, applies the $10 response to it). This is why step 1 stores the request_hash. On a repeat, compare hashes: equal → it is a true retry, return the cached response; unequal → reject with 422, because "same key, different request" is definitionally a client bug, and guessing which one they meant is how you double-charge or lose a charge. The hash turns the key from a blind cache lookup into a verified identity claim.

3. The outbox — why you don't call the PSP inside the transaction

Step 3a created the intent and an outbox row in one transaction, then a worker calls the PSP. Why not just call the PSP directly while the row is being written? Because the PSP call is a side effect outside the database's transaction, and you get the dual-write problem: if you charge first then the DB commit fails, you have a charge with no record (and the retry double-charges); if you commit first then crash before charging, you have a record with no charge. There is no ordering that is safe, because two systems can't share one atomic commit.

The transactional outbox (DDIA Ch. 11 — the outbox as log-based change capture; Ch. 9's exactly-once = at-least-once + dedup) breaks the deadlock: the intent and the "please charge" message are written to the same database in one transaction, so they commit or roll back together. A separate worker reads unsent outbox rows and calls the PSP — at-least-once (it may retry after a crash), which is exactly why the PSP call also carries an idempotency key so the PSP itself dedups. The DB is the single source of truth; the outbox is the durable, replayable intent-to-act. This is the messaging-side counterpart to C27's delivery-semantics: C27 owns the concept (why exactly-once is at-least-once + dedup); this case owns the concrete request/payment key implementation.

Trade-offs

The sharpest one is the last: read-then-write looks correct in every single-threaded test and leaks duplicates only under the exact concurrency the retry storm guarantees. It is the difference between code that passes review and code that survives production — and the entire reason the UNIQUE constraint, not application logic, owns the dedup.

Failure modes

Interview Q&A

Related lessons

This case is the anchor for idempotency-key and outbox mechanics — insert-if-absent, key TTL, request-hash. For the concept of delivery semantics and why effectively-once is at-least-once + dedup, see C27 (delivery semantics); it owns the messaging concept, this case owns the request/payment-key implementation. The double-entry side of money movement — what the charge debits and credits — lives in C33 (double-entry ledger); C32 (orchestration) and C34 (reconciliation) both link here for the dedup key. Foundation: lesson 12 (idempotency, outbox, sagas) for the primitives, and lesson 13 for the backoff-with-jitter that keeps the retry storm from becoming a thundering herd.