Design a payment system

A payment is not an API call — it's a long-lived state machine wrapped around money that moves through a system you don't control. The whole design exists to survive the gap between "we asked the provider to charge the card" and "we know for certain whether it happened."

The hinge: you don't own the truth

The moment that makes this problem hard is not throughput — payment QPS is small. It's that the operation crosses a trust boundary into a system with its own latency, its own retries, and its own idea of what happened. Card authorization through a provider takes on the order of 1–2 seconds end to end (network to the acquirer, the acquirer to the card network, the issuing bank's fraud and funds check, and back). During that window your request is in flight: you have committed to charging the user but you do not yet know the outcome. If the user's connection drops, or your process crashes, or the response is lost on the wire, you are left holding a record that says "maybe." The architecture is the set of mechanisms that make "maybe" safe.

Clarify the contract

Treat the prompt as a product contract before a box diagram. The system must:

• Create a payment intent (one business purchase, one amount, one currency).
• Confirm a payment — request a charge from a provider — at most once per intent no matter how many times the client retries.
• Ingest provider webhooks (authorized, captured, failed, refunded, disputed) idempotently and out of order.
• Maintain a ledger that is the source of truth for money movement.
• Reconcile internal state against the provider's settlement reports daily.

And explicitly what it need not do here: it need not be the provider — we do not touch the card network, run fraud scoring, or settle funds between banks. It need not store raw PAN (card numbers) — that is the provider's tokenization job and a PCI scope we deliberately avoid. And the two mechanisms most likely to dominate this conversation — idempotency-key plumbing and the double-entry ledger — are taught in depth elsewhere; we link to them and cover only the payment-flow delta (see Related lessons).

Put numbers on it

The numbers don't size a cluster here — they size the in-flight problem, which is the real constraint. Take a busy merchant platform at 1000 payments/s and a provider auth latency of 2 s. By Little's Law, the number of charges simultaneously waiting on the provider is:

in-flight PENDING = arrival rate × auth latency = 1000/s × 2 s = 2000 concurrent attempts

So at any instant there are ~2000 records that say "we asked, we don't know yet." That is the population your timeout/retry/reconciliation logic must manage — not a cache to be evicted but money in limbo. Two more numbers fall out. Webhook delivery is at-least-once, so empirically 1–5% of deliveries are duplicates (a provider retries when it doesn't see your 2xx fast enough): design for 1000/s × 0.03 ≈ 30 duplicate webhooks/s that must be no-ops. And the ledger appends 2 entries per payment (double-entry: one debit, one credit), so the ledger write rate is 1000/s × 2 = 2000 rows/s — append-only, never updated.

The takeaway the numbers force: throughput is trivial (a single Postgres handles 2000 append rows/s easily), but the 2000 in-flight uncertain charges and the steady drizzle of duplicate webhooks are what the design must be built around. Storage is cheap relative to financial ambiguity — keep every event forever.

Data model & API

The data model is the architecture. The central decision is separating intent (what the customer wants — one purchase) from attempt (what we tried — possibly several provider calls).

API surface, kept small so it expresses the product op and not the internals: POST /intents (create), POST /intents/:id/confirm (carries an Idempotency-Key header), POST /webhooks/provider (the provider calls us), POST /intents/:id/refund.

Linearized design

Follow one payment from tap to settled, exposing the bottleneck (the in-flight gap) where it naturally appears.

1. Create intent. Client calls POST /intents; we persist a CREATED intent with amount and the client's idempotency key.
2. Confirm (idempotent). Client calls confirm with Idempotency-Key. We INSERT-if-absent on that key (the mechanics live in C35). First time: create a payment_attempt and move the intent to PENDING. A retry with the same key returns the same in-flight result — no new attempt, no second charge.
3. Charge the provider — from a durable record. Only after the attempt row is committed do we call the provider. This ordering matters: if we crash mid-call, the durable attempt lets us recover and ask "what happened to this one?" rather than losing it.
4. Return immediately; user sees PENDING. We do not block on the 2 s auth. The user gets a "processing" state; this is the async choice from the hinge.
5. Webhook resolves the outcome. Provider posts charge.authorized / captured / failed to our webhook. We dedup on the provider event id, advance the state machine, and on success/capture append the ledger entries.
6. Reconcile. A daily job pulls the provider's settlement file and compares it against our intents + ledger, surfacing any intent we think is PENDING but the provider actually charged (or vice versa).

The self-loops are the point. Because delivery is at-least-once, the same charge.captured event will arrive more than once. Dedup on the provider event id collapses every duplicate into a no-op, so a state is only ever entered once and the ledger is only ever written once per real transition. This is DDIA Ch. 11's event sourcing: the provider events are an ordered (well, mostly — see below) log, and our intent state is a fold over that log.

Deep dives

1. Sync confirmation vs. async pending — why you almost always go async

The seductive design returns the charge result on the confirm response: the user clicks Pay, you call the provider, you hold the HTTP request open for the 1–2 s auth, and you return "succeeded." Two things kill it. First, latency coupling: lesson 02's tail dominates — if the provider's p99 auth is 5 s, your p99 checkout is 5 s, and a provider hiccup becomes a checkout outage. Second, and worse, the lost response: if the network drops the provider's reply, or your server crashes after the provider charged but before you recorded it, the synchronous model has nowhere to put "we charged but don't know it." The user retries, you charge again.

The async model decouples the two. Confirm persists a PENDING attempt and returns instantly; the user sees "processing." The real outcome arrives later by webhook, and the UX shows the final state (a push notification, a polled status, an emailed receipt). The user's latency is now bounded by your system, not the provider's tail. And because the attempt is durable before the provider call, a crash is recoverable: on restart you have a record that says PENDING and you go ask the provider (or wait for reconciliation) instead of guessing. The cost is real — you must build pending UX and handle the small window where the user has paid but hasn't seen confirmation — but it's the cost of a system that survives the 1% of charges that go wrong, and that's the whole job.

2. The ledger is the source of truth — status is not

A tempting shortcut is a mutable balance column and a status field on the intent. Don't. Balances must be derived from immutable, append-only double-entry ledger entries, not from a mutable counter. The reason is auditability: when finance asks "why does this account show $40," the answer must be a list of entries that sum to $40, each traceable to a payment, refund, or correction — not "because the column says so." Status is for workflow (drive the state machine, show the user a spinner); the ledger is for money truth. They serve different masters and must not be conflated.

The payment-flow delta is just when we append: a single ledger transaction of two entries (debit the customer's receivable, credit the merchant payable, for instance) is written exactly once, on the first-time captured transition, inside the same DB transaction that advances the state machine (DDIA Ch. 7 atomicity — both or neither). A refund appends a new compensating pair; we never edit or delete the original. The double-entry mechanics — debit/credit invariants, available vs. posted balances, snapshotting — are the subject of C33; here we only need that the append is atomic with the state transition and happens once.

3. Reconciliation against provider events

Webhooks are best-effort; reconciliation is the safety net that makes the system actually correct. Every provider gives you a daily settlement report — the authoritative list of what they charged, captured, and settled. A batch job (DDIA Ch. 10 territory) joins that report against your intents and ledger and classifies every discrepancy:

• Provider charged, we show PENDING — a webhook was lost. Advance our state and append the missing ledger entries.
• We show CAPTURED, provider has nothing — we processed a phantom or a test event. Investigate; likely a correction entry.
• Amounts differ — partial capture, currency rounding, or a fee we didn't model. Correction entry.

The philosophy: webhooks make the system fast (near-real-time updates), reconciliation makes it correct (eventually, every charge is accounted for). You need both. A design that relies on webhooks alone is wrong because at-least-once delivery still permits zero deliveries when a webhook endpoint is down for an hour — only the daily reconcile catches that.

The at-least-once dedup, concretely

Static walk-through of why the dedup gate matters. Fire N duplicate confirms (or N copies of one webhook) and watch whether the card gets charged once or N times:

The pattern is identical on both edges: an INSERT-if-absent on a stable key (the client's idempotency key inbound, the provider's event id outbound) turns "at-least-once" into "effectively-once." This is DDIA Ch. 9's recipe — you can't get exactly-once delivery, so you get at-least-once delivery plus idempotent processing, which is observably equivalent.

Trade-offs

The sharpest of these is the first, and it's worth narrating: candidates reach for sync because it demos cleanly. The senior signal is to say "I'd return PENDING and resolve async, because the provider owns the latency and the truth, and a synchronous design has nowhere to stand when the response is lost — which at 1000/s and a 2 s auth is happening to ~2000 charges at any instant." Naming the in-flight number is what separates a memorized answer from a derived one.

Failure modes

Interview prompts you should be ready for

Related lessons

This case is deliberately a satellite: two of its hardest mechanisms are anchored elsewhere, and you should link rather than re-derive. The idempotency-key and outbox mechanics — INSERT-if-absent, key TTL, request-hash to catch a reused key with a different body, transactional outbox relay — are taught in full in C35 (idempotency anchor); here we used the key only as a gate and pointed at it. The double-entry ledger — debit/credit invariants, available-vs-posted balances, balance snapshotting — is anchored in C33 (ledger anchor); we needed only that the append is atomic with the state transition. The general delivery-semantics / exactly-once-via-dedup framing is anchored in C27, which is where at-least-once + idempotent processing is derived once for the whole set. On foundations: lesson 12 (distributed transactions) gives the saga/outbox spine this case leans on, and lesson 09 (consistency) is why webhooks are eventually-consistent and reconciliation is the convergence mechanism.