Microsecond-latency exchange architecture
When the budget is single-digit microseconds, you do not "add" a fast path — you subtract until nothing slow is left. Architecture becomes the disciplined removal of every hop, copy, allocation, lock, and syscall between the wire and the match.
1. The hinge — latency is a budget you spend by subtraction
The discipline this enforces is the one foundation lesson 14 calls optimisation-by-removal: you do not speed up a hop, you delete the hop. The cost of an operation in this regime is dominated by things that are invisible in normal service design — a kernel crossing, a cache line that lives on another core, a heap allocation that might trip the garbage collector. The architecture is whatever survives after you have subtracted all of those. That is why the right mental model for this page is a waterfall: start with the wall-clock you are allowed, subtract each unavoidable cost, and what remains is the slack you must protect.
2. Clarify the contract
Treat the prompt as a product contract before drawing boxes. The matching engine must:
- Minimise wire-to-wire order-entry latency — the time from the order's last bit arriving on the NIC to the acknowledgement's first bit leaving it.
- Match deterministically — identical input sequence ⇒ identical output, so the book can be replayed and audited (the C39 / DDIA Ch. 11 property).
- Have no unpredictable pauses — no GC, no page fault, no lock convoy on the hot path. This is the real spec; "fast" is downstream of it.
- Be durable for accepted orders — an acknowledged order must survive a process crash, without the journal sitting synchronously in the budget.
- Publish market data fast and in order.
What it explicitly need not do: be a general-purpose service, scale horizontally per book (one book = one writer — see §6), tolerate dynamic allocation in steady state, or use a rich RPC stack. Those are the very things we will subtract.
3. Put numbers on it — the latency budget table
This is the load-bearing calculation of the whole case. Take a target of <10 µs wire-to-wire for order entry (a credible "fast but not exotic" target; the lowest tier of houses push under 1 µs with FPGAs, which we treat as the extreme below). Now write down what each unavoidable mechanism costs, and subtract.
| Mechanism on the path | Typical cost | Verdict / what you do about it |
|---|---|---|
| NIC → kernel network stack → userspace (normal sockets) | ~10 µs each way | Catastrophic — it alone blows the whole budget. Subtract it: kernel bypass. |
| Kernel bypass (DPDK / Solarflare / Onload, NIC DMA straight to userspace) | ~1 µs each way | The replacement for the row above. This is the single largest saving you make. |
| Cross-core / cross-NUMA cache miss (data owned by another core's cache) | ~100 ns (≈0.1 µs) | Per hop. Pin the book to one core; keep its hot state in that core's L1/L2 (single-writer, §6). |
| Lock contention (uncontended CAS is ~tens of ns; a contended lock that parks) | hundreds of ns → µs+ if it sleeps | Subtract entirely: no locks on the hot path. One writer per book means nothing to contend. |
| Main-memory access (last-level cache miss to DRAM) | ~100 ns | Preallocate and keep the working set resident; never fault in pages on the hot path. |
| GC pause (managed runtime stop-the-world) | ~milliseconds | 1000× over budget. Not a tax — a cliff. Either no allocation on the path, or no managed runtime (§6). |
Now sum the budget by subtraction from the 10 µs target, one direction (inbound):
10 µs target − 1 µs (NIC bypass in) − ~0.5 µs (parse binary msg, risk check) − ~0.2 µs (sequencer) − ~0.3 µs (match: a few cache-resident lookups + book update) − ~0.5 µs (encode + journal handoff) − 1 µs (NIC bypass out) ≈ 3.5 µs spent, ≈ 6.5 µs of slack.
Two things fall out of that arithmetic immediately. First, the NIC crossings dominate: 2 µs of the ~3.5 µs spent is just getting bits on and off the wire — which is why kernel bypass is non-negotiable, not an optimisation. Second, the budget has no room for a single millisecond-scale event. A lone GC pause (≈1 ms) is ~285× the entire spent budget; one page fault or one contended lock that sleeps is the same story. The table does not say "GC is slow," it says one GC pause = ~285 orders' worth of budget vaporised in one go. That is the number that forces every later decision.
4. Data model & API
The data model is the latency design. Every hot-path object is preallocated at startup so steady-state allocation is zero, and laid out for cache locality. (The book structure itself — price levels, FIFO queues per level — belongs to C39; here we care only that it lives entirely in one core's cache.)
| API / operation | Why it exists / latency note |
|---|---|
NewOrder (binary, fixed-layout) | Compact binary over a persistent session — no JSON, no per-message connection setup, parseable without allocation. |
Cancel / Replace | Same fast path; cancels are often the latency-sensitive ones in market-making. |
MarketData feed | Published from the same ordered event stream that drove matching — one source of truth, sequence-numbered. |
Halt (admin) | Control plane; deliberately off the hot path. |
5. Linearized design — follow one order, and the colocated layout
Walk the order in the order it actually moves. The whole pipeline lives in a single colocated rack (clients colocate too — physical distance is light-speed latency you cannot subtract), and the matching stage is pinned to one core per book.
- 1. NIC, kernel-bypassed. The NIC DMAs the order frame straight into a userspace ring buffer (DPDK/Solarflare). No kernel, no copy, no context switch. ~1 µs.
- 2. Gateway parse + risk. A busy-spinning thread reads the ring, parses the fixed-layout binary message in place (no allocation), and runs pre-trade risk (credit, size limits) against preallocated state. ~0.5 µs.
- 3. Sequencer. The order is stamped with a monotonic sequence number — the total order that makes deterministic replay possible (DDIA Ch. 11). This, not the wall clock, is "truth."
- 4. Matching core (single writer). One thread, pinned to one core, owns this book's entire state in its L1/L2. It matches against the resident book, mutating preallocated objects. No lock (nobody else writes), no allocation, no cache miss to another core. ~0.3 µs.
- 5. Journal handoff. The sequenced event is handed (lock-free, via a ring/queue) to a journal writer that appends it sequentially. The ack does not wait on fsync (see §6.3); durability runs alongside, not inside, the budget.
- 6. Encode + NIC out. The ack is encoded into a preallocated buffer and DMA'd back out through the bypassed NIC. Market data is published off the same ordered stream. ~1.5 µs.
6. Deep dives
6.1 Architecture as subtraction
Re-read the waterfall as a list of things that used to be there. A normal service would have, on this path: a kernel socket (subtracted → DPDK), a load balancer hop (subtracted → clients connect directly to the colocated gateway), JSON/Protobuf parsing with allocation (subtracted → fixed-layout binary parsed in place), an RPC call to a separate matching service (subtracted → matching is in-process, one core away via a ring), a lock around the shared book (subtracted → single writer, nothing to lock), a heap allocation per order (subtracted → preallocated object pool), and a synchronous database write (subtracted → async sequential journal, §6.3). Each subtraction removes both a chunk of mean latency and, more importantly, a source of variance. The finished design is not "a service with optimisations bolted on"; it is the residue left after you delete everything that can stall. This is exactly the removal-not-addition discipline of lesson 14 taken to its limit.
6.2 Managed runtime vs manual memory; spin vs interrupt
The trade-off table lists "managed runtime vs manual memory" with pause risk as the cost and matching core as the place it bites — and §3 tells you why: one stop-the-world GC pause (~1 ms) is ~285× the spent budget. You have two honest ways out. (a) Use a non-managed language (C++/Rust) on the matching core so there is no collector at all. (b) Keep a managed runtime (Java is common — LMAX Disruptor is the canonical example) but engineer zero steady-state allocation: preallocate every hot object at startup, reuse from pools, and the GC has nothing to collect, so it never runs on the hot path. Both reach the same place — no allocation in steady state — which is the real invariant; the language is secondary.
The second mechanical-sympathy choice is busy-spin vs interrupt. A normal thread blocks on a socket and is woken by an interrupt — but the wakeup itself costs microseconds and is jittery (it depends on the scheduler). So the hot threads busy-spin: a pinned thread polls its ring buffer in a tight loop, burning a whole CPU core to stay warm and never pay a wakeup. You trade power and a dedicated core for the elimination of scheduler-induced variance. That is the same trade everywhere on this path: spend a fixed, predictable resource (a core, RAM for preallocation) to delete an unpredictable cost (a wakeup, an allocation, a fault).
6.3 The p99.99 tail is the real enemy
This is the point foundation lesson 02 and DDIA Ch. 1 hammer, and it is the heart of this case. DDIA's own framing: users do not experience your mean or even your p99 — they experience your tail, and in a system that fires millions of orders, p99.99 (the worst one in 10,000) happens constantly. A p50 of 5 µs alongside a p99.99 of 2 ms is not "fast with a rare blip"; it is a slow system that is occasionally fast, because the 2 ms outliers are the trades that get picked off. So the engineering target is not "lower the mean" — the mean is easy — it is "collapse the distance between p50 and p99.99." Every subtraction in §6.1 is really a tail-killer: allocation, GC, page faults, lock contention, cross-NUMA misses, and syscalls each produce a small fraction of slow events that never show in the mean but are the tail. You attack the tail by removing its causes one by one, then proving it with histograms (HdrHistogram-style, full distribution — never averages), per lesson 16.
6.4 Determinism, sequence numbers, and recovery
Low latency does not excuse you from durability — it just dictates how. The single-writer matching core processing one ordered input stream is precisely DDIA Ch. 11's deterministic stream processing: because there is exactly one writer and a total order, the same event sequence always yields the same book, so the journal is a replayable source of truth. On crash you replay the journal to reconstruct the book — no distributed consensus needed on the hot path. The sequence number, not the clock, is truth: DDIA Ch. 8 warns that physical clocks across machines are unreliable and non-monotonic, so you never order trades by timestamp; the sequencer's monotonic counter is the canonical order, and timestamps are advisory metadata only. The journal write is kept off the budget by handing the event to an async writer over a lock-free ring — but you can choose to make the ack wait for the journal (sync) when the regulatory/risk requirement outweighs the latency (the §7 sync-vs-async row).
7. Trade-offs
| Choice | Buys | Costs | Choose when |
|---|---|---|---|
| Kernel bypass vs normal sockets | ~9 µs saved per crossing; removes scheduler jitter | operational complexity, special NICs, busy-spin burns cores | always, in this regime — it is the single biggest line item |
| Busy-spin vs interrupt-driven | no wakeup cost, predictable latency | a dedicated core at 100% + power | hot threads (gateway, matching) — not the control plane |
| Managed runtime vs manual memory | developer speed, safety | GC pause risk (~1 ms cliff) | managed is fine only with zero-allocation discipline; else C++/Rust on the matching core |
| Sync journal vs async journal | sync: order is durable before ack | sync: fsync (ms-scale) lands inside the budget | async by default; sync only when regulation/risk demands ack-after-durable |
The sharpest of these is the journal one, because it is where the latency story and the durability story collide. An fsync to even an NVMe device is tens to hundreds of microseconds — orders of magnitude over the whole order-entry budget — so putting it inside the ack path destroys the latency you spent the rest of the design earning. The standard resolution is to make durability concurrent: hand the sequenced event to a journal writer over a lock-free ring and let the ack return immediately, accepting a tiny window where an acked order could be lost on a hard crash before it lands. When that window is unacceptable (and for an exchange it often is), you do not synchronously fsync per order — you batch the journal (the §6.3 batching/throughput row) or replicate the sequenced stream to a hot standby in the same rack, so "durable" means "received by the replica," which is far cheaper than a disk fsync. The latency is preserved by making durability a parallel concern, never a serial one.
8. Failure modes
| Failure | Mitigation |
|---|---|
| Latency outlier (the tail) | Profile the full distribution at p99.99 (histograms, not means); hunt and remove the cause — allocation, GC, page fault, lock contention, cross-NUMA miss, or syscall (§6.3). |
| GC pause on the matching core | Zero steady-state allocation (preallocated pools) or a non-managed runtime; pre-touch and lock pages so nothing faults in. |
| Clock skew between machines | Never order by wall clock (DDIA Ch. 8). The sequencer's monotonic seq# is truth; timestamps are advisory only. |
| Dropped market-data packet | Sequence-number every message so gaps are detectable; provide a separate recovery/snapshot channel to refill. |
| Hot symbol overload | Partition books across cores, but keep exactly one writer per book — never split a single book across cores (that reintroduces locks and cross-core misses). |
| Matching process crash | Replay the sequential journal to rebuild the in-memory book deterministically (DDIA Ch. 11), or fail over to a hot standby replaying the same ordered stream. |
9. Interview Q&A
- Your p50 is 5 µs but your p99.99 is 2 ms — where does that tail come from and how do you kill it? (senior answer) A 2 ms tail next to a 5 µs median is a millisecond-scale event, not gradual slowness — so look for the handful of causes that produce ms-scale stalls: a GC pause, a major page fault (memory not resident), a lock that parked a thread, a cross-NUMA access, or a syscall/scheduler wakeup. I'd capture the full histogram and correlate the slow samples with GC logs, page-fault counters, and core/NUMA placement. The kills, in order: eliminate steady-state allocation (or drop the managed runtime) to remove GC; pre-touch and mlock pages so nothing faults; busy-spin pinned threads so there's no scheduler wakeup; single-writer-per-core so there's no lock and no cross-core miss. The tail isn't "the system being slow" — it's a specific event class, and you remove the classes one at a time.
- Why is kernel bypass non-negotiable here? (senior answer) Normal NIC→kernel→userspace is ~10 µs each way, which alone exceeds a sub-10 µs wire-to-wire budget — and it's jittery because of context switches and scheduling. DPDK/Solarflare DMAs frames straight into a userspace ring (~1 µs), removing both the mean cost and a major variance source. It's the single biggest line item on the waterfall, so it's a starting assumption, not an optimisation.
- Why single-writer per book instead of a thread pool with a lock? (senior answer) A lock on the book reintroduces exactly what we're subtracting: contention (hundreds of ns, µs+ if it sleeps) and, worse, unpredictable tail latency under contention. One thread pinned to one core owns the whole book in its L1/L2 — no lock to take, no cross-core cache miss, and as a bonus the processing is deterministic, which is what makes journal replay correct (DDIA Ch. 11). You scale by partitioning different books to different cores, never by sharing one book.
- Can you use Java/managed languages at all? (senior answer) Yes — LMAX runs the Disruptor pattern in Java in production. The trick is zero steady-state allocation: preallocate every hot object at startup and reuse from pools, so the GC never has work to do on the hot path. If you can't guarantee that discipline, use C++/Rust on the matching core. The invariant is "no allocation in steady state," not "no managed runtime."
- How do you stay durable without putting an fsync in the budget? (senior answer) Hand the sequenced event to an async journal writer over a lock-free ring so the ack doesn't wait on disk; or, when an order must be durable before ack, replicate the sequenced stream to a hot standby in the same rack and define "durable = received by replica," which is far cheaper than a disk fsync, and batch the disk journal behind that. Durability runs parallel to the hot path, never serial to it.
- Why busy-spin and burn a whole core? (senior answer) Blocking-and-interrupt wakeups cost microseconds and are jittery — they depend on the scheduler, which is a tail-latency source. Spinning on the ring keeps the thread hot and removes the wakeup entirely. You're trading a fixed, predictable resource (a dedicated core + power) for the removal of an unpredictable cost. In this regime predictability is worth more than the core.
- Why sequence numbers and not timestamps for ordering? (senior answer) Physical clocks across machines drift and aren't monotonic (DDIA Ch. 8), so ordering trades by timestamp is incorrect and unauditable. The sequencer assigns a single monotonic position to every event — that total order is the truth, it's what makes deterministic replay and fair price-time priority possible, and timestamps become advisory metadata only.
- Where does this case stop and C37/C39 begin? (senior answer) C37 is the exchange overview (sessions, gateways, the book as a concept); C39 is the matching kernel's algorithm (price-time priority, deterministic replay mechanics). This case (C38) is specifically the latency budget and the mechanical engineering — bypass, pinning, no-alloc, spin, async journal — that lets the C39 kernel run inside the C37 system at microsecond scale. They're one arc, not three overlapping designs.
Related lessons
This page is the latency satellite of the exchange arc: C37 is the system overview/anchor (read it first for the order book and session model), and C39 owns the matching kernel and deterministic replay (the algorithm that runs inside this budget). It leans hardest on foundation lesson 02 for the percentile/tail framing and lesson 14 for the optimisation-by-removal discipline, with lesson 16 for measuring the tail honestly with histograms.