Prefill/decode disaggregation

Two phases with opposite bottlenecks share one GPU. The fix: don't. Run them on different hardware pools and pay one KV transfer for the privilege.

The asymmetry, one more time

Lesson 01 established it; here it drives the architecture. The same model, the same kernels, two phases:

One run, opposite bottlenecks. That's the architectural smell. The two phases want different hardware, and forcing them to share a stream costs you on both ends.

The co-located failure mode

Default vLLM does both phases on the same GPU, same stream. A long prefill blocks the queue for as long as it takes — hundreds of milliseconds for a 4k-token prompt — and every decode in flight waits.

CO-LOCATED, ONE GPU, ONE STREAM

t=0     t=80    t=120    t=320         t=400
├─decode─┤├arr.─┤├──── prefill ────────┤├─decode─
   r0,r1     r2       r2 (4000 tok)        r0,r1,r2
                ↑                       ↑
                r0, r1 stalled here     TTFT(r2) = ~280 ms
                TTFT(r1) ← punished     TTFT(r0,r1) ← also punished

The cost lands on tail latency. p50 looks fine; p99 is awful because every request unlucky enough to arrive behind a long prefill picks up the prefill's full duration as queuing delay.

Chunked prefill (lesson 10) softens this — chunk the prompt into 512-token slices and pack each slice with one decode step in the same forward. But each chunk still steals compute from decode. The interference is reduced, not removed.

The disaggregated architecture

Run the two phases on two distinct GPU pools. Route based on phase, not on request.

                   ┌───────────────┐
                   │  router / LB  │  (short-prompts can skip pool A)
                   └───────┬───────┘
                           │
              ┌────────────┴────────────┐
              ▼                         ▼
        ┌───────────┐             ┌───────────┐
        │ PREFILL   │             │  DECODE   │
        │ pool A    │             │  pool B   │
        │           │             │           │
        │  few GPUs │             │ many GPUs │
        │ big FLOPs │  KV xfer    │ lots HBM  │
        │           │ ──────────► │           │
        │ batched   │             │ small per │
        │ long-ctx  │             │ step batch│
        │ prefill   │             │ many seqs │
        └───────────┘             └───────────┘
            ↑                          │
            └──── request arrives      └──► streams tokens to client

Key property: a long prefill on pool A does not block any decode on pool B. The two queues are independent. Tail latency on the decode side decouples from prompt-length distribution on the prefill side.

The flow per request:

1. Router admits to prefill pool A.
2. Pool A runs the prompt through the model, producing one full KV cache.
3. KV cache transfers to a GPU on pool B.
4. Pool B owns the request; runs decode steps; streams tokens to the client.

The cost: KV transfer

You've now added a network hop. The KV cache for a T-token prompt is exactly what lesson 01 quantified:

For a 7B fp16 (32 layers, 32 kv heads, d_head 128): ~500 KB/tok (or ~60 KB/tok on a GQA model with fp8 KV — see lesson 09; this 8× reduction is what makes cross-node disagg practical at scale). A 4096-token prompt is 2 GB. Transfer time depends entirely on the fabric:

This is the central tradeoff. On NVLink the transfer is free; disagg almost always wins. Over RDMA the transfer competes with the interference it's meant to save — so disagg only wins on workloads with enough long prefills.

Optimizations that make disagg practical

The systems

Interactive · bimodal-workload timeline

Below: a sequence of requests arrives on a Poisson-ish schedule, with prompts drawn from a bimodal distribution — most short (50-200 tokens), some long (2000-4000 tokens). Compare two architectures on the same workload:

• Top timeline. Co-located: one GPU, serial. Prefills (orange) block decodes (blue).
• Bottom timeline. Disaggregated: prefill pool runs to the left of the slash, decode pool to the right. The two pools are independent.

Things to try:

• Default settings, NVLink. Disagg wins on p99 TTFT cleanly — the long prompts don't poison the queue for everyone else.
• Same settings on PCIe. p99 still improves, mean E2E gets slightly worse — the transfer fee shows up.
• Drop long-prompt % to 0. Now disagg adds the transfer cost without saving any interference. Co-location is faster.
• Crank long-prompt % to 60%. p99 TTFT on the co-located timeline becomes unusable. Disagg flattens to within a factor of ~2 of the mean.

// Per-token costs (ms):
const PREFILL_PER_TOK = 0.5;            // compute-bound, batch-of-1
const DECODE_STEP_MS  = 10;             // one decode step over the active set
const XFER_PER_TOK = {                  // KV transfer bandwidth → per-token cost
  nvlink: 0.005,
  pcie:   0.10,
  rdma:   0.12,
};
// Co-located: prefill and decode share one stream, serialized.
// Disagg: prefill pool FIFO + decode pool independent loop; KV transfer adds latency
//         before the request is eligible for decode.

Takeaways