Part III - Production inference

Disaggregated Serving

Lesson 09 gave you autoscaling and routing that react to the real signals of LLM serving — tokens in flight, KV-cache occupancy, queue time. But all of that assumed a single kind of worker doing the whole job: read the prompt, then emit tokens. Inside that one worker live two phases with opposite appetites. Prefill chews through the whole prompt at once and is starved for raw compute; decode dribbles out one token at a time and is starved for memory bandwidth and steady, low-jitter timing. Disaggregated serving is the decision to stop running both phases on the same GPU — to split them onto separate, specialized pools — and this lesson is about when that split raises utilization and when it just hands you a distributed-systems problem with a state-transfer boundary in the middle.

1 · Two phases, one GPU, and the interference between them

Recall the two phases of autoregressive generation. Prefill takes the entire input prompt and runs it through the model in a single forward pass, in parallel across all prompt tokens. Its by-product is the KV cache: for every layer and every attention head, the key and value vectors computed for each prompt token, stored in GPU memory so later steps do not recompute them. Prefill is compute-bound — it does a large matrix multiply over the whole prompt at once, so it saturates the GPU's arithmetic units. Decode then generates the answer one token at a time: each step reads the entire KV cache, attends over it, produces one new token, and appends that token's K and V to the cache. Decode is memory-bandwidth-bound — each step moves a lot of cached state through the chip but does comparatively little arithmetic, so it is gated by how fast the GPU can stream memory, and it is sensitive to jitter because a stall shows up directly as a stutter in the user's token stream.

In a colocated server — the normal case, where one replica does both phases — these two appetites share one GPU and one scheduler. That sharing is the problem. When a 4,000-token prompt arrives, its prefill monopolizes the compute units for a chunk of time, and the decode steps of other in-flight requests have to wait behind it. That wait is head-of-line blocking: a long, compute-heavy job at the front of the line stalls the short, latency-sensitive jobs behind it. The visible symptom is inter-token latency that spikes whenever a big prompt lands — users mid-stream see their tokens hitch even though their own request is tiny.

2 · What disaggregation actually does — and the artifact it creates

Disaggregated serving puts prefill on one pool of GPUs and decode on another. A request now flows across a boundary:

request --> [ prefill pool ]  build KV cache
                  |
                  |  KV transfer over the interconnect
                  v
            [ decode pool ]  read KV, emit tokens --> stream to user

The payoff is specialization: size and tune each pool for its bottleneck, isolate prefill bursts from decode jitter, and scale the two queues independently. The price is a brand-new distributed artifact. In a colocated server the KV cache is a local object — it is built and consumed on the same GPU and never moves. The moment you split the phases, the KV cache must be transferred: every byte of key/value state that prefill produced has to travel to whichever decode worker will continue the request. KV transfer is that copy, and its cost — in bandwidth consumed and latency added before the first token — is the tax you pay for the split. Disaggregation pays only when the utilization you reclaim is worth more than this tax.

3 · The KV-transfer test, worked out

The size of a request's KV cache is not a vibe — it is a formula. For every prompt token you store, per layer, a key and a value vector across all attention heads:

KV bytes = 2 (K,V) × layers × heads × head_dim × tokens × bytes_per_elem

Take a Llama-3-8B-class model: 32 layers, 32 heads, head_dim 128 (so hidden size 4096), serving in FP16 (2 bytes per element). A 4,000-token prompt produces:

2 × 32 × 32 × 128 × 4000 × 2 B = 2 × 32 × 4096 × 4000 × 2 B ≈ 2.1 GB of KV state for that single request.

That 2.1 GB is what has to move from a prefill GPU to a decode GPU before the first answer token can be produced. Now put it on a link:

The break-even reasoning falls straight out of those numbers. Colocating wastes some GPU time to interference; call the reclaimed time per request ΔT_save (the prefill/decode stall you avoid plus the utilization you regain from right-sized pools). Disaggregating adds the transfer time T_kv plus scheduling/coordination overhead T_sched. Disaggregation pays only when ΔT_save > T_kv + T_sched. On NVLink T_kv is ~5 ms and almost any interference saving wins; over a contended network T_kv balloons to hundreds of ms or seconds and there is no realistic ΔT_save that beats it. This is why the first design rule is placement: the prefill and decode pools must sit where the interconnect between them is fast.

// KV bytes = 2 * layers * hidden * tokens * bytes_per_elem  (heads*head_dim = hidden)
const kvBytes = 2 * 32 * 4096 * tokens * 2;          // FP16, Llama-3-8B-class
const kvGB    = kvBytes / 1e9;
const tKv_ms  = (kvGB / bwGBs) * 1000;               // transfer time over the link
const net_ms  = reclaimedSave_ms - tKv_ms;           // > 0 => disaggregation pays
// add a small fixed scheduling overhead to T_kv in practice (T_sched)

4 · The trade, and the multi-pod shape that runs it

Note the coupled-queue trap. A disaggregated system has two autoscalers — one for the prefill pool, one for the decode pool — but the two queues are not independent: every prefill completion becomes a decode arrival. If the prefill autoscaler scales up aggressively, it floods the decode queue; if decode is the bottleneck and back-pressures, prefill work piles up with nowhere to go. The two control loops can oscillate against each other. The fix is that admission control must see both queues: you do not admit a request the decode pool cannot eventually absorb, no matter how much prefill capacity is free.

The LeaderWorkerSet-style shape

Multi-pod serving — whether disaggregated, or tensor-parallel where one model is sharded across several GPUs (lesson 07) — needs the platform to treat a group of pods as one logical serving unit. The common Kubernetes shape is leader-worker: a leader pod owns the external endpoint, readiness, and coordination, while worker pods run shards or phase-specific execution. The hard part is lifecycle: every member must agree on model version, runtime config, and topology. A rollout cannot update the leader and leave workers on a mismatched artifact; a readiness check must mean the whole group is healthy, not just that the leader process is alive; and a failure policy must decide whether losing one worker drains the group, restarts it, or routes around it.

This is exactly what LeaderWorkerSet-style abstractions encode — group identity and group lifecycle — so model-parallel and disaggregated serving are not hand-stitched from unrelated Deployments that have no idea they belong together. The rule of thumb: if a request's correctness depends on several pods being in lockstep, you want a group abstraction, not N independent Deployments.

5 · Where it goes wrong, and what to check

Checkpoint exercise

Where this points next

Every failure mode above shares one root cause: a request that used to live on one GPU now crosses two pools, a transfer, and two control loops — and you cannot fix what you cannot see. A KV transfer eating the gain, two autoscalers oscillating, a request lost at the boundary, a leader-worker group skewed by a bad rollout — each is invisible until you have a trace that follows one request end to end and metrics that distinguish prefill time from decode time from transfer time. Lesson 11, Observability for LLM Serving, builds exactly that machinery: the LLM-specific metrics (TTFT, inter-token latency, KV occupancy, queue time), the distributed traces that span pools, and the signals that let you tell whether a disaggregated split is paying off or quietly costing you.

Interview prompts

• Why do prefill and decode interfere on a colocated GPU? (§1 — prefill is compute-bound and processes the whole prompt at once; a long prefill monopolizes the compute units and head-of-line-blocks the latency-sensitive, bandwidth-bound decode steps of other in-flight requests.)
• What new artifact does disaggregation create, and why is it the central cost? (§2 — the KV cache, which was a local object on one GPU, must now be transferred from a prefill GPU to a decode GPU over the interconnect; that transfer's bytes and latency are the tax the split must beat.)
• Size the KV cache for a 4,000-token prompt on a Llama-3-8B-class model and explain what it depends on. (§3 — 2 × 32 layers × 4096 hidden × 4000 tokens × 2 B ≈ 2.1 GB; it scales linearly with prompt length, layer/hidden size, and bytes-per-element, so longer prompts and higher precision cost more to move.)
• State the break-even condition for disaggregation. (§3 — it pays only when reclaimed interference ΔT_save > T_kv + T_sched; on NVLink T_kv ≈ 5 ms so almost any saving wins, but across a contended network T_kv reaches hundreds of ms or seconds and erases the gain — so placement comes first.)
• Why can two autoscalers fight, and what fixes it? (§4 — prefill and decode queues are coupled because each prefill completion is a decode arrival, so independent control loops oscillate; admission control must see both queues and refuse work the decode pool cannot absorb.)
• What does a LeaderWorkerSet-style abstraction give a disaggregated or model-parallel deployment? (§4 — group identity and group lifecycle: every pod agrees on model version, config, and topology, readiness means the whole group is healthy, and a failure policy is defined — so the unit is not hand-stitched from unrelated Deployments.)
• When should you NOT disaggregate? (§3, §4 — when bottlenecks are not actually opposite, when the pools would sit across a slow/contended link so T_kv dominates, or when the platform cannot trace a two-pool request and handle partial-request failure; start colocated and measure first.)