Part III - Production inference

Benchmarking and Runtime Tuning

Lesson 07 spread a model across several GPUs and showed how interconnect topology decides whether that split helps or hurts. You now have a serving deployment that runs — but someone files a ticket that says "the model feels slow," and you have no idea whether that means the queue is backed up, the prompts got longer, the batch is too big, or a quantization change last week quietly broke the task. This lesson turns that vague complaint into a controlled experiment. The core discipline is treating a benchmark as a contract: a number like "1,800 tokens/s" means nothing until you also state the prompt length, output length, concurrency, sampling settings, and hardware it was measured on. We will pin down that contract, walk the coupled runtime knobs that move the numbers, and prove with arithmetic why a tidy synthetic benchmark can lie about production by a factor of sixty.

Source coverage

PDF Chapter 4: model and runtime tuning, language-model evaluation, benchmark, vLLM tuning, and startup time.

Linear position

Prerequisite: Lesson 07 (Multi-GPU inference and topology) — you can place a model on one or many GPUs and reason about where its memory and bandwidth go. Lessons 04-06 gave you the serving runtime (vLLM, PagedAttention, the KV cache) and the latency vocabulary.
New capability: A repeatable method for measuring serving performance under a stated workload, and for tuning the runtime knobs as a coupled set while watching latency, throughput, and task quality together — so you optimize the thing the user feels, not a benchmark artifact.

The plan

Five moves. (1) Define the workload contract — the parameters without which a throughput number is decoration. (2) Re-anchor TTFT vs TPOT and the roofline split: prefill is compute-bound, decode is memory-bandwidth-bound, so prefill-heavy and decode-heavy workloads must be benchmarked apart. (3) Work the 60x number — how a 64-token synthetic prompt mispredicts a 4,000-token production prompt. (4) Treat the vLLM knobs as one coupled system — max-num-batched-tokens, max-num-seqs, KV-cache dtype, quantization, prefix caching, chunked prefill — and show with the widget how chasing peak throughput blows up p99 TTFT. (5) Pair every speed change with a quality check, then the failure modes, checklist, and hand-off to autoscaling.

1 · A benchmark is a workload contract

Imagine two teams both report "we get 2,000 tokens/second on an H100." One measured it sending 64-token prompts and asking for 8 tokens out, 256 requests at once. The other measured 4,000-token prompts asking for 1,000 tokens out, 4 requests at once. These are not the same model behaving inconsistently — they are two entirely different workloads that happen to share a hardware label. Comparing their tokens/s tells you nothing. The number is meaningless until the workload is pinned down.

So the first artifact of any tuning effort is not a graph — it is a workload contract: the full set of parameters that, once fixed, make a measurement reproducible and comparable. Change any one of them and you are measuring a different thing.

Input length

Tokens per prompt (and its distribution — mean is not enough; tails drive tail latency). Sets the prefill cost.

Output length

Tokens generated per request. Sets how long decode runs and how much KV cache each request holds.

Concurrency

How many requests are in flight at once. This is the knob that trades throughput against tail latency; it must be a controlled variable, never an accident of the load tool.

Sampling settings

Greedy vs temperature/top-p, and especially whether generation stops early. Early stops shorten real output and inflate apparent tokens/s.

Hardware + topology

GPU model, count, interconnect (NVLink vs PCIe — lesson 07), and tensor-parallel degree. Identical config on a different topology is a different result.

Model + runtime version

Weights, quantization, vLLM/engine version, and the knob values below. A silent version bump is the most common reason a benchmark stops reproducing.

The mental model: a benchmark result is a function of all of these, written down together, or it is an anecdote. A throughput figure presented without its contract is the serving equivalent of a price with no currency.

2 · Two clocks: TTFT vs TPOT, and why the workload shape matters

Earlier lessons introduced the two latencies that matter to a user. We need them sharp here because they are governed by different physical resources, which is the whole reason a single benchmark cannot characterize a model.

→TTFT (time to first token) — the wait from request arrival until the first output token. It is dominated by queue time, scheduling, and prefill: the one-shot forward pass that reads the entire prompt and fills the KV cache.

→TPOT (time per output token) — the steady-state gap between successive output tokens once generation starts. It is dominated by decode: one forward pass per token, each re-reading the model weights and the growing KV cache.

Prefill is compute-bound; decode is memory-bandwidth-bound. That single sentence is the roofline intuition behind everything in this lesson. Prefill processes all prompt tokens in parallel through dense matrix multiplies, so it saturates the GPU's arithmetic units — more FLOPs, more tensor cores help. Decode generates one token at a time, so each step does very little arithmetic but must stream the full weight matrices and the KV cache out of HBM (high-bandwidth memory — the GPU's on-package RAM). Decode is gated by how fast you can move bytes, not multiply them. (The fix for that idleness — batching many decode steps together so one weight read serves many requests — is exactly why concurrency raises throughput.)

The consequence for benchmarking is direct: a prefill-heavy workload (long prompts, short answers — RAG, summarization, classification) and a decode-heavy workload (short prompts, long answers — chat, code generation) stress different ceilings. A knob that helps one can be irrelevant or harmful to the other. So you benchmark them separately, with their own contracts, and you never average them into one headline number.

Track the full vector, not one scalar

Every run must record together: TTFT (p50 and p99), TPOT, end-to-end tokens/s, GPU utilization, HBM occupancy, queue/wait time, and a task-quality score. Reporting tokens/s alone hides the three things that actually get you paged — a tail-latency blowup, a KV cache that overflowed and forced preemptions, and a quality regression that no speed metric can see.

3 · The 60x lie — when a synthetic benchmark mispredicts production

Here is the worked number that justifies the whole "contract" discipline. Prefill compute scales roughly linearly with prompt length (each token's attention also grows, but the dominant term for these sizes is the per-token matmul work). Suppose your engineering team benchmarks with neat 64-token prompts, while production traffic — a RAG assistant stuffing retrieved documents into context — averages 4,000-token prompts.

Worked number — prefill cost and the TTFT flip

The prefill work ratio is 4000 / 64 ≈ 62×. If a 64-token prefill takes ~5 ms, the 4,000-token prefill takes on the order of 5 ms × 62 ≈ 310 ms — and that lands entirely in TTFT, before the user sees anything. In the synthetic benchmark, prefill was negligible and TTFT looked great (~15 ms including queue); decode dominated the experience. In production, TTFT is now ~325 ms and prefill dominates the user-perceived latency. The synthetic run optimized decode batching and declared victory; the real bottleneck — first-token wait on long prompts — was never measured. Same model, same GPU, two contracts, opposite conclusions about what to tune.

There is a redeeming twist that the synthetic benchmark also misses, in the other direction. Production RAG prompts often share a long, identical prefix — the same system prompt, the same retrieved boilerplate. With prefix caching (the engine keeps the KV-cache blocks for a prefix it has already computed and reuses them across requests), the expensive 4,000-token prefill is paid once and skipped on every subsequent hit. A short-prompt benchmark has no shared prefix, so it can neither see the 60x penalty nor the cache that rescues it. Only a benchmark whose contract mirrors the real prompt structure — length and reuse — predicts production at all.

4 · The runtime knobs are one coupled system

vLLM-style engines expose a handful of knobs that look independent and are not. Move one and the others' best settings shift. Define them once, then see why they couple.

Knob	What it does	Improves	Can hurt
`max-num-seqs`	Cap on concurrent sequences in a batch	Throughput, GPU utilization	p99 TTFT; KV pressure → preemption
`max-num-batched-tokens`	Token budget per scheduler step (prefill + decode)	Prefill throughput, fairness	Big prefills monopolize a step, stalling decode
KV-cache dtype (fp16 → fp8)	Bytes per cached key/value	~2x more concurrency in same HBM	Small quality loss on some tasks
Weight quantization (AWQ/GPTQ/int4)	Bytes per weight	HBM footprint, sometimes decode speed	Task quality, runtime/kernel compatibility
Prefix caching	Reuse KV blocks for shared prefixes	Repeated prefill cost (huge for RAG)	Wasted memory when reuse is low
Chunked prefill	Slice long prefills, interleave with decode	Mixed-workload utilization, decode fairness	More complex latency tuning

Quantization means storing weights (or the KV cache) in fewer bits than the trained fp16 — int8, int4, fp8. It shrinks HBM use and can speed memory-bound decode, but it is lossy: the model's answers can shift. Pruning is the sibling technique — removing weights (or whole structures) deemed redundant — same trade: smaller and faster, at some quality cost. Both are bets that the task you care about tolerates the approximation, and that bet must be measured, never assumed.

Why they couple: raise max-num-seqs and each extra sequence wants KV-cache space, so you hit HBM limits sooner — unless you switched KV dtype to fp8 (frees room) or enabled prefix caching (frees room when prefixes repeat) or quantized the weights (frees room for cache). Push max-num-batched-tokens high for prefill throughput and a single 4,000-token prefill can eat an entire scheduler step, stalling every other request's decode and spiking TPOT — which is exactly what chunked prefill exists to prevent. You cannot tune these one at a time; you fix the workload contract, sweep one knob, measure the full vector, and only then move the next.

Peak throughput is a trap for p99

The seductive failure: crank max-num-seqs to maximize tokens/s. Throughput climbs because bigger batches amortize the memory-bound weight reads of decode — but every request now waits behind a fuller queue and a heavier prefill step, so p99 TTFT explodes. A config that wins the throughput headline can violate your latency SLO on the very tail of traffic that users complain about. The widget below makes this curve draggable.

Concurrency sweep — throughput vs p99 first-token latency

Drag max concurrency (the effective max-num-seqs) and watch the two curves diverge. Throughput (tokens/s) rises with batch size because decode is memory-bandwidth-bound — one weight read serves many sequences — but saturates as the GPU's compute and bandwidth fill. p99 TTFT climbs roughly with queue depth and explodes once you exceed what the hardware can keep in flight. The dashed line is your SLO. The sweet spot is the largest concurrency that still sits under it — not the peak of the throughput curve.

max concurrency: 32 p99 TTFT SLO (ms): 400

Throughput

—

p99 TTFT

—

SLO status

—

Max safe concurrency

—

Show the core model

// throughput saturates (Amdahl-ish): more batch helps, with diminishing returns.
tput  = TPUT_MAX * c / (c + K_HALF);            // tokens/s, saturating
// p99 TTFT: a floor (prefill+overhead) plus queue cost that turns superlinear
//           once concurrency exceeds what the GPU keeps in flight.
ttft  = TTFT_FLOOR + A*c + B*Math.max(0, c-C_KNEE)**2;  // ms
// pick the LARGEST c with ttft(c) <= SLO — not argmax(tput).

5 · Pair every speed number with a quality number

The single rule that turns runtime tuning from folklore into engineering: a performance experiment that does not also measure quality is incomplete. Quantization, pruning, and aggressive batching can all silently move the model's answers. The platform must be able to make a sentence like: "this fp8 KV-cache config is 20% cheaper and raises concurrency 1.9x, and it changes our task-eval score from 0.84 to 0.83 on the regression suite." Without the second clause you have not made a decision — you have made a guess that looks like a decision.

So run a fixed task evaluation (the held-out set that represents what users actually need — correctness, format adherence, safety, refusal behavior) before and after every compression or knob change, and report the delta next to the speed delta. The dangerous quality losses are rarely uniform: int4 might be fine on chat but wreck a math or code task, fp8 KV cache might be invisible on short answers but drift on long ones. The eval set is the only instrument that catches it; tokens/s never will.

The experiment loop

(1) Fix the workload contract (prefill-heavy and decode-heavy as separate contracts). (2) Establish a baseline: full metric vector + quality score. (3) Sweep one knob at a time. (4) Measure latency (p50/p99) + throughput + HBM + quality together. (5) Keep the change only if it improves the target metric and stays inside the p99 SLO and does not regress quality past your bar. Then move to the next knob, re-baselining because the knobs couple.

6 · Startup time is part of the workload too

One number the synthetic benchmark forgets entirely: how long a replica takes to become ready. A serving pod must pull a multi-gigabyte image, download or mount tens-of-GB of weights, allocate the KV cache, and warm the engine before it serves a single token — often 60-300 seconds. That latency is invisible at steady state but decisive the moment you autoscale (next lesson): if cold start is 4 minutes and a traffic spike arrives in 1, the new replicas are useless for the spike. So startup time belongs in the tuning budget — bake weights into the image or use a fast model-streaming mount, pre-allocate the cache, and measure cold-start as a first-class metric, not an afterthought.

7 · Where it breaks and what to check

Failure modes

Synthetic short-prompt benchmark predicts production badly. Real prompts are long and often share a cacheable prefix; a 64-token benchmark misses both the 60x prefill penalty and the prefix-cache rescue. Signal: p99 TTFT in production is many times the benchmark's, and it tracks prompt length, not concurrency.
Quantization improves cost but silently degrades the task. int4/fp8 shrinks HBM and may speed decode, but task accuracy slips on exactly the workload you care about. Signal: dashboards are all green while user-reported answer quality or eval scores fall — because nothing was watching quality.
Knobs tuned for peak throughput blow up p99. A huge max-num-seqs wins the tokens/s headline while violating the latency SLO on the tail. Signal: mean latency looks fine, p99 TTFT is 5-10x p50, and complaints cluster at peak traffic.
Knobs tuned in isolation. Raising concurrency without freeing KV space (fp8/prefix cache/quant) triggers cache overflow and preemptions. Signal: throughput drops as concurrency rises, with preemption/recompute counters climbing.
Early stops inflate tokens/s. Generation that hits a stop token at 50 tokens instead of the contracted 500 makes throughput look great while measuring a different workload. Signal: measured output length far below the contract.

Implementation checklist

Is there a written workload contract — input/output length distributions, concurrency, sampling, hardware, model + runtime version — attached to every result?
Do you benchmark prefill-heavy and decode-heavy workloads separately, never averaged into one number?
Does every run record the full vector together: TTFT (p50/p99), TPOT, tokens/s, GPU util, HBM occupancy, queue time, and a quality score?
Are max-num-batched-tokens, max-num-seqs, KV-cache dtype, quantization, and prefix caching tuned as a coupled set, re-baselining after each change?
Is there a p99 SLO that any throughput-maximizing config must respect, and do you pick the largest safe concurrency rather than the peak of the throughput curve?
Do you run a fixed task eval before and after every compression and report the quality delta beside the speed delta?
Is cold-start time measured as a first-class metric, since autoscaling depends on it?

Checkpoint exercise

Try it

Write two workload contracts for one model: a prefill-heavy RAG workload (e.g. ~3,500-token prompts, 200-token answers, shared system prefix, concurrency 8) and a decode-heavy chat workload (~80-token prompts, 600-token answers, no shared prefix, concurrency 32). For each, predict which clock (TTFT or TPOT) dominates and which knob you would sweep first (prefix caching? chunked prefill? max-num-seqs? KV dtype?). Then state the p99 SLO and the one quality metric you would gate the change on. Justify why a single combined benchmark would have hidden the right answer for at least one of them.

Where this points next

You can now characterize a serving deployment under a stated load and tune it without fooling yourself. But all of that assumes a fixed number of replicas. Real traffic breathes — and the moment you add or remove replicas, two things you measured here become load-bearing: the cold-start time that decides whether new capacity arrives before the spike passes, and the latency/throughput curve that tells the autoscaler what "saturated" even means for a GPU (CPU and request-count signals lie for LLM serving). Lesson 09 — Autoscaling and AI-Aware Routing — builds the machinery that scales replicas on GPU-aware signals like queue depth and KV-cache pressure, and routes requests to exploit the prefix caching and warm replicas you tuned here.

Takeaway

A benchmark is a workload contract: input/output length, concurrency, sampling, hardware, and model+runtime version, or the throughput number is decoration. The two clocks are governed by different resources — TTFT/prefill is compute-bound, TPOT/decode is memory-bandwidth-bound — so prefill-heavy and decode-heavy workloads must be measured separately and the full metric vector (TTFT p50/p99, TPOT, tokens/s, GPU util, HBM, queue time, quality) recorded together. A short-prompt synthetic benchmark can mispredict a long-prompt cacheable production workload by ~60x on prefill cost and flip which latency dominates. The runtime knobs — max-num-batched-tokens, max-num-seqs, KV-cache dtype, weight quantization, prefix caching, chunked prefill — are one coupled system: tune them together, sweep one at a time, keep p99 inside the SLO instead of chasing the peak of the throughput curve, and pair every speed change with a before/after task-eval delta. Quantization and pruning trade quality for cost; that trade is only honest when it is measured.

Interview prompts

Why is "2,000 tokens/s on an H100" a meaningless number on its own? (§1 — throughput is a function of the workload contract; without input/output length, concurrency, sampling, and hardware+version it describes two different workloads sharing a label.)
Why must prefill-heavy and decode-heavy workloads be benchmarked separately? (§2 — prefill is compute-bound and decode is memory-bandwidth-bound; they stress different ceilings, so a knob that helps one is irrelevant or harmful to the other and an averaged number hides both.)
How can a synthetic benchmark mispredict production by 60x? (§3 — prefill cost scales with prompt length; 4,000/64 ≈ 62x, turning a ~5 ms prefill into ~310 ms that lands in TTFT, flipping which latency dominates. A short-prompt run also misses the shared-prefix cache that rescues it.)
Why can't you tune the vLLM knobs one at a time and stop? (§4 — they couple: more concurrency needs KV space freed by fp8/quant/prefix caching; a big batched-token budget lets one prefill stall decode unless chunked prefill intervenes. Sweep one, re-baseline, repeat.)
Why is maximizing tokens/s the wrong objective? (§4 — bigger batches raise throughput but blow up p99 TTFT; the right target is the largest concurrency that stays under the latency SLO, not the peak of the throughput curve.)
What makes a quantization or pruning decision honest? (§5 — a before/after task-eval delta reported beside the speed/cost delta; the loss is non-uniform across tasks, so only the eval set catches it — tokens/s never will.)
Why does startup time belong in the tuning budget? (§6 — cold start (image pull, weight load, cache alloc) is 60-300 s and invisible at steady state, but decisive for autoscaling: if cold start outlasts the spike, new replicas are useless. Measure it as a first-class metric.)