Part III - Production inference

Observability for LLM Serving

Lesson 10 split prefill and decode onto separate pools so each could scale on its own bottleneck. But the moment you disaggregate, the question "is it healthy?" stops having a single answer: prefill can be starved while decode is idle, or the GPUs can be pinned at 100% utilization while users wait twenty seconds for a first token. A green HTTP 200 tells you almost nothing about any of that. This lesson builds an observability stack that sees an LLM workload in its own language — token timing, queue depth, GPU memory pressure, and output quality — so that when something degrades, the telemetry points at the exact path that broke.

Source coverage

PDF Chapter 5: model observability.

Linear position

Prerequisite: Lessons 06-10 (the serving runtime, batching, KV cache, autoscaling, and the disaggregated prefill/decode split) — you now run a multi-stage inference path whose stages can fail independently.
New capability: An observability stack — logs, metrics, traces, and quality signals — that decomposes a request along four paths and tells you which one is out of bounds, instead of one aggregate "up/down".

The plan

Five moves. (1) Reframe "healthy" for an LLM service as four paths in bounds, not one status code. (2) Define the token-path metrics that traditional monitoring lacks — TTFT and TPOT — and show with arithmetic why a single p50 latency is useless. (3) Walk the GPU path: what DCGM exposes and why GPU utilization alone lies. (4) Build the trace: spans for gateway, queue, prefill, decode, and streaming, tagged with model + version, and why segmenting beats averaging. (5) Add the quality path, then the failure modes, checklist, and the hand-off into guardrails.

1 · "Healthy" is four paths, not one status code

A classic web service is healthy when requests return 2xx quickly and the error rate is low. That model breaks for LLM serving because a single request is not one operation — it is a multi-stage pipeline that produces output incrementally over seconds. The HTTP status fires once, at the start, when the server accepts the streaming connection. Everything that actually matters to the user — how long until the first token, how smoothly the rest stream, whether the answer is correct — happens after the 200 is already sent.

So the mental model for the rest of this lesson: an LLM request is healthy only when four independent paths are all inside their bounds. Each can fail while the others look fine, and each has its own telemetry.

Request path (HTTP / queue)

The classic layer: arrival rate, HTTP status, and — new for LLMs — queue depth and queue wait time. A request can be accepted (200) yet sit in the runtime's queue for seconds because all batch slots are full. Queue metrics are how you see saturation that the status code hides.

Token path

TTFT (time to first token), TPOT (time per output token), and throughput (tokens/sec). This is the layer the user actually feels: TTFT is the wait before anything appears; TPOT is how fast the rest streams. Defined in §2.

GPU path (DCGM)

DCGM = Data Center GPU Manager, NVIDIA's metrics agent: SM (streaming-multiprocessor) occupancy, HBM (high-bandwidth memory) usage, utilization, power, and thermals. Tells you whether the expensive accelerators are doing useful work or stalled. Detailed in §3.

Quality path

Eval scores, refusal rate, and hallucination rate. The path with no analogue in classic ops: a model can serve fast, within every latency bound, and still regress to wrong answers. Covered in §5.

The discipline this lesson teaches is to instrument all four and to correlate them by the same dimensions — model, version, tenant, route, prompt length, output length — so that "TTFT spiked" can be joined to "HBM hit 95%" on the same trace.

2 · The token path — and why a single latency is a lie

Two metrics carry the token path. TTFT (time to first token) is the elapsed time from when the request is accepted to when the first output token is emitted; it is dominated by queue wait plus prefill (the one-shot forward pass over the whole prompt, lesson 06). TPOT (time per output token) is the average gap between successive tokens once generation has started; it is dominated by decode (one token at a time, memory-bandwidth bound). End-to-end latency for a streaming response is therefore:

total = TTFT + TPOT × (output_tokens − 1)

This decomposition is the whole point, because the two terms move for completely different reasons. TTFT grows when the queue is deep or the prompt is long (prefill cost scales with prompt length). TPOT grows when decode is contended — more concurrent sequences sharing the GPU, or KV-cache pressure forcing evictions. A single aggregate latency number cannot tell these apart.

Worked number — why p50 latency hides the diagnosis

Take one request: TTFT = 200 ms, then 500 output tokens at TPOT = 30 ms each. Total = 200 + 30 × 499 ≈ 200 + 14,970 = 15.2 s. Now a second request: a long 8k-token prompt pushes TTFT to 2,000 ms, but it only emits 40 tokens at the same 30 ms TPOT, so total = 2,000 + 30 × 39 ≈ 3.2 s. Their total latencies (15.2 s vs 3.2 s) tell you nothing actionable — the slow one is slow because it generated a lot, not because anything is wrong. But the decomposed view is diagnostic: request two's 2 s TTFT screams "prefill / prompt-length problem," while request one's healthy 200 ms TTFT and 30 ms TPOT say "this request is fine, it's just long." If you only logged end-to-end p50, you would chase the wrong layer. The fix is to report TTFT and TPOT as separate distributions, each segmented — TTFT by prompt length, TPOT by output length — because averaging across a 100-token prompt and an 8k-token prompt blends two unrelated populations into a meaningless middle.

Throughput (tokens/sec, aggregated across all concurrent sequences) is the third token-path metric and the one that pays the bills: it is the denominator of your $/1M-tokens cost. Critically, throughput and per-request TPOT trade against each other — larger batches raise total tokens/sec but can raise each individual user's TPOT. You cannot reason about that trade if you only watch one of them, which is exactly the knob the widget in §4 makes draggable.

3 · The GPU path — why utilization alone lies

The instinct from VM/container ops is to watch one number: GPU utilization. DCGM (Data Center GPU Manager) is NVIDIA's host-level metrics exporter — it runs as a DaemonSet on every GPU node and scrapes per-GPU counters into Prometheus. The trap is that DCGM's headline "GPU utilization" (DCGM_FI_DEV_GPU_UTIL) only means "a kernel was running on the device during the sample window." It can read 100% while the GPU does almost no useful token work — for example, when the batch is tiny and decode is memory-bandwidth bound, or when tensor-parallel ranks (lesson 07) spend the window blocked in a collective waiting on a slow peer.

So the GPU path needs more than one counter:

→SM occupancy → fraction of the streaming multiprocessors actually busy with compute. The honest "are we doing work?" signal, vs. the binary GPU-util flag.

→HBM usage → high-bandwidth memory consumed. This is where weights and the KV cache live; when it approaches the ceiling, the runtime starts evicting cache and rejecting or queueing requests.

→power & thermals → a GPU throttling on power/temperature silently lowers clocks, which shows up as rising TPOT with no code change.

→tokens/sec ÷ utilization → the derived ratio that catches the "100% busy, low output" pathology that no single counter reveals.

Concrete reading — "high GPU, low tokens"

An H100 has 80 GB of HBM. Suppose weights for a served model take 16 GB and DCGM reports utilization at 98%, HBM at 78/80 GB, but tokens/sec is half of last week's baseline. The high HBM is the tell: the KV cache has filled memory, the runtime is evicting and re-prefilling, and the GPU is "busy" doing wasted recompute rather than new tokens. Utilization alone said "healthy and maxed out"; the combination of near-full HBM + collapsed tokens/sec said "cache pressure — shed load or add a replica." This is why GPU metrics must be correlated with the token path, never read in isolation.

4 · The trace — spans, segmentation, and the streaming gap

Metrics tell you a path is out of bounds in aggregate; a trace tells you where a single slow request spent its time. A trace is a tree of spans — timed intervals — and the LLM-specific work is to emit a span for each stage of the inference pipeline, each tagged with model and model_version attributes so you can slice by deployment:

gatewayAuth, routing, request validation. Cheap; if it dominates, the problem is the front door, not the model.

queueTime waiting for a batch slot in the runtime. A fat queue span is the saturation signal the HTTP 200 concealed.

prefillThe forward pass over the whole prompt. Scales with prompt length; this span + queue ≈ TTFT.

decodeToken-by-token generation. Its duration ÷ tokens ≈ TPOT; segment by output length.

streamTokens flushed to the client over the wire. Easy to forget to instrument — and that omission is its own failure mode.

A decomposed trace for the 15.2-second request from §2 reads like this — and immediately localizes the cost:

request (15.2 s)
├─ gateway      10 ms   auth + route, model=llama-70b v=2026-05
├─ queue       150 ms   waited for a decode slot   ┐
├─ prefill      50 ms   120-token prompt           ┘ queue + prefill ≈ TTFT 200 ms
├─ decode    14.97 s    500 output tokens @ ~30 ms each (499 gaps)   ← the time lives here
└─ stream    (overlaps decode)   tokens flushed to client as they are produced

The lesson the trace teaches: this request is not a prefill or queue problem — 14.97 of 15.2 seconds is legitimate decode for 500 tokens. No amount of adding prefill replicas would help. Contrast that with a request whose queue span is 3 s: same total latency, completely different fix (the runtime is saturated — autoscale decode, lesson 09). Without the spans you cannot tell these apart from the outside.

Standardize the attributes — OpenTelemetry GenAI

Do not invent ad-hoc attribute names. The OpenTelemetry GenAI semantic conventions define a gen_ai.* namespace — e.g. gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons — so your spans are portable across tools and your dashboards keep working when you swap a backend. Adopting the convention is what lets a generic trace UI know that this span is an LLM call and render token counts, not just a black-box duration. Caveat for 2026: this is an actively evolving spec — most of the GenAI conventions are still marked Development (experimental), so attribute names and the opt-in capture of prompt/response content can still shift between releases. Pin the convention version you target and expect to migrate, rather than treating today's names as frozen.

The single most common tracing bug here is dropping the streaming span. If the trace ends when the request is accepted (or when the first token leaves), the seconds of decode that the user actually experiences are invisible — TTFT looks fine and the trace looks short, yet users complain. You must keep the span open across the whole streamed response and record first-token time within it.

Latency decomposition — prompt length feeds TTFT, output length feeds TPOT

Drag prompt length (which grows prefill, and thus TTFT) and output length (which multiplies TPOT). The bar splits total latency into its TTFT and decode parts. Watch how a long prompt and a long output produce the same total for completely different reasons — the split is the diagnosis, the total is not.

prompt length (tokens): 512 output length (tokens): 500

TTFT (queue + prefill)

—

Decode (TPOT × (out−1))

—

Total latency

—

Bottleneck path

—

5 · The quality path — green metrics, wrong answers

The first three paths are about the request being served; the fourth is about it being correct. This path has no equivalent in classic ops, and it is the one that ships silent regressions, because operational dashboards stay green while the model gets worse. A prompt-template edit, a retrieval index that went stale, a quantized weight swap, or a base-model version bump can all leave TTFT, TPOT, and HBM untouched while answer quality falls off a cliff.

So the quality path needs its own continuously-tracked signals, joined to the same model + version attributes as the spans: eval scores on a held-out task set run against live traffic samples; refusal rate (fraction of requests the model declines — a spike often means an over-tightened safety prompt, a drop means guardrails loosened); and hallucination rate (sampled, judged by a rubric or a stronger judge model). These are sampled, not measured on every request, and they are the bridge into the next lesson — guardrails are the enforcement layer that acts on what the quality path observes.

Privacy is a constraint on the quality path, not an afterthought

The richest quality signal is the raw prompt and completion — and logging them is exactly what creates a data-leak incident. Prompt/completion capture must be governed: sample rather than log everything, redact PII before storage, gate access by role, set an explicit retention window, and respect per-tenant policy. Observability that exfiltrates user data is not observability; it is a new incident class. Treat the privacy policy as a hard input to what the quality path is even allowed to record.

6 · Failure modes and the checklist

Failure modes

Only node-level GPU utilization is visible. You see a green DCGM utilization dashboard and conclude the system is healthy, but queue wait and token latency are uninstrumented — so saturation and a degraded TTFT stay mysterious. Signal: users report slowness while every GPU graph looks maxed-out-and-fine. Add queue depth, TTFT, and TPOT.
Tracing drops the streaming span. The trace ends at request acceptance or first token, so the seconds of decode the user lives through never appear, and TTFT/TPOT can't be reconstructed from the trace. Signal: short, healthy-looking traces alongside real complaints about response speed. Keep the span open across the whole stream.
Quality regressions ship because ops stays green. A template/index/weight change tanks answer quality with zero movement in latency or GPU metrics, so it passes every operational gate. Signal: rising thumbs-down or support tickets with no latency anomaly. Track eval/refusal/hallucination rates per version.
Averaging across segments. A single TTFT distribution blends 100-token and 8k-token prompts into a meaningless middle that hides the prefill problem. Signal: a stable aggregate metric while specific cohorts suffer. Segment TTFT by prompt length and TPOT by output length.
Metrics that can't be joined. GPU, token, and request metrics carry inconsistent labels, so "TTFT spiked" can never be correlated with "HBM hit 95%." Signal: every dashboard looks plausible in isolation and no one can build the causal chain. Tag everything with the same model/version/tenant/route dimensions.

Implementation checklist

Do you emit request, queue, prefill, decode, and streaming spans, each tagged with model and model_version (ideally via the OpenTelemetry gen_ai.* conventions)?
Are TTFT and TPOT tracked as separate distributions, segmented by prompt length and output length respectively?
Is the streaming span kept open for the full response, with first-token time recorded inside it?
Do you correlate GPU utilization, SM occupancy, HBM usage, and cache pressure with the token-path and queue metrics on shared labels?
Is there a derived tokens/sec-per-utilization signal to catch the "100% busy, low output" pathology?
Are eval, refusal, and hallucination rates sampled and tracked per model version, joined to the operational metrics?
Is prompt/completion logging governed by sampling, redaction, access control, retention, and tenant policy?

Checkpoint exercise

Try it

Sketch the span tree for one slow request in your serving stack and label each span with the metric it feeds (queue → queue wait, prefill → TTFT, decode → TPOT). Then write the three dashboard panels you would put on call: pick which segmentation each uses (prompt length vs. output length), which DCGM counter you correlate against, and the exact alert that would have caught last quarter's silent quality regression without firing on a merely-long generation.

Where this points next

This lesson built the seeing layer — telemetry that decomposes a request into request, token, GPU, and quality paths and tells you which one is out of bounds. The quality path in particular only observes: it reports refusal and hallucination rates, but it does not stop a bad output from reaching the user. Lesson 12, Safety, Quality, and Guardrails, adds the acting layer — input and output filters, policy enforcement, and quality gates that consume the very signals you just instrumented and block, rewrite, or escalate before the response ships. For the broader monitoring discipline behind all of this, see ML system monitoring.

Takeaway

For an LLM service, "healthy" means four paths are all inside bounds — the request path (HTTP plus queue depth/wait), the token path (TTFT = queue + prefill, TPOT = per-token decode, throughput), the GPU path (DCGM: SM occupancy, HBM usage, power, not just the lying utilization flag), and the quality path (eval, refusal, hallucination rates). One green HTTP 200 sees none of them. Because total latency is TTFT + TPOT × (out − 1), a single p50 blends two unrelated causes; you must split TTFT from TPOT and segment each by prompt and output length, then emit per-stage spans (gateway → queue → prefill → decode → stream) tagged with model + version under the OpenTelemetry gen_ai.* conventions so GPU, token, and quality signals can be correlated. Never drop the streaming span, never read GPU utilization alone, and govern prompt/completion logging by privacy policy — observability that leaks data is just a new incident.

Interview prompts

Why is an HTTP 200 insufficient to call an LLM request healthy? (§1 — the status fires once at stream acceptance; TTFT, TPOT, GPU pressure, and answer quality all happen afterward, so the four paths must be instrumented independently.)
Define TTFT and TPOT and write end-to-end latency in terms of them. (§2 — TTFT = accept-to-first-token (queue + prefill); TPOT = average inter-token gap (decode); total = TTFT + TPOT × (output_tokens − 1).)
Why is a single p50 latency useless for diagnosis? (§2 — it blends prefill-bound and decode-bound requests; a long-prompt request and a long-output request can share a total but need opposite fixes. Split and segment TTFT by prompt length, TPOT by output length.)
Why is GPU utilization a misleading health metric, and what do you watch instead? (§3 — it only means "a kernel ran"; it reads 100% during memory-bound decode or collective stalls. Watch SM occupancy, HBM usage, power/thermals, and tokens/sec ÷ utilization.)
What spans should an LLM trace emit, and which one is most often missed? (§4 — gateway, queue, prefill, decode, stream, tagged with model + version; the streaming span is the one most commonly dropped, hiding the decode time users feel.)
What does the OpenTelemetry GenAI convention buy you? (§4 — a standard gen_ai.* attribute namespace for model, token counts, and finish reasons, making traces portable across tools and dashboards stable across backend swaps.)
How can a release pass every operational gate yet still be a regression? (§5 — quality lives on a separate path; a template/index/weight change can leave latency and GPU metrics untouched while answers degrade. Track eval/refusal/hallucination per version, governed by privacy policy.)