Part VI — Shipping the compiled model

Debugging compiled models

Lesson 16 produced something that actually runs in production: a cache of compiled, quantized kernels, a memory plan, and a runtime that replays the whole launch sequence as one driver call. The model is fast and it fits. There is just one problem you've been quietly accumulating since lesson 04 — it no longer computes exactly what the eager model computed. Fusion reorders arithmetic. Reassociation (which lesson 04 planted as the one rewrite that isn't bit-exact) reorders reductions. Quantization (lesson 15) is deliberately approximate. A different fused kernel accumulates in a different order. And graph-replay can silently fall back to recompiling without telling you. The whole track kept promising "preserve semantics" — and then spent four lessons legitimately moving bits. This lesson is how you tell an acceptable float drift from a real bug, find which pass caused it, and decide when the answer is "ship it" versus "that's broken."

1 · Why compiled ≠ bit-identical, and the test that tells drift from a bug

Start by accepting the thing nobody tells you: a correct compiler does not produce the same bits as eager. It produces a result that is mathematically equivalent on the reals but rounded differently. Four mechanisms, all of which you've already met:

So "the numbers differ" is not a bug report — it's the normal state of affairs. The real question is how much they differ, and the only honest way to ask it is against a reference. The reference is the eager model run in fp32 (or fp64 if you're paranoid): the highest-precision computation of the same math, which is your best proxy for the true real-number answer. You then compare the compiled output to it with a two-part tolerance, exactly the test torch.allclose / numpy.allclose implements:

atol (absolute tolerance) is the floor — it forgives small differences near zero, where relative error explodes (|0.0001 − 0.0| is "infinitely" wrong relatively but absolutely tiny). rtol (relative tolerance) scales with the magnitude of the value — it forgives a larger absolute gap on a large number, because large floats have larger ulps. A reasonable starting pair for bf16/fp16 work is rtol ≈ 1e-2, atol ≈ 1e-3; for fp32, something like rtol ≈ 1e-5. The verdict — within tolerance? — is your "drift vs bug" classifier:

The trap is tolerance choice itself. Set the tolerance too loose and a genuine bug slips through as "close enough"; set it too tight and benign reassociation drift trips it on every run and you waste a week chasing a non-bug. The discipline is to anchor the tolerance to a downstream metric wherever you can — perplexity, top-1 accuracy, a generation-quality eval — because the last bits of a logit almost never matter, but a clipped channel that drops accuracy 3 points does. Raw-tensor allclose is the fast screen; the task metric is the verdict that ships.

2 · Bisection and the minifier: let the tool find the pass

Suppose the verdict is "real bug": the compiled model's error blows past any sane tolerance. The graph has thousands of ops and a dozen passes ran over it. Where is the fault? The brute-force answer is bisection — disable half the passes (or half the graph), re-test, and recurse into whichever half still reproduces. That's log₂ of the problem size in steps, and it's exactly what a minifier automates.

A minifier is a tool that takes a large failing program plus a check ("does the compiled output diverge from the reference?") and automatically shrinks it — deleting ops, narrowing the graph, simplifying inputs — to the smallest subgraph that still triggers the failure, while also identifying the pass after which the divergence first appears. In torch.compile you arm it with an environment variable:

# reproduce-after a given compiler stage, on an accuracy divergence
TORCHDYNAMO_REPRO_AFTER="aot"        # bisect after AOTAutograd (the joint fwd/bwd graph)
TORCHDYNAMO_REPRO_LEVEL=4            # 4 = accuracy minifier (vs crash minifier)
python train.py
# → writes repro.py: a ~20-line graph that still diverges,
#   plus which lowering reproduced it. Hand that to a bug report.

Two flavors matter. The crash minifier shrinks a graph that makes the backend error (a codegen assertion, an unsupported op) down to the offending node. The accuracy minifier (the one you want here, REPRO_LEVEL=4) shrinks a graph whose output diverges from the reference beyond tolerance, isolating the few ops whose lowering introduced the divergence. The output is a tiny, self-contained repro.py — the difference between "the 70B model is wrong somewhere" and "this 8-op subgraph, after fusion, produces a NaN on this one input shape." That artifact is what turns a vague divergence into a fixable, reportable bug. REPRO_AFTER picks the stage to bisect from — "dynamo" (the captured FX graph), "aot" (after autodiff + functionalization, lesson 12), or the Inductor lowering — so you can localize the fault to capture, the joint graph, or codegen.

The same bisection logic works by hand when you don't have a minifier: compare against the reference at each lowering stage. If the FX graph (capture) already diverges, the bug is in your model or capture; if it's exact after capture but wrong after Inductor, the bug is in fusion/codegen; if it only appears under reduce-overhead (CUDA graphs, lesson 16), it's a stale-buffer or stable-address problem, not a math bug. Each stage you can mark "matches reference" cuts the search space in half.

3 · Read the compiler: graph-break storms and recompile storms

The other failure mode isn't wrong — it's slow, and silently so. Two diseases, both diagnosable only by making the compiler talk. You make it talk with structured logs: torch.compile exposes TORCH_LOGS, a comma-separated list of channels, and tlparse, a tool that turns the raw log dump into a navigable HTML report of every compiled region, guard, and break.

TORCH_LOGS="graph_breaks,recompiles,guards" python serve.py
# graph_breaks → every place Dynamo fell back to eager and why
# recompiles    → every guard that fired and forced a recompile
# guards        → the exact assumptions each compiled graph is specialized on

A graph-break storm (callback to lesson 02) is when Dynamo can't trace some construct — a data-dependent .item(), a print, an unsupported op (lesson 14), a Python side effect — and graph-breaks back to eager at that point, splitting your one big fusable graph into many small islands. Fusion can't cross a break (lesson 05's seam problem), so the win evaporates: instead of one fused kernel you get the eager launch storm lesson 01 priced. The log tells you exactly where and why:

[graph_breaks] torch._dynamo: BREAK in forward() at model.py:88
    reason: Tensor.item() called — data-dependent value escapes the graph
    → graph split here; 2 subgraphs, fusion cannot cross the seam
[graph_breaks] torch._dynamo: BREAK at model.py:142
    reason: unsupported builtin print() with tensor arg

The fix is to remove the break (avoid .item() in the hot path, wrap unsupported ops as custom ops per lesson 14, or set fullgraph=True to make a break a hard error you must resolve rather than a silent fallback). A recompile storm (callback to lesson 11) is different: the graph compiled fine, but a guard — the assumption set the compiled graph is specialized on (shapes, dtypes, the value of a Python flag, an object id) — keeps failing, so the runtime recompiles on nearly every call. The classic cause is shape specialization: you didn't mark a dimension dynamic, so every new sequence length recompiles. The log makes it obvious:

[recompiles] torch._dynamo: recompiling forward() (cache size 7)
    guard failed: L['x'].size()[1] == 512   (now 537)
    → tip: mark batch/seq dynamic with torch._dynamo.mark_dynamic(x, 1)
[recompiles] ... (cache size 8)  guard failed: L['x'].size()[1] == 537 (now 489)

A climbing "cache size" with a shape guard in every entry is the signature: you are compiling, not running. The fix is the lesson-11 toolkit — symbolic/dynamic=True shapes so one graph covers a range, or bucketing so only a few sizes ever compile, plus the recompile_limit guardrail that converts an out-of-control storm into a loud error. tlparse stitches all of this into one report: how many distinct graphs you compiled, what each is guarded on, and which guard is thrashing — the whole-program view that a single exception never gives you. (For kernel-level profiling once the graph is stable, the deeper treatment is gpu_kernels · 20 the profiling toolkit and 21 reading a profile against the roofline.)

4 · Determinism: where the nondeterminism lives, and what the switch costs

Sometimes the bug is that the model gives a different answer each run with no code change — which makes everything above impossible, because you can't bisect a moving target. Determinism means: same inputs, same code, bit-identical outputs, every run. Compiled (and even eager GPU) code is nondeterministic by default for performance reasons, from three sources:

The frameworks expose one switch that forces deterministic algorithms everywhere they exist and errors where they don't, so a hidden nondeterministic kernel can't sneak through:

torch.use_deterministic_algorithms(True)   # deterministic kernels, or raise
torch.backends.cudnn.deterministic = True
torch.backends.cuda.matmul.allow_tf32 = False   # full fp32, no TF32 shortcut
import os; os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"  # deterministic cuBLAS

This is a real trade, not a free correctness dial. Forcing deterministic algorithms can cost 10–30% wall-clock (sometimes worse): the deterministic scatter-add replaces fast atomics with a serialized or sorted reduction, and disabling TF32 drops matmul throughput by ~2–3× on tensor cores. So you turn determinism on while you debug — to get a stable target to bisect against and to make a divergence reproducible — and turn it off for production, where a few ulps of run-to-run noise are harmless and the speed is not. The same reproducibility-vs-speed tension as fast-math in lesson 04, now stated as an operational switch. The mistake is leaving it on in production (paying 20% for nothing) or expecting bit-identical results without it (chasing "nondeterminism bugs" that are just atomics).

5 · The debugging flowchart

Put it together as a triage. Every symptom of a compiled model routes to one of three questions, and each question has its tool. Don't reach for the minifier when the problem is a recompile storm, or for TORCH_LOGS when the output is wrong — match the tool to the symptom.

6 · Drive it: the divergence detective

The widget runs a compiled block against its fp32 reference and reports the two numbers that decide everything: max absolute error and max relative error. Toggle fault sources — aggressive fusion-reassociation, int8 quantization, TF32 — and watch each add a characteristic amount of drift that sits comfortably under a sane tolerance. Then flip on a real bug (a wrong accumulation dtype that loses bits every add) and watch the error blow past any reasonable line. Move the rtol/atol slider to set your tolerance and read the within tolerance? verdict. The lesson is in the knob that breaks: tighten the tolerance far enough and the benign drift trips the verdict too — false-positive triage — which is exactly why you anchor tolerance to a task metric, not to "as tight as possible."

The shape of the readout is the whole skill. With sane tolerances (rtol = 1e-2), every drift source — fusion, TF32, even int8 quant — reports "within tolerance," while the real bug always fails: that's the classifier working. Then drag rtol down toward 1e-5 and watch fusion drift flip to "FAIL" even though nothing is broken. The compiler didn't get worse; your tolerance got unrealistic. A correct debugging process is a correct tolerance plus a correct reference — get either wrong and the verdict lies.

Failure modes & checklist

Checkpoint

Where this points next

You can now do the thing the whole production half was missing: not just build a fast, fitting model, but trust it. You can tell an acceptable float drift (reassociation, TF32, a different kernel — uniform, tiny, ship it) from a real bug (growing error, NaN, a clipped channel — minify it); you can let the accuracy minifier hand you a 20-line repro tagged with the guilty pass; you can read TORCH_LOGS/tlparse to catch a graph-break or recompile storm before it eats your throughput; and you can force determinism to get a stable target, knowing what it costs. That closes the loop the track has been building since lesson 04 — every "preserve semantics" promise now has a way to verify the preservation and a defined boundary (fast-math, quant) where it's relaxed on purpose. With build (00–13), ship (14–16), and trust (17) all in hand, you've seen every part of the machine in isolation. The remaining question is how the real systems wire them together — and whether torch.compile, XLA, TVM, MLIR, and TensorRT are really the same chain under different names. That's lesson 18, The real stacks.

Interview prompts

• Why doesn't a correct compiler produce bit-identical results, and how do you tell drift from a bug? (§1 — fusion/reassociation (04) reorder non-associative float adds, different kernels accumulate differently, quant (15) is deliberately approximate; compare to an fp32 reference with |out−ref| ≤ atol + rtol·|ref| — uniform tiny error is benign drift, error that grows per layer or yields NaN is a real bug.)
• What's the difference between rtol and atol, and why do you need both? (§1 — atol is an absolute floor that forgives differences near zero where relative error explodes; rtol scales the allowed gap with the value's magnitude because large floats have larger ulps; the combined bound handles both small and large values.)
• What does the accuracy minifier do, and how is it different from the crash minifier? (§2 — the crash minifier shrinks a graph that makes the backend error down to the offending node; the accuracy minifier (REPRO_LEVEL=4) shrinks a graph whose output diverges from the reference to the few ops that introduced the divergence, plus the pass; REPRO_AFTER picks the stage — dynamo/aot/inductor — to bisect from.)
• You turn on torch.compile and the model gets slower. How do you diagnose it? (§3 — TORCH_LOGS=recompiles,graph_breaks + tlparse: a climbing cache size with a shape guard in each entry is a recompile storm (mark dynamic/bucket, 11); many small kernels with breaks is a graph-break storm (remove breaks / custom-op, 02/14); if the graph is stable and still slow it's a kernel problem, profile it.)
• Where does nondeterminism in compiled GPU code come from, and what does forcing determinism cost? (§4 — atomic/reduction order (run-varying float add order), TF32 reduced-precision matmuls, and reduced-precision accumulation; use_deterministic_algorithms(True) + disabling TF32 forces deterministic kernels or errors, costing ~10–30% wall-clock — keep it on to debug, off to ship.)
• A quantized model passes torch.allclose but eval accuracy drops 3 points. What went wrong? (§1 — you anchored tolerance to raw tensors instead of the task metric; quant drift on its accuracy budget can pass a loose allclose while a clipped/outlier channel still hurts the downstream metric — the metric is the verdict, allclose is only a screen.)
• Why diff against an fp32 reference rather than the eager bf16 model? (§1, Failure modes — eager bf16 is itself imprecise and drifts, so a bf16-vs-bf16 diff can't say which is right; the fp32 (or fp64) run of the same math is the best proxy for the true real-number answer and gives a stable target.)