Part VI — Shipping the compiled model

Quantization & precision lowering

Lesson 14 closed the last coverage gap: every op in the graph now lowers to a generated or vendor kernel, so fusion runs end to end with no graph-break seams. But step back and look at the currency we have spent the whole track chasing — bytes moved across HBM. Every pass so far moved fp16/bf16 bytes: fusion cut how many times we move them, layout decided their order, planning decided where they live. Not one pass changed how wide a number is. That is the single biggest remaining lever, and it has been sitting untouched: spend precision. Halve every weight and activation from 16 bits to 8 and you halve HBM traffic and double tensor-core throughput in one move. This lesson makes that a compiler pass — and it is the one pass in the entire stack that does not preserve the math exactly.

The previous step left this broken

After lesson 14 the compiler can generate a kernel for every op, and lessons 05–10 made those kernels as tight as fp16 allows: fused, planned, laid out, scheduled, autotuned. But re-read the roofline from lesson 01. A memory-bound op's wall-clock is bytes ÷ bandwidth, and a GEMM that is bandwidth-limited (the decode-time matmul, which reloads the whole weight matrix to produce one token) is paying for 16-bit weights it does not numerically need. Tensor cores on an H100 do bf16 at ≈990 TFLOP/s but int8 at ≈1,980 TOPS and fp8 at ≈1,980 TFLOP/s — 2× — and the weights themselves are half the bytes. The middle end never reached for any of this because every pass it ran had to be a provable no-op on the numbers, and lower precision is not a no-op. That self-imposed rule left a clean 2–4× on the table.

Linear position

Forced by: even a fully fused, scheduled, autotuned fp16 graph leaves 2–4× unclaimed — int8/fp8 halve HBM traffic and double tensor-core throughput, and weight-only int8/int4 directly attacks the bandwidth-bound decode GEMM (lesson 01's roofline).
New idea: quantization as a graph pass — insert quantize/dequantize (QDQ) nodes, pick per-tensor or per-channel scale/zero-point by calibration (PTQ) or by learning them (QAT), then fuse the dequant into the GEMM epilogue (lesson 05) so the matmul runs in int8/fp8 and only widens at the end; decide which ops stay fp16 (precision assignment, the dual of layout assignment in lesson 07). The twist on the track's first motif: this is the one pass that is deliberately not bit-exact — it preserves the math approximately, on an accuracy budget.
Forces next: you have now generated and quantized a cache of kernels — artifacts. But an artifact is inert. What actually launches them at run time, allocates their buffers, and amortizes launch overhead? That is the runtime (lesson 16).

The plan

Five moves. (1) The bytes argument: int8 vs fp16 traffic and tensor-core throughput, and why weight-only int8/int4 is the right tool for bandwidth-bound decode. (2) The QDQ representation: quantize → int op → dequantize as graph nodes; per-tensor vs per-channel scale/zero-point; symmetric vs affine. (3) How you choose the scales: PTQ calibration (min-max / percentile / MSE on a small set) vs QAT (fake-quant in the forward, straight-through estimator). (4) The compiler's job: propagate quant, fuse dequant into the matmul epilogue, and assign which ops stay fp16. (5) fp8 (e4m3/e5m2), int4 weight-only (AWQ/GPTQ-style), 2:4 structured sparsity — and the honest caveat about outliers and the accuracy budget. Then drive the precision dial until int4 with bad calibration falls off the accuracy cliff.

1 · The bytes argument: why precision is the biggest remaining lever

Quantization is representing a tensor of real numbers with a lower-precision integer (or narrow-float) type plus a small amount of metadata — a scale and optionally a zero-point — so that the original value is recovered approximately as x ≈ scale · (q − zero_point), where q is the stored integer. Going from bf16 (2 bytes) to int8 (1 byte) does two things at once, and both attack the roofline.

Half the bytes. A 7-billion-parameter model in bf16 is ≈14 GB; in int8 it is ≈7 GB; in int4 it is ≈3.5 GB. During autoregressive decode the model reads its entire weight matrix from HBM to produce a single token — the matmul is tiny (batch 1, one query vector) but the weight load is the whole model, so decode is weight-bandwidth-bound. Halving the weight bytes nearly halves decode latency directly. This is the lesson-01 roofline turned into money: time ≈ weight_bytes ÷ bandwidth, so cutting the bytes cuts the time.

Double the FLOPs. Tensor cores have dedicated low-precision modes. Approximate H100 peaks: bf16 ≈990 TFLOP/s, fp8 (e4m3) ≈1,980 TFLOP/s, int8 ≈1,980 TOPS, and on Blackwell the gap widens again with fp4. For the prefill / training GEMMs that are compute-bound, int8/fp8 is a straight 2× on the math throughput. So quantization helps both regimes: it cuts traffic for the bandwidth-bound decode and lifts throughput for the compute-bound prefill.

format	bytes/elem	7B model size	tensor-core peak (H100, ≈)	best for
fp16 / bf16	2	≈14 GB	≈990 TFLOP/s	baseline; no accuracy risk
fp8 (e4m3 / e5m2)	1	≈7 GB	≈1,980 TFLOP/s	training & inference; keeps a float exponent
int8	1	≈7 GB	≈1,980 TOPS	inference; needs calibration
int4 (weight-only)	0.5	≈3.5 GB	dequant→fp16 matmul	memory-bound decode; accuracy-sensitive

The crucial distinction the table hides: weight-only vs weight+activation. Weight-only quant (store weights int4/int8, dequantize them to fp16 just before the matmul, compute in fp16) wins purely on weight bandwidth — perfect for decode, where the weight load dominates and the actual matmul is tiny. Weight+activation quant (both operands int8, compute in int8 on the int8 tensor cores) is what unlocks the throughput 2×, but it is harder because activations have outliers (more on that in §5). For a decode-bound LLM, int4 weight-only is often the single highest-leverage knob; for a compute-bound prefill or training step, int8 weight+activation is the prize.

2 · The QDQ representation: quantize and dequantize as graph nodes

The compiler does not need a parallel "int8 graph." It represents quantization with two new node types inserted into the ordinary graph: a quantize node (fp → int, dividing by the scale and rounding) and a dequantize node (int → fp, multiplying by the scale). The pattern is QDQ: wrap a region you want to run in low precision with quantize on its inputs and dequantize on its outputs. The op in the middle sees integers; the rest of the graph still sees floats.

FX-style graph with QDQ nodes around a linear layer (per-channel weight, per-tensor act):

  %x        : f16[B, K]
  %w        : f16[N, K]                 # weight, known at compile time
  # --- quantize the weight once, at compile time, per output channel ---
  %w_scale  : f16[N]      = compute_per_channel_scale(%w)   # one scale per row
  %w_q      : i8[N, K]    = quantize(%w, %w_scale, zp=0, axis=0)   # symmetric
  # --- quantize the activation at run time, per tensor ---
  %x_scale  : f16[1]      = calibrated_scale                 # from PTQ, baked in
  %x_q      : i8[B, K]    = quantize(%x, %x_scale, zp=0)
  # --- the matmul runs in int8 → i32 accumulator ---
  %acc      : i32[B, N]   = int8_matmul(%x_q, %w_q)          # int8 tensor cores
  # --- dequantize: scale back to f16 (fused into the epilogue, see §4) ---
  %y        : f16[B, N]   = dequantize(%acc, %x_scale * %w_scale)   # per-channel
  %out      : f16[B, N]   = gelu(%y + %bias)                 # epilogue, stays f16

Three choices define a QDQ scheme, and the compiler must pin each one:

scale & zero-point

Scale maps the float range to the integer range (e.g. fp range ±6 → int8 ±127 ⇒ scale ≈ 0.047). Zero-point is the integer that represents float 0; needed when the range is not centered on 0.

symmetric vs affine

Symmetric: zero-point = 0, range is ±max, one multiply to dequant — ideal for weights (roughly zero-centered). Affine (asymmetric): a nonzero zero-point fits a one-sided range like a post-ReLU activation, at the cost of an extra add.

per-tensor vs per-channel

Per-tensor: one scale for the whole tensor — cheapest, but one fat channel forces a coarse scale on all. Per-channel: a scale vector along the output axis, so each row/channel gets its own range — much better accuracy for weights, the standard for int8 linear layers.

static vs dynamic

Static: activation scale fixed ahead of time by calibration (one less op at run time). Dynamic: compute the activation's scale per call from its actual min/max — more accurate for spiky activations, a small runtime cost.

Why does per-channel matter so much? A weight matrix can have one output channel with 10× the dynamic range of its neighbors. A single per-tensor scale must cover that fat channel, so every other channel wastes most of its 256 int8 codes on a range it never uses — quantization error explodes. Give each output channel its own scale (a length-N vector, negligible metadata) and every channel uses its full code range. Per-channel weight quant is essentially free in storage and is the reason int8 linear layers hold accuracy.

3 · Choosing the scales: PTQ calibration vs QAT

The scales are the whole game. Set them too tight and you clip the tails; too loose and you waste resolution on empty range. Two families pick them.

Post-training quantization (PTQ) takes a trained fp model and chooses scales without any retraining, by running a small calibration set (a few hundred representative inputs) through the network and observing each tensor's actual value distribution. From that histogram you pick the scale by one of three rules:

min–maxScale to cover the absolute min/max seen. Simplest; but one outlier activation stretches the range and starves the bulk of values of resolution.

percentileClip to, say, the 99.99th percentile and let the rare tail saturate. Trades a little clipping error for far better resolution on the common values — usually a win.

MSE / KLSearch the clipping threshold that minimizes reconstruction error (mean-squared, or KL-divergence between the fp and quantized histograms, as in TensorRT's entropy calibrator). Most accurate, most search.

PTQ is fast (minutes, no gradients, a small dataset) and is the default for int8 inference. Its ceiling is that it cannot recover from the error it introduces — it only chooses the least-bad scales for a fixed model.

Quantization-aware training (QAT) goes further: it simulates quantization during training (or a fine-tune) so the weights learn to be robust to it. The mechanism is fake-quant — in the forward pass, insert dequantize(quantize(x)) so the network actually sees the rounded/clipped values and its loss reflects the quantization error. The problem is that quantize is a step function: its derivative is zero almost everywhere, so gradients cannot flow back through it. The fix is the straight-through estimator (STE): on the backward pass, pretend the rounding was the identity (gradient = 1) within the representable range, so gradients pass through the fake-quant unchanged. The model trains against the quantized forward while still receiving usable gradients. QAT typically recovers most of the accuracy PTQ loses, at the cost of a training run — so you reach for it only when PTQ's drop exceeds your budget (commonly for int4, or int8 on a sensitive model).

	PTQ	QAT
cost	minutes; a few hundred calibration samples; no gradients	a training/fine-tune run with fake-quant + STE
accuracy	good for int8; degrades for int4 / sensitive models	recovers most of the loss; needed for aggressive bit-widths
when	default first attempt	when PTQ blows the accuracy budget

4 · The compiler's job: propagate, fuse the dequant, assign precision

Given a QDQ-annotated graph, the compiler does three things, and each rhymes with a pass you have already seen.

Propagate the quant. A bare QDQ pattern is wasteful: quantize right before an op and dequantize right after it means you constantly cross back to fp16 between layers. The pass slides the quant boundaries outward so a run of ops stays in int8 — the output of one int8 matmul feeds the next int8 op without a round trip through fp16. This is exactly the propagation idea from layout (lesson 07): pick a region's precision and conform the agnostic ops inside it, instead of choosing op-by-op.

Fuse the dequant into the epilogue. The headline optimization, and the direct callback to lesson 05. A naive lowering computes the int8 matmul into an int32 accumulator, writes the int32 result to HBM, then a separate dequantize kernel reads it back, multiplies by the scale, and writes fp16. That is two extra round trips — it would undo the whole point. Instead the compiler fuses the dequantize (the per-channel scale multiply, plus the bias-add and activation) into the matmul's epilogue: while the int32 output tile is still hot in registers, multiply by x_scale · w_scale, add bias, apply GELU, and write fp16 once. The quantization machinery costs zero extra HBM traffic — it rides the trip the matmul was making anyway. Without epilogue fusion, quantization is often a net loss; with it, it is the win the table promises.

Assign which ops stay fp16. Not everything should be quantized. Softmax, layernorm, the residual stream, and the final logits are numerically delicate — quantizing them buys little (they are not the big GEMMs) and risks a lot. So the compiler solves a precision-assignment problem: choose a precision for each tensor that minimizes bytes/latency subject to staying inside the accuracy budget, keeping sensitive ops in fp16 and the heavy linear layers in int8/fp8. This is the structural twin of layout assignment (lesson 07) — a global per-tensor labeling where a locally greedy choice (quantize everything) is globally wrong (it breaks accuracy), so you anchor on what must stay fp16 and conform the rest. The objective is identical in shape; only the label being assigned (precision, not layout) differs.

And here is the motif, stated plainly. Every other pass in this track was bound by preserve semantics — a provable no-op on the numbers (lessons 03–04, 05, 07). Quantization breaks that rule on purpose. It does not preserve the math exactly; it preserves it approximately, trading a bounded, measured accuracy drop for bytes and FLOPs. The invariant changes from "bit-exact" to "within the accuracy budget" — which is why lesson 17 will need a whole debugging discipline to tell acceptable quantization drift from a real bug. Of all the passes, this is the one that requires you to decide, in advance, how much correctness you are willing to spend.

5 · fp8, int4 weight-only, 2:4 sparsity — and what breaks

Three production directions push past int8, each with its own bargain.

fp8. Two 8-bit float formats trade exponent for mantissa: e4m3 (4 exponent, 3 mantissa bits — more precision, smaller range, used for weights and activations) and e5m2 (5 exponent, 2 mantissa — more range, less precision, used for gradients). Because fp8 keeps an exponent, it tolerates the wide dynamic range of activations far better than int8 does, which is why fp8 is viable for training (with per-tensor or finer scaling factors) where int8 activations would clip. Hopper and Blackwell have native fp8 tensor cores; the cost is the bookkeeping of the scaling factors.

int4 weight-only. For decode-bound LLMs the prize is shrinking the weights as far as possible while computing in fp16. GPTQ quantizes weights one column at a time, using second-order (Hessian) information to compensate the remaining weights for each rounding error. AWQ (activation-aware) observes that a small fraction of weight channels are salient (the ones multiplied by large activations) and scales them to protect their precision before quantizing. Both push to 4 bits with surprisingly small loss — but at int4 you are near the cliff, and the method and calibration matter enormously (cross-link distillation — the compression sibling track, which treats weight compression and low-rank/quant methods in depth).

2:4 structured sparsity. A different axis: instead of narrower numbers, fewer of them. 2:4 sparsity means that in every contiguous group of 4 weights, exactly 2 are forced to zero. NVIDIA's sparse tensor cores (Ampere onward) skip the zeros and run such a matmul at ≈2×, storing only the nonzeros plus a 2-bit index per group. It composes with quantization (int8 + 2:4 stacks both wins) but requires retraining/pruning to the 2:4 pattern, and the structured constraint costs more accuracy than unstructured pruning would.

The honest caveat. Quantization spends an accuracy budget, and the bill comes due at outliers. A handful of activation channels in large transformers have magnitudes 10–100× the rest; a per-tensor scale that covers them clips everything else into a few codes, and accuracy collapses. This is why activation quant is harder than weight quant, why methods like SmoothQuant migrate scale from activations to weights, and why fp8's exponent helps. What breaks, concretely: per-tensor int8 on outlier-heavy activations; int4 without GPTQ/AWQ-grade calibration; quantizing the delicate softmax/norm/logit ops; and any scheme where the dequant is not fused into the epilogue (then it is slower and less accurate). The pass is real leverage, but it is the one place in the compiler where "I made it faster" must always be paired with "and I measured what it cost."

6 · Drive it: the precision dial

The widget is a matmul-heavy block (a transformer linear layer). Pick a precision — fp16, fp8, int8, or int4 weight-only — and set the calibration quality. Watch the four numbers the pass actually trades: HBM bytes, model size, estimated speedup, and a simulated accuracy drop. The point to feel: int8 is a near-free ≈2× (small drop even at mediocre calibration); int4 is a bigger win but its accuracy depends sharply on calibration — drop calibration to "poor" at int4 and the accuracy falls off a cliff. That cliff is the accuracy budget made visible.

Precision dial — bytes vs the accuracy budget

Lower precision halves bytes and lifts throughput, but spends accuracy. The knob that breaks: set precision to int4 and slide calibration toward poor — the accuracy drop blows past the budget while int8 at the same calibration stays fine. Good calibration tames int4; nothing tames a too-low bit-width with bad scales.

precision: calibration quality: 85 / 100 weight count: 7 B params

model size

—

HBM bytes / decode step

—

est. speedup vs fp16

—

accuracy drop

—

accuracy retained—

—

The shape of the result is the lesson: fp16 is the safe, expensive baseline; int8 buys ≈2× for a fraction of a point of accuracy that good calibration nearly erases; int4 buys ≈4× on size but lives on the edge — calibrate it well (GPTQ/AWQ-grade) and it holds; calibrate it poorly and it falls off the cliff. The compiler's job is to ride that curve to the lowest precision that stays inside the accuracy budget, never past it.

Failure modes & checklist

Failure modes

Per-tensor scale on outlier-heavy activations. One fat channel forces a coarse scale on all. Signal: int8 accuracy collapses on transformers; switching to per-channel weights / SmoothQuant or fp8 recovers it.
Dequant not fused into the epilogue. A separate dequant kernel reads int32 from HBM and writes fp16. Signal: quantization made it slower — the extra round trips ate the win (lesson 05).
int4 with PTQ min–max calibration. Too few bits and naive scales. Signal: the accuracy cliff — large quality drop where GPTQ/AWQ would have held; the widget's broken knob.
Quantizing the delicate ops. int8 on softmax / layernorm / final logits. Signal: big accuracy loss for tiny byte savings — those ops were never the heavy GEMMs.
No accuracy gate in the pipeline. Shipping a quantized model without measuring drift. Signal: a silent quality regression in production that the bytes/latency dashboard never showed (forces lesson 17).

Checklist

Profile the regime first. Decode-bound → weight-only int4/int8 (bandwidth). Compute-bound prefill/training → int8/fp8 weight+activation (throughput).
Per-channel for weights. One scale per output channel — nearly free metadata, big accuracy win.
Calibrate, don't min–max. Use percentile or MSE/entropy on a representative set; reach for QAT only when PTQ blows the budget.
Always fuse the dequant. Fold the per-channel scale + bias + activation into the GEMM epilogue; never a standalone dequant kernel.
Keep sensitive ops fp16. Anchor softmax/norm/logits at fp16; quantize the heavy linear layers — it's precision assignment, not all-or-nothing.
Set an accuracy budget and gate on it. Define the tolerated drop up front and measure every quantized build against an fp reference.

Checkpoint

Try it

Open the widget at 7 B params, int8, calibration 85. Note model size (≈7 GB, half of fp16's ≈14 GB), the ≈2× speedup, and the small accuracy drop. (1) Slide calibration down to 0: int8's drop grows but stays modest — int8 is forgiving. (2) Switch to int4 at calibration 85: size drops to ≈3.5 GB, speedup climbs, accuracy drop is still acceptable — good calibration holds int4. (3) Now slide calibration toward 0 at int4: watch the accuracy drop fall off the cliff and the verdict turn red — same bit-width, ruined by bad scales. (4) Confirm by hand: at int4, weight bytes are ¼ of fp16, so a weight-bandwidth-bound decode step should be ≈4× faster — does the HBM-bytes KPI match? That gap between int8's gentle slope and int4's cliff is the accuracy budget you must respect.

Where this points next

You have now built the last of the make-it-fast passes: a graph that is captured, canonicalized, fused, planned, laid out, scheduled, autotuned, coverage-complete, and now quantized — a cache of compiled kernels, each in the precision the accuracy budget allows, with their dequant folded into the epilogue. But that cache is inert. Nothing in lessons 02–15 said what happens at run time: who allocates the buffers the memory planner sized, who dispatches the kernels in order, who keeps lesson 01's launch-overhead ceiling from reappearing now that you have a pile of tiny quantized kernels to fire. The compiler produced an artifact; an executor has to run it, with low overhead, honoring the plan. That is the other half of the compile-time/run-time split: 16 · The runtime & executor.

Takeaway

Every pass so far moved fp16 bytes; the biggest untouched lever is to make the numbers narrower. Quantization stores weights/activations as int8/int4 or fp8 plus a scale (and zero-point), recovering x ≈ scale · (q − zp) — halving HBM traffic (≈14 GB → ≈7 GB at int8 for a 7 B model), doubling tensor-core throughput (≈990 → ≈1,980 TFLOP/s for fp8/int8), and directly attacking the weight-bandwidth-bound decode GEMM with weight-only int4. The compiler represents it as QDQ nodes, picks per-channel (not per-tensor) scales by PTQ calibration (min–max/percentile/MSE) or QAT (fake-quant + straight-through estimator), then propagates the quant, fuses the dequant into the GEMM epilogue (lesson 05) so the int8 matmul costs zero extra round trips, and solves a precision-assignment problem (the dual of layout, lesson 07) that keeps softmax/norm/logits in fp16. fp8 (e4m3/e5m2) keeps an exponent for training; int4 weight-only (GPTQ/AWQ) and 2:4 sparsity push further. The defining twist: this is the one pass that is deliberately not bit-exact — it preserves the math approximately, on an accuracy budget, and outliers are where that budget breaks. With kernels generated and quantized, the artifact is built — but inert, which forces the runtime that launches it.

Interview prompts

Why does quantization help both decode and prefill, by different mechanisms? (§1 — decode is weight-bandwidth-bound, so halving weight bytes (int8/int4 weight-only) nearly halves latency; prefill/training is compute-bound, so int8/fp8 tensor cores at ≈2× throughput lift the FLOPs. Bytes for one, FLOPs for the other.)
What is a QDQ node and what does the dequant fuse into? (§2,4 — quantize (fp→int) and dequantize (int→fp) nodes wrap a low-precision region; the dequant (per-channel scale multiply + bias + activation) fuses into the GEMM epilogue while the int32 tile is hot, so it costs zero extra HBM traffic — lesson 05. Without that fusion, quant can be a net loss.)
Per-tensor vs per-channel scale — why does per-channel matter for weights? (§2 — one fat-range output channel forces a per-tensor scale that wastes most int8 codes on the other channels; a per-channel scale vector (negligible metadata) lets every channel use its full range, which is why int8 linear layers hold accuracy.)
PTQ vs QAT, and what is the straight-through estimator? (§3 — PTQ picks scales from calibration on a small set with no retraining (fast, good for int8); QAT inserts fake-quant in the forward and trains against it. quantize has zero gradient, so the STE pretends rounding is the identity (gradient 1) on the backward so gradients flow. QAT recovers accuracy PTQ can't, for int4/sensitive models.)
How is quantization a counterexample to "preserve semantics"? (§4 — every other pass is a provable no-op on the numbers; quantization is deliberately approximate, trading a bounded, measured accuracy drop for bytes/FLOPs. The invariant changes from bit-exact to "within the accuracy budget," which is why it needs a debugging discipline (lesson 17).)
Why is int8 activation quant harder than weight quant, and how does fp8 help? (§5 — activations have outlier channels 10–100× the rest; a per-tensor int8 scale clips everything else. fp8 keeps an exponent so it spans wide dynamic range without clipping, which is why fp8 (not int8) is viable for training activations; SmoothQuant migrates scale to weights as another fix.)
Why is int4 weight-only the right tool for LLM decode, and what's the risk? (§1,5 — decode reloads the whole weight matrix per token (bandwidth-bound) with a tiny matmul, so int4 weights ≈4× less weight bytes ≈ ≈4× faster decode, computing in fp16. Risk: 4 bits is near the accuracy cliff, so it needs GPTQ/AWQ-grade calibration — naive min–max falls off.)