Mixed precision — bf16, fp16, fp8

Lower precision halves memory and roughly doubles tensor-core throughput. Doing it without losing convergence is a small set of careful rules — and the rules are different for each format.

What "precision" buys, exactly

A floating-point number has three pieces: a sign bit, an exponent (range), and a mantissa (precision within an exponent). The IEEE/standard ML formats:

Two observations make most of the rest of this lesson:

• bf16 and fp32 have the same exponent (8 bits). Same range; bf16 just has worse precision within each exponent. This is why bf16 nearly drop-in replaces fp32 — gradients can't overflow or vanish.
• fp16 has only 5 exponent bits. Its real problem is the bottom of the range: small gradients (≈ 10⁻⁷ and below) underflow and flush to zero, so updates silently vanish. The narrow 5-bit exponent also means very large values can overflow past ~65k, but underflow is the dominant failure mode. fp16 needs a "loss scaler" to push gradients up into a healthy range. bf16, with fp32's full 8-bit exponent, doesn't.

There's also a hidden default on the matmul path: TF32. Since Ampere (A100), tensor cores run "fp32" matmuls in TF32 mode by default — a 19-bit internal format with fp32's 8 exponent bits but only 10 mantissa bits, giving near-fp32 throughput on the tensor cores at fp16-like precision. So even code that never touches autocast is already running its matmuls in reduced precision unless you explicitly set torch.backends.cuda.matmul.allow_tf32 = False.

2D · bit-format visualiser

Side-by-side bit layouts. Type a number and watch how each format encodes it: the sign bit, the exponent bits (shifted by bias), and the mantissa bits. Notice how bf16 truncates fp32's mantissa but keeps the exponent, while fp16 keeps mantissa precision at the cost of range.

Why bf16 won

In 2018, the only widespread 16-bit format on GPUs was fp16, and people invented elaborate machinery (loss scaling, gradient unscaling, dynamic scaler adjustment) to make it work. NVIDIA's A100 (2020) added bf16 hardware support. By 2022, bf16 became the default for training. The reason is just the range: with bf16 you can take any fp32 training recipe, swap the activations and gradients to bf16, and it works. With fp16 you have to manage a scaler — and even then, the loss landscape's tail events sometimes overflow.

The cost of bf16 is precision. With 7 mantissa bits, accumulating a long sum loses information rapidly — adding a small number to a large one rounds to the large one (the small number is below the LSB of the large one's mantissa). The workaround is one of the foundational rules:

The master-weights trick (recap from lesson 01)

Even with accumulate-in-fp32 inside ops, the parameter update step still loses precision in bf16:

For a typical LR η = 10⁻⁴ and gradient g ~ 1, the update is 10⁻⁴. A bf16 weight near 1.0 has LSB ≈ 2⁻⁷ ≈ 8 × 10⁻³. The update rounds to zero. Updates simply don't accumulate, training stalls.

The fix: keep a master copy of weights in fp32. The bf16 weight is a cast of the master. Forward and backward use bf16. The optimizer step updates the fp32 master, then the bf16 working copy is refreshed from it. This is the "12 bytes of optimizer state + master" in lesson 01's bytes-per-param accounting.

The standard alternative — the one that lets some recipes drop the fp32 master copy entirely — is stochastic rounding. Instead of round-to-nearest, you round up or down with probability proportional to where the value falls between the two representable bf16 numbers: an update that lands 20% of the way to the next LSB rounds up 20% of the time. The 10⁻⁴ update that deterministically rounds to zero now nudges the weight upward one LSB on roughly 10⁻⁴ / (8 × 10⁻³) ≈ 1.3% of steps, so in expectation the small update is preserved even though no single step can represent it. That makes bf16-only — and increasingly fp8-only — training viable without a separate fp32 parameter copy, trading a little per-step noise for the saved bytes.

fp16 and the loss scaler — the historical wart

fp16's main problem is underflow: small gradients fall below fp16's smallest normal value (~6 × 10⁻⁵) and flush to zero, so updates silently vanish. (Overflow past ~65k is the secondary, rarer failure.) The fix is the loss scaler:

1. Multiply the loss by some scale factor S (typically 2¹⁵) before backward.
2. Backward proceeds; gradients are S× their natural magnitude — but now in a healthy fp16 range, away from the underflow floor.
3. Before the optimizer step, "unscale" by dividing gradients by S.
4. Optimizer step in fp32 as usual.

If any gradient overflows to inf during step 2, PyTorch's GradScaler detects it, skips the step entirely, and halves S for next time. If many steps pass without overflow, S is doubled back up. This is the dynamic loss-scaling algorithm. bf16 deletes all of this machinery.

fp8 — two formats, training vs inference

fp8 lives in the next-precision-down regime. NVIDIA's Hopper (H100) and Blackwell (B100/B200) have tensor cores that consume fp8 inputs and produce fp16 or fp32 accumulators. Two fp8 formats:

• E4M3 (4 exponent, 3 mantissa). Range ~448, more mantissa. Used for weights and activations.
• E5M2 (5 exponent, 2 mantissa). Range ~57k, less mantissa, more like fp16 in range. Used for gradients.

The asymmetry is intentional. Weights and activations have small dynamic range (post-LayerNorm), so a tight-range high-precision format works. Gradients can span many orders of magnitude (back-propagated through long networks), so a wide-range low-precision format works. This E4M3 (forward) × E5M2 (gradients) split is NVIDIA's Transformer Engine convention. Not everyone follows it: DeepSeek-V3 reported using all-E4M3 everywhere — including gradients — and recovered the lost range with fine-grained (per-tile / per-block) scaling rather than E5M2, explicitly moving away from the wide-range gradient format.

fp8 training combines:

• fp8 matmul with an fp32 accumulator. The Transformer Engine recipe is E4M3 × E4M3 for the forward and E4M3 × E5M2 for the backward; DeepSeek-V3 instead uses E4M3 throughout with finer-grained scales.
• fp32 master weights and Adam state still dominate the bytes — that's 12 bytes per param (4 master + 4 m + 4 v) on its own. Adding fp8 weights + fp8 grads gives ~14 B/param, plus a handful of bytes for per-tensor scale factors. So fp8 training trims roughly 2 B/param off the lesson-01 mixed-precision figure, not half of it — the optimizer state is what makes the parameter budget mostly format-independent.
• Scaling factors that adjust dynamically (otherwise the limited range causes weight or activation overflow). The granularity varies: classic Transformer Engine uses one scale per tensor; DeepSeek-V3 uses much finer per-tile / per-block scales.
• Some "stay in higher precision" ops: norms, the LM head, the embedding table.

The serving win is bigger than the training win: fp8 weights are half the size of bf16, so HBM traffic per decode token halves, so decode latency halves (lesson 14's HBM-bound math). H100's fp8 throughput is roughly 2× bf16; B200 pushes it further with fp4 in places.

2D · E4M3 vs E5M2, the same value placed on each

Two log-scale axes side-by-side. Every representable positive value of each fp8 format is shown as a dot. Drop a value with the slider and see which format keeps it in range and what the rounding error looks like. The two formats handle out-of-range values differently: E5M2 has an Inf encoding and saturates to ±inf, while OCP E4M3 has no infinity — its largest value is 448 (it reserves codes only for NaN), so big values saturate to 448. E4M3 has more dots clustered in a tight range; E5M2 covers more orders of magnitude but spaces them further apart.

Autocast — the developer ergonomic

You don't manually cast every tensor. PyTorch's torch.autocast(device='cuda', dtype=torch.bfloat16) context manager wraps ops in a dispatcher layer (the "Autocast" key from lesson 16) that automatically casts inputs to the right type:

• bf16-safe ops (matmul, conv, attention): inputs cast to bf16.
• fp32-needed ops (softmax, LayerNorm, sum reductions, loss functions): inputs cast to fp32.
• Preserved ops: simple elementwise pass-through, follow input type.

The cast is cheap (a fused kernel can do it for free), but the rules are opinionated: PyTorch decides for you which ops are precision-sensitive. Most of the time it's right. For fp8 training, NVIDIA's Transformer Engine extends this with per-tensor scales.

Animated · autocast data flow through one step

Forward + backward of a single layer with autocast enabled. Each arrow is a tensor; its color encodes the dtype the framework actually stores. Watch the matmul take bf16 inputs but produce an fp32 accumulator before casting back, the gradient flowing back in bf16, and the optimizer step happening on the fp32 master weights.

The numerics traps that survive autocast

Even with autocast on, a few classes of bugs reappear:

• Reductions over many elements in low precision. A custom x.sum(dim=-1) on a bf16 tensor with 10k elements drops bits at the end of the accumulation. Cast to fp32 first.
• Pseudo-fp32 work in user code. Anything you do with torch.tensor directly, especially RoPE / position-embedding math, may not be autocast-aware. Stay in fp32 for the math, cast result back.
• Layer norms. The accumulator inside LayerNorm should be fp32 — modern PyTorch does this, but third-party implementations occasionally don't.
• Custom CUDA / Triton kernels. You're responsible for the accumulator dtype yourself.

The decision table

Interactive · feel the range & precision tradeoff

Pick a value and a format; the widget shows where it lands (the nearest representable number), the relative error, and what happens to the sum a + b when b ≪ a. Try entering 65000 in fp16: it survives. Try 70000: overflows to inf. Try summing 1.0 + 1e-4 in bf16: the small term disappears.