Profiling, MFU & finding the bottleneck

The whole series taught you to predict where the bottleneck lives. This lesson is how you measure it on a real run and score the prediction. MFU is the single number; the trace is how you find which named cause — each from an earlier lesson — is eating the gap.

The one number, and what it really asks

Every previous lesson ended with a hammer: FSDP for the memory wall, overlap for the comm wall, fusion for memory-bound kernels, CUDA graphs for launch overhead. You now have a toolbox and a habit of guessing which tool a given run needs. This lesson closes the loop. There is exactly one scalar that scores the whole run, and it has a name: Model FLOPs Utilization — MFU.

The definition starts from the same accounting we did in lesson 01. A transformer step does, for the useful forward+backward math, about 6 · N FLOPs per token, where N is the parameter count (2 for forward, 4 for backward — the 6ND rule). So the useful work per training step is:

where B · T is the number of tokens processed in one optimizer step (global batch × sequence length, summed across all data-parallel replicas). Divide that useful work by what the hardware could have done in the wall-clock time the step actually took, across every GPU you paid for:

That denominator is the honest one. peak_FLOPS is the chip's datasheet number — for an H100 SXM, 989 TFLOP/s dense bf16 (the marketing "1979 TFLOP/s" is the 2:4-sparse figure; never use it for MFU). Multiply by num_gpus and by the measured step_time and you have the total FLOP-budget the cluster had available during that step. MFU is the fraction of that budget that went to math your model actually needed.

Three rules of thumb worth burning in: 35–45% MFU is typical for well-tuned large-model training, >50% is very good (frontier labs publish numbers in the 40–55% range), and 100% is physically impossible. Why impossible? Because the 6ND count ignores everything that isn't a big matmul — attention softmax, LayerNorm, the optimizer step, embeddings, the LM head — and those run at well below peak because they are memory-bound (next section). Even a perfect, zero-overhead, zero-communication run spends real time in kernels the tensor cores can't keep busy. MFU is asymptotic to something like 60–70%, not 100%, on a transformer.

It helps to see where the FLOPs actually go. In the 6ND accounting, the dominant terms are the four big GEMMs per layer — the QKV projection, the attention output projection, and the two MLP matmuls — and these are exactly the kernels that sit comfortably above the roofline ridge and run near peak. But a real step also pays for things the 6ND count politely ignores: the softmax inside attention, two LayerNorms per layer, the residual adds, the SwiGLU/GELU nonlinearity, dropout, the embedding lookup, the final LM-head projection over a 100k+ vocabulary, and the optimizer update over every parameter. Individually each is cheap in FLOPs; collectively they are a meaningful slice of wall-clock precisely because they run at single-digit-percent of peak. That is the structural reason MFU tops out well below 1.0: the denominator counts time the numerator's 6ND formula refuses to credit.

Make the ceiling concrete by budgeting the wall-clock of one clean step — no exposed comm beyond what's unavoidable, no data stall, just the irreducible mix. Account for where the seconds go, not the FLOPs:

• ~75% in the big GEMMs at near-peak. The four projections and the two MLP matmuls dominate FLOPs and run at ~90–95% of peak when well-tiled. Call it ~0.75 of wall-clock delivering ~0.92 efficiency → contributes 0.75 × 0.92 ≈ 0.69 to MFU.
• ~15% in memory-bound ops at ~10% of peak. Softmax, the two LayerNorms, residual adds, the nonlinearity, dropout, and the optimizer update are all far below the ridge (next section). They take ~0.15 of wall-clock but, running at ~0.10 efficiency, contribute only 0.15 × 0.10 ≈ 0.015 to MFU — they spend time, not useful FLOPs.
• ~10% in exposed comm + launch gaps at ~0 useful FLOPs. Even a well-overlapped run leaves a residue — a TP AllReduce on the critical path, the tail of an FSDP AllGather, kernel-launch gaps between small ops — that the tensor cores sit idle through. Contributes ≈ 0 to MFU.
• The GEMMs themselves leave a little on the table. Wave quantization (a GEMM whose tiles don't evenly fill the SMs leaves a partial final wave) and imperfect tiling are already folded into the ~0.92 above; that is the gap between the 75% slice and a perfect 100%-of-peak slice.

Summing the contributions: 0.69 + 0.015 + 0 ≈ 0.70. That is the ~60–70% ceiling, derived: a transformer step is ~three-quarters near-peak GEMM and ~one-quarter time the tensor cores can't be kept busy, so even the perfect run asymptotes to ~0.7, not 1.0. The denominator counts all the wall-clock; the numerator credits only the GEMM-shaped fraction of it.

One more subtlety on the numerator. The strict 6ND count is the matmul-only approximation; some teams add the attention score-and-context FLOPs (the QKᵀ and ·V products, which scale with T² rather than T) into "useful" work, which nudges the reported MFU up at long sequence lengths. Both conventions are defensible — what is not defensible is comparing your MFU against someone else's without knowing which convention each of you used. When you read "we hit 52% MFU" in a paper, the first question is always "6ND, or 6ND plus attention?" The widget below uses the clean 6ND convention so the numbers are comparable across model sizes.

MFU vs HFU — the recompute distinction

There is a second utilization number, and confusing the two is the most common profiling mistake. HFU — Hardware FLOPs Utilization — counts every FLOP the hardware executed, including the ones you did twice on purpose.

Recall lesson 10: activation checkpointing throws away activations in the forward pass and recomputes them during backward to save memory. That recompute is real forward work the GPU does a second time — roughly +33% more FLOPs (one extra forward on top of the fwd+bwd, so 6ND becomes ~8ND of executed work). Those extra FLOPs are hardware work but not useful model work — the gradient would be identical without them; you only pay them to fit in memory.

So with full activation recompute, HFU ≈ 1.33 × MFU. The practical reading:

• MFU answers "how efficiently is my run turning GPU-seconds into model progress?" — the number you report and budget against. Tokens/dollar lives here.
• HFU answers "how busy are the tensor cores?" — the number that tells you whether the chip is saturated. If HFU is ~70% but MFU is ~50%, the gap is recompute, and that's a memory-vs-compute trade you chose (lesson 10), not a bug to chase.

The roofline — the per-kernel lens

MFU scores the whole run. To understand why a kernel is slow you drop to the roofline from lesson 01. Every kernel is bound by one of two ceilings: how fast you stream bytes from HBM, or how fast the tensor cores chew FLOPs already in SRAM. The crossover is the ridge point — the arithmetic intensity (FLOPs per byte) at which the two ceilings meet:

That number — ~295 FLOP/byte for an H100 (989 TFLOP/s over 3.35 TB/s of HBM3) — is the line that decides everything. A kernel whose arithmetic intensity sits above 295 is compute-bound: it can in principle reach peak FLOP/s. A kernel below 295 is memory-bound: it will finish reading its bytes before the tensor cores are busy, and no amount of faster math helps.

Work one kernel explicitly to make the ridge concrete. Take an MLP matmul [M, K] × [K, n] with M = B·T rows, K = d contraction, n = 4d output. The FLOPs are 2·M·K·n. The bytes moved (bf16, 2 B/elem) are roughly the two inputs plus the output: 2·(M·K + K·n + M·n). The arithmetic intensity is the ratio, and for a square-ish problem it grows with M — i.e. with how many tokens you push through at once. Plug in real numbers with K = d = 8192, n = 4d = 32768. The intensity is 2MKn / [2(MK + Kn + Mn)] = MKn / (MK + Kn + Mn).

At M = 1 (single-token decode) the matmul is a matrix-vector product: the K·n weight term (~268M elements) swamps the denominator, so you read the entire weight (megabytes) to do 2·K·n FLOPs — intensity ≈ 1, ~300× below the ridge, deep on the orange slope. Now push the batch to a training tile:

The same weight is read once but reused across 4096 rows, the M·K and M·n activation terms grow while the dominant K·n weight term stays fixed, and intensity vaults to ~2500 — an order of magnitude above the 295 ridge, firmly compute-bound. Between M=1 and M=4096 the kernel crosses the ridge: that is the whole story in two substitutions. This single ratio — reuse of the weight across the batch dimension — is why training prefers big tiles and why decode is doomed to be memory-bound (the entire content of lesson 14 and the motivation for continuous batching).

Two consequences you will see in every trace:

• Big training GEMMs sit above the ridge. A [B·T, d] × [d, 4d] MLP matmul at training batch sizes has intensity in the hundreds — comfortably compute-bound, running near peak. This is the work MFU rewards, and it's why big batches help.
• Decode and elementwise ops sit far below it. A single-token decode matmul reads the whole weight matrix to do one matrix-vector product: intensity ≈ 2 FLOP/byte (lesson 14's punchline). LayerNorm reads a tensor, does a handful of FLOPs per element, writes it back — a few FLOP/byte. These run at a tiny fraction of peak no matter what, which is exactly why kernel fusion (lesson 19) pays: fusing LayerNorm+GELU+bias into one kernel doesn't add FLOPs, it removes round-trips to HBM, moving the kernel up its memory-bound slope.

Why MFU < 100% — the subtraction list

MFU is a budget, and the gap from 100% (really, from the ~60–70% ceiling) is a sum of named subtractions. Each one is a lesson you already read; the trace tells you which subtraction is large right now:

The mental model: MFU is the scoreboard that tells you how much total headroom remains. The trace tells you which row above is currently the big one. You don't optimize the list top-to-bottom; you profile, find the dominant gap, fix that, re-measure, and repeat. Optimizing a subtraction that's already small is wasted work.

The profiler — what a trace actually is

Two tools cover almost everything. torch.profiler emits a Chrome trace (open it in chrome://tracing or Perfetto) and gives you per-operator CPU and CUDA timings with autograd attribution. Nsight Systems (nsys) gives a lower-level system timeline — NVTX ranges, NCCL kernels, CUDA API calls, even the PCIe/NVLink traffic — and is the tool of choice when you suspect communication or driver-level stalls.

The minimal torch.profiler incantation, which you should keep in muscle memory:

import torch
from torch.profiler import profile, ProfilerActivity, schedule

sched = schedule(wait=1, warmup=1, active=3, repeat=1)  # skip warmup steps

with profile(
        activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
        schedule=sched,
        on_trace_ready=torch.profiler.tensorboard_trace_handler("./tb"),
        record_shapes=True, with_stack=True) as prof:
    for step, batch in enumerate(loader):
        train_step(batch)
        prof.step()                      # advances the schedule

# prints the kernels eating the most GPU time:
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=15))

Three details in that snippet are load-bearing and routinely gotten wrong. First, the schedule with a warmup phase: the first few steps of any run pay one-time costs — CUDA context creation, the caching allocator growing its pools (lesson 18), cuDNN/cuBLAS algorithm autotuning, and (if you use it) torch.compile's first-call compilation (lesson 21). Profiling those steps gives you a trace dominated by costs you'll never pay again; always skip them. Second, record_shapes and with_stack are not free — stack capture in particular adds real overhead and can distort the very timings you're measuring, so turn them on only when you specifically need to attribute a kernel back to a Python line. Third, CUDA is asynchronous: the CPU launches a kernel and moves on, so a naive CPU-side timer measures launch time, not execution time. The profiler's CUDA activity lane records the real on-device duration; if you ever hand-roll timing, you must torch.cuda.synchronize() or use CUDA events, or your numbers are fiction.

The key_averages().table() print is the fast first read before you even open the visual trace. Sorted by cuda_time_total, the top rows tell you where the GPU spends its seconds: if the top entries are ampere_sgemm/cutlass GEMM kernels, your time is going to matmuls (healthy — you're compute-bound on the right thing). If the top entries are elementwise, vectorized_, reduce, or a long tail of tiny named ops, you're memory-bound and fusion is the lever. If a nccl kernel (e.g. ncclAllReduce) is near the top and the run isn't overlapping it, communication is exposed. The table answers "what" in five seconds; the visual trace answers "why is the GPU idle," which the table cannot show because idle time has no kernel to attribute.

Whatever the tool, a trace is the same picture: time on the x-axis, and a set of horizontal lanes stacked vertically. The four lanes that matter:

• CPU / Python lane — the training loop, autograd, and the op-dispatch + kernel-launch calls (lesson 16's dispatcher lives here). When this lane is the only busy one, the CPU is the bottleneck.
• GPU compute stream — the actual kernels (GEMMs, attention, elementwise). The blocks here are what you want packed wall-to-wall.
• NCCL / comm stream — collective kernels (AllReduce, AllGather, ReduceScatter, AllToAll), usually on their own CUDA stream so they can overlap with compute.
• The gaps — the white space on the GPU compute stream. Every gap is the bottleneck for the duration of that gap, and its signature — what's busy on the other lanes while compute is idle — is the diagnosis.

The MFU calculator

This is the scoreboard. Set your run's parameters and read the MFU live. The lower stacked bar lets you attribute the gap from peak to named causes (bubble, exposed comm, data stall) — drag them until the bar's leftover "unexplained" slice shrinks to zero, which is the same accounting you do when you read a trace. The KPIs report achieved TFLOP/s per GPU, MFU %, and how much headroom remains to the 50% "very good" mark.

Play with it: bump the micro-batch from 1 and watch MFU climb (you're walking up the roofline slope). Push the step time up by 2× and watch it halve — that's what an unhidden AllReduce or a starving dataloader does to the same useful work. The headroom KPI is the prize: it's the number a quarter of optimization work is chasing.

Reading a trace — the diagnostic decision tree

This is the heart of the lesson. You have the scoreboard (MFU is low); now you open the trace and ask one question repeatedly: while the GPU compute stream is idle, what is busy? The answer is the signature, and each signature has exactly one first response:

Walk the five rows, because the distinctions are exactly what's hard the first time you stare at a real trace:

• Launch-bound. The tell is the ratio of lane activity: the CPU lane is a dense wall of small dispatch calls while the GPU compute lane is mostly red gaps punctuated by tiny kernels. The CPU literally cannot issue launches fast enough to keep the GPU fed — each kernel runs for, say, 8 μs but takes 12 μs of Python+dispatch to launch (lesson 16's overhead). This is the dominant failure mode of decode (tiny per-token kernels) and of any model with thousands of small ops. The fix is to stop launching per-op: capture the whole step as a CUDA graph and replay it (lesson 22), or let torch.compile fuse and cut the launch count (lesson 21).
• Comm-bound. The tell is vertical alignment: a fat orange NCCL kernel on the comm lane with a red gap on the compute lane directly above it, for the same time window. The collective is on the critical path — compute is waiting for it. The usual culprits are the TP AllReduce that sits in the forward critical path (hard to hide — lesson 13 explains why), an FSDP AllGather that wasn't prefetched early enough, or a gradient AllReduce that's too small to overlap. Levers: prefetch earlier, grow the bucket so the collective is bandwidth-bound rather than latency-bound, and keep collectives intra-node where NVLink is ~18× faster than IB (lesson 03).
• Data-starved. The tell is the absence of activity: compute idle, comm idle, and the CPU lane showing one long blocking call instead of its usual dispatch ticks. Nothing on the GPU is waiting on the GPU — it's waiting on the host to produce the next batch. Often the H2D copy isn't pinned, or there are too few dataloader workers, or the dataset decode (image resize, tokenization) is CPU-bound. Lesson 25 is the whole toolbox; the first move is almost always "add workers and prefetch."
• Memory-bound but packed. A fully packed compute lane feels like success — the GPU is never idle! — but if those packed kernels are memory-bound elementwise ops or skinny matmuls, you're busy doing low-FLOP work and MFU stays low. The trace alone won't catch this (no gaps to see); you catch it by cross-referencing with MFU and with the op table, where the top entries are elementwise/reduce kernels rather than GEMMs. That's the case the roofline diagnoses and fusion fixes: the lane is full, just full of the wrong thing.
• Pipeline bubble. The giveaway the others lack is periodicity: the gaps repeat with the cadence of the pipeline fill/drain, identical every iteration — unlike a one-off allocator hiccup (lesson 18) or a stochastic data stall. Once you see the same-shaped gap at every iteration boundary, you know it's structural, and the fix is a better schedule — interleaved 1F1B or zero-bubble (lesson 07) — not anything you can tune away with workers or buckets.

The trace reader

This is the payoff. Pick a scenario; the canvas draws a synthetic three-lane trace (CPU/launch, GPU-compute, GPU-comm) with that scenario's characteristic block-and-gap signature. It computes an illustrative utilization and prints the diagnosis plus the lesson number to go fix it. Compare "healthy" against the four failure modes — the goal is to make the signatures recognizable on sight, so that when you open a real Perfetto trace your eye already knows what it's looking for.

Notice how different the five pictures are. Healthy: compute packed, comm tucked underneath it, CPU launching ahead. Launch-bound: tiny compute blocks marooned in a sea of red idle while the CPU lane churns. Comm-bound: a fat orange NCCL block on the comm lane with the compute lane idle right above it. Data-starved: everything idle, including comm, while the CPU blocks on the loader. Bubble: the regular, periodic gaps. Once these shapes are in your head, diagnosing a real trace is pattern-matching.

Five ways to compute MFU wrong

Because MFU is the number you report, budget against, and compare to other teams, it is worth knowing how it gets miscalculated — every one of these produces a number that looks plausible and is wrong:

• Using the sparse peak. The H100 datasheet headlines ~1979 TFLOP/s — that is the 2:4 structured-sparse figure. Dense bf16 is 989. Using the sparse number halves your apparent MFU and makes a healthy 44% run look like a broken 22%. Always use the dense peak for the precision you're actually training in.
• Forgetting that the precision changes the peak. fp8 on an H100 doubles the dense peak again (~1979 dense fp8). If you train in fp8 (lesson 17) you must put the fp8 peak in the denominator, or your MFU is inflated 2×.
• Counting recompute as useful (MFU vs HFU). If you compute the numerator with the recompute FLOPs included, you've computed HFU and labeled it MFU. The two differ by ~1.33× under full checkpointing, which is a huge gap to misattribute.
• Timing only one step, including a stall. Step times are noisy — a checkpoint write, an allocator defrag, or a slow batch can double a single step. Measure the median over many steady-state steps, after warmup, not a single sample.
• Getting tokens-per-step wrong with gradient accumulation. With accumulation (lesson 11), the optimizer step covers accum_steps × micro_batch × seq × dp tokens, and the step_time in the denominator must be the time for the whole accumulated step, not one micro-batch. Mixing the micro-batch token count with the full-step time (or vice versa) is the most common off-by-a-factor in the whole calculation.

The workflow, in one breath

1. Measure MFU. One number. If it's 40%+, you're probably fine — go do something else. If it's low, continue.
2. Open a trace (torch.profiler for op-level, nsys for system-level). Find the largest gap on the GPU compute stream.
3. Read the signature — what's busy while compute is idle — and match it to the decision tree. That names the cause and the lesson.
4. Apply that lesson's fix. Just the one for the dominant gap.
5. Re-measure MFU. Did it move? If the dominant gap shifted to a new signature, repeat from step 2.

The discipline that matters: never optimize before profiling. Half of "obvious" optimizations target a subtraction that was already small, and you can't know which subtraction is large without the trace. The series gave you the hammers; this lesson is the X-ray that tells you which nail to hit.