Part I — The gap

The eager baseline and its ceiling

Lesson 00 made the claim that eager execution wastes the chip: one tidy line y = gelu(x @ w + b) becomes several kernels and several round trips to HBM, and the hardware wanted one fused kernel. A claim is not a number. Before you can trust a single compiler pass, you have to price the thing it replaces — measure where eager's time actually goes — so every later pass has a concrete number to beat. This lesson builds that cost model and shows the two distinct ceilings eager slams into.

The previous step left this broken

Lesson 00 asserted that the gap between a hardware-agnostic op graph and a chip that only rewards big fused kernels is where performance leaks. But it argued by picture. You cannot optimize what you have not priced, and you cannot tell which optimization to reach for until you know whether you are losing time to launching kernels or to moving bytes — they are different diseases with different cures. So we put a stopwatch on eager: what does one PyTorch op actually cost, and where does the millisecond go when you run a whole block of them one at a time?

Linear position

Forced by: you cannot optimize what you have not measured — lesson 00's "eager wastes the chip" needs a cost model with real microseconds and bytes.
New idea: the eager cost model = per-op dispatch overhead (Python + the C++ dispatcher, ~1–8 µs/op) + kernel-launch latency (~3–10 µs) + the usually-dominant HBM round trip (bytes ÷ bandwidth). For the memory-bound ops that fill a model, total time ≈ Σ round trips. From it fall the two ceilings: launch-bound (many tiny ops, time = launches) and bandwidth-bound (each op re-reads HBM, time = traffic).
Forces next: both ceilings are whole-program faults — fewer launches and one shared round trip can only be found by seeing all the ops at once, which eager Python never hands you. You must first capture a graph (lesson 02).

The plan

Five moves. (1) Trace one x @ w from Python down to the kernel launch and HBM, naming each layer's cost. (2) Put microseconds and bytes on it: dispatch ~1–8 µs, launch ~3–10 µs, a 256 MB activation round trip @3.3 TB/s ≈ 160 µs. (3) Recap arithmetic intensity and the ridge point — why most non-GEMM ops sit far below it and are memory-bound. (4) Name the two ceilings — launch-bound vs bandwidth-bound — and why CUDA graphs fix only the first. (5) Lay out the scoreboard (launches · bytes · peak memory) the rest of the track fills in. Then drive the profiler until the same block flips from bandwidth-bound to launch-bound.

1 · One op, end to end: Python → dispatcher → ATen → kernel → HBM

Start with the smallest unit: you write y = x @ w and some time later an SM finishes a tensor-core multiply. Between those two events sits a fixed chain of CPU-side work that runs on every single op, no matter how big the tensors are. Dispatch is the act of resolving which concrete kernel to run: PyTorch is a thin Python facade over a large C++ dispatcher that, on each call, looks up the right implementation for this device, dtype, memory layout, and autograd state. The library of those concrete tensor kernels is ATen (PyTorch's C++ tensor operator library — "A Tensor library"); the dispatcher's job is to map (op, dispatch-key) → an ATen function pointer.

Python bytecodeCPython resolves __matmul__, pushes args, crosses the pybind11 Python↔C++ boundary. ~1–2 µs.

autograd layerIf requires_grad, record a node for the backward and bump version counters. ~1–3 µs.

ATen dispatchLook up (matmul, CUDA·bf16·strided) → kernel fn ptr; backend (cuBLAS) picks an algorithm. ~1–4 µs, cached after first call.

kernel launchcudaLaunchKernel enqueues grid + block + args onto a stream. ~3–10 µs of CPU + driver.

HBM round tripThe kernel reads its inputs from HBM, computes, writes the result back. Bytes ÷ bandwidth — usually the dominant term.

The first four steps are the per-op tax: roughly 5–25 µs of CPU-side work, fixed regardless of tensor size. For a 4,096×4,096 GEMM that tax is under 1% of the runtime and vanishes. For a tiny pointwise add on a batch of 1, that tax is the runtime. That single fact — a fixed tax paid per op, against a variable amount of real work — is the seed of both ceilings. This dispatch path is dissected op-by-op in gpu_kernels · 18 — PyTorch op → kernel launch; here we lift it to the whole-program cost model.

2 · Putting microseconds and bytes on it

Numbers make the tax concrete. Take a residual-stream activation of shape (B, T, d) = (8, 4096, 4096) in bf16 (2 bytes): that is 8 · 4,096 · 4,096 · 2 ≈ 256 MB. On an H100 at ≈3.3 TB/s of HBM bandwidth, reading it once costs 256 MB ÷ 3.3 TB/s ≈ 80 µs, and a read-plus-write elementwise op costs ≈160 µs. The arithmetic for, say, a bias add and a GELU — a handful of FLOPs per element, ≈ 1 GFLOP total — finishes on ~1,000 TFLOP/s tensor cores in well under a microsecond. The op spends ≈99% of its wall-clock waiting on memory.

cost component	where it lives	order of magnitude	scales with
dispatch overhead	Python + C++ dispatcher (per op)	≈1–8 µs / op	number of ops
kernel-launch latency	`cudaLaunchKernel` + driver (per op)	≈3–10 µs / op	number of ops
HBM round trip	read inputs + write output	≈160 µs / 256 MB op	bytes moved ÷ bandwidth
the math (FLOPs)	SM tensor/CUDA cores	< 1 µs for elementwise	FLOPs ÷ peak FLOP/s

So a single eager op costs about (dispatch + launch) + (bytes ÷ bandwidth), and the FLOPs term is, for everything except a large GEMM or conv, negligible. Now chain the ops the way a model does. The toy block gelu(dropout(bias(x @ w))) followed by an add is, eagerly, five separate launches: matmul, bias, GELU, dropout, add. Each non-matmul op reads the whole ≈256 MB activation from HBM and writes it back — four memory-bound ops, ≈4 × 160 µs ≈ 640 µs of pure traffic, plus 5 × ≈15 µs ≈ 75 µs of tax. The math itself, summed across the block, is a rounding error. The wall-clock is launches + traffic, not FLOPs — and that is the single sentence the whole track is built to exploit.

t_eager ≈ Σ_ops (dispatch + launch) + Σ_ops (bytes_op ÷ bandwidth)

3 · Why most ops are memory-bound: arithmetic intensity and the ridge point

Whether an op is limited by the math or by the memory is decided by one ratio: its arithmetic intensity, I = FLOPs ÷ bytes moved (FLOPs per byte read+written from HBM). The roofline says an op runs at min(peak FLOP/s, I × bandwidth). The crossover — the ridge point — is where those two are equal, I* = peak FLOP/s ÷ bandwidth. On an H100 that is roughly 1,000 TFLOP/s ÷ 3.3 TB/s ≈ 300 FLOPs/byte (bf16). An op below the ridge is memory-bound (compute-bound is its complement — limited by FLOP/s); above it, compute-bound.

op	arithmetic intensity (FLOPs/byte)	vs ridge ≈300	verdict
bias add / GELU / dropout (elementwise)	≈0.5–2	far below	memory-bound — pays the round trip, does almost no math
LayerNorm / softmax (reduction)	≈2–10	far below	memory-bound
large square GEMM (4,096³)	≈hundreds–thousands	above	compute-bound — the one op that uses the tensor cores
attention during decode (batch 1)	≈1	far below	memory-bound

The headline: nearly every op a model runs other than its big matmuls sits an order of magnitude or two below the ridge, so it is memory-bound and its time is set by bytes. A fused version that reads the activation once and does bias + GELU + dropout while the data is hot in SRAM/registers raises the intensity and pays the bytes a single time. This is the same roofline currency that gpu_kernels · 10 — kernels from first principles derives for one kernel and that cs336 · 07 — kernels & FlashAttention turns into fusion; this track lifts it to the whole program — every byte you avoid re-reading is a byte the compiler is about to save you.

4 · The two ceilings — and why CUDA graphs fix only one

Read off the cost model and two distinct failure regimes drop out, depending on which term dominates the sum.

launch-bound (the GPU bubble)

Many tiny ops on small tensors. The fixed ≈5–25 µs per-op tax is paid hundreds of times while each kernel's actual work finishes in a microsecond. The GPU sits idle waiting for the CPU to dispatch the next op — "Python is the bottleneck." A transformer layer in eager can launch 80–200 kernels per token; at ≈15 µs each that is ≈1–3 ms of pure overhead per token.

bandwidth-bound

A chain of memory-bound ops on big tensors. Each op re-reads the whole activation from HBM and writes it back, so the block pays ≈2 × bytes per op when it could pay ≈2 × bytes for the entire chain. Time = traffic ÷ bandwidth, and the tensor cores starve.

The two ceilings demand different cures, which is exactly why measuring first matters. CUDA graphs — a captured, replayable sequence of kernel launches enqueued in one shot — erase the per-op CPU dispatch/launch tax: capture the launch sequence once, replay it with a single driver call, and the launch-bound bubble collapses. But a CUDA graph replays the same kernels in the same order; it moves the same bytes across HBM. It does nothing for the bandwidth-bound ceiling. To cut traffic you must change which kernels exist — fuse the chain into one round trip — and that is a graph rewrite, not a replay. Two diseases: launch overhead wants fewer launches; traffic wants fused kernels. A compiler does both; CUDA graphs alone do only the first.

5 · The scoreboard the rest of the track fills in

The cost model gives us exactly three quantities worth tracking, and every later pass moves at least one of them. This is the scoreboard you will see collapse, line by line, all the way to the capstone.

launchesNumber of kernels dispatched. Attacked by fusion (05) and graph capture. Sets the launch-bound term.

HBM bytesTotal traffic across HBM. Attacked by fusion (05), layout (07), memory planning (06). Sets the bandwidth-bound term.

peak memoryMax bytes live at once. Attacked by memory planning (06) and the recompute partitioner (12). Decides whether the program fits the device.

An eager baseline of the toy block reads roughly: 5 launches, ≈10 × the activation in HBM traffic, peak ≈ sum of all live intermediates. A fully compiled version of the same block reads roughly: 1–2 launches, ≈2 × traffic, peak ≈ the single largest concurrent buffer. The gap between those two rows — measured here, in microseconds and bytes — is the entire reason the next fifteen lessons exist. Hold the eager numbers; they are the figure every pass is graded against.

6 · Drive it: the eager cost profiler

The widget runs the cost model on the fixed toy block — matmul + bias + GELU + dropout + add. Slide the tensor size, the number of elementwise ops in the chain, the per-op dispatch cost, and the HBM bandwidth, and watch the estimated wall-clock split into a launch bar and a memory bar. At default (big tensors) the block is bandwidth-bound: memory dwarfs launches. Flip the small-model regime toggle — it shrinks the tensors so each round trip is tiny — and the same five ops become launch-bound: the fixed per-op tax now dominates and the bars swap. That swap, with nothing changed but tensor size, is the proof that time is launches + traffic, not FLOPs.

Eager cost profiler — where the block's milliseconds go

The toy block is fixed: add(x, dropout(gelu(bias(x @ w)))) = 1 matmul + N elementwise ops, each its own eager kernel. Slide the knobs and watch wall-time split into launch (dispatch + launch tax × launches) and memory (Σ HBM round trips ÷ bandwidth). The knob that flips it: turn on small-model regime and the same block goes from bandwidth-bound to launch-bound.

tensor (M×K elements, ×10⁶): 16 elementwise ops N: 4 dispatch µs/op: 4 bandwidth (TB/s): 3.3 small-model regime (tiny tensors)

total launches

—

HBM bytes moved

—

est. wall-time

—

bottleneck

—

launch

memory

Notice what does not move the verdict: the FLOPs. Both regimes run the same arithmetic; only the tensor size changes, and that alone flips which ceiling you hit. That is the cost model's whole thesis, and it is why the rest of the track never optimizes FLOPs — it optimizes launches and bytes.

Failure modes & checklist

Failure modes

Counting FLOPs to predict eager time. The math is nearly free for non-GEMM ops; the wall-clock is launches + traffic. Signal: your FLOP-based estimate is 10–100× too fast, and the profiler shows kernels spending their time on memory, not compute.
Reaching for CUDA graphs against a bandwidth-bound block. Graphs remove launch tax, not traffic. Signal: you captured a graph, the launch bubble vanished, and the step is barely faster because each op still re-reads HBM.
A .item() or print(loss) in the loop. Each forces a sync, serializing the async launch queue. Signal: a 50 ms step balloons to 200 ms with no kernel getting slower — the gaps are sync stalls.
Benchmarking without a warmup or sync. First call pays one-time algo selection and compile; without torch.cuda.synchronize() you time the launch, not the kernel. Signal: impossibly fast or wildly variable numbers.
Calling a small-batch model "GPU-bound." At batch 1 the per-op tax dominates and you are launch-bound. Signal: the GPU utilization is low while the CPU is pinned dispatching.

Checklist

Price the op as tax + traffic. ≈5–25 µs dispatch+launch per op, plus bytes ÷ bandwidth for the round trip.
Compute arithmetic intensity first. Below the ≈300 FLOPs/byte ridge → memory-bound → the cure is fewer bytes.
Diagnose the ceiling before the cure. Launch-bound wants fewer launches; bandwidth-bound wants fusion. Don't apply the wrong one.
Always warm up and synchronize around any GPU timing.
Record the baseline scoreboard: launches, HBM bytes, peak memory — the three numbers every pass is graded against.

Checkpoint

Try it

Open the profiler and leave it at the defaults (tensor 16 ×10⁶, N = 4, dispatch 4 µs, 3.3 TB/s). Read the memory bar — it should dwarf the launch bar; this block is bandwidth-bound. Now compute the traffic by hand: a 16 ×10⁶-element bf16 tensor is ≈32 MB; one read+write is ≈64 MB; the matmul moves ≈3× the tensor (read x, read w, write out) and each of the four elementwise ops moves ≈2×, so (3 + 4·2) · 32 MB ≈ 352 MB, and 352 MB ÷ 3.3 TB/s ≈ 107 µs — confirm the widget agrees within rounding. Then flip small-model regime and watch the bars swap: with tiny tensors the ≈(4 + 6) µs tax × 5 launches ≈ 50 µs now beats the few-µs of traffic. Finally, ask: which knob would a CUDA graph shrink? (The launch bar only.) Which would fusion shrink? (The memory bar — and the launch count.)

Where this points next

You now have a priced baseline and a diagnosis. Eager's wall-clock is the sum of a fixed per-op tax (launches) and a per-op HBM round trip (traffic); below the ≈300 FLOPs/byte ridge, traffic wins and the chip starves. The two ceilings — launch-bound and bandwidth-bound — are both whole-program faults: to launch fewer kernels you must know which ops follow which, and to share one round trip you must see a whole chain at once. But eager Python hands the runtime exactly one op at a time and then forgets it; there is no program to optimize, only a stream of isolated calls. The very first thing a compiler must do, before any rewrite, is recover the missing object — a graph of the whole computation. That is lesson 02, Capturing the graph.

Takeaway

You cannot optimize what you have not priced, so we put a stopwatch on eager. Every PyTorch op pays a fixed CPU-side tax — dispatch through the C++ dispatcher into the ATen kernel library (~1–8 µs) plus a kernel launch (~3–10 µs) — and then the usually-dominant HBM round trip: a 256 MB activation @3.3 TB/s costs ≈160 µs to read and write while its FLOPs finish in under a microsecond. So eager wall-clock ≈ Σ(dispatch+launch) + Σ(bytes ÷ bandwidth) — launches + traffic, not FLOPs. Arithmetic intensity decides which term wins: below the ridge point (≈300 FLOPs/byte on an H100) an op is memory-bound, and nearly every non-GEMM op is. From the model fall the two ceilings: launch-bound (many tiny ops drowning in the per-op tax) and bandwidth-bound (each op re-reading HBM). A CUDA graph erases the launch tax but not the traffic, so it fixes only one ceiling — fusing kernels is the other cure. The baseline scoreboard — launches, HBM bytes, peak memory — is the figure the next fifteen lessons drive down. Both ceilings are whole-program problems, which forces the first compiler move: capture the graph.

Interview prompts

Decompose the cost of one eager PyTorch op. (§1–2 — fixed per-op tax: Python/pybind11 + autograd + ATen dispatch + cudaLaunchKernel ≈5–25 µs; plus the HBM round trip = bytes ÷ bandwidth, which dominates for memory-bound ops; FLOPs are negligible for non-GEMM.)
Why is eager sometimes 5× slower than compiled when the same matmul runs at the end? (§1,4 — the extra time is dispatch + launch tax paid per op (launch-bound) and re-reading HBM per op (bandwidth-bound); the compiler fuses many ops into fewer kernels and one round trip, the GEMM is unchanged.)
What is the ridge point and roughly where is it on an H100? (§3 — I* = peak FLOP/s ÷ bandwidth ≈ 1,000 TFLOP/s ÷ 3.3 TB/s ≈ 300 FLOPs/byte bf16; ops below it are memory-bound, above it compute-bound.)
Name the two ceilings eager hits and the different cure for each. (§4 — launch-bound (many tiny ops, fixed tax dominates) → fewer launches via CUDA graphs/fusion; bandwidth-bound (each op re-reads HBM) → fusion into one round trip. Wrong cure for the wrong ceiling does little.)
Do CUDA graphs fix the bandwidth-bound ceiling? (§4 — no; a graph replays the same kernels moving the same bytes, so it only erases the per-op CPU launch tax. Cutting traffic requires changing which kernels exist — fusion.)
What is MFU and why is an eager small-batch model's MFU low? (MFU = model FLOPs utilization, achieved FLOP/s ÷ peak. It is low when the GPU is starved — launch-bound by dispatch tax or bandwidth-bound re-reading HBM — so the tensor cores sit idle and realized FLOP/s is far below peak.)
Which three numbers form the baseline scoreboard, and what attacks each? (§5 — launches (fusion, graph capture), HBM bytes (fusion, layout, memory planning), peak memory (memory planning, recompute partitioner).)