Part II — Systems

GPUs — the hardware you're renting

Lesson 05 quoted two numbers as if they were given to us: a "peak FLOP/s" the runtime estimate divided by, and an "MFU" of ~50% that quietly halved your calendar. Neither was explained — they were placeholders for the machine. The whole budget is denominated in GPU-hours, yet we have not once looked at what a GPU actually is: how it moves bytes, how it does math, and why those two rates are wildly out of balance. This lesson opens the box. The single concept inside — arithmetic intensity, plotted on the roofline — explains where the 50% comes from and which operations in your model are throwing tensor cores away.

1 · The memory hierarchy: a bandwidth cliff

A GPU is not one pool of memory; it is a steep pyramid, and the price of moving a byte changes by orders of magnitude as you climb it. Take an NVIDIA H100 (SXM) as the running example — the chip most frontier runs are actually rented on.

Read the table as a cliff. The 80 GB of HBM is where everything you allocated in PyTorch actually sits, and it is the slow tier: 3.3 TB/s sounds enormous until you compare it to the compute rate in section 2. On-chip SRAM is roughly 20× faster per byte but holds only tens of megabytes total — you cannot park a 7B model there, you can only stage a tile of it. Registers are faster still and essentially free, but a thread has only a handful of kilobytes. The entire performance game on a GPU is: get data into SRAM/registers once, do as much math on it as possible, and write back to HBM as little as you can. Every byte you re-read from HBM that you could have kept on-chip is bandwidth you set on fire.

  registers   few KB/thread     fastest          <- ALUs eat from here
      ^
  SRAM        ~228 KB/SM        ~20x HBM BW       <- staging tile, shared within a block
  (L1/shared) ~50 MB L2
      ^
  HBM         ~80 GB            ~3.3 TB/s         <- everything lives here; the slow tier
      ^
  host/PCIe   TB                ~tens of GB/s     <- off-GPU; avoid in the hot loop entirely

One concrete consequence to keep in your pocket: an operation that just reads a big tensor from HBM and writes it back — say, adding a bias — is paying full HBM latency for almost no math. Whether that op is fast has nothing to do with how many TFLOP/s the chip has. It is gated entirely by 3.3 TB/s. Hold that thought; it is the whole lesson.

2 · Compute: ~1,000 TFLOP/s, and the gap that is the whole story

Now the other rate. An H100's tensor cores — dedicated matrix-multiply-accumulate units, separate from the general CUDA cores — do roughly 990 TFLOP/s of bf16 (call it ~1 PFLOP/s; with FP8 and sparsity the marketing numbers roughly double). The CUDA cores that handle elementwise math (an add, a GELU, a comparison) are far slower — on the order of tens of TFLOP/s of fp32. So "the GPU's compute" is really "the tensor cores doing matmuls," and everything else is a sideshow that mostly exists to feed them.

Put the two rates side by side and the central tension of GPU programming appears:

That ratio is the crux of Part II. The hardware can perform on the order of 300 floating-point operations in the time it takes to read a single byte from HBM. So unless an operation does ≥300 FLOPs of useful work for every byte it pulls from HBM, the tensor cores spend most of their cycles idle, waiting on memory. The TFLOP/s number on the spec sheet — the one you bragged about in the runtime estimate — is reachable only by operations dense enough to clear that bar. Most operations in a language model are not. The gap between TFLOP/s and TB/s is not a footnote; it is the reason MFU is 50% and not 95%, and the reason every remaining lesson in Part II exists.

3 · Arithmetic intensity and the roofline

To make "does this op clear the bar" precise, define one number per operation. Arithmetic intensity (AI) is the ratio of useful work to data traffic:

It is a property of the algorithm and its data layout, not of the chip. An op with low AI touches a lot of memory per unit of math (bandwidth-hungry); an op with high AI reuses each loaded byte for many operations (compute-hungry). Now overlay the two hardware rates. The time an op takes is bounded below by whichever ceiling is binding:

Turn that into achievable throughput as a function of AI and you get the roofline: a curve that rises linearly while bandwidth is the limit, then flattens once compute takes over.

The two regimes meet at the ridge point — the arithmetic intensity at which a perfectly bandwidth-bound op would exactly saturate compute:

The verdict is binary and unforgiving. An op whose AI is below ≈300 can never reach peak FLOP/s no matter how good your kernel is — it is memory-bound, capped at AI × 3.3 TB/s, and the tensor cores idle. An op whose AI is above ≈300 is compute-bound — it can in principle saturate the tensor cores, and is the only kind of op that earns the spec-sheet TFLOP/s. The roofline is the single picture that says, for any operation, the best speed it could possibly hit on this chip and which resource you would have to buy more of to go faster.

4 · Classifying the LM op zoo

Now walk the operations a Transformer actually runs and place each on the roofline. The pattern is stark: exactly one family is compute-bound, and it is the one you want to dominate the budget.

The lesson of the table: a big training GEMM has arithmetic intensity in the hundreds-to-thousands because a tile of the weight matrix, once in SRAM, is multiplied against many rows of activations — the loaded bytes are reused heavily, so the math piles up while the traffic does not. That is why matmuls are the only ops that hit the tensor-core ceiling, and why you want them to be the overwhelming majority of your FLOPs. Everything else — every norm, every activation, every softmax, and decode-time attention — does only a few FLOPs per byte and is pinned to the 3.3 TB/s wall, the tensor cores sitting idle the whole time.

This is precisely why real training MFU lands at ~40–55% rather than near 100% (lesson 5): even though GEMMs dominate the FLOP count, the memory-bound ops between them consume wall-clock time during which the tensor cores do nothing. The chip is fast at the one thing and starved on everything else, and "everything else" still has to run.

5 · The execution model, briefly — and where MFU leaks

You do not need the full programming model to reason about the budget, but you need the vocabulary, because the leaks in MFU live here. A GPU runs a kernel (one function) as a grid of thread blocks; each block is scheduled onto one streaming multiprocessor (SM — an H100 has ~132) and runs as groups of 32 threads called warps that execute in lockstep. Threads in a block share that SM's fast SRAM and can cooperate; threads in different blocks cannot. Occupancy is how many warps an SM keeps resident at once — enough resident warps let the scheduler hide memory latency by switching to a ready warp while another waits on HBM. Low occupancy (too few warps, too many registers per thread) means the SM stalls with nothing to run during a memory wait.

That gives the four reasons MFU comes in under 100%, all visible from this lesson:

• Memory-bound ops (§3–4): norms, softmax, elementwise, decode-attention can't saturate the tensor cores by construction — they're below the ridge, so the cores idle while HBM is read.
• Kernel-launch + HBM round-trips: running each op as its own kernel pays launch overhead and re-reads the tensor from HBM every time — the exact waste lesson 7's fusion removes.
• Low occupancy / small ops: tensors too small to fill the SMs, or too register-hungry, leave warps idle and latency un-hidden.
• Communication stalls: once work is split across GPUs (lessons 8–9), devices wait on each other's all-reduces — compute idles during comm that isn't overlapped.

The depth here is its own track. For the line-level programming model — threads, warps, divergence, the launch — see gpu_kernels · 01 (the execution model); for the memory hierarchy and coalesced access, gpu_kernels · 02; and for the kernel-engineering mindset that turns a roofline into a fast kernel, gpu_kernels · 10 (kernels from first principles). This lesson only needs you to hold the roofline and know which side of the ridge your ops fall on.

6 · Drive the roofline

The plot below is the H100 roofline: the diagonal is the bandwidth ceiling (AI × 3.3 TB/s), the flat top is the compute ceiling (990 TFLOP/s), and they meet at the ridge (~300 FLOP/byte). Pick a preset op or drag the slider to set an arithmetic intensity. The marker lands on the roofline at that AI, and the readout shows the best achievable throughput and what fraction of peak that is. Drag below the ridge and the verdict turns red: memory-bound — you're wasting tensor cores. Drag past it and the op finally reaches the flat ceiling where the spec-sheet TFLOP/s lives. Notice how far left every real LM op except big GEMM sits.

The widget makes the spine of Part II visible: the great majority of an LM's distinct operations sit on the bandwidth slope at single-digit percentages of peak, and no kernel rewrite can lift a single-pass op off that line. The only way to recover the wasted tensor cores is to change what gets moved — to stop round-tripping each intermediate through HBM. That is fusion.

Failure modes & checklist

Checkpoint

Where this points next

The roofline just delivered an uncomfortable verdict: the overwhelming majority of an LM's distinct operations — every norm, every activation, every softmax, attention at decode — live far below the ridge and can reach only a few percent of the chip's compute, because they do only a handful of FLOPs per byte they pull from HBM. And the way frameworks run them makes it worse: naive PyTorch launches each op as its own kernel, so every intermediate tensor is written to HBM by one kernel and immediately read back by the next, paying full 3.3 TB/s round-trips for math that takes nanoseconds. The tensor cores starve between every step. The forced fix is to stop moving data we don't need to move — to load a tile once into SRAM and do a whole chain of operations on it before writing back, so many ops share one HBM round-trip. That is kernel fusion, and its headline case is FlashAttention. Next: 07 · Kernels — fusion and FlashAttention.

Interview prompts

• Define arithmetic intensity and the roofline; what is the ridge point on an H100? (§3 — AI = FLOPs ÷ bytes moved to/from HBM; achievable FLOP/s = min(peakCompute, AI×peakBW); the ridge = peakCompute ÷ peakBW ≈ 990 ÷ 3.3 ≈ 300 FLOP/byte, the AI above which an op can saturate compute.)
• Why is a LayerNorm memory-bound and a large matmul compute-bound? (§4 — a norm does ~2–4 FLOPs per element read (AI≈2), so it's capped at AI×3.3 TB/s ≈ <1% of peak; a GEMM reuses each loaded tile across many outputs (math ∝ M·N·K, traffic ∝ M·K+K·N), giving AI in the hundreds, above the ridge.)
• Walk the H100 memory hierarchy and the cost of moving a byte at each level. (§1 — HBM ≈80 GB @ ≈3.3 TB/s (everything lives here, slow); SRAM tens of MB @ ~20× HBM (staging tile, on-chip); registers tiny/fastest. The game is load to SRAM once, compute a lot, write back rarely.)
• Why is real training MFU ~50% and not ~100%? (§4–5 — GEMMs dominate the FLOP count but the memory-bound ops between them (norms, softmax, elementwise) consume wall-clock with the tensor cores idle; add kernel-launch/HBM round-trips, low occupancy, and comm stalls.)
• Estimate the best achievable throughput of an RMSNorm on an H100. (§4 — AI ≈ 2 FLOP/byte → 2 × 3.3 TB/s ≈ 6.6 TFLOP/s, under 1% of 990 TFLOP/s peak; no kernel beats it, only fusing away the HBM round-trip helps.)
• How does switching from bf16 to FP8 change the roofline? (§3 caveat — peakCompute roughly doubles, so the ridge climbs from ~300 to ~600 FLOP/byte; compute-bound GEMMs speed up, but memory-bound ops fall further below the ridge — harder to feed the faster cores.)
• Why is attention at decode memory-bound while training attention is not? (§4, lesson 18 — decode processes one query token against the whole KV cache: a matrix-vector with no reuse, each cached byte read once → low AI; training attention is dense matmuls over the full sequence with reuse → higher AI.)