Part I — The gap

The semantic gap

There is no previous lesson — this is where the chain begins, so it begins with the wall the whole track exists to climb. You wrote a model as a few lines of tensor math; the GPU is a bandwidth-and-FLOPs machine that only rewards a few large, fused, specialized kernels. Between those two facts sits a wide gap, and naive "eager" execution falls straight into it — leaving the most expensive chip you own mostly idle. People wave at torch.compile or XLA as a magic box that "makes it fast." It is not magic. It is a stack of forced rewrites, each one demanded by a single question. This lesson names the gap, makes it concrete in bytes, and lays out the chain of forced decisions that the next 15 lessons walk down.

The premise this track starts from

Treating the compiler as a black box has a cost: you cannot tell whether it helped, why it didn't, or what to change when it gives you a 1.0× speedup. The premise here is the opposite — every pass a real ML compiler runs is derivable. It is the unique forced fix for the inefficiency the previous step left behind. The root inefficiency, the one that starts the chain, is the semantic gap: you write a hardware-agnostic graph of coarse tensor ops, but the hardware is a memory hierarchy that is starved by anything except a few big fused kernels. Run your graph the obvious way — one kernel per op, eager — and you pay the gap on every single line. y = gelu(x @ w + b) looks like one expression; eagerly it is three or more kernels and three or more full round trips to off-chip memory, when the hardware wanted exactly one.

Linear position

Forced by: the wish to understand and trust what torch.compile / XLA / TVM do rather than worship them — which means seeing the one constraint (the semantic gap) that every pass exists to attack.
New idea: the engine of the whole track — a compiler is a stack of semantics-preserving rewrites that lower a high-level graph through levels of representation, trading portability for hardware speed, and the scoreboard it plays against is bytes moved across HBM and kernels launched. Five stages: capture → represent → optimize → lower → codegen.
Forces next: to attack a cost you must first price it — so lesson 01 measures the eager baseline (dispatch, launch, HBM round trips) to give every later pass a concrete number to beat.

The plan

Five moves. (1) Define a compiler, then specialize the classic source→IR→opt→asm picture to ML, and name why ML changes the game (coarse ops, a memory hierarchy, the same shapes a million times). (2) Make the semantic gap concrete in bytes on one line of code. (3) Lay out the five stages every ML compiler runs — the track's map. (4) Draw the linear spine: a wall→fix→new-wall chain naming all 20 lessons. (5) State the three motifs that recur. Then drive the gap meter until a long eager chain wastes an order of magnitude in HBM traffic while the fused cost stays flat.

1 · What a compiler is, generalized to ML

A compiler translates a program written for humans into a program a machine runs fast, without changing what it computes. The classic shape is four steps: parse the source into an intermediate representation (an IR — a structured, in-memory form of the program that is easy to analyze and rewrite, unlike raw text), run optimization passes over the IR (a pass is one self-contained transformation — constant folding, dead-code removal), then lower it toward the hardware (lowering = rewriting into a more machine-specific, less portable form) and emit assembly. C does exactly this: clang turns C into LLVM IR, optimizes, lowers to your CPU's instruction set.

An ML compiler does the same dance, but the inputs and the stakes differ enough to change every decision:

source = a tensor program

Not C. A Python (or JAX) function over whole tensors: x @ w, gelu(...), softmax(...). The "instructions" are coarse — each op already touches millions of elements.

IR = a typed dataflow graph

A directed graph of tensor values (shape + dtype + device in the type), not a linear list of scalar ops. Optimization means rewriting the graph; lesson 03 builds it.

target = a memory hierarchy

A GPU is registers + SRAM (fast, tiny) over HBM (huge, ~100× slower). Performance is decided by how few bytes cross HBM and how few kernels (GPU programs launched from the host) you fire.

workload = the same shapes, a million times

A trained model runs the identical op graph on identical shapes for billions of tokens. So spending minutes of compile-time search to shave microseconds per run pays for itself — a luxury a C compiler rarely has.

That last point is the quiet reason ML compilers can be so aggressive — autotuning, exhaustive search, learned cost models (lesson 10). The same kernel runs ~10⁶–10¹² times, so even a 10-minute compile amortizes to nothing. Two more terms you will need immediately. Fusion: merging several ops into one kernel so their shared data stays in fast on-chip memory instead of bouncing through HBM between ops — the single highest-leverage transform in the whole stack (lesson 05). AOT vs JIT: ahead-of-time compilation runs the whole pipeline before the program starts (XLA, TensorRT); just-in-time compilation captures and compiles at the first real call and caches the result (torch.compile). Both run the same five stages — they differ only on when.

2 · The gap, in bytes, on one line

Take the most ordinary line in any transformer block:

y = gelu(x @ w + b)

Read it as a graph and there are three coarse ops after the matmul produces an intermediate: an add (bias), a gelu (activation), and — counting the matmul output write — already several tensors crossing memory. Eager execution runs each op as its own kernel: launch, read the input tensor from HBM, do a few FLOPs per element, write the result back to HBM, repeat. The arithmetic is trivial. The bytes are not.

Put numbers on the elementwise tail. Say the activation after the matmul is (B, T, d) = (8, 2,048, 4,096) in bf16 (2 bytes): that is 8 · 2,048 · 4,096 · 2 ≈ 134 MB. Each elementwise op reads that tensor and writes it — one round trip, ≈ 268 MB across HBM. On an H100 at ≈3.3 TB/s, one round trip costs 268 MB / 3.3 TB/s ≈ 81 µs; the FLOPs (a couple per element, ≈0.27 GFLOP) finish in well under a microsecond. The op is ≈99% spent waiting on memory.

execution	kernels launched	HBM round trips (bias + gelu tail)	bytes moved (≈)	est. memory time (≈)
eager — one kernel per op	add, gelu = 2 (+ matmul)	2 — each op re-reads + re-writes	≈ 536 MB	≈ 162 µs
fused — bias+gelu in the matmul epilogue	1 (folded into the matmul)	1 — read once while hot, write once	≈ 268 MB	≈ 81 µs

Fusion did the exact same math and moved half the bytes on this two-op tail — and the win grows with the chain. Add dropout and a residual add and a norm and eager pays a fresh round trip for each, while the fused kernel still reads once and writes once. That is the gap: the math says "one expression," the hardware says "one kernel," and eager execution says "as many kernels and round trips as you have ops." Everything in this track is machinery to close that gap while computing bit-for-bit the same result. For the kernel-level view of why one op = one launch = one round trip, see gpu_kernels · 18 PyTorch → kernel and gpu_kernels · 10 kernel first principles.

3 · The five stages of any ML compiler

Strip away the product names and every ML compiler — torch.compile, XLA, TVM, TensorRT, IREE — runs the same five-stage pipeline. This is the track's map; each stage is one or more lessons.

captureTurn imperative Python into a graph. Tracing / bytecode (Dynamo). Lesson 02.

representA typed SSA dataflow IR with rules, in levels (high → low). Lesson 03.

optimizeHardware-agnostic rewrites: fold, fuse, plan memory, lay out. Lessons 04–07.

lowerPick the loop nest, emit machine code, search the config. Lessons 08–10.

codegenReal PTX / Triton, cached and replayed. Lesson 09, used everywhere.

Three ML-specific complications sit on top of this pipeline and get their own part: dynamic shapes (every static-shape assumption above breaks under variable batch and growing KV cache — lesson 11), autodiff (training needs a backward graph the compiler must build itself — lesson 12), and distribution (the graph outgrows one device — lesson 13). The synthesis lessons (14–15) then show that torch.compile/XLA/TVM are literally this pipeline with different IRs and bets, and walk one block source-to-kernel.

4 · The linear spine — wall → fix → new wall

This track is not a tour of features; it is one chain. Each lesson hits a wall, derives the forced fix, and that fix exposes the next wall. Read the spine as a single sentence per link:

00 → 01The gap is real → but you can't optimize what you can't price, so measure the eager baseline: dispatch µs, launch µs, HBM round trips.

01 → 03The ceilings are whole-program → but Python hands you one op at a time, so capture a graph (02) and give it rules — a typed SSA IR in levels (03).

04 → 07A raw graph is redundant and unfused → rewrite it (04), then fuse the central win (05), then plan memory (06) and lay out tensors (07).

08 → 10An optimized graph still only says what → choose the loop nest (08), generate code (09), and autotune the huge config space (10).

11 → 13Everything assumed static, single-device, forward-only → handle dynamic shapes (11), build the backward graph (12), and shard across devices (13).

14 → 17Every pass assumed kernels are the finish line → ship the compiled model: cover the long tail of ops (14), quantize precision (15), launch it at run time (16), and debug & trust the result (17).

18 → 19You've derived and shipped every pass → see the real stacks as one pipeline (18), then walk one block source-to-kernel end to end (19).

By the end you should be able to look at any pass in any compiler and name the inefficiency that forced it — and sketch the stack yourself.

5 · The three motifs

Three ideas recur in every lesson; we will call them back by file number. Hold them now.

preserve semantics

Every transform must be a provable no-op on the math. This correctness invariant is what bounds what a compiler may legally do — and float non-associativity makes it sharper than it sounds (03–04, 05, 12).

abstraction ↔ specialization

A high IR is portable but slow; a low IR is fast but locked to one chip. Lowering is spending portability to buy speed — the trade that defines every stage (03, 08, 09, 14).

memory traffic is the real cost

For the memory-bound ops that dominate a model, wall-clock ≈ bytes moved / bandwidth. Fusion (05), memory planning (06), and layout (07) all attack one number: bytes across HBM.

6 · Drive it: the gap meter

The widget makes §2 a knob. Pick a chain of elementwise ops on a tensor whose size you set. Eager runs one kernel per op: each op is one launch and one HBM round trip (read the whole tensor, write it back). Compiled fuses the whole chain into one kernel: one launch, one read, one write — flat, no matter how long the chain. Watch the eager bars climb linearly with chain length while the fused bars stay pinned, and read the bytes-moved ratio: it tracks the chain length, because eager moves ≈ 2 · n tensor-widths and fusion moves 2.

The gap meter — eager (kernel per op) vs compiled (one fused kernel)

The knob that proves it: push the chain length right. Eager #kernels, HBM round trips, and bytes all rise with the chain; the fused kernel stays at 1 launch, 1 read, 1 write. Bigger tensors scale both costs but only widen the gap in absolute bytes. The ratio ≈ chain length is the whole lesson.

chain length (elementwise ops): 4 tensor size: 134 MB

eager — kernels

—

eager — HBM round trips

—

eager — bytes moved

—

compiled — kernels

compiled — bytes moved

—

bytes ratio (eager / fused)

—

eager

—

compiled

—

The shape is the point: eager is a staircase that climbs with every op you add; compiled is a floor. That flat line is what a compiler buys, and §1's amortization argument is why paying compile-time to find it is worth it. But notice the meter assumes the bytes-moved model — to trust the win you need real numbers on a real baseline, which is exactly what forces lesson 01.

Failure modes & checklist

Failure modes

Treating the compiler as magic. Calling torch.compile and assuming a speedup. Signal: a measured 1.0× and no idea why — usually graph breaks or a library-bound model with nothing to fuse (lesson 18).
Counting FLOPs instead of bytes. Optimizing the matmul math when the elementwise tail is the cost. Signal: the profiler shows tiny pointwise kernels dominating wall-clock, not the GEMM.
"Fusing" a compute-bound op. Fusion attacks memory traffic; a big GEMM is already compute-bound. Signal: you fused everything and the matmul kernel still dominates — it was never bandwidth-limited.
Assuming a rewrite is free. Forgetting that float addition is non-associative, so reassociating a sum can change results. Signal: a "compiled vs eager" mismatch beyond float noise after an aggressive fast-math pass (lesson 04).
Confusing AOT and JIT costs. Surprised by a multi-second first call under JIT. Signal: the first request after deploy times out — that was compile time, not inference (lessons 10–11).

Checklist

Name the gap. A hardware-agnostic op graph vs a memory hierarchy that only rewards few, large, fused kernels.
Score in two currencies. Bytes moved across HBM and kernels launched — the numbers every pass moves.
Know the five stages. Capture → represent → optimize → lower → codegen; place any feature you read about in one.
Hold the three motifs. Preserve semantics; abstraction ↔ specialization; memory traffic is the cost.
Define the words. IR, pass, lowering, kernel, fusion, AOT vs JIT — you'll use all six in lesson 01.
Trust numbers, not vibes. Any claimed win must show fewer kernels or fewer bytes, not just a faster wall-clock once.

Checkpoint

Try it

Open the gap meter and set the tensor to 134 MB. Slide the chain length from 1 to 8 and watch the eager bytes climb while the fused bytes stay flat — read the ratio at each end (it should run from ≈1× at length 1 to ≈8× at length 8, because eager moves ≈ 2 · n tensor-widths and fusion moves 2). Now answer by hand: for a chain of 5 elementwise ops on a 134 MB tensor, how many HBM round trips does eager pay, and how many does the fused kernel pay? (Eager: 5 — one per op. Fused: 1.) Then double the tensor size and confirm the ratio doesn't move — fusion's advantage is set by the chain length, not the tensor size. Predict what lesson 01 must add to make this trustworthy: a real per-op dispatch and launch cost, so the meter isn't bytes-only.

Where this points next

You now have the frame: a compiler is a stack of semantics-preserving rewrites that lower a graph through levels of IR, trading portability for speed, scored in bytes-moved and kernels-launched — and the gap meter shows the prize. But the meter is a model. It asserts that eager wastes bandwidth; it doesn't price the other two eager costs that bite real models — per-op dispatch overhead (Python plus the dispatcher) and kernel-launch latency — and it doesn't tell you which regime you're in: launch-bound (many tiny ops) or bandwidth-bound (big chains re-reading HBM). You cannot optimize what you haven't measured. So the next move is forced: build a concrete cost model of eager execution and read the scoreboard the rest of the track will fill in. That is lesson 01, The eager baseline and its ceiling.

Takeaway

A neural network is a hardware-agnostic graph of coarse tensor ops; a GPU is a memory hierarchy that only rewards a few large, fused, specialized kernels. Between them sits the semantic gap, and naive eager execution — one kernel and one HBM round trip per op — falls into it on every line: gelu(x @ w + b) becomes 3+ kernels and 3+ round trips where the hardware wanted one. A compiler closes the gap as a stack of semantics-preserving rewrites that lower the graph through levels of IR, spending portability to buy hardware speed, and it plays against one scoreboard: bytes moved across HBM and kernels launched. Every ML compiler runs the same five stages — capture, represent, optimize, lower, codegen — plus the ML-specific hard parts (dynamic shapes, autodiff, distribution). Because the same op graph runs ~10⁶–10¹² times, heavy compile-time search amortizes to nothing, which is why these compilers can be so aggressive. The three motifs that recur: preserve semantics, abstraction ↔ specialization, and memory traffic is the real cost. The chain starts here; to attack the gap you must first price it on a real eager baseline — lesson 01.

Interview prompts

What is the "semantic gap" an ML compiler exists to close? (§2 — between a hardware-agnostic graph of coarse tensor ops and a memory hierarchy that only rewards few large fused kernels; eager execution pays one kernel and one HBM round trip per op, so one expression becomes many round trips when the hardware wanted one.)
How does compiling a tensor program differ from compiling C? (§1 — coarse whole-tensor ops vs scalar instructions; a typed dataflow graph IR vs a linear list; a memory-hierarchy target where bytes and launches dominate; and the same shapes run ~10⁶–10¹² times, so heavy compile-time search amortizes — a luxury C rarely has.)
Name the two currencies an ML compiler optimizes and why those, not FLOPs. (§1/§5 — bytes moved across HBM and kernels launched; for the memory-bound ops that dominate a model wall-clock ≈ bytes/bandwidth and the FLOPs are nearly free, so the levers are fewer round trips and fewer launches.)
List the five stages of an ML compiler and put torch.compile's components in them. (§3 — capture (Dynamo), represent (FX/typed IR), optimize (Inductor fusion/planning/layout), lower (schedule + Triton codegen), codegen (Triton → PTX, cached); plus dynamic shapes / autodiff / distribution as ML-specific add-ons.)
Define lowering and explain the abstraction ↔ specialization trade. (§1/§5 — lowering rewrites the IR into a more machine-specific, less portable form; a high IR is portable but slow, a low IR is fast but locked to one chip, so descending the IR ladder spends portability to buy speed.)
What is fusion and why is it the highest-leverage pass? (§2 — merging ops into one kernel so shared data stays on-chip instead of round-tripping HBM between ops; it directly cuts the dominant cost (bytes moved) for memory-bound chains, turning ≈2·n tensor-widths of traffic into ≈2.)
AOT vs JIT — same pipeline or different? (§1 — same five stages; they differ only on when: AOT (XLA, TensorRT) compiles before the program runs, JIT (torch.compile) compiles at the first real call and caches — which is why a JIT'd model has a slow, sometimes multi-second, first call.)