Part VI — Shipping the compiled model

The runtime & executor

Everything since lesson 02 has produced a thing, not an act. Capture, lowering, fusion, the memory plan from lesson 06, codegen, autotuning, and now quantization (lesson 15) all hand you the same deliverable: an inert artifact — a cache of compiled kernels plus a recipe that says which buffer goes where. An artifact does not run. Something has to walk in, allocate the buffers the plan describes, push the kernels onto the GPU in order, and do it without paying the per-op tax that lesson 01 priced and that has been sitting on the critical path this whole time. That something is the runtime. This lesson is the run-time half of a track that has, until now, been entirely about compile time.

The previous step left this broken

Quantization (lesson 15) was the last pass that touches kernels: by lesson 16 the compiler has finished. What it leaves behind is a static bundle — Triton or PTX cubins keyed by shape and dtype (lesson 09, 10), a memory plan that says "buffer 7 reuses buffer 3's slot, the arena is 412 MB" (lesson 06), and a list of guards (lesson 11) that say which inputs the bundle is valid for. None of that allocates a single byte or launches a single kernel. Worse, lesson 01's launch-bound ceiling is still standing: even a perfectly fused, quantized, scheduled artifact, if you dispatch its kernels one Python call at a time, pays ≈5–25 µs of CPU tax per launch and the GPU starves between them. The compiler made the kernels fast; nothing yet makes launching them fast, or even possible.

Linear position

Forced by: the compiler produces an inert artifact — kernels plus a memory plan plus guards — and nothing in it allocates buffers, dispatches kernels in order, or amortizes the per-op launch tax that lesson 01 left on the critical path.
New idea: the runtime / executor as the run-time counterpart to the compiler — streams & async dispatch (a launch queue the GPU drains while the CPU runs ahead, overlapping kernels with H2D/D2H copies via events), a caching allocator / memory pool that realizes the lesson-06 plan by serving buffers from one preallocated arena instead of calling cudaMalloc on the hot path, and CUDA graph capture & replay — record the entire launch sequence once, replay it with a single driver call, which is the real fix for lesson 01's launch ceiling. Plus the AOT path (torch.export → AOTInductor .so, a serialized engine) for Python-free serving.
Forces next: graph replay needs stable shapes and stable addresses, so it fights the dynamic shapes of lesson 11; and fusion, quantization, and replay all silently change results or recompile. You now need to trust and debug the compiled artifact — lesson 17.

The plan

Five moves. (1) Draw the line between the compiler artifact and the runtime: who owns what. (2) Streams & async dispatch — the launch queue, overlap with copies, events, and why the launch tax is still on the critical path for small models. (3) The caching allocator that realizes the lesson-06 plan without hot-path cudaMalloc; fragmentation and OOM; pool segments. (4) CUDA graphs — capture/replay, the one-call launch, mode="reduce-overhead", cudagraph trees, and the stable-shape/stable-address catch that collides with lesson 11. (5) The AOT/serving path — torch.export → AOTInductor .so, serialized engines, the guard-check-then-dispatch loop, and the code/graph cache for warm starts. Then drive the widget until graph-replay collapses N launches into ≈1 driver call — and dynamic shapes break it.

1 · Two halves of the machine: the artifact vs the runtime

It is worth saying plainly because the whole track has blurred it: a compiler and a runtime are different programs, run at different times, that own different things. The compiler runs once, ahead of (or just before) serving, and its output is a compiled artifact — an inert data structure. The runtime (also called the executor) runs on every request and is the live process that turns that data structure into GPU work.

concern	compiler (compile time) — owns the artifact	runtime / executor (run time) — owns the act
kernels	generates & caches them (lesson 09), tunes configs (lesson 10), quantizes (lesson 15)	looks them up by shape/dtype and launches them in order
memory	computes the plan: liveness, reuse, arena size (lesson 06)	realizes the plan — allocates the arena, hands out slices, recycles them
shapes	specializes & emits guards (lesson 11)	checks guards on each input; dispatches or recompiles on a miss
overhead	nothing it can do at compile time about per-launch CPU tax	amortizes it — async streams, then CUDA graph replay
when it runs	once, offline or on first call (JIT)	every forward pass, on the critical path of latency

The reason this split is the natural close of the track: every pass you derived attacked a compile-time quantity — fewer kernels, fewer bytes, smaller peak. But the model's actual latency is paid at run time, and three run-time costs survive every compile-time win — buffer allocation, kernel dispatch, and the launch tax. The runtime is the only place those die. Lesson 06's plan is the clearest example: it is a brilliant assignment of buffers to slots that, on its own, allocates nothing. The caching allocator (§3) is the runtime component that makes the plan real.

2 · Streams & async dispatch: running the CPU ahead of the GPU

The first thing the runtime does to hide the launch tax is refuse to wait. A stream is a FIFO queue of GPU work; operations enqueued on the same stream execute in order, but the enqueue call returns to the CPU immediately, before the kernel runs. So cudaLaunchKernel is asynchronous: the CPU pushes a launch onto the stream and races ahead to dispatch the next op while the GPU is still chewing on the last one. If the CPU can stay ahead of the GPU, the per-op dispatch+launch tax is hidden behind kernel execution and only the kernels' own time is on the critical path.

CPU dispatchesPython → dispatcher → cudaLaunchKernel enqueues kernel k onto the stream, returns in ≈5–25 µs (lesson 01).

CPU runs aheadWhile the GPU executes k, the CPU is already enqueuing k+1, k+2, … building a backlog the GPU drains.

copies overlapA separate stream does H2D / D2H cudaMemcpyAsync; with pinned host memory it overlaps the compute stream instead of serializing it.

events synchronizeA cudaEvent recorded on one stream and waited on by another expresses "kernel B needs A's output" without a full device sync.

This is genuine overlap, and it is why a well-fed training step can be compute-bound despite a slow Python loop. But it has a hard limit, and the limit is exactly lesson 01's launch-bound ceiling. If the GPU drains the queue faster than the CPU can refill it — many tiny kernels, each finishing in ≈1–2 µs while dispatch costs ≈6 µs — then the GPU runs dry: it idles waiting for the next launch, the CPU never catches up, and the launch tax is back on the critical path. A transformer layer at small batch can launch 80–200 kernels per token; at ≈6 µs each that is ≈0.5–1.2 ms of pure CPU dispatch per token that no amount of async hides, because there is nothing to hide it behind. Async dispatch overlaps the tax with work; when there is not enough work, the tax wins.

Two more runtime hazards belong here. A device synchronization — anything that forces the CPU to wait for the GPU — drains the pipeline: a stray .item(), print(loss), or cpu() in the loop serializes the whole async queue and re-exposes every launch. And the launch queue itself is finite; if the CPU runs too far ahead the enqueue call blocks. The runtime's job is to keep the CPU exactly far enough ahead to hide the tax and no further.

3 · The caching allocator: realizing the lesson-06 plan

Lesson 06 produced a memory plan: a static assignment of every intermediate buffer to an offset in one arena, sized to the maximum concurrent live bytes. The caching allocator (also memory pool) is the runtime component that turns that plan into pointers. The motivation is the same fact lesson 06 opened on: a raw cudaMalloc costs on the order of a few microseconds, can synchronize the whole device, and fragments the heap. Calling it on the hot path — thousands of times per step — would dwarf the kernels you fused so carefully.

reserve in big segments

On startup the allocator grabs a few large slabs from the driver (PyTorch's caching allocator pulls 2 MB / 20 MB segments). Per-tensor allocations are then served from these segments with pointer arithmetic — no driver call, no sync.

free means return-to-pool

A free does not call cudaFree; it returns the block to the pool so the next allocation of that size reuses it. This is exactly lesson 06's "two non-overlapping buffers share a slot," made dynamic at run time.

stream-aware

Because frees are async, a block freed on a stream cannot be reused until that stream's work finishes. The allocator records a stream event and only recycles the block once it is safe — otherwise you'd hand live data to a new tensor.

fragmentation & OOM

Segments split into mismatched sizes leave gaps too small to use: you OOM with gigabytes "free" but unusable. expandable_segments and empty_cache() fight it; the lesson-06 plan, when honored, sidesteps it entirely.

The connection to the plan is the payoff. A pure runtime allocator (eager PyTorch) discovers buffer lifetimes as it goes and does a good but greedy job. A compiled artifact ships the plan, so the runtime can preallocate exactly the arena the compiler computed and hand out slices with zero discovery and zero fragmentation. Compile-time planning (lesson 06) and run-time allocation (here) are the two halves of one mechanism: one decides where bytes go, the other puts them there. And — foreshadowing §4 — CUDA graph replay requires the buffer addresses to be stable across replays, which is only possible if the allocator hands out the same pointers every time. So the graph-safe path needs a dedicated, captured memory pool.

4 · CUDA graphs: collapse N launches into one

Async streams hide the launch tax behind work; they do not remove it, and when there is not enough work (§2) the tax is exposed. To actually remove it you stop issuing N launches. A CUDA graph is a recorded DAG of GPU operations — kernels, copies, events — captured once and then replayed as a unit. Capture runs the model once in a special mode that, instead of executing launches, records them; replay submits the entire recorded sequence to the driver with a single call. The per-launch CPU cost — Python, dispatch, cudaLaunchKernel — collapses from N × ≈6 µs to ≈1 driver call's worth of overhead for the whole graph.

warmupRun the block a few times on a side stream so cuBLAS/cuDNN pick algorithms and the allocator settles. Capturing un-warmed code records the wrong kernels.

capturecudaStreamBeginCapture → run the block → cudaStreamEndCapture yields a graph object. No kernels actually ran; their launches were recorded.

instantiatecudaGraphInstantiate compiles the graph into an executable form once.

replaycudaGraphLaunch submits all N kernels in one call. Run it every step; the CPU is essentially idle.

In PyTorch this is what torch.compile(model, mode="reduce-overhead") turns on — Inductor captures the compiled region into a CUDA graph automatically. Because real models have regions that can be captured and regions that cannot (dynamic control flow, CPU-side logic), the runtime keeps cudagraph trees: a forest of captured subgraphs sharing one memory pool, so the decode loop's stable inner kernels replay from a graph while the changing outer logic stays eager. Worked number: a small-batch decoder spending ≈1 ms/token in launch overhead and ≈0.4 ms in actual kernel time can drop to ≈0.4–0.5 ms/token once the launch tax collapses — a 2× step speedup with zero change to any kernel, purely from killing dispatch.

The catch — and it is the whole reason lesson 17 exists

A CUDA graph is a recording. It replays the same kernels reading from the same memory addresses in the same order. That imposes two hard requirements that collide with everything dynamic:
Stable shapes. The captured kernels are specialized to the shapes seen during capture. A new sequence length or batch size needs a different graph — so dynamic shapes (lesson 11) either force re-capture per shape (a capture storm, the runtime cousin of lesson 11's compile storm) or force you to pad/bucket every request up to a captured size, paying lesson 11's wasted-FLOPs tax to keep the cache hit.
Stable addresses. Replay writes to the exact pointers recorded at capture. The caching allocator must therefore serve the captured region from a dedicated, private pool that hands out identical addresses every replay — and inputs must be copied into those fixed buffers rather than passed as fresh tensors. Forget this and the graph reads stale or wrong memory: a silent correctness bug, not a crash.

So CUDA graphs are the cleanest fix for lesson 01's launch ceiling and one of the most error-prone runtime features: the speedup is real and large, but it is bought with rigidity that fights lesson 11 directly and demands a cooperating allocator from §3.

5 · The AOT / serving path: Python-free dispatch

Everything above still assumes a live Python process driving the runtime. JIT compilation (Dynamo capturing on first call, lesson 02) is convenient for development but carries a Python interpreter, a warmup compile on cold start, and graph-break risk into production. The ahead-of-time (AOT) path removes Python from the serving loop entirely: compile the artifact offline, serialize it, and ship a self-contained executable the server loads and calls.

exporttorch.export traces the model to a sound, ahead-of-time graph (a stricter capture than Dynamo — no graph breaks allowed, dynamic dims declared up front, lesson 11).

compile to artifactAOTInductor lowers that graph to a shared library — a .so bundling the generated kernels, the memory plan, and a tiny C++ launcher. TensorRT does the analogous thing: build a serialized engine.

load & serveThe server dlopens the .so (or deserializes the engine) — no Python, no warmup compile. Each request runs the same guard-check → dispatch loop.

At serve time the loop is mechanical and matches the artifact-vs-runtime table from §1: check the guards (do these input shapes/dtypes match what was compiled? lesson 11) → on a hit, dispatch the cached kernels (optionally as a replayed CUDA graph, §4) → on a miss, fall back (recompile in JIT, or reject/pad in a static AOT engine that cannot recompile). The economics are the lesson-10 amortization argument, now applied to the whole artifact rather than one kernel: a compile cache (Inductor's FX-graph cache, on disk and shareable across processes) means a cold start reloads a previously compiled artifact instead of re-running capture → fuse → plan → tune. Pay the compile once; replay the warm artifact a million times.

For production LLM serving the runtime is the whole game, and dedicated serving engines push these ideas much further — paged KV-cache allocators, continuous batching that keeps the launch queue full, and CUDA-graph-captured decode steps. That depth lives in vLLM · lessons and SGLang · lessons; this lesson is the compiler-runtime boundary those engines sit on top of.

6 · Drive it: the launch-overhead killer

The widget runs the §2–§4 model on a chain of small kernels. Set the kernel count and the per-launch CPU tax (lesson 01's profiler numbers), then switch the dispatch mode: eager pays the full tax on every launch; streams overlaps the tax with kernel work but still exposes whatever the GPU cannot absorb; CUDA-graph replay collapses all N launches into ≈1 driver call. Watch total launch overhead and wall-time fall as you move down the modes — and watch the graph-capture valid? flag: flip on dynamic shapes and it turns FALSE, because a recording cannot replay against a shape it never captured. That flag turning red is lesson 11 reaching forward to break lesson 16's best trick.

Launch-overhead killer — eager vs streams vs CUDA-graph replay

A chain of N small kernels, each ≈1–2 µs of real work but paying a fixed CPU launch tax (lesson 01). Eager = tax on every launch. Streams = tax overlapped with work, but the GPU runs dry when the queue empties. CUDA-graph replay = all N launches → ≈1 driver call. The knob that breaks it: turn on dynamic shapes and graph capture goes invalid (a recording can't replay an unseen shape — lesson 11).

kernels in chain N: 120 per-launch CPU tax µs: 6 per-kernel GPU work µs: 2 dispatch mode: dynamic shapes (shape changes every call)

total launch overhead

—

wall-time / step

—

driver calls

—

graph-capture valid?

—

CPU launch tax GPU kernel work one driver call (graph)

The lesson the widget makes physical: streams help only while there is enough kernel work to hide the tax behind — push N up or work down and async stops saving you, exactly lesson 01's launch-bound regime. Graph replay does not depend on having work to hide behind; it removes the launches outright. But it is the one mode the dynamic-shapes switch can disable, which is precisely the tension lesson 17 and lesson 11 have to manage.

Failure modes & checklist

Failure modes

Expecting the compiler to fix launch overhead. Fusion cuts launches; it does not remove the dispatch tax on the launches that remain. Signal: a fused model is still slow at batch 1 and the GPU trace shows gaps between kernels — that is the runtime's job (graphs), not the compiler's.
A sync in the hot loop. .item(), print(loss), .cpu(), or a Python if on a tensor value drains the async queue. Signal: wall-time jumps with no kernel getting slower; the timeline shows the GPU idling at each sync.
Capturing a CUDA graph against dynamic shapes. Each new shape needs a re-capture — a capture storm — or silently replays the wrong-sized kernel. Signal: memory balloons as graphs pile up, or outputs are wrong only for off-size inputs.
Passing fresh input tensors to a replayed graph. Replay reads the captured addresses; new tensors live elsewhere, so the graph reads stale memory. Signal: a silent correctness bug that vanishes the moment you copy inputs into the static buffers.
Hot-path cudaMalloc defeating a good plan. Calling a raw allocator (or empty_cache()) inside the loop reintroduces the per-alloc sync and fragmentation. Signal: microsecond-scale gaps before kernels and a sawtooth memory profile.
Shipping JIT to production. First request pays a warmup compile; a graph break mid-serve drops to slow eager. Signal: p99 latency spikes on cold start or on rare inputs — the fix is the AOT export path.

Checklist

Separate artifact from runtime. Kernels + plan + guards are inert; the executor allocates, dispatches, and amortizes.
Keep the CPU ahead of the GPU. Async streams, pinned memory for copies, no stray syncs in the loop.
Serve buffers from a pool. Realize the lesson-06 plan from one preallocated arena; never cudaMalloc on the hot path.
Use CUDA graphs when shapes are stable. reduce-overhead for launch-bound regions; pad/bucket to a captured shape, and feed inputs through fixed buffers.
Reach for AOT to ship. torch.export → AOTInductor .so (or a serialized engine) for Python-free, warmup-free serving; rely on the compile cache for warm starts.

Checkpoint

Try it

Open the widget at its defaults (N = 120 kernels, 6 µs tax, 2 µs work, CUDA-graph mode). Read the wall-time, then switch to eager: the overhead should jump to ≈120 × 6 µs ≈ 720 µs of pure tax, dwarfing the ≈240 µs of kernel work — this is lesson 01's launch-bound ceiling, alive in a compiled model. Switch to streams: overhead drops because the tax overlaps with work, but it does not vanish — the GPU still runs dry whenever 6 µs of tax outruns 2 µs of work. Now push per-kernel GPU work up to 40 µs and watch streams nearly catch graph mode: with enough work to hide behind, async is almost as good. Drop work back to 2 µs and only graph replay stays fast. Finally, in graph mode, tick dynamic shapes and watch graph-capture valid? flip to FALSE and the overhead snap back to the eager number — the runtime falls off its best trick exactly where lesson 11 said it would.

Where this points next

You can now ship: the compiler's inert artifact has a runtime that allocates its buffers from a pool, dispatches its kernels asynchronously, and — when shapes hold still — replays them as a single CUDA-graph call that finally buries lesson 01's launch ceiling, all packaged as an AOT .so the server loads without Python. But look at what shipping required. CUDA graphs replay a fixed recording, so they break under the dynamic shapes of lesson 11. The caching allocator hands out reused storage, so a stale address is a silent wrong answer. And the kernels themselves arrived here already willing to move bits: fusion reassociated floats (lesson 04), quantization is deliberately approximate (lesson 15), and a recompile can swap in a different-accumulating kernel without telling you. The whole track kept promising "preserve semantics," yet half of Part VI legitimately changes the output. So the last question before synthesis is the one production actually asks: how do you tell acceptable float drift from a real bug, and find which pass — or which silent recompile — caused it? That is lesson 17, Debugging compiled models.

Takeaway

The compiler's job ends with an inert artifact: cached kernels (lessons 09–10, 15), a memory plan (lesson 06), and guards (lesson 11). It allocates nothing and launches nothing. The runtime / executor is the run-time other half that makes it run. Streams dispatch kernels asynchronously so the CPU races ahead and the per-op launch tax overlaps with kernel work — but only while there is enough work to hide behind; small models run the GPU dry and lesson 01's launch ceiling returns. The caching allocator realizes the lesson-06 plan by serving every buffer from one preallocated arena, never calling cudaMalloc on the hot path, recycling freed blocks the way the plan said two non-overlapping buffers share a slot. CUDA graphs are the real fix for the launch ceiling: capture the whole launch sequence once, replay it with ≈1 driver call (mode="reduce-overhead", cudagraph trees), collapsing N × ≈6 µs of tax to nearly nothing — at the price of stable shapes (it fights lesson 11) and stable addresses (it needs a graph-safe pool). The AOT path (torch.export → AOTInductor .so, serialized TensorRT engines) drops Python from serving entirely: a guard-check-then-dispatch loop over a compile-cached artifact, the lesson-10 amortization argument applied to the whole model. Compile time decides; run time does. And because replay, fusion, and quantization all bend the output or hide a recompile, the next step is learning to trust and debug it.

Interview prompts

What does a deep-learning compiler actually produce, and what runs it? (§1 — an inert artifact: shape/dtype-keyed kernels, a memory plan, and input guards. A separate runtime/executor allocates the buffers, checks guards, and dispatches the kernels every forward pass — compile time vs run time.)
If cudaLaunchKernel is async, why is a model still launch-bound at batch 1? (§2 — async hides the per-op tax only by overlapping it with kernel work. When kernels finish faster than the CPU can dispatch (tiny tensors), the GPU drains the queue and idles; the ≈5–25 µs tax is back on the critical path.)
Why does the caching allocator exist, and how does it relate to the memory-planning pass? (§3 — a raw cudaMalloc is slow, syncs the device, and fragments; the allocator serves buffers from preallocated segments with no driver call. It realizes the compile-time plan from lesson 06 at run time — planning decides the layout, the allocator places the bytes.)
How does a CUDA graph remove launch overhead, and what does mode="reduce-overhead" do? (§4 — it records the whole launch DAG once and replays it with a single cudaGraphLaunch, collapsing N CPU launches to ≈1; reduce-overhead makes Inductor capture the compiled region into a graph automatically.)
Why do CUDA graphs conflict with dynamic shapes, and what are the two fixes? (§4 — a graph replays kernels specialized to the captured shape from captured addresses. A new shape needs a re-capture (capture storm) or padding/bucketing up to a captured size (wasted FLOPs, lesson 11); addresses must come from a fixed graph-safe pool with inputs copied in.)
What is the difference between the JIT and AOT serving paths? (§5 — JIT (Dynamo) compiles on first call, keeps Python in the loop, risks warmup spikes and graph breaks; AOT (torch.export → AOTInductor .so / TensorRT engine) compiles offline into a self-contained library the server runs with no Python and no warmup compile, backed by a compile cache.)
You captured a CUDA graph and outputs are wrong only for some inputs. What likely happened? (§4 — you passed fresh input tensors instead of copying into the static captured buffers, so replay reads stale addresses; or the shape changed and the graph replayed a wrong-sized kernel. Fix: stable addresses + stable shapes, or re-capture per bucket.)