Part I - Inference foundations

Getting Model Weights Into Pods

Lesson 04 gave each model version an immutable identity in a registry — a content-addressed artifact you can name, pin, and roll back. Knowing which bytes you want is only half the job. The runtime cannot serve weights it cannot read, and a 100 GB checkpoint does not arrive the way a 4 KB ConfigMap does. This lesson is about the delivery path: how the bytes physically travel from a registry or object store onto a node and into the pod's filesystem before the server reports ready. That path — not the model code — decides how long a scale-up takes, how much egress every replica burns, and whether a rollback is reproducible.

1 · A model is a supply-chain artifact, not configuration

The instinct from ordinary backend work is to treat "the model" like any other piece of app config: bake it in, mount it, ship it. That instinct breaks at scale. A ConfigMap is kilobytes; a serving checkpoint for a mid-sized open model is tens to hundreds of gigabytes. Llama-3 70B in 16-bit weights is about 70 × 10⁹ params × 2 bytes ≈ 140 GB; even quantized to 8-bit it is ~70 GB. Those bytes do not live in your container image by default, they do not fit in etcd, and they are far too large to re-fetch casually. So the right mental model is the one this lesson keeps returning to:

The single distinction that organizes every option is this: distributing the artifact (getting the bytes near the pod) is a different problem from reading the artifact (the runtime opening files and loading tensors into GPU memory). A path can be excellent at one and poor at the other. An OCI registry is built to distribute immutable, content-addressed layers and to benefit from node-level layer caching — but how the runtime then reads those bytes depends on whether they are unpacked to disk or mounted. A network filesystem is built to expose one copy of the bytes to many readers — but its read throughput under concurrent load is now on your critical path. Keep the two questions separate and the paths stop looking interchangeable.

2 · The cold-start equation — and why scale-up is a network event

"Cold start" is the time from a pod being scheduled to the model server reporting ready (passing its readiness probe so it can take traffic). It is not one thing. Decompose it:

cold start  =  image pull  +  model fetch  +  model read  +  runtime init  +  warmup
              └──────────── platform concern ───────────┘   └─ model-server concern ─┘

• image pull — pulling the container image (runtime + server code) if it is not already on the node.
• model fetch — moving the model bytes onto the node / into a volume the pod can read (this is the term the delivery path controls).
• model read — the server opening the files and loading tensors from disk/network into host RAM and then GPU HBM (the GPU's high-bandwidth on-package memory).
• runtime init — CUDA context creation, allocator setup, building/compiling kernels.
• warmup — running a few dummy requests so lazily-compiled kernels and the KV cache allocator (the per-request store of attention keys/values the server reuses across decode steps) are hot before real traffic.

The first three terms are platform concerns and the last two are model-server concerns. The mistake is letting the platform terms hide. Here is why they dominate on a cold node. Suppose you scale from zero and the only path to the bytes is fetching a 100 GB model over a 10 Gbit/s network link:

The fix is to make model fetch approach zero on the common path, which is exactly what a warm node cache does. If the model bytes are already on the node — because a prior pod pulled the OCI layers, or because a PV already holds them, or because you pre-pulled before scaling — then for the next pod model fetch is a local-disk read at GB/s instead of a network transfer at Gbit/s. The same 100 GB read from a 5 GB/s local NVMe is 100 / 5 = 20 seconds, and four-fifths of cold start vanishes. This is the whole game: cold-start economics are dominated by whether the bytes are already there, and your delivery path is what decides cache-hit behavior. Drag the knobs below to feel the curve.

// misses fetch over a shared link and serialize on it; hits read from local disk.
var misses   = Math.round(N * (1 - hitRate));
var bits      = sizeGB * 8;                 // gigabits per model
var fetch1    = bits / bwGbps;              // seconds, one miss alone
var readLocal = sizeGB / LOCAL_GBps;        // local NVMe read (cache hit path)
var ready1    = fetch1 + READ_INIT_WARMUP;  // single missing replica
// the burst of misses shares the egress link -> they serialize:
var tail      = misses * fetch1 + READ_INIT_WARMUP;
var egressGB  = misses * sizeGB;

3 · The four delivery paths and what each trades

The paths are not ranked — they trade who pays for bandwidth, how rollback works, and whether scale-up is predictable. Pick by model size and rollout pattern.

Notice how each path answers the two questions from §1 differently. The PV is shared distribution with shared reading — one copy, many concurrent readers, so it shines when reads parallelize and dies when they contend. The init copy is per-pod distribution with fast local reading — great read path, but it re-pays fetch on every restart. The modelcar and image volume both get distribution and node caching from the OCI registry; the image volume is simply the modelcar idea promoted into the platform so you no longer babysit a sidecar.

4 · Measure model load as its own metric

The operational rule that ties this together: model load is its own metric, never hidden inside pod startup. If all you have is "pod took 6 minutes to become ready," you cannot act. A platform that has done this right can decompose that number and tell you exactly which term in the cold-start equation was slow:

Emit each of these as a separate timing (a span or a labeled histogram), distinct from the HTTP process-start time. The payoff is that an incident becomes a lookup instead of a guess: you read the slow term and go straight to its fix, rather than re-running the deploy and hoping.

5 · Where image volumes stand in 2026

The book's chapter was written when OCI image volumes were an early, experimental idea, so its caveats need refreshing. As of the current Kubernetes task documentation, image volumes are stable in recent Kubernetes (alpha in v1.31, beta in v1.33, stable and enabled by default by v1.35/v1.36). That changes the design conversation. The question is no longer "is this experimental, do I dare?" — it is the more concrete supply-chain question: does my cluster version, my container runtime, my registry policy, and my artifact format all support the path I want? A stable API does not mean your container runtime implements it, that your registry serves the artifact format, or that your image-pull policy allows it. Keep the external reference because runtime and registry support can still lag the API feature state, and the docs track the runtime support matrix.

6 · Failure modes and a checklist

Checkpoint exercise

Where this points next

You can now get the right bytes onto a node and into a pod, and you can say why a cold start was slow. But every term in the cold-start equation assumed a node was there and capable — that it had the GPU, the HBM, the driver, and the room to mount the volume. Nodes with accelerators are not interchangeable commodity compute; they are a scarce, typed, expensive resource with their own scheduling and lifecycle rules. The next lesson — 06 - GPU Nodes as a Platform Layer — treats the GPU node as a platform abstraction: how Kubernetes discovers, labels, taints, and schedules accelerators, and why "just add a node" is its own multi-minute event that compounds the cold start you just learned to measure.

Interview prompts

• Why treat a model as a supply-chain artifact rather than configuration? (§1 — it is tens-to-hundreds of GB, does not fit in etcd or a ConfigMap, and its delivery path — not its code — controls startup time, node pressure, rollback reproducibility, and cache-hit rate.)
• What two distinct questions does every delivery path answer? (§1 — how the bytes are distributed near the pod vs how the runtime reads them; a path can be good at one and poor at the other, e.g. a PV distributes one copy but its concurrent read throughput is now your critical path.)
• Decompose cold start, and which terms are the platform's? (§2 — image pull + model fetch + model read + runtime init + warmup; the first three are platform concerns, only read+init+warmup are the model server's, and the platform terms dominate on a cold node.)
• Work the cold-start number for a 100 GB model. (§2 — 100 GB = 800 Gbit; at 10 Gbit/s that is an 80 s transfer floor before any read/init; a 20-replica burst is 2 TB of egress that serializes to ~27 min on a shared 10 Gbit/s link unless a warm node cache makes fetch ~free.)
• Compare PersistentVolume vs init copy vs image volume, and when each wins. (§3 — PV: one shared copy, many readers, wins when reads parallelize and storage is fast; init copy: per-pod fast local read but re-fetches every restart; image volume: native OCI distribution + node cache + read-only mount with reproducible digest rollback, on a Kubernetes recent enough for the stable feature — v1.35/v1.36.)
• Why measure model load separately from pod startup? (§4 — so an incident is a lookup: you can say it was slow because the node lacked the layer / storage was slow / CUDA init was slow / the runtime was warming, and go straight to the fix instead of re-running the deploy.)
• How do you prevent a scale-up from becoming a thundering-herd egress event? (§6, §2 — node cache / PV / pre-pull so only the first pod on a node fetches, reference bytes by digest, and alert on egress bytes spiking toward N × model_size per scale-up.)