Part I - Inference foundations

Model Artifacts and Registries

Lesson 03 gave you a model server and a controller that reconciles a Deployment for it — the machinery that runs a model. But that controller pointed at a model by name as if the name were enough, and it is not. The same string can resolve to different weights on different days, and "the model" is never just the weights: it is a tokenizer, a config, a license, a quantization, a base-model lineage, and an eval record that must all travel together. This lesson turns a giant opaque download into a versioned, signed, discoverable platform artifact — one your serving stack can load efficiently and your release process can promote with confidence. It is the difference between "we ran a model" and "we can name, reproduce, and roll back exactly what we ran."

Source coverage

PDF Chapter 2: model data formats and model registries — Safetensors, GGUF/GGML, ONNX, Hugging Face, MLflow, Kubeflow Model Registry, and OCI registries.

Linear position

Prerequisite: Lesson 03 (Model servers and controllers) — you can stand up an inference server and have a controller reconcile a Deployment toward it. The controller assumed "the model" was already a coherent, loadable thing.
New capability: Treat a model version as a first-class, versioned artifact — identity, format, and lineage made explicit — so it can be discovered, signed, promoted, and rolled back like any other production dependency.

The plan

Five moves. (1) Establish that a "model version" is far more than parameters, and why that distinction is operational, not academic. (2) Walk the three storage formats the chapter names — safetensors, GGUF/GGML, ONNX — as compatibility contracts with the runtime, defining each at first use. (3) Treat the registry as a control plane and compare the four paths the book lists (Hugging Face, MLflow, Kubeflow, OCI) in a trade table. (4) Name the central trade-off — velocity vs governance — and make it concrete with an artifact-identity record and a worked size-by-quantization number. (5) Failure modes, a checklist, and the hand-off into getting weights into pods.

1 · A "model version" is more than parameters

The instinct from ordinary software is to treat a model as a file: download model.bin, mount it, point the server at it. That instinct breaks immediately, because a running model depends on several artifacts that are not the weights and that must match the weights exactly. Mismatch any one and you do not get an error — you get silently wrong output.

A complete model version is the bundle of:

→parameters (the weights) — the tensors themselves, the multi-gigabyte payload everyone pictures

→tokenizer — the exact vocabulary and merge rules that turn text into the token IDs the weights were trained on; a different tokenizer maps the same prompt to different IDs and corrupts every response

→config — architecture, context length, special tokens, dtype assumptions the runtime needs to build the model correctly

→license + lineage — what you are legally allowed to do with it, and which base model it descends from

→quantization + adapter relation — whether the weights are full-precision or compressed, and whether this is a base model or a LoRA adapter that only makes sense layered on a specific base

→intended runtime + eval record — which server can load it efficiently, and the measured quality you are promising callers

To make the "silently wrong" claim concrete: suppose you update the weights to a new fine-tune but the pod still mounts last week's tokenizer. The server starts cleanly — both files are valid — but the tokenizer now maps the word " Kubernetes" to token ID 4711 while the new weights were trained expecting ID 9023 for that same string. Every prompt is quietly fed token IDs the model never saw in training. There is no stack trace, no 500; the model just emits plausible-looking nonsense at a slightly elevated rate. The only signal is a quality metric, and only if you are watching one. This is why the bundle, not the weights, is the unit of versioning.

The mental model for the lesson: the registry is the source of truth for promotion; the format is the compatibility contract with the runtime; the cluster mount path is only the last mile. These three concerns are independent. The registry decides which artifact is approved and where it sits in dev/staging/prod. The format decides whether a given runtime can load it and how fast. The mount path (the volume in lesson 05) only decides how the bytes physically reach the pod. Confusing them is the root of most artifact incidents: people pin a mount path, think they have pinned a version, and discover the underlying bytes moved.

Quantization appears repeatedly below, so define it once: quantization stores each weight in fewer bits than the original training precision — typically fp16 (16 bits) down to 8-bit or 4-bit integers — trading a small, measurable quality loss for a large drop in file size and memory footprint. It is a property of the artifact, not the runtime, which is exactly why it must live in the version's identity.

2 · The format is a contract with the runtime

The storage format answers one question: can this runtime load these weights efficiently, and safely? It is a contract, not a preference. Pick a format the runtime cannot read and the model simply will not serve. The chapter names three that matter, each born from a different deployment world.

safetensors

A simple tensor container designed to be safe to load — it stores raw tensors plus a small JSON header, with no executable code, unlike the older pickle-based checkpoints that could run arbitrary code on load. It is also memory-mappable, so a server can load only the slices it needs. This is the de-facto standard for transformer serving (vLLM, TGI, Hugging Face). Default choice unless the runtime forces otherwise.

GGUF / GGML

The format family from the llama.cpp world. GGML was the original; GGUF is its successor, a single self-describing file that packs weights, metadata, and tokenizer together and supports many quantization levels. Built for local, CPU-only, and CPU/GPU-hybrid inference where you want one portable file and aggressive quantization. Common on laptops and edge nodes, less so in datacenter GPU serving.

ONNX

Open Neural Network Exchange — a framework-neutral computation-graph format. Instead of just storing tensors, it serializes the model's operations so an ONNX runtime can execute it across backends. The win is graph portability and runtime optimization; the cost is that not every model exports cleanly, and you need an ONNX-serving stack to consume it.

The throughline: safetensors and GGUF are weight containers (the runtime still knows the architecture); ONNX additionally captures the graph. For the GPU transformer serving this track is about, safetensors is the path of least resistance. GGUF earns its place when you serve on CPU or commodity hardware with heavy quantization; ONNX earns its place when portability across runtimes, or a specific optimizing runtime, is the goal.

The safety property in safetensors is not cosmetic. The previous standard, PyTorch .bin/.pt checkpoints, are Python pickle files — and unpickling executes arbitrary code embedded in the file. Loading a pickled checkpoint from an untrusted source is equivalent to running a stranger's script as root on your GPU node, a class of supply-chain attack that has shown up in the wild on public model hubs. safetensors removes the attack surface entirely: the file is just a length-prefixed JSON header followed by raw tensor bytes, with no code path that can execute. That is precisely why a governed pipeline (§4) prefers it: the format itself is part of your supply-chain posture, not only a performance choice. The memory-map property matters operationally too — because the loader can mmap the file and let the OS page in tensors on demand, a multi-replica node can share one page cache across pods instead of each pod allocating its own copy, which lesson 05 turns into a cold-start optimization.

3 · The registry is a control plane, not a download mirror

If the format is the runtime contract, the registry answers the orthogonal question: which artifact is approved, discoverable, and promotable? A registry is not just a place to fetch bytes — it is where a model acquires an identity, a lifecycle state, and a promotion history. The book lists four paths, and the right one depends less on taste than on which system your release process already trusts.

Registry path	Strength	Risk
Hugging Face	The discovery and distribution hub — broadest ecosystem, model cards, easiest pull for exploration and research.	An external dependency: availability, licensing terms, and promotion control all sit outside your cluster. A name can point at moved bytes.
MLflow Model Registry	Lifecycle states (staging/production) plus experiment lineage — it ties a model back to the run, data, and metrics that produced it.	Strong on lineage and metadata, weaker on distributing very large artifacts efficiently; you often pair it with separate object storage.
Kubeflow Model Registry	Kubernetes-native metadata store — fits cleanly when the rest of your ML platform is already Kubeflow-aligned.	Most of its value assumes you are committed to the Kubeflow platform; less compelling outside it.
OCI registry	Treats the model as a content-addressed artifact (referenced by immutable digest), reusing the container ecosystem's promotion, caching, signing, and access-control — the exact workflow your images already use.	Requires packaging discipline: someone must define how a model bundle becomes an OCI artifact and keep that consistent.

"Content-addressed" is the key phrase for the OCI path. A digest like sha256:9f2c… is computed from the bytes themselves, so it is immutable by construction: the same digest can only ever mean the same bytes. A tag like :latest or :prod is mutable — it can be repointed at new bytes at any time. Promoting by digest is what makes a rollback honest; promoting by tag is how versions silently change underneath you (failure mode 1 in §5).

Concretely, an OCI artifact is just an OCI image manifest whose layers happen to hold model files instead of a root filesystem (the OCI spec lets a manifest carry an arbitrary artifactType and arbitrary layer blobs). So the same registry, the same push/pull, the same digest math, and the same RBAC that already guard your container images now guard the model — no second system to secure. Tools like ORAS push the weights, tokenizer, and config as layers under a model media type; cosign then signs the manifest digest, producing the supply-chain attestation referenced in the identity record below. The "packaging discipline" cost is real but bounded: someone has to define the bundle layout once and wire it into CI, after which a model promotes exactly like an image.

4 · Velocity vs governance — the central trade-off

Every artifact decision is a point on one axis: velocity versus governance. At the fast end, you pull a model straight from an open hub by name — perfect for exploration, where the cost of being wrong is a wasted afternoon. At the governed end, you promote a signed, versioned, content-addressed OCI artifact through dev, then staging, then prod — slower, but reproducible, auditable, and rollback-safe. Research legitimately starts at the fast end; production almost always needs the governed end. The skill is knowing where the boundary is and crossing it deliberately, not by accident.

What "governed" actually buys you is captured in an artifact identity record — the metadata that turns a download into a production artifact. Make it queryable by humans and by automation:

model:
  name: support-assistant
  version: 2026.06.0
  artifact_digest: sha256:9f2c1a7b4e…       # immutable, content-addressed
  base_model: meta-llama/Llama-3.1-8B        # lineage
  is_adapter: true                            # this is a LoRA, not a full model
  adapter_base_digest: sha256:0ab4…           # the base it must layer on
  tokenizer_hash: sha256:c41d…                # must match the weights exactly
  format: safetensors
  quantization: int8
  license: llama-3.1-community                # + export constraints reviewed
  intended_runtime: vllm>=0.6
  eval_id: eval-2026-06-12-support-v2         # the quality you are promising
  signature: cosign://…                       # provenance / supply-chain attestation

Now a worked number for why quantization belongs in that record — the same 8-billion-parameter model in three formats. A rough rule: bytes ≈ parameters × bytes-per-parameter, plus a few percent overhead.

fp16 (16-bit)

8e9 × 2 bytes = 16 GB. Full-precision serving; needs a GPU with ≥ ~20 GB HBM (high-bandwidth memory, the GPU's on-card RAM) once you add the KV cache.

int8 (8-bit)

8e9 × 1 byte = 8 GB. Half the footprint and download; fits comfortably on a 24 GB card with room for the working set, at a small quality cost.

4-bit GGUF

8e9 × 0.5 bytes ≈ 4 GB (plus quant overhead, ~4.5 GB in practice). Runs on modest hardware and even CPU; the largest quality trade-off of the three.

So the same model ships as a 16 GB, an 8 GB, or a ~4.5 GB artifact depending only on quantization — a 3.5× swing in download time, cache footprint, and minimum GPU. Put a number on the download too: over a 10 Gbit/s (≈ 1.25 GB/s) node link, the fp16 artifact takes 16 / 1.25 ≈ 13 s to pull cold, the int8 about 6.5 s, and the 4-bit about 3.6 s — and every replica on a fresh node pays that toll until the bytes are cached (the cold-start problem lesson 05 attacks). At 8B these are seconds; the same arithmetic on a 70B model is the difference between a 13-second and a 110-second scale-up. That is why "which 8B model" is an underspecified question: without the quantization and format in the identity record, you cannot predict where it fits, what it costs to cache, what a scale-up will cost in egress, or whether two nodes are even running the same thing.

Thinking drill — is this a production artifact yet?

Before deploying any model, answer five questions. (1) Can I name the exact artifact digest — not a tag? (2) Can I reconstruct precisely which tokenizer was used? (3) Can I state the license and any export constraints? (4) Can I roll back without relying on a mutable tag? (5) Can I correlate production traffic to a specific eval report? If any answer is "no," what you have is a download, not a production artifact — and the gap is exactly the work this lesson is about.

5 · Where artifact handling goes wrong

Failure modes

The version moved but the image tag didn't. Your Deployment pins the same container image and a model reference like :prod, but the bytes behind that mutable tag were repointed. The pod count, the YAML, and the dashboard all look unchanged — yet output quality shifts overnight. Signal: eval metrics drift with no deploy event. The fix is to pin by digest, never tag.
Tokenizer and weights drift apart. They were promoted as separate artifacts and a tokenizer update shipped without a matching weight update (or vice versa). There is no load error — the model just produces subtly garbled or off-distribution text. Signal: rising rate of malformed/nonsensical completions with healthy infra metrics. The fix is to bind them in one versioned bundle with a recorded tokenizer_hash.
License/export constraint found too late. A model's license or export restriction is discovered after it has already been pulled and cached onto production nodes — now you have a compliance problem on running infrastructure, not a code review comment. Signal: legal/security review flags a model already serving traffic. The fix is to gate license review at promotion, before the artifact can reach prod nodes.
Adapter without its base. A LoRA adapter is promoted as if it were a standalone model; the serving node loads it against the wrong base (or no base), producing weights that are valid bytes but meaningless. Signal: coherent-looking model, incoherent answers. The fix is to record and enforce adapter_base_digest.
Unqueryable metadata. The identity lives in a wiki page or someone's head, so automation cannot check it and humans cannot audit it at 2 a.m. Signal: nobody can answer "which exact artifact is in prod?" during an incident. The fix is a machine-queryable registry record, populated before promotion.

Implementation checklist

Does every model carry an identity record — name, version, digest, license, base model, quantization, tokenizer hash, intended runtime, eval id?
Is the artifact referenced by immutable digest in the Deployment, never by a mutable tag?
Are weights, tokenizer, and config promoted as one bundle so they cannot drift apart?
Do you default to safetensors for transformer GPU serving, switching to GGUF/ONNX only when the runtime genuinely requires it?
Do you use an OCI registry when distribution, caching, signing, and promotion should mirror your existing container workflow?
Is license and export review a gate at promotion, before the artifact can be cached on prod nodes?
Is the metadata queryable by both humans and automation before anything reaches production?

Checkpoint exercise

Try it

Take one open model you would actually serve and write its full artifact identity record (use the YAML in §4 as the template): pin the digest, name the base model and tokenizer hash, state the format and quantization, and cite the license. Then answer the five thinking-drill questions against it. For any "no," write the one concrete step — sign it, repackage as an OCI artifact, bundle the tokenizer, run an eval — that would turn that "no" into a "yes." That list is your promotion gate.

Where this points next

You now have a model version that is named, signed, and discoverable — an artifact your release process can promote and roll back with confidence. But the registry record is still just a pointer; the bytes are not yet on the GPU node. Lesson 05 — Getting Model Weights Into Pods — builds the last mile: how those gigabytes physically reach a pod (init containers, CSI volumes, image-packaged weights, sidecar pullers), the latency and cold-start math of each path, and how to keep the digest you promoted here intact all the way to the mount point. The identity record from this lesson is what that delivery machinery is obligated to honor.

Takeaway

A model version is never just parameters — it is weights plus tokenizer, config, license, lineage, quantization, intended runtime, and eval record, and those must travel together. Three concerns stay separate: the registry is the source of truth for promotion, the format is the compatibility contract with the runtime (safetensors for transformer GPU serving, GGUF for local/CPU-hybrid, ONNX for graph portability), and the mount path is only the last mile. The registry is a control plane, not a mirror: Hugging Face for discovery, MLflow for lineage, Kubeflow for platform fit, and OCI for content-addressed, signable, promotable artifacts. The governing trade-off is velocity versus governance — pull-by-name for exploration, promote-by-digest through dev/staging/prod for production — and the test of which side you are on is whether you can name the exact digest, reconstruct the tokenizer, explain the license, roll back without a mutable tag, and correlate prod traffic to an eval. If any of those is "no," it is a download, not a production artifact.

Interview prompts

Why is a "model version" more than the weights file? (§1 — it must include the tokenizer, config, license, base-model lineage, quantization, adapter relation, intended runtime, and eval record; mismatch any one and you get silently wrong output, not an error.)
What question does the storage format answer, and when do you pick each? (§2 — "can this runtime load it efficiently and safely?" safetensors is the standard for transformer GPU serving; GGUF/GGML for local/CPU-hybrid with heavy quantization; ONNX for graph portability across runtimes.)
Why is "content-addressed" the operative property of an OCI registry path? (§3, §5 — a digest is computed from the bytes, so it is immutable; promoting and pinning by digest makes rollback honest, whereas a mutable tag lets versions change underneath you.)
State the central trade-off and which side production lives on. (§4 — velocity vs governance: pull-by-name from an open hub is fast for exploration; promoting a signed, versioned, content-addressed artifact through dev/staging/prod is slower but reproducible, and production needs it.)
The same 8B model ships at 16 GB, 8 GB, or ~4.5 GB — why, and why does it matter? (§4 — fp16 vs int8 vs 4-bit GGUF; a 3.5× swing in download, cache footprint, and minimum GPU, so quantization must be in the identity record or "which 8B model" is unanswerable.)
How does a model version change while the Deployment looks untouched? (§5 — the image tag and YAML stay the same but a mutable model tag was repointed at new bytes; eval drifts with no deploy event. Pin by digest to prevent it.)
What five questions tell you whether something is a production artifact? (§4 — can you name the exact digest, reconstruct the tokenizer, explain the license, roll back without a mutable tag, and correlate prod traffic to an eval report? Any "no" means it is still just a download.)