Training pipeline, registry, and lineage

A model artifact is only meaningful if you can explain where it came from and safely move it into production. The pipeline turns data snapshots into checkpoints; the registry turns checkpoints into deployable versions; lineage connects every deployed prediction back to code, data, features, labels, config, and eval. Without that graph, rollback and debugging are guesswork.

The production invariant

For any prediction in production, you should be able to answer: which model version produced it, trained from which data snapshot, with which feature definitions, code commit, hyperparameters, and eval gate? If not, you do not have an ML system. You have a model-shaped script.

Plain English dictionary

A pipeline is the repeatable factory that turns data into a model artifact. A registry is the controlled shelf of model versions allowed to serve traffic. Lineage is the receipt that proves what data, code, features, labels, and tests produced that artifact. A gate is a test the artifact must pass before it can move closer to users.

0 · Think of release, not training

Beginners often stop at "train a model." Production thinking starts with a stricter question: what would make this model safe to release and easy to roll back? The answer is linear:

Freeze the training inputs: data snapshot, labels, feature definitions, code, config.
Run training and produce an artifact: weights plus preprocessing, thresholds, tokenizer or schema, and serving config.
Evaluate the artifact against offline metrics, slices, calibration, cost, and safety gates.
Promote it through staging, canary, and production only when each prior promise is true.
Rollback the whole bundle if a promise breaks, not just the weights.

This lesson is the release discipline that keeps ML from becoming a pile of unrepeatable experiments.

1 · The artifact graph

General ML lifecycle design is a lineage problem. The output artifact is not just weights; it is a bundle of dependencies that must travel together.

Artifact	Why it must be versioned	Common bug if missing
Code commit	Training logic and preprocessing define the learned function	Metric cannot be reproduced
Data snapshot	Rows change, late labels arrive, deletes happen	Rollback model retrains differently
Feature definitions	Feature code is part of the model	Train/serve skew after a feature patch
Model config	Hyperparameters and seeds affect artifact	"Same" run produces incompatible output
Eval report	Promotion needs a gate and audit trail	Bad model ships on anecdote
Serving contract	Input schema, output schema, thresholds, calibration	Model scores change meaning in production

2 · Pipeline stages and gates

A robust training pipeline is a sequence of gates. The point is not bureaucracy; it is to fail early while the blast radius is small.

Ingest gate: schemas valid, source freshness inside SLO, required fields present.
Feature gate: point-in-time joins pass, train/serve parity checks pass, feature distributions inside expected ranges.
Label gate: label maturity window satisfied, label coverage and class balance within bounds.
Training gate: run completed, no NaNs, expected loss curve shape, resource budget respected.
Offline eval gate: task metrics improve or hold, segment regressions bounded, calibration checked.
Release gate: shadow/canary/A/B guardrails clear, rollback path exists.

Why the gates are linear

A downstream gate cannot repair an upstream broken contract. A high AUC cannot forgive leaked features. A green canary cannot make an unknown training dataset reproducible. Each gate consumes the guarantees before it and adds one new guarantee.

Sample release checklist

A candidate can enter staging only if the dataset snapshot is immutable, the feature definitions are versioned, the labels are mature, the run has no NaNs, and the offline metric beats the current production model on the primary slice with no critical slice regression. It can enter canary only if shadow traffic meets p99 latency and output-schema checks. It can enter production only if online guardrails hold: error rate, latency, cost, calibration, and business-critical segments stay inside declared bounds. Each item is pass/fail, not vibes.

3 · Experiment tracking vs model registry

These are often conflated, but they answer different questions.

System	Question answered	Writes many or few?	User
Experiment tracker	What did we try, and what happened?	Many runs	Researchers / MLEs
Model registry	Which artifact is allowed to serve traffic?	Few promoted versions	Production / release
Feature registry	Which feature definitions are valid and owned?	Moderate	Data + ML platform
Dataset registry	Which immutable snapshots can train or evaluate?	Moderate	Training + audit

A model can be a great experiment and a bad registry candidate. Promotion requires more than a metric: lineage, serving compatibility, calibration, safety, cost, and rollback.

4 · Promotion is a state machine

Treat model lifecycle as explicit state transitions, not a folder naming convention.

candidate │ offline eval + schema + lineage ▼ staging │ shadow traffic + load test + calibration check ▼ canary │ online guardrails + p99 + cost + segment metrics ▼ production │ ├── rollback → previous production artifact └── retire → archived, still reproducible

Transition	Must prove	Rollback object
candidate -> staging	Offline metrics, schema compatibility, lineage complete	Do not promote
staging -> canary	Serving p99, resource cost, output contract, shadow no-op safety	Remove canary route
canary -> production	Online primary metric wins; guardrails hold	Previous prod model + previous feature set
production -> retired	No active traffic, audit retention satisfied	Archived artifact

5 · The reproducibility budget

Perfect bitwise reproducibility is expensive and sometimes unnecessary. But the level should be chosen deliberately:

Level	Guarantee	Enough for
Audit reproducible	Can explain data/code/config/eval lineage	Most product ML
Metric reproducible	Retrain lands within expected metric band	Regression debugging
Artifact reproducible	Same weights/logits from same inputs	Regulated or safety-critical workflows

Do not pay artifact-level complexity if audit-level is enough. But do not accept "we think this was the data" for any model that gates money, safety, or user access.

Decision rule

Use audit reproducibility for low-risk personalization where rollback is easy. Use metric reproducibility when a regression can cost real money or trust and you must debug why a metric moved. Use artifact reproducibility when regulation, safety, finance, or legal review requires proving the exact output. The higher the harm and the harder the rollback, the more reproducibility you buy.

What carries forward

Lineage is the backbone of MLOps. A model artifact without data, feature, code, config, and eval lineage cannot be safely promoted or debugged.
Gates are ordered. Data correctness before model quality; offline before online; canary before production.
Registries are release systems. Experiment trackers remember attempts; registries decide what is allowed to serve.
Rollback includes dependencies. Reverting only weights while leaving a new feature definition or calibration map in place is not a rollback.