Monitoring, drift, and governance beyond LLM serving

Lesson 11 taught production for GPU serving: queue depth, KV utilization, p99 TTFT/TPOT, rollback, and cost. General ML adds another class of incidents: the service is fast and healthy, but the data changed, features went stale, calibration drifted, the ANN index aged, or one segment regressed. These are silent failures unless you monitor the material flow, not only the API.

The observability rule

Monitor every contract the system depends on: input schema, feature freshness, feature parity, prediction distribution, label distribution, calibration, segment metrics, index freshness, model version, and cost. API uptime says the code ran; ML monitoring says the decision still means what it meant yesterday.

Plain English dictionary

Monitoring asks whether yesterday's promises are still true today. Drift means the world, labels, users, or policy changed. Train-serve skew means the model saw one kind of input in training and another kind in production. Governance means privacy, fairness, rollback, audit, and cost rules are enforced by the system, not remembered by humans.

0 · The incident thinking loop

When a production ML metric moves, do not jump straight to retraining. Retraining before diagnosis can hide the bug. Work linearly:

Detect: which contract moved: schema, freshness, feature values, scores, labels, slice metric, index, latency, or cost?
Localize: did it move for all traffic or one segment, feature owner, model version, route, region, or device?
Compare: replay the same examples against the previous model/feature/index bundle.
Stabilize: rollback, use a feature fallback, rebuild an index, adjust a threshold, or shed a risky route.
Learn: retrain only after you know whether the problem is data drift, concept drift, calibration drift, or a broken pipeline.

1 · The monitor stack

Layer	Leading signal	Incident it predicts
Data quality	Schema violations, null spikes, enum drift	Feature computation or upstream logging broke
Freshness	Max feature age, stream lag, index age	Decisions made from stale world state
Train-serve parity	Online/offline feature diff on shadow examples	Model sees different inputs in prod than training
Prediction drift	Score distribution shift, entropy, top-K churn	Input distribution or model behavior changed
Calibration drift	Reliability curve moves after labels mature	Thresholds and expected values are wrong
Outcome drift	Delayed labels, conversion, complaints, chargebacks	True product quality changed
Slice health	Metrics by cohort, region, language, device	Aggregate hides subgroup regression

2 · Drift comes in four different species

"Drift" is too vague to debug. Name the species before picking a response.

Diagnosis order

First ask whether the inputs changed: that is data drift. If inputs are similar but mature labels got worse, suspect concept drift. If ranking/order is still good but probabilities no longer match reality, suspect calibration drift. If the system's own choices changed which examples receive exposure or labels, suspect policy drift. This order prevents the common mistake of retraining for a logging bug or recalibrating for a changed fraud pattern.

Drift type	What moved	Example	Response
Data drift	P(x)	More mobile users, new merchant category	Check feature distributions; maybe retrain
Concept drift	P(y\|x)	Fraudsters change tactics	Fresh labels, faster retrain, new features
Calibration drift	Score-to-probability mapping	0.8 risk now means 0.5 actual rate	Recalibrate or adjust thresholds
Policy drift	The system changed its own data	Ranker overexposes one content type	Holdout/exploration traffic, bias correction

2.5 · Drift statistics — measure movement before naming a cause

Once you know which distribution might have moved, use a statistic that matches the object. For binned features and score distributions, three common first-pass monitors are:

PSI = Σ_i (cur_i − base_i) · ln(cur_i / base_i)
KS = max_x |F_cur(x) − F_base(x)|
KL(cur || base) = Σ_i cur_i · ln(cur_i / base_i)

Worked example: baseline bins (.25,.25,.25,.25), current bins (.40,.30,.20,.10). Then PSI≈.228, KS=.20, and KL≈.106 nats. Read that as meaningful movement, not automatic model failure.

Thresholds are heuristics, not truth

PSI <.1 is often treated as small, .1–.25 as warning, and >.25 as major. KS should be read with effect size and the two-sample critical value 1.36·sqrt((n+m)/(n·m)). KL needs smoothing for empty bins and is best thresholded against historical p95/p99. These statistics tell you something moved; mature labels and slice metrics tell you whether the decision got worse.

Labels arrive late

Concept and calibration drift often cannot be confirmed immediately because labels arrive later. Monitor proxies in real time, but do not confuse proxies with truth. Fraud, ads conversion, retention, and safety appeals all have delayed labels; mature-label dashboards need their own cadence.

3 · Train-serve skew monitor

The most direct skew test is to log online requests, replay the same request through the offline feature pipeline, and diff the feature vectors.

skew_rate = count(features_online != features_offline_replay) / count(features_checked)

For numerical features, use tolerances; for categorical and embedding features, version and exact-match semantics matter. Alert on both global skew and skew by feature owner. A single hot feature drifting can damage every model that consumes it.

Skew symptom	Likely cause	Fix
Online value missing, offline present	Online store lag or TTL expiry	Freshness alert, backfill, fallback policy
Aggregate off by a window	Event-time vs processing-time mismatch	Shared window definition and watermark
Embedding dimensions/version differ	Embedding model and index not upgraded together	Versioned embedding/index bundle
String/category mismatch	Different normalization code	Single feature definition, parity tests

4 · Retrieval/index monitoring

For search/RAG/recsys, the index is part of the model. Monitor it like a model artifact.

Metric	Meaning	Bad day
Index freshness	Age of newest included document/item/user embedding	New content invisible; deleted content still retrievable
Recall@K on probe set	ANN approximation quality	Latency optimization silently drops relevant items
Lookup p99	Serving tail for retrieval stage	Ranking SLO breaks before model score runs
Filter/ACL miss rate	Permissions and business constraints	Unauthorized or unavailable items appear
Embedding version mix	Whether index and query encoder match	Semantic space mismatch kills recall

5 · Governance is operational, not decorative

Governance means the system can enforce constraints automatically. It is not a PDF saying "be careful."

Constraint	Operational enforcement	Failure if informal
Privacy / retention	PII redaction, TTL, deletion propagation, consent flags	Training on data user opted out of
Fairness / slices	Predeclared slice guardrails and release blocks	Aggregate win harms subgroup
Explainability / audit	Store reason codes, features, model version, threshold	Cannot explain adverse action
Rollback	Versioned model + feature + threshold bundle	Weights revert but feature contract stays broken
Cost	$/prediction and capacity dashboards	Traffic grows faster than unit economics

Incident playbook

When a metric moves: freeze deploys, identify the layer (data, feature, model, decision, label), compare slices, replay shadow traffic against previous artifact bundle, then choose rollback, threshold change, feature fallback, index rebuild, or retrain. Retrain is rarely the first move; it is the move after you know what changed.

What carries forward

Monitor contracts, not only uptime. Fast wrong decisions are still incidents.
Name the drift species. Data, concept, calibration, and policy drift require different responses.
Train-serve skew is measurable. Replay online examples through offline features and diff them.
Indexes are model artifacts. Version, monitor, rebuild, and rollback them with the model.
Governance must be enforceable. Privacy, fairness, explainability, rollback, and cost need system hooks.