Deployment & serving

A model that wins offline has done nothing yet. It earns its keep only when it answers a live request correctly, inside a latency SLA, on traffic it has never seen — and keeps doing so after you stop watching.

The latency budget is the real spec

A ranker answers under a hard wall-clock deadline — typically 100–300 ms end-to-end for the whole funnel, of which the ranking serving call gets maybe 20–80 ms. That budget is not advice; it is the spec. A model that is 0.3% more accurate but 2× slower is, in production, a worse model: it either misses the deadline (and a degraded fallback gets served instead) or forces you to shrink the candidate set, losing more accuracy than you gained.

So before you touch the model, decompose where the milliseconds go. A single ranking request is not "the forward pass" — it is a chain, and the surprise is which link dominates.

Why P99, not the mean

Your SLA is written on a tail percentile (P99, sometimes P999), not the average, because a user is one request, not a distribution. With a 200 ms budget and a fan-out of, say, 30 parallel sub-calls, the request finishes when the slowest sub-call returns — so even a 1-in-100 slow feature read becomes a near-certain slow request. This is the tail-at-scale effect: as fan-out grows, the request's latency tracks the tail of the component latency, not its mean. Mitigations: hedged requests (issue a duplicate after a delay, take the first), tight per-call timeouts with a fallback, and trimming the number of dependencies on the critical path.

Batching — the one free lever

A ranking request already scores 100–1000 candidates; that is a batch, and it is what makes GPU serving efficient — the weights are read once and reused across candidates (the arithmetic-intensity argument). Across requests, dynamic batching (Triton, TF-Serving) groups concurrent requests arriving within a few-ms window into one forward pass, trading a tiny latency add for a large throughput gain. The tuning knob is the max queue delay: too small and you lose the throughput; too large and you blow the tail.

Export & the serving runtime

The training artifact — a PyTorch .pt graph, a TF SavedModel — is built for flexibility (autograd, eager Python). The serving artifact wants the opposite: a frozen, optimized, language-agnostic graph that a high-throughput C++ runtime executes with no Python in the loop. That is what export buys you.

Two optimizations happen during export and recur in 19 · Large-scale optimization: operator fusion (collapse matmul → bias-add → activation into one kernel, killing memory round-trips) and quantization (FP32 → INT8 cuts memory ~4× and speeds up integer math — but is also the first place numerical results drift from training, which is exactly the next section's problem).

What each optimization actually buys — with numbers

"Quantize the model" is not one lever; it is a family, and the headline you quote in a design review should be the measured one, not the hopeful one. The numbers below are representative of a multi-head ranker (an MMoE-style model) served on a commodity inference GPU.

Serving a billion-row embedding table

The ranker's parameters are dominated by one object: the embedding table — one learned row per categorical id (every user, item, ad, query token). At industrial scale this table alone is hundreds of GB to terabytes, far past what one box holds in RAM, and it is read on every request and written continuously by online learning. The dense MLP on top is a rounding error by comparison. So "serve the model" is really "serve a TB-scale, constantly-mutating key-value store under a few-ms read SLA" — a storage problem wearing a model's clothes. (The sharding and ANN side of this lives in 19 · Large-scale optimization and 03 · Embeddings & ANN; here we cover the serving mechanics.)

Hot/cold storage tiering

Access to embedding rows is brutally Zipfian — a few popular items and active users are read constantly; the long tail almost never. So you tier the storage to match: hot rows in memory (or GPU VRAM for the hottest), cold rows on local SSD, full table in a distributed KV store (the Angel / BytePS class of system). A read checks RAM first, falls to SSD on a miss. The win is that the few-ms read SLA only has to hold for the hot set; the tail tolerates an SSD hop. Prefetching the embeddings for a user's recent-history items off the critical path hides most of the remaining latency.

Consistent-hash sharding with locality

What doesn't fit on one box is sharded across a parameter-server cluster, with each id mapped to a shard by consistent hashing — chosen over plain modulo because adding or removing a node remaps only ~1/N of keys instead of nearly all of them, so a scale-up doesn't trigger a table-wide reshuffle (and a cold cache stampede) mid-traffic. Locality-aware scheduling then co-locates a request's likely ids on as few shards as possible, because a lookup that fans out to 30 shards pays the tail-at-scale penalty from 17 · Real-time & streaming: the request waits on the slowest shard.

Double-buffer atomic swap — updating a table you're reading

The table is read on every request and written by retraining/online learning. Lock a row for a write and reads queue behind it; the tail latency blows. The fix is a double buffer: keep two copies of the table (or shard), serve all reads from buffer A while the update lands in buffer B, then flip a single pointer atomically. In-flight reads drain on A, new reads hit B — no read ever waits on a write. It is the same atomic-pointer-flip discipline as the model hot-swap below, applied to the parameter store, and it is what makes "update embeddings every few minutes without a latency glitch" possible. The cost is the obvious one: 2× memory for the buffered table (the same factor the worked numbers above already budgeted for).

Feature admission and eviction — don't store a row you can't learn

A brand-new id has been seen a handful of times; its embedding is essentially noise, yet a naive table allocates a full row for it the instant it appears. Across a long tail of millions of one-shot ids this is pure waste — memory spent on rows the model can't actually fit. Feature admission (特征准入) is the gate: admit an id into the table only after it has accumulated N exposures (e.g. seen ≥ 10 times), so you pay for a row only once there's enough signal to train it. On the other end, eviction reclaims the tail — LRU drops embeddings unread for a long window. Together they hold the table at a working set of ids that earn their memory; the worked numbers above used a 20%-admitted figure, and that one policy did more than the quantization did.

Per-feature 8-bit quantization

The last lever is to store low-frequency feature embeddings at int8 (1 byte/dim) rather than FP32 (4 bytes/dim) — a 4× shrink on those rows. It is applied per feature, not globally, on purpose: head features that drive most predictions keep their precision; the long-tail rows, which are noisy anyway, lose little from coarse quantization. This is the storage-side cousin of the model-weight quantization above, and it composes with admission — admit fewer rows, and store the rows you admit more cheaply.

Train–serve skew — the silent killer

This is the single most important idea in the lesson, and the one most likely to cost you a launch. Train–serve skew is when a feature is computed one way during training and a subtly different way at serving time. The model was fit to the training distribution; at serving it receives a shifted distribution and silently produces worse predictions. Nothing crashes. No alert fires by default. The offline AUC was great. The online metric is flat or down, and you have no idea why.

Where skew creeps in:

• Two implementations of one feature. Training computes avg_watch_time_7d in a Spark batch job; serving computes it in Java from a streaming aggregate. A boundary condition (timezone, inclusive vs exclusive window, null handling) differs by a hair → systematic skew.
• Time-window misalignment. The training label window and the online feature window don't line up; the model effectively trained on slightly future information it can't have online — a cousin of leakage. Aligning offline/online windows is exactly why 17 · Real-time & streaming uses one streaming job (Flink) to compute both.
• Missing real-time features. A just-occurred click that exists in the replayed training log hasn't propagated to the online store yet → the feature is null online but present offline.
• Numerical drift. FP16/INT8 export changes outputs slightly; usually harmless, but for a multi-head model the per-head score scales can drift enough to need online calibration.

The fix: one pipeline, not two

The structural cure is to not have two implementations. A feature store serves the same materialized feature values to both training (via point-in-time correct historical reads) and serving (via low-latency online reads), computed by shared transformation code. The same streaming job that updates the online store also writes the offline feature logs, so the window logic is identical by construction. Where you can't fully unify, you add a guard: log the exact feature vector the model saw online, periodically replay those same entities through the offline pipeline, and diff.

skew check: compare online_log(f) vs offline_pipeline(f) on the same entities → alert if Δmean, Δp50, Δp99, or PSI exceeds threshold

The distance metric of choice is the Population Stability Index — the same statistic that monitors drift in production, just applied to a different baseline. Skew and drift are one measurement against two reference distributions.

PSI = Σᵢ (online_pctᵢ − ref_pctᵢ) · ln(online_pctᵢ / ref_pctᵢ)

Rule of thumb: PSI < 0.1 stable · 0.1–0.25 watch · > 0.25 significant shift, investigate. Data-validation libraries (TFDV, Great Expectations, Deequ) automate the distribution checks; the log-and-replay loop is the gold standard.

The consistency toolchain — what to actually wire up

"One pipeline plus PSI" is the principle; the question an interviewer follows with is "with what tools?" Each tool below targets a specific way the train and serve paths diverge, and the point is the mechanism each one catches, not the brand name.

The distance metric these checks lean on — PSI — is the one already derived above; TFDV and the KS test are just two ways of computing a distribution diff (PSI is the recsys default; cross-link, don't re-derive). The KS test answers a slightly different question than PSI: it asks "are these two samples drawn from the same distribution?" and gives a p-value, which is exactly what you want when comparing the new model's scores to the old model's in shadow mode.

Safe rollout — shadow → canary → ramp → rollback

You never flip 0% → 100%. The whole point of rollout is to bound the blast radius: at every stage the worst case is that a small, recoverable slice of traffic saw a bad model for a short time. Four stages, each answering a different question.

• Shadow (mirror). 100% of traffic is copied to the new model, which scores it but its output is logged, not returned. This catches the things that have nothing to do with quality: does it crash, does it meet latency, does it match the old model where it should, is there a feature-skew gap? Shadow validates correctness and cost at full scale with zero user risk.
• Canary. Route a tiny slice (1–5%) of real traffic to the new model, bucketed by user-ID hash so a user is consistently in one arm. Now you watch for harm: guardrails (P99 latency, error rate, crash rate) and a first read on the core metric. The slice is small enough that even a disaster is contained and rollback is instant.
• Ramp / A/B. This is where rollout meets 08 · Evaluation's A/B machinery. Step the traffic up — 5 → 20 → 50% — and at each level run a properly powered experiment on the core metric (e.g. avg watch time) plus guardrails. Advance only when the lift is statistically significant and no guardrail regressed. (Randomization unit, MDE/power, Simpson's paradox, novelty effect, network interference — all live in 08.)
• Full + standby. At 100%, keep the previous model warm as a hot standby (e.g. 24 h) so rollback is a traffic switch, not a redeploy.

Rollback is a feature, and it must be automatic

The single most valuable piece of deployment infrastructure is a one-click — better, automatic — rollback. Define tripwires in advance and wire them to the traffic router.

The statistics of a canary

A canary at 1% traffic is also a statistical object: with little traffic you can only detect a large regression quickly. The minimum detectable effect scales like MDE ∝ σ / √n, so a 1% slice needs a bigger true regression — or more time — to flag than a 50% ramp does. That is by design: the canary's job is to catch catastrophes fast (large and obvious), and to hand the subtle "is it 0.5% better?" question off to the powered A/B at higher traffic. Don't ask a canary to prove a small win.

The incident playbook — when it's already on fire

Auto-rollback handles the cases you pre-registered. Real incidents are messier, and at 2 a.m. you want a decision tree, not improvisation. The discipline is always the same order: stop the bleeding first, diagnose second — isolate the bad version and switch to the stable backup before you start root-causing, because every minute spent debugging in production is a minute users see the bad model.

Two-stage rollback — features first, then model

A model and its feature-engineering version are coupled: the model was fit to features produced a specific way. So a rollback is not one switch — you roll back in two stages, feature-engineering version first, then the model architecture. The reason is the partial-rollback trap from the auto-rollback tripwires: revert the model but leave new features (or vice-versa) and you've created a new untested combination — its own outage. Staging features-then-model lets you check after stage one whether the feature change alone caused the regression (often it did — a failure knowledge base will tell you feature issues like leakage are a large share of rollbacks) before you also touch the heavier model artifact.

Memory-leak diagnosis — the slow-burn incident

Not every incident is a sharp metric drop; a memory leak is a slow climb in RSS that ends in an OOM kill and a latency cliff. Same discipline, stop-then-diagnose:

• Stop the bleed. Use the service mesh to switch traffic off the leaking instance and circuit-break it, so the climbing instance stops taking requests before it OOMs. Capacity drops; the service stays up.
• Collect. Grab GC logs, RSS over time, thread stacks — the evidence of where the growth is.
• Diagnose with a profiler, not logs. Heap-analyze with pprof / Valgrind (native) or jmap (JVM). The recsys-specific prime suspect: tensor-release logic in the inference path — a held reference to per-request tensors (or an unbounded feature cache) that never frees. Verify the predict path releases what it allocates.
• Re-ship safely. Reproduce in isolation, fix, then re-deploy via the ramp (5 → 20 → 100%) with a memory-watermark alert wired this time. The prevention that would have caught it: a 24 h sustained-peak load test in the pre-launch checklist.

Model versioning & hot updates

Three things must version together, because a mismatch between any two is an outage: the model weights, the feature-pipeline code that produces its inputs, and the serving/API contract. A model registry (MLflow Models, TF-Serving's versioned dirs, a custom registry) tracks this triple and is what rollback rewinds to.

Hot-swap matters because recommenders retrain constantly — sometimes hourly, with online/incremental learning (17 · Real-time) updating embeddings continuously. You can't take the service down to load a new model. The pattern: load the new version into memory alongside the old, validate it (health check, a few shadow requests), then atomically flip a pointer so in-flight requests drain on the old graph and new ones hit the new graph. Done wrong, the flip causes a latency glitch — a spike while the new graph's caches warm — which is why you warm it up with synthetic requests before flipping, and ramp rather than slam.

Online / nearline / offline — three update tiers, not one

"Retrain constantly" hides a real architectural decision: different parts of the model update at different speeds, because freshness and model complexity trade off against each other. You cannot fully retrain a heavy DNN every second, and you don't need to — most of its knowledge is stable. So production splits the update into tiers, each with its own cadence, model class, and transport. The rule of thumb: the faster the tier, the lighter the model.

Feeding all three is a feature pipeline held to a < 1 s end-to-end SLO (Kafka → Flink → online store), because a second-level model tier is pointless if the features it reads are minutes stale — the same window-alignment discipline as 17 · Real-time & streaming. A concrete real-time feature: the user's short-term interest vector as the mean of their last few interaction embeddings, blended with the long-term vector.

Monitoring — the metrics that predict an incident

Shipping is not the end; the model now lives in a non-stationary world — user behavior drifts, content changes, upstream pipelines break. Monitoring is layered, and the art is watching the upstream signals that move before the business metric, so you get a warning instead of an autopsy.

Alerting without crying wolf

The failure mode of monitoring is too many alerts — on-call fatigue makes a noisy system worse than none. The levers: dynamic thresholds (a baseline from a time-series forecast like Prophet, not a static line, so weekend/holiday cycles don't trip it), windowed triggers (fire only if the condition holds N minutes, not on one spiky sample), correlated suppression (one root cause → one alert, not fifty), and severity tiers (P0 page-now for "service down" vs P3 next-day for "minor wobble"). Audit rules periodically and retire any whose precision drops below ~80%.

Interactive · latency-budget allocator

You have a fixed serving budget. Drag the per-stage shares and pick the cuts you'd ship. The widget recomputes a toy P99 against the SLA and diagnoses what you optimized — and whether you optimized the right stage. The numbers are illustrative; the diagnoses are the point.

Interview prompts you should be ready for