The universal ML system design frame

Lessons 01-03b gave you the LLM arithmetic: TTFT, TPOT, tokens/s, KV, GPUs, dollars. This page widens the lens. A production ML system is not "a model behind an API." It is a decision system: raw events become features, features feed a model, the model changes a product decision, the decision creates labels, and those labels flow back into the next model. Design the loop, not the checkpoint.

The first-principles object

A model is a lossy function f(x) -> y_hat. A production ML system is the machinery that makes x correct at inference time, makes y trustworthy at training time, and keeps the decision useful as the world changes. The hard failures are rarely "the neural net cannot compute." They are stale features, delayed labels, training-serving skew, bad thresholds, hidden feedback loops, and monitors that watch the wrong distribution.

Plain English dictionary

A feature is an input fact the model can use, like "transactions in the last hour." A label is the answer learned later, like "this transaction became a chargeback." A decision is the product action, like block, rank, approve, answer, or forecast. A model is only the scoring function inside that larger decision loop. Start there and the system stops feeling like magic.

0 · The beginner mental model

When the domain is unfamiliar, do not start by naming tools. Start with a five-question ladder. Each answer unlocks the next one:

What decision is being made? A user sees an item, a card transaction is approved, an answer is generated, inventory is ordered.
What facts are known before the decision? These are serving-time features; they must exist fast enough to use.
What truth is learned after the decision? These are labels; they may arrive seconds, days, or months later.
What can go wrong if the decision is wrong? This gives the metric, threshold, guardrails, and human-review needs.
Which promise breaks first at scale? Freshness, latency, recall, calibration, label delay, cost, or GPU memory becomes the first wall.

That ladder is the linear thinking habit. You can apply it before you know whether the final model is a tree, embedding model, transformer, or LLM.

1 · Start from the decision, not the model

The design prompt usually names a product: fraud detection, feed ranking, demand forecasting, image moderation, RAG search, or an LLM assistant. The first move is to translate that product into a concrete decision:

Prompt	Decision	Prediction target	Action surface
Fraud detection	Approve, hold, or challenge a transaction	P(fraud \| transaction, user)	Block, step-up auth, manual review
Recommendation feed	Order candidate items	Expected utility: click, watch, hide, return	Rank, diversify, explore
Forecasting	Allocate inventory or capacity	Demand over horizon H	Buy, route, staff, throttle
LLM assistant	Generate or route a response	Task success subject to safety and latency	Answer, retrieve, call tool, refuse

This move prevents model-first overengineering. If the decision is a thresholded fraud hold, calibration and review capacity may matter more than AUC. If the decision is a feed ranking, latency and candidate recall may matter more than the final ranker's architecture. If the decision is an LLM answer, output length may dominate the entire GPU bill.

2 · The six flows every ML design must place

After the decision is clear, draw the material flows. Do this before naming a technology. Every production ML system has the same six flows, even when the model class changes:

Flow	Question to answer	Typical wall
Request path	What must happen synchronously before a decision?	Latency budget, fan-out, tail p99
Feature path	Which facts must be fresh online and reproducible offline?	Freshness, point-in-time correctness
Label path	When does ground truth arrive, and who/what creates it?	Delay, bias, missing labels
Training path	How do examples become reproducible model artifacts?	Shuffle, lineage, compute, leakage
Serving path	How does a model decision meet latency and cost SLOs?	Feature fetch, model latency, scale-out
Feedback path	How do production decisions change tomorrow's data?	Selection bias, drift, Goodhart

3 · The general design loop

The LLM loop still holds, but the currencies broaden. Instead of only FLOPs, bytes, bandwidth, and dollars, general ML adds data correctness, label delay, feature freshness, and decision cost.

Decision requirements. What action will the system take, how fast, at what volume, and what is the cost of a false positive / false negative / stale decision?
Data contract. Define entities, event time, schemas, labels, feature freshness, and ownership. If the contract is vague, the model will learn a moving target.
Model and serving topology. Start with the simplest model and serving path that could plausibly satisfy the decision; escalate only when a measured requirement fails.
Bottleneck. Name the wall: feature freshness, online lookup p99, candidate recall, model latency, label delay, calibration, drift, GPU cost, or human review capacity.
Iterate on the binding wall. Add a feature store only for feature consistency/freshness, ANN only for candidate recall, distillation only for latency/cost, active learning only for label scarcity.

Model-choice ladder

Start with a baseline that is cheap to train, explain, and serve. Use a linear or tree model when tabular features carry the signal. Add embeddings or a two-tower retrieval model when the system must search a huge item space. Add a deeper ranker when feature interactions matter and latency allows it. Use a transformer or LLM when the input is language, code, images, long context, or generation. Use an ensemble only when the quality gain beats the extra latency, cost, and operational complexity. The ladder is not about prestige; it is about the first model that satisfies the decision contract.

Linearized answer pattern

In an interview, say: "The decision is X. The synchronous path has these steps and this p99 budget. The offline path creates these features and labels with this delay. The first bottleneck is Y, so I choose mechanism Z. I will monitor A, B, C because they move before users feel pain." That is the general ML version of requirements -> arithmetic -> topology -> bottleneck -> iterate.

4 · What changes by domain

Different ML systems are the same loop with a different first wall. The table below is the fastest way to avoid cargo-culting an LLM serving design onto a ranking or fraud system.

System	Usually binds first	Load-bearing mechanism	Primary eval trap
Fraud / spam	Label delay + class imbalance + review capacity	Cost-sensitive threshold, calibrated score, human queue	AUC high, deployed precision/recall bad
Search / recsys	Candidate recall under tight latency	Retrieve -> rank -> rerank cascade, ANN, feature store	Offline NDCG wins do not transfer online
Ads	Calibration and auction economics	pCTR/pCVR calibration, pacing, bid x quality score	Good ranking, bad expected value
Forecasting	Horizon and data freshness	Backtesting, time-split validation, exogenous features	Random split leaks future
LLM serving	Output length, KV memory, GPU bandwidth	Continuous batching, paging, TP, caching, quantization	Fast wrong answer or cost explosion

These walls are derivable, not memorized. Fraud binds on label delay because true fraud may be known weeks later while the decision happens in milliseconds. Recsys binds on candidate recall because the ranker cannot choose an item retrieval never returned. Ads bind on calibration because a probability becomes money in an auction. Forecasting binds on horizon because the decision is about the future, so random splits leak information. LLM serving binds on output length and KV because each generated token spends memory bandwidth and cache capacity.

What carries forward

Design the decision system, not the model. The model is one stage in a loop from events to actions to labels.
Every ML system has six flows: request, feature, label, training, serving, feedback. Missing one is how silent bugs enter production.
The bottleneck vocabulary is broader than GPUs: freshness, leakage, label delay, calibration, drift, human review, and online lookup p99 can bind before compute.
The right mechanism follows the wall. Feature stores answer training-serving parity; ANN answers candidate recall; registries answer reproducibility; drift monitors answer changing data.