Orientation — start here

This page is the map. It tells you what "ML systems design" means in this track, why it deserves its own series separate from the mechanism tracks, and the single loop you will run across LLM serving, general ML, training, evaluation, production, and case studies. Read it once; come back to it whenever a later lesson feels like a pile of disconnected facts.

The gap this track fills

The other series on this site each teach a mechanism deeply: how a kernel uses the GPU, how FSDP shards a model, how RadixAttention reuses a prefix, how GRPO drops the critic. Each answers a how.

None of them answers the question you actually get asked on the job, or in a staff-level interview:

The question this track is about

"Here is a product decision, a model, data, traffic, a budget, and a latency target. Design the system." What decision is being made? Which facts must be fresh? Which labels arrive late? How many GPUs or CPUs? Replicate or shard? What breaks first when traffic triples? What does it cost per million predictions or tokens, and which single change halves that?

That is a different skill from knowing the mechanisms — the way knowing the rules of chess is different from playing well. This track is the playing-well layer. It assumes the mechanisms exist (and links down to them) and spends its attention on selection under constraints.

If you know little ML systems vocabulary

Read every lesson as a sequence of questions, not as a glossary to memorize. A feature is an input fact. A label is the answer learned later. A model is the scoring function. A serving path is what must happen before the product responds. A bottleneck is the first promise that breaks: latency, memory, freshness, label delay, recall, calibration, reliability, or cost.

The loop, in one page

Every design — inference, general serving, training, RL, a data plane, monitoring, or a recommender — is the same five steps. You will see this loop in lesson 01 stated formally, then repeat it until it feels automatic.

Decision and requirements. Turn vague asks into a concrete action plus numbers: SLOs, throughput, workload shape, scale, budget, utility, and cost of mistakes. A design without a decision and a number is an opinion. 03, 03c
Material contract. Name what moves through the system: tokens, features, labels, model artifacts, indexes, logs, feedback. Put clocks on them: decision time, freshness, label delay, and replayability. 03d, 03e
Arithmetic. Estimate the cost in the currencies that bind ML systems: FLOPs, bytes, dollars, milliseconds, and data freshness. This is back-of-the-envelope, done before any code. 02, 03a, 03b
Topology. Lay the work onto systems: which parallelism, how many replicas, which features are online, which stages are cached, what is synchronous vs async, how data flows. 04–09
Bottleneck and iterate. Find which promise breaks first: memory, bandwidth, network, compute, feature p99, candidate recall, label delay, calibration, drift, human capacity, or dollars. Apply the mechanism that moves that wall, then re-run the loop.

The discipline that makes it "linearized"

You never apply a mechanism (paging, quantization, feature store, ANN index, pipeline parallel, speculative decoding, recalibration) until the loop has named the wall it removes. Optimizations are answers to measured questions. A design that lists ten optimizations without naming which bottleneck each one targets is a cargo cult, and an interviewer will catch it in one follow-up question: "why that one, and what does it buy you?"

Three numbers to carry in your head

You will derive these in lesson 02, but seeing them now makes the early lessons concrete. For a dense transformer with N parameters:

Quantity	Rule of thumb	Why it matters
Forward FLOPs / token	≈ 2N	Sets inference compute and prefill cost
Training FLOPs / token	≈ 6N	Fwd + bwd; sets pretraining GPU-months
Weights memory	2N bytes (fp16/bf16)	The floor before you've served a single token

A 70B model is therefore ~140 GB of weights — already more than one 80 GB H100. That single fact forces tensor parallelism or quantization before you've thought about anything else. The whole track is fact-after-fact like this: a number forces a structural decision.

How the lessons depend on each other

This is a mostly linear chain. Each lesson tries to teach the next question you should ask, not just the next tool you should know.

01 what is different ↓ 02 numbers ─→ 03 requirements ─→ 03a latency ─→ 03b throughput ↓ ↓ 03c decision frame ─→ 03d data/features/labels ─→ 03e pipeline/registry ↓ 04 one LLM replica ─→ 05 scale ─→ 06 optimize ─→ 06a general serving ↓ 07 training ─→ 08 data plane ─→ 09 RL loop ↓ 10 eval ─→ 10a task metrics/bias ─→ 11 production ─→ 11a drift/governance ↓ 12 capstone ─→ 13–19 case studies

Read 01–03e carefully and in order. They teach the method and the vocabulary. After that, each block applies the same method to a different subsystem. If a later lesson feels hard, ask: what is the decision, what material is moving, what promise has a number on it, and which promise breaks first?

What this track is not

Not a mechanism tutorial. We won't derive FlashAttention or implement PagedAttention. We decide when you need them and link to the track that builds them.
Not classic web-system design. Load balancers, sharded SQL, and CAP theorem have their place, but the binding constraints here are GPU memory bandwidth and interconnect, not database round-trips. Lesson 01 is about exactly this difference.
Not framework documentation. vLLM, SGLang, Megatron, and veRL are instances of these designs. We reason about the design space they all live in, so you can evaluate the next framework too.

A note on the numbers

Every figure here is a 2024–2025-era rule of thumb (H100/H200 class hardware, dense and MoE transformers). Hardware moves; the method doesn't. When the H300 ships, you re-run the same arithmetic with new constants. Memorize the loop, not the gigabytes.