Part I — Basics

The efficiency game

There is no previous lesson to repair — this is where the whole derivation begins. Most coverage of large language models treats them as artifacts to be admired or feared: a model "knows" things, "reasons," "hallucinates." This track rejects that framing entirely. A frontier model is the output of an optimization problem with exactly one binding constraint, and every engineering decision in the next nineteen lessons — the tokenizer, the attention kernel, how big to train, what data to feed it, how to align and serve it — falls out as the forced answer to a single question. This lesson states the question, names the constraint, and lays out the chain of walls we will hit on the way.

The premise this track starts from

You are handed a fixed compute budget — a cluster of G GPUs for t days — and told: build the most capable language model this buys. That is the real job. Not "build a smart model" in the abstract; build the best model per FLOP, because FLOPs are the thing you actually ran out of. Nobody at the frontier is compute-unconstrained. Once you accept that the budget is fixed, every downstream choice stops being a matter of taste and becomes arithmetic: it either buys more capability per FLOP or it does not. Efficiency is not one chapter near the end. It is the spine the entire course hangs on.

Linear position

Forced by: the desire to understand and reproduce a frontier model rather than worship it — which means reasoning about every decision from first principles, not folklore.
New idea: the budget thesis. Compute is the currency, C ≈ 6·N·D FLOPs is the exchange rate (N = parameters, D = training tokens), and the course is the ordered chain of decisions that spend C well — then serve the result cheaply.
Forces next: you cannot spend a single FLOP until text becomes the integer token ids a model actually consumes. That is tokenization (01).

The plan

Five moves. (1) What "from scratch" means and why building beats admiring. (2) The budget identity C ≈ 6ND, GPU-hours as the real currency, and the order of magnitude of a real run. (3) The map: the five efficiency frontiers, each a different way to buy capability per FLOP. (4) The track as a dependency chain — wall → forced fix → new wall. (5) The three motifs that recur for twenty lessons, then a budget-allocator widget you drive yourself.

1 · What "from scratch" means

Stanford's CS336 makes an unfashionable bet: you only understand the modern LLM stack if you build it — the tokenizer, the Transformer, the training loop, the parallelism, the data pipeline, the alignment, the server — rather than calling someone's API and narrating the output. The bet pays off because the stack is not a bag of tricks. It is a single chain of forced consequences, and the only way to feel the force is to hit each wall yourself.

This track is the rigorous, full-stack sibling of the six-lesson Mini GPT track. Mini GPT is the compact tour: one tiny model, the whole train-then-align arc in miniature. CS336 is the from-scratch course: the same arc, but at the scale where the budget actually bites and every decision has to be paid for in FLOPs and bytes. Where Mini GPT shows you that SFT is "just a loss mask," this track shows you why you could only afford that model after a tokenizer cut your sequence length 4×, a FlashAttention kernel kept attention from melting your HBM, and FSDP let the optimizer state fit across eight GPUs.

The reward for finishing is not trivia. It is that you can take any single decision in a frontier system — "why GQA?", "why 20 tokens per parameter?", "why is decode memory-bound?" — and re-derive it from the budget, on a whiteboard, without looking it up. That is what "from scratch" buys.

2 · The budget identity: C ≈ 6ND

Two numbers define a training run. N, the parameter count — the size of the weight matrices. D, the number of tokens you train on. Lesson 05 derives it carefully, but the punchline is worth stating on day one, because it is the equation the whole course orbits:

C ≈ 6 · N · D FLOPs

Where does the 6 come from? A forward pass through the network costs about 2N floating-point operations per token — each of the N weights participates in roughly one multiply and one add. The backward pass costs about twice the forward (you compute gradients with respect to both activations and weights), so ≈ 4N per token. Forward plus backward is ≈ 6N per token, and you do this for all D tokens: C ≈ 6ND. It ignores attention's quadratic-in-context term and assumes the matmuls dominate — true for every model worth the name. Hold onto it; we call it back in 05, plan with it in 12, and spend it in 19.

FLOPs are the abstract currency, but you pay in GPU-hours. An H100 delivers on the order of ≈ 1×10¹⁵ bf16 FLOP/s at peak, and you never get peak — realistic model FLOPs utilization (MFU) is ≈ 40–55% (lesson 05). So the time to burn a budget is roughly:

t ≈ C / (G · F_peak · MFU)

Plug in real numbers to feel the scale. GPT-3 was N ≈ 175×10⁹ parameters trained on D ≈ 300×10⁹ tokens, so C ≈ 6 × 175e9 × 300e9 ≈ 3.15×10²³ FLOPs. On 1,000 H100s at 45% MFU that is 3.15e23 / (1000 × 1e15 × 0.45) ≈ 7×10⁵ seconds ≈ 8 days — a useful sanity check on the form of the equation, though GPT-3 itself trained on slower A100-era hardware. A current frontier run sits at C ≈ 1×10²⁵ to 1×10²⁶ FLOPs — two to three orders of magnitude more, which is months on tens of thousands of GPUs and tens of millions of dollars of electricity and rent. At that price, a 1% efficiency gain anywhere in the stack is a real number with a dollar sign on it. That is why efficiency is the spine.

N — parameters

Size of the weights. Sets capability and the per-token FLOP cost (2N forward). Lessons 02–03 design them; 08–10 fit them across GPUs.

D — tokens

How much text you train on. The other half of the budget. Lesson 12 picks it; lesson 13 makes them worth training on.

C ≈ 6ND — compute

The budget. Fixed before you start. Every lesson is a way to convert a slice of C into more capability.

GPU-hours — the bill

What C costs in the real world, via t ≈ C / (G · F · MFU). MFU is where kernels and parallelism (Part II) earn their keep.

3 · The map: five efficiency frontiers

Given a fixed C, there are exactly five places to find more capability per FLOP, and they are the five parts of this course. Each is a distinct lever on the same budget.

Frontier	The lever — capability per FLOP comes from…	Lessons
Basics	An architecture and optimizer that learn more per FLOP and per parameter, and that you can predict the cost of before you launch.	00–05
Systems	Raising MFU — fusing kernels and splitting work across GPUs so the FLOPs you paid for actually do useful math instead of waiting on memory — then keeping a weeks-long run alive through the hardware failures and loss spikes that would otherwise burn the budget.	06–11
Scaling	Spending C on the right N and D — you get one shot at the big run, so predict the optimum instead of guessing.	12
Data	Quality as a hidden multiplier — the same FLOPs on better tokens buy a far better model. And measuring whether any of it worked.	13–14
Post-training & inference	Extending the trained model's reach to long context, turning a text-completer into a useful assistant, then serving each token cheaply so the model you paid for is actually affordable to run.	15–18

Read the column again: every row is the same sentence with a different lever. That sameness is the point. You are never learning "a new topic"; you are learning a new way to not waste the budget.

4 · The track as a dependency chain

This course is linear thinking by derivation. Each lesson opens by naming the exact wall the previous one hit, derives the next idea as the forced fix, and ends by exposing the new wall that forces the lesson after it. You should never feel a topic was introduced because "it comes next in the textbook." It comes next because the prior step broke and this is the only repair. Here is the whole chain in one view — read it top to bottom as wall → fix → new wall:

  WALL                                  FORCED FIX                          owns
  ----------------------------------------------------------------------------------
  a model eats integers, not text   ->  byte-level BPE tokenizer            (01)
  need a parallel next-token fn     ->  decoder-only Transformer            (02)
  vanilla GPT-2 block is unstable   ->  modern recipe: RMSNorm/RoPE/...     (03)
  weights don't move themselves     ->  AdamW + warmup-cosine + bf16        (04)
  will it even fit? how long?       ->  resource accounting (6ND, memory)   (05)
       |
       v  ledger says 7B won't fit on one GPU -> cross the systems wall
  what is MFU, really?              ->  the GPU: roofline, intensity        (06)
  most ops are memory-bound         ->  kernel fusion, FlashAttention       (07)
  training state > one GPU          ->  data parallel: DDP -> ZeRO -> FSDP  (08)
  a layer too big for one device    ->  tensor / pipeline / 3D parallel     (09)
  params and FLOPs rise together    ->  Mixture of Experts (sparse)         (10)
  the machine exists, but a weeks-  ->  running the job: checkpoint/restart, (11)
    long run dies on a failed node      loss-spike recovery
       |
       v  too many knobs, one shot at the run -> cross the scaling wall
  can't grid-search at frontier     ->  scaling laws, Chinchilla-optimal    (12)
       |
       v  compute-optimal token COUNT says nothing about token QUALITY
  garbage tokens waste the budget   ->  data pipeline: filter/dedup/mix     (13)
  every change CLAIMS improvement   ->  evaluation: ppl, benchmarks, contam (14)
       |
       v  a measured base model still can't reach past its short window
  trained context window is fixed   ->  long context: position interp/YaRN  (15)
       |
       v  a long-reaching base model still won't follow instructions
  base model only completes text    ->  SFT: completer -> assistant         (16)
  SFT can't say A is better than B  ->  preferences: RLHF / DPO / GRPO      (17)
  a trained model is useless if     ->  inference: prefill/decode, KV cache (18)
    each token costs too much
       |
       v  every stage built in isolation; the point is they are ONE chain
  the chain, end to end             ->  capstone synthesis                  (19)

Notice the places the chain crosses a part boundary (the indented arrows): the memory ledger in 05 is what forces the entire systems half; once the machine is built and the run survives (11), the knob explosion forces scaling; compute-optimal token count in 12 says nothing about token quality, forcing the data half; and a measured base model still can't reach past its trained window or follow instructions, forcing the post-training half (long context, then alignment). Those hinges are the backbone of the whole story. If you remember nothing else, remember why each part begins.

5 · Three motifs you will see for twenty lessons

Three ideas recur so often we name them now and call them back by file number whenever they reappear.

① the 6ND budget Compute is fixed; capability is what you buy with it. Introduced here, derived in 05, planned with in 12, spent end-to-end in 19. Whenever a decision looks arbitrary, ask what it does to C.

② memory-bound vs compute-bound An op is limited either by how fast the chip does math or by how fast it moves bytes. Most LM ops are bytes-limited. This is the master concept of 06, recurs in kernels (07), communication (09), and decode (18). It is why MFU is hard.

③ params vs active FLOPs Capability scales with parameters; you pay FLOPs only for the parameters a token actually touches. Dense models couple them; 10 MoE decouples them, and 18 inference re-uses the split. The whole "free parameters" trick lives here.

6 · Drive the budget yourself

The widget below makes the thesis concrete with the simplest possible toy. You hold a fixed compute budget C. The constraint C = 6ND means every parameter you add costs tokens, and every token you add costs parameters — the slider just trades one for the other along the constraint. A standard scaling-law form predicts the loss:

L(N, D) = E + A / N^α + B / D^β

Here E is the irreducible loss (the entropy of language itself — no amount of compute beats it), and the two power-law terms shrink as you grow N and D. Because of the C = 6ND tether, you cannot grow both; you allocate. Drag the slider and watch the predicted loss trace a U: starve N and the model is too small to fit the data; starve D and a big model never sees enough text. The minimum lands near ≈ 20 tokens per parameter — the Chinchilla result we derive properly in 12. You are previewing the single most important budgeting decision in the course.

Budget allocator — split a fixed C between N and D

Compute C is fixed by the cluster. The slider moves you along C = 6ND: left = few big-model parameters trained on many tokens, right = a huge model starved of data. The predicted loss L = E + A/N^α + B/D^β is U-shaped; its minimum sits near 20 tokens/param — proving that for a given budget there is one best size, not "bigger is better." Foreshadows lesson 12.

compute budget C (FLOPs): 10²³·⁵ allocation (N ↔ D): 20 tok/param

Parameters N

—

Tokens D

—

Tokens / param

—

Predicted loss L

—

Loss vs tokens/param (this C)

● your allocation · dashed = compute-optimal ≈ 20 tok/param

Two things to take away from playing with it. First, the optimum barely moves as you slide C up and down the range — the best tokens-per-param ratio is roughly scale-free, which is exactly why a cheap small-scale fit can predict a frontier run (lesson 12). Second, GPT-3's actual ratio was about 300e9 / 175e9 ≈ 1.7 tokens per parameter — far left of the optimum, badly under-trained. The same compute spent at 20 tok/param would have bought a markedly better model. That mistake, worth tens of millions of dollars, is the single cleanest proof that the budget thesis is not philosophy.

Failure modes & checklist

Failure modes

Treating "bigger model" as the goal. Under a fixed budget, a bigger N means a smaller D; past the optimum, capability falls. Signal: you can name your model's parameter count but not its tokens-per-param ratio.
Ignoring MFU when estimating cost. Pricing a run at peak FLOP/s underestimates wall-clock by 2× or more. Signal: your t ≈ C / (G·F) estimate has no MFU factor and the run takes twice as long as planned.
Optimizing one stage in isolation. A clever kernel that raises MFU 10% is worthless if your tokens-per-param is wrong by 10×. Signal: effort concentrated on one frontier while another sits at an obviously-broken default.
Forgetting the bill is GPU-hours, not FLOPs. FLOPs are abstract; the constraint that actually binds is hardware × time × utilization. Signal: reasoning about "compute" with no GPU count or day count attached.

Checklist

State the budget first. G GPUs × t days × MFU → C. Every decision is downstream of this number.
Know your C ≈ 6ND triangle. Given any two of C, N, D, you can compute the third on a whiteboard.
Ask "FLOPs per what?" of every choice. Capability per FLOP is the only objective; if a change doesn't improve it, it's decoration.
Aim for ≈ 20 tokens/param unless you have a serving reason to over-train (lesson 12, 18).
Track the three motifs. When stuck, ask: what does this do to the budget, is it memory- or compute-bound, and am I paying for active or total params?

Checkpoint

Try it

Set the widget's budget to C = 10²³ (close to GPT-3). Slide the allocation to 1.7 tokens/param — GPT-3's actual ratio — and read off the predicted loss. Now slide to 20 tokens/param and read it again: the loss drops noticeably for the same compute. In one sentence, explain to a teammate why "we trained the biggest model we could afford" was the wrong objective. Then, by hand: a cluster of 2,000 H100s (≈ 1e15 FLOP/s each) runs for 30 days at 45% MFU — what C does that buy, and at 20 tok/param what N and D does it imply? (Answer shape: C ≈ 2000 × 1e15 × 0.45 × 30 × 86400 ≈ 2.3×10²⁴; then solve C = 6ND with D = 20N.)

Where this points next

The thesis is set: a fixed C ≈ 6ND, and a chain of forced decisions that spend it. But we have been blithely writing "tokens" as if they were a given. They are not. A neural network does not consume characters or words — it consumes integers indexing a finite vocabulary, fed in as vectors. Before a single FLOP of that 6ND can be spent, raw text must be turned into a sequence of integer token ids, losslessly and compactly, because the sequence length you produce directly sets how many FLOPs each document costs. That conversion is the first wall, and the first real engineering decision of the course: 01 · Tokenization — BPE from raw bytes.

Takeaway

A frontier language model is not magic; it is the solution to one optimization problem under one binding constraint — a fixed compute budget. Compute is the currency and C ≈ 6ND FLOPs is its exchange rate: roughly 2N per token forward, 4N backward, over all D tokens. You pay it in GPU-hours via t ≈ C / (G·F·MFU); GPT-3 was ≈ 3.15×10²³ FLOPs and a frontier run is ≈ 10²⁵–10²⁶. Every lesson in this track is a way to buy more capability per FLOP — through architecture and optimizer (basics), utilization and a run that survives to the end (systems), the right N and D (scaling), better tokens (data), and longer reach plus cheap useful output (post-training & inference) — and each is forced by the wall the previous one hit. Three motifs recur throughout: the 6ND budget, memory-bound vs compute-bound, and params vs active FLOPs. The toy allocator already proves the punchline of lesson 12 — for a given budget there is one best model size (≈ 20 tokens/param), and "the biggest model we could afford" is a multimillion-dollar mistake. Spending the budget starts with turning text into tokens.

Interview prompts

Where does the 6 in C ≈ 6ND come from? (§2 — ≈2N FLOPs/token forward, ≈4N backward, summed over D tokens; ignores the attention quadratic term, which is small when matmuls dominate.)
GPT-3 was 175B params on 300B tokens — estimate its training FLOPs and say what's wrong with that allocation. (§2, §6 — C ≈ 6 × 175e9 × 300e9 ≈ 3.15e23; only ~1.7 tokens/param, far under the ~20 compute-optimal ratio, so it was badly under-trained for its compute.)
Why is "build the biggest model we can fit" the wrong objective? (§3, §6 — under fixed C, larger N forces smaller D; past the optimum the loss rises. The objective is capability per FLOP, which has an interior optimum near 20 tok/param.)
Translate a budget of 4,000 GPUs for 60 days into FLOPs. (§2 — C ≈ G · F_peak · MFU · t; with F≈1e15 FLOP/s, MFU≈0.45, t in seconds: ≈ 4000 × 1e15 × 0.45 × 60 × 86400 ≈ 9.3e24 FLOPs.)
Name the three recurring motifs and one lesson each pays off in. (§5 — 6ND budget (→12 scaling), memory- vs compute-bound (→06 roofline / 18 decode), params vs active FLOPs (→10 MoE).)
Why does this course claim each lesson is "forced" by the last rather than a menu of topics? (§4 — it is a derivation: lesson N opens on N−1's wall and derives the only repair; e.g. 05's memory ledger forcing the systems half, 12's token-count result forcing the data half.)
If a small-scale loss fit can predict a frontier run, what property of the optimum makes that work? (§6, →12 — the compute-optimal tokens-per-param ratio is roughly scale-invariant, so the curve fit at low C extrapolates to high C.)