Part III — Scaling laws

Scaling laws — Chinchilla-optimal

Parts I and II handed you a working dense or sparse model and the systems to train it at any size you like. That is exactly the problem. You now have a huge knob space — parameters N, training tokens D, MoE experts and top-k, depth versus width — and one fixed compute budget C that buys you exactly one big run. You cannot grid-search at frontier scale; the first time you find out a layout was wrong is a few hundred thousand GPU-hours later. This lesson is the discipline that lets you pick N and D on paper before you spend a FLOP: scaling laws.

The previous step left this broken

Lessons 06–11 took you all the way to a running job: you can now build a distributed training run (parallelism, kernels, FSDP, tensor/pipeline splits, MoE) and lesson 11 just showed you how to survive and execute one — launch it, watch the loss, ride out a spike, restart from a checkpoint. So execution is no longer the gap. The gap is that nothing in that whole stack tells you how big to make the model in the first place. You face a single budget C ≈ 6ND (derived in lesson 5) and a huge knob space — dense parameters N, MoE experts E and top-k, training tokens D — collapsing to one fork: a bigger model on fewer tokens, or a smaller model trained longer. Both burn the same FLOPs; one is meaningfully worse. And at frontier scale you can run the job flawlessly yet still get exactly one attempt — you cannot grid-search a frontier model across that knob space, because each point costs the whole budget. Pick blind and you waste the entire run; this is precisely how GPT-3 left a large fraction of its budget on the table.

Linear position

Forced by: lessons 06–11 let you build and survive the run, but the budget is still fixed, the knob space is huge (dense N, experts E, top-k, tokens D), and you get one shot — you can execute the job, but you cannot grid-search a frontier model, so you must predict the best (N, D) split before committing the run.
New idea: scaling laws — loss is a power law in N, D, and C that is a straight line on log-log axes, so you fit a cheap ladder of small models and extrapolate; the Chinchilla result then gives the compute-optimal N* and D* for any budget (≈20 tokens/param), correcting Kaplan-2020's under-trained recipe.
Forces next: compute-optimal D* is a count of tokens — pure quantity. It says nothing about their quality, and quality is where most of the remaining loss actually hides. That forces the data pipeline, lesson 13.

The plan

Five moves. (1) The empirical power laws L(N), L(D), L(C) — why they are straight lines on log-log, and the irreducible floor E. (2) Kaplan-2020 vs Chinchilla-2022: the IsoFLOP sweep that finds N*, the N* ∝ C^0.5, D* ∝ C^0.5 result, and the ≈20 tokens/param ratio. (3) How you actually fit a law — a ladder of small runs, extrapolated to the target — and the pitfalls that quietly break it. (4) Why you deliberately deviate from compute-optimal: inference cost — over-train a smaller model if you will serve it a lot. (5) Drive the IsoFLOP explorer, then close the loop back to the lesson-00 budget widget.

1 · The three power laws, and why log-log is straight

Train a family of language models and measure the cross-entropy loss L on held-out text. Three startlingly clean regularities show up, each spanning many orders of magnitude. Hold data and compute effectively unbounded and grow only the parameter count N; the loss falls as a power law. Do the same growing only the training tokens D. And along the compute-optimal frontier — the best loss achievable for each compute budget C — the loss falls as a power law in C too.

L(N) ≈ E + A / N^α · L(D) ≈ E + B / D^β · L(C) ≈ E + (C₀ / C)^γ

Read the pieces. E is the irreducible loss — the entropy of the text itself, the floor you cannot cross no matter how much you scale, because language has genuine residual unpredictability (a fair coin is unpredictable even to an oracle). A/N^α and B/D^β are the reducible loss: the part you buy down with more parameters or more data. The exponents are small and similar — for the Chinchilla fit, α ≈ 0.34 and β ≈ 0.28 — so progress is brutally diminishing: each halving of the reducible loss costs roughly 2^1/α ≈ 7.7× the parameters.

Why is the curve a straight line on log-log axes? Take logs of the reducible part: log(L − E) = log A − α·log N. That is a line in (log N, log(L−E)) with slope −α — any pure power law plots straight on log-log. But that is only algebra; it does not explain why the measured losses obey a power law in the first place. Be honest about what is known here: the power-law form L ≈ E + A/N^α is empirical. There is no accepted first-principles derivation of it — it is a regularity read off the data, not a theorem, and the exponents α, β, γ are fitted, not predicted from the architecture.

What we do have is intuition for why a power law is the natural shape. Think of each new parameter or training token as buying a little more resolution on the data distribution — a finer partition of the space of contexts, so the model can tell apart cases it used to blur together. The first units of resolution are hugely valuable (they separate the grossest, most common distinctions); each additional unit refines distinctions that are rarer and matter less, so its marginal payoff shrinks. When the marginal value of resolution decays as a fixed fraction of how much resolution you already have, integrating that decay gives exactly a power law — diminishing returns with no special scale, which is why it looks the same from a 10M-param toy to a 100B-param frontier model. And the floor E is not mysterious either: it is the data's own intrinsic entropy — the genuine residual unpredictability of text (the coin-flip floor from above) that no amount of resolution can remove. So the takeaway is not a proof — it is knowing precisely what is measured (the form, the exponents, the six-plus decades of straightness) versus what is merely motivated (this resolution story). That measured stability — not the story — is what licenses extrapolation: you fit the slope from points you can afford and read off a point you cannot.

Subtract the floor first

It is the reducible loss L − E that is straight, not L itself. Plot raw L on log-log and it bends as it approaches the floor E — which is exactly the regime a frontier run lives in. A common rookie error is to fit a line to raw loss, miss the floor, and extrapolate to an impossibly low loss. You must fit E jointly with the slope.

2 · Kaplan vs Chinchilla: the IsoFLOP sweep

The first influential scaling-laws paper (Kaplan et al., 2020) concluded that, given more compute, you should spend most of it on a bigger model and comparatively little on more data — roughly N ∝ C^0.73, D ∝ C^0.27. Models built on that advice, GPT-3 (175B params) among them, were trained on only ≈300B tokens — about 1.7 tokens per parameter. The Chinchilla paper (Hoffmann et al., 2022) showed that recipe leaves loss on the table, and the reason was partly methodological: Kaplan held the learning-rate schedule fixed instead of re-tuning it per run length, which systematically handicapped the longer-trained (more-data) models and biased the optimum toward big-and-undertrained.

Chinchilla's fix is the IsoFLOP sweep, and it is the cleanest experiment in the field. Fix a compute budget C. Along the curve C = 6ND, every choice of N pins D = C / (6N) — so a bigger model is automatically trained on fewer tokens, same FLOPs. Train several models along that curve and plot final loss against N: you get a parabola in log N (a valley). Too small N wastes compute re-reading data the model is too small to absorb; too large N starves the model of tokens. The bottom of the valley is the compute-optimal N* for that budget. Repeat for several budgets and the valley bottoms trace out the law itself.

fix C → sweep N, with D = C/(6N) → loss is a valley in log N → bottom = N*(C)

Doing this across budgets, Chinchilla found the optimal exponents are each almost exactly one half:

N* ∝ C^0.50 · D* ∝ C^0.50 ⇒ D* / N* ≈ constant ≈ 20 tokens / parameter

The symmetry is the headline: parameters and tokens should grow together, in lockstep, as the square root of the budget. The constant ratio falls out — if both scale as C^0.5, their ratio is flat, and the empirical fit puts it near 20. (It is a soft optimum: anywhere from ~10 to ~40 tokens/param costs only a fraction of a percent of loss, which is why nobody agonizes over hitting exactly 20.)

	Kaplan 2020	Chinchilla 2022
How the optimum was found	Fit L(N,D), fixed LR schedule	IsoFLOP sweep, LR retuned per run
N exponent in C	≈ 0.73	≈ 0.50
D exponent in C	≈ 0.27	≈ 0.50
Tokens / param target	≈ 1–2	≈ 20
Verdict on GPT-3 (175B, 300B tok)	well-sized	badly under-trained

The punchline that reset the field: with the same C GPT-3 used, Chinchilla (70B params, 1.4T tokens — ≈20 tok/param) was smaller yet beat GPT-3 across the board. GPT-3 had spent its budget on parameters it never had the tokens to train. At ≈20 tok/param it should have been a ~3.5T-token model; it saw under a tenth of that. The lesson is exact: for a fixed budget, an under-trained giant loses to a right-sized model fed properly.

3 · How you actually fit a law

You never have the budget to sweep at target scale — that is the whole point. So you build a ladder: a handful of small models you can afford, spanning maybe 10M → 1B parameters and a range of token counts, each trained to convergence with its hyperparameters properly tuned for its size. Fit E, A, α, B, β to those points, then plug in your real target budget C and read off N* and D*. CS336's Assignment 3 is exactly this: you are given a fixed FLOP budget for the ladder, must allocate it across small runs, fit the law, and submit a prediction for the optimal config at a much larger budget.

1 · ladderTrain ~10–30 small models spanning sizes/token-counts you can afford, each with its own tuned LR and schedule.

2 · fitRegress the loss surface to L(N,D) = E + A/N^α + B/D^β (or fit the IsoFLOP valley bottoms). Recover the floor E jointly.

3 · extrapolatePlug the target C into N* ∝ C^0.5; read off N*, D* = C/(6N*), and the predicted loss. Commit the one big run.

Three pitfalls quietly poison the extrapolation, all of them about fairness across the ladder:

LR not re-tuned per scale. The single biggest one, and Kaplan's error. The optimal peak learning rate and the cosine decay length both depend on run length and width. Reuse one schedule across the ladder and the longer/larger runs are handicapped — your fitted exponents drift and the optimum lands wrong. Every rung needs a schedule tuned to its size and token count.
Fitting range too narrow or too small. A power law fit over one decade extrapolated across four is a leap of faith. The ladder must span enough orders of magnitude that the slope is pinned. And the smallest models can sit in a different regime (warmup-dominated, batch-size effects) — they bend the line.
Ignoring the floor and the embedding terms. Fit raw L instead of L − E and you bend off the line near the floor (§1). For small N, embedding/un-embedding params (≈ V·d) are a large, non-quadratic fraction of N, so "params" must be counted consistently (Chinchilla counts non-embedding params).

4 · Why you deviate: the train-vs-serve budget

Compute-optimal answers one question — "what is the lowest training loss for a fixed training budget?" — and it is the wrong question if you are going to serve the model. Chinchilla minimizes loss per training FLOP. It does not price the millions or billions of forward passes you will run at inference time, and those are dominated by N: every served token costs ≈2N FLOPs (lesson 5), forever, for the life of the deployment.

So the real objective is total lifetime FLOPs = training + serving:

C_total ≈ 6·N·D + 2·N·D_inf

where D_inf is the number of tokens you expect to generate over the model's life. If D_inf is large — a model behind a popular API, served for months — the 2N·D_inf term dwarfs training, and you should shrink N below Chinchilla-optimal and push the saved budget into more training tokens on the smaller model. This is deliberate over-training: train past 20 tok/param to recover, at a smaller N, the quality a bigger model would have had — and then pay the smaller N on every served token for the rest of the deployment.

This is exactly the Llama strategy. Llama-1 7B was trained on 1T tokens (≈140 tok/param, ~7× past compute-optimal); Llama-2 7B on 2T; Llama-3 8B on ~15T tokens (≈1,900 tok/param). Those models are wildly "over-trained" by Chinchilla's training-only accounting — and that is the point. Meta knew they would serve them at enormous volume, so a slightly higher training bill buys a permanently cheaper inference bill. The cross-link is direct: lesson 18 · Inference is where that per-token 2N serving cost becomes the wall, and this is the knob that lowers it before you ever train.

The bitter lesson, in budget form

Step back to the thesis of lesson 00. The "bitter lesson" (Sutton) is that general methods that scale with compute beat hand-engineered cleverness in the long run. Scaling laws are that lesson made quantitative and predictive: they say not just "scale wins" but exactly how loss falls with compute, and exactly how to split a budget to ride that curve. The frontier is not magic and it is not artisanal — it is a power law you can fit on a laptop's worth of small runs and then cash in at scale. Every choice in this course is "spend C well"; this lesson is the one that tells you, numerically, where the bottom of the valley is.

5 · Drive the IsoFLOP explorer

The widget is the IsoFLOP sweep made live, and it closes the loop opened by the lesson-00 budget widget. The first slider sets the training compute budget C. The canvas sweeps N along C = 6ND and plots the loss L(N | C) using the toy law L = E + A/N^α + B/D^β — you will see the valley, and a marker sits on its minimum N*. Read off N*, the implied D* = C/(6N*), and the ratio D*/N*, which lands near 20 for every budget — that constancy is the Chinchilla result. Then drag the second slider, inference tokens you will serve: it switches the objective to total lifetime FLOPs and re-marks the optimum, which slides left to a smaller N with more D. Crank it up and watch compute-optimal drift into Llama-style over-training. For clarity the explorer uses equal exponents (α = β = 0.34) — a teaching simplification that makes the tok/param ratio exactly budget-independent, unlike the real asymmetric fit (α ≈ 0.34, β ≈ 0.28) in §1.

IsoFLOP explorer — find N*, then let serving move it

Sweep N along the fixed-budget curve C = 6ND and watch the loss valley. The white marker is the training-optimal N* (ratio lands ≈20 tok/param, the Chinchilla result). Drag inference tokens into the upper third (so 2N·D_inf rivals C) to see the optimum move — the effective optimum (orange) slides to smaller N / more D, deliberate over-training to cut serving cost. The knob that "breaks" compute-optimal: heavy serving makes the training-optimal model the wrong choice.

loss L(N | C) training-optimal N* serving-aware optimum

log₁₀ C (FLOPs): 22.0 log₁₀ inference tokens served: 0

Training-optimal N*

—

Training-optimal D*

—

D* / N* (tok/param)

—

Serving-aware N

—

Serving-aware tok/param

—

Notice the two markers coincide when inference tokens are zero — with no serving cost, compute-optimal is optimal. The moment serving FLOPs rival the training budget, the right choice drifts smaller. That single drift is the entire reason a 70B Chinchilla-optimal model and an 8B over-trained Llama can both be "correct" — they answer different budgets.

Failure modes & checklist

Failure modes

Under-training a giant (the GPT-3 trap). Spending the budget on parameters you have no tokens to train. Signal: tokens/param far below ~20 and a smaller model from the same budget beats you on every eval.
Fitting raw loss, ignoring the floor E. The reducible loss is straight on log-log; raw L bends near the floor. Signal: your extrapolation predicts a loss below the irreducible entropy of text — physically impossible.
One LR schedule across the ladder. Reusing a fixed schedule handicaps longer runs and biases the optimum toward big-and-undertrained (Kaplan's error). Signal: your fitted N-exponent comes out near 0.7 instead of 0.5.
Extrapolating from too narrow a ladder. Fitting one decade and reading off four. Signal: the prediction is wildly off when you finally run the big model; small confidence interval, wrong answer.
Compute-optimal when you will serve heavily. Minimizing training FLOPs while ignoring 2N·D_inf. Signal: serving bill dominates lifetime cost; a smaller over-trained model would have been cheaper end to end.

Checklist

Build a ladder, don't guess. Span several orders of magnitude in N and D; tune each rung's LR and schedule to its own size.
Fit the floor jointly. Recover E with the exponents; check the reducible loss is linear on log-log before trusting the line.
Target ≈20 tok/param for pure compute-optimal. Then sanity-check: N* ∝ C^0.5, D* = C/(6N*).
Price inference before committing N. Estimate D_inf; if serving is heavy, deliberately over-train a smaller model.
Treat the optimum as soft. 10–40 tok/param costs <1% loss; don't over-engineer the exact ratio — engineer the LR fairness instead.

Checkpoint

Try it

Open the widget and set log₁₀ C = 22 (≈10²² FLOPs). Read off N*, D*, and confirm the ratio lands near 20. Now raise C by 2 decades to 24 and check that N* grows by ≈10× (since N* ∝ C^0.5, two decades of C is one decade of N) — and that the ratio stays ≈20. Then, at C = 24, drag inference tokens from 0 up to 13 and watch the serving-aware marker slide left: estimate how much smaller the model should be if you will serve it ~10¹³ tokens. Finally, in one sentence, explain why GPT-3's ≈1.7 tok/param was the wrong end of the valley for its budget.

Where this points next

Scaling laws hand you a target: train a model of size N* on D* tokens (adjusted down in N if you will serve it heavily). But D* is a count — it says you need, say, 1.4 trillion tokens, and assumes every one of them is worth training on. They are not. The same compute spent on raw, deduplicated-but-unfiltered web text buys a dramatically worse model than the same compute on a well-curated mix; quality is the hidden multiplier the loss curve never sees. The power law tells you the quantity of data; it is silent on the quality, and that is where most of the remaining loss is actually hiding. Turning a token target into trillions of good tokens is an engineered pipeline — source, extract, filter, dedup, decontaminate, mix — and it is lesson 12 · Data.

Takeaway

Loss is a power law: L ≈ E + A/N^α + B/D^β, straight on log-log once you subtract the irreducible floor E — which is why a slope fit from a cheap ladder of small models extrapolates to a budget you cannot afford to test. Chinchilla's IsoFLOP sweep (fix C, vary N along C = 6ND, find the valley bottom N*) gives N* ∝ C^0.5 and D* ∝ C^0.5, so parameters and tokens grow together and the optimum lands near 20 tokens per parameter — correcting Kaplan-2020, whose fixed LR schedule biased the answer toward big-and-undertrained and left models like GPT-3 (≈1.7 tok/param) badly under-trained for their compute. But compute-optimal minimizes training loss per FLOP; the moment you will serve the model, the right objective is lifetime FLOPs 6ND + 2N·D_inf, and heavy serving pushes you to deliberately over-train a smaller model (the Llama strategy) so every one of billions of served tokens costs less. This is the bitter lesson made numerical: the frontier is a power law you can fit and spend against, the way every choice in this course spends the lesson-00 budget. It tells you how many tokens — not how good they must be, which forces data.

Interview prompts

Why is the loss curve a straight line on log-log axes, and what must you subtract first? (§1 — a pure power law L − E = A/N^α has log(L−E) = log A − α·log N, a line of slope −α; you must subtract the irreducible floor E or the raw loss bends near the floor and you extrapolate to an impossible value.)
Describe the IsoFLOP sweep and what it produces. (§2 — fix budget C, vary N along D = C/(6N), train several models, plot loss vs log N; you get a valley whose bottom is the compute-optimal N*; repeating across budgets traces the law.)
What did Chinchilla correct in Kaplan, and what changed numerically? (§2 — Kaplan held the LR schedule fixed, biasing the optimum toward big-and-undertrained (N ∝ C^0.73, ~1–2 tok/param); Chinchilla retuned per run and found N* ∝ C^0.5, D* ∝ C^0.5, ≈20 tok/param.)
Why was GPT-3 under-trained, and how would you have sized it? (§2 — 175B params on only ~300B tokens ≈1.7 tok/param, far below ~20; at compute-optimal it should have seen ~3.5T tokens, or been a much smaller model trained properly on the same budget — and a right-sized model (Chinchilla 70B/1.4T) beat it.)
How do you actually fit a scaling law, and name two pitfalls. (§3 — train a ladder of small models spanning sizes/token-counts each with tuned LR, fit E, A, α, B, β, extrapolate to target C; pitfalls: reusing one LR schedule across the ladder, fitting too narrow a range, or ignoring the floor / embedding-param counting.)
When and why should you train past compute-optimal? (§4 — when you will serve the model a lot; the real objective is 6ND + 2N·D_inf, so a large D_inf makes you shrink N and over-train (Llama: 7B–8B on 1T–15T tokens) to cut the permanent per-token 2N serving cost — see lesson 18.)
The ≈20 tokens/param ratio falls out of which fact, and how soft is it? (§2 — both N* and D* scale as C^0.5, so their ratio is budget-independent; empirically ≈20, but the valley is shallow — anywhere ~10–40 tok/param costs under ~1% of loss.)