Part IV — Data & evaluation

Evaluation — measuring what you can't see

Lesson 13 turned raw web text into a clean, deduplicated, decontaminated token stream and claimed each filter made the model better. Every lesson before it made the same kind of claim: RMSNorm is more stable, GQA is nearly free, Chinchilla picks a better (N, D), a better data mix lifts quality. You have been taking all of it on faith. The whole stack is a pile of changes that each assert an improvement, and you have no instrument to confirm a single one. Without honest measurement you are flying blind: you cannot compare two runs, cannot tell a real gain from noise, cannot even tell whether your model is good. This lesson builds the instrument — and shows why the instrument itself is the hardest, least-solved part of the field.

The previous step left this broken

Lesson 13 built a pipeline of stages — source, extract, filter, dedup, decontaminate, mix — and every stage was justified by "this improves the model." But improves against what number? Loss on the training set falls trivially as you memorize; that tells you nothing about a model you will deploy on text it has never seen. You need a measurement that (a) generalizes to held-out data, (b) is comparable across architectures, data mixes, and scales, and (c) actually tracks the thing you care about — usefulness. The naive instinct, "just look at the loss," gives you (a) and almost (b) but fails (c) badly, and the moment money rides on a benchmark number, the number itself starts to lie. Nothing in parts I–III is trustworthy until this is fixed.

Linear position

Forced by: a stack of changes each claiming improvement, with no instrument to confirm, compare, or rank them — you cannot improve what you cannot measure.
New idea: the eval toolkit — perplexity (intrinsic, exp(loss), clean but loosely coupled to usefulness) and downstream benchmarks (MMLU, HellaSwag, GSM8K, HumanEval), scored either as multiple-choice (argmax of option log-likelihoods) or generation (exact-match / pass@k) — wrapped in the hard truths of the eval crisis: contamination, prompt/format sensitivity, harness irreproducibility, and Goodhart.
Forces next: even a well-measured base model has a hidden ceiling your instrument never tested. Every number so far was scored on short inputs, because the model was trained at a short context window — the activation budget of lesson 05 forced that — and it is strong only within that window. Real use — long documents, whole codebases, multi-step agents — needs 10–100× more context, where the naive position handling of lesson 03's RoPE degrades sharply past its training length. Extending the window is the next forced step → lesson 15, Long context.

The plan

Six moves. (1) Perplexity = exp(loss): what it measures, why it is comparable on held-out text, and why it is only loosely coupled to usefulness. (2) The benchmark families and exactly how each is scored — MC by option log-likelihood, generative by exact-match and pass@k. (3) The contamination problem: leaked test data inflates scores by memorization, not skill; decontamination, canary strings, n-gram overlap. (4) Prompt and format sensitivity, few-shot vs zero-shot, and the reproducibility mess across harnesses. (5) Eval during training — a basket of metrics tracked over steps — and LLM-as-judge for open-ended outputs. (6) Goodhart's law: once a benchmark becomes a target it stops being a measurement. Then drive the contamination inflator and watch fake skill appear.

1 · Perplexity: the intrinsic ruler

The cheapest honest number you can compute is the model's own training objective evaluated on data it never saw. Take a held-out corpus, run the model, and average the per-token cross-entropy loss L (in nats, or bits if you use log base 2). Perplexity is just its exponential:

PPL = exp(L) = exp( −(1/T) · Σ_t log p_θ(x_t | x_<t) )

The interpretation is concrete: perplexity is the effective branching factor — the model is, on average, as uncertain about the next token as if it were choosing uniformly among PPL equally-likely options. A perplexity of 1 means perfect prediction; a perplexity equal to the vocabulary size V means the model learned nothing (uniform guessing over V ≈ 50,257 tokens). Real models on held-out web text land in the low tens: GPT-2 small ≈ 37 PPL on WikiText-103, modern models on clean text in the single digits to mid-teens depending on tokenizer and domain.

Perplexity has three virtues. It is cheap — one forward pass, no generation, no grading. It is dense — every token contributes signal, so it has low variance and resolves tiny differences between runs that a coarse accuracy number cannot. And it is the exact quantity scaling laws predict (lesson 12): the loss curve you fit and the perplexity you measure are the same object, so it is the natural metric to watch a run against its predicted trajectory.

Perplexity is not comparable across tokenizers

Cross-entropy is per token, and a token is whatever your BPE merges decided (lesson 1). A model with a 256k vocab packs more text into each token, so its per-token loss looks lower than a 50k-vocab model's even at identical real-world quality. Two perplexities are only comparable when the tokenization is identical, or when you normalize to bits-per-byte (divide total loss by the number of underlying UTF-8 bytes, not tokens). Quote PPL across different tokenizers and you are comparing nothing.

The fatal limitation is the third leg of the instrument: perplexity is only loosely coupled to usefulness. It rewards being unsurprised by text, which is necessary for competence but nowhere near sufficient. A model can shave half a nat off held-out loss by getting better at predicting boilerplate, whitespace, and common function words — tokens that dominate the corpus by count — while its ability to do multi-step arithmetic or follow an instruction barely moves. Two models can sit within 0.1 PPL of each other and differ by 15 points on a reasoning benchmark. Perplexity tells you the model compresses text well; it does not tell you the model is good at things you want. For that you must measure the things directly.

2 · Benchmarks, and how each is actually scored

A benchmark is a fixed set of tasks with known answers. The subtlety — the part that trips up everyone reading a leaderboard — is that the same model produces wildly different scores depending on how you turn its next-token distribution into an answer. There are two scoring regimes, and which one a benchmark uses is part of the benchmark's definition.

Benchmark	What it tests	Format	How it is scored
MMLU	57 subjects of knowledge (law, biology, math…)	4-way multiple choice	argmax of per-option log-likelihood
HellaSwag	commonsense sentence completion	4-way multiple choice	argmax of (length-normalized) log-likelihood
GSM8K	grade-school multi-step arithmetic	free-form generation	exact-match on the final number
HumanEval	Python function synthesis from a docstring	free-form code generation	pass@k — does generated code pass hidden unit tests

Multiple-choice = argmax of option log-likelihoods

For MMLU or HellaSwag you do not ask the model to "type A, B, C, or D" and parse its reply. A base model is a completer; it may not emit a clean letter at all. Instead you score each option by the log-probability the model assigns to the option's text given the question, and pick the largest. With prompt q and options {o₁, …, o₄}:

# MMLU-style scoring: no generation, just K forward passes
for each option o_i:
    seq = prompt + o_i
    logp[i] = sum( log p_theta(token_t | token_<t) for token_t in o_i )
prediction = argmax_i logp[i]          # the option the model finds least surprising

This is purely a likelihood computation — no sampling, no decoding, deterministic. One wrinkle bites hard: length normalization. Raw summed log-prob favors shorter options (fewer negative terms to add), so HellaSwag, whose options differ in length, divides by token count (or byte count) before the argmax. because MMLU's option texts also vary in length, harnesses disagree on whether to length-normalize — and that single choice can swing a reported MMLU number by several points (more on this in §4).

Generative = exact-match and pass@k

GSM8K and HumanEval cannot be scored by likelihood because the answer space is open. You actually generate. For GSM8K you sample a chain-of-thought, extract the final number with a regex, and check exact equality to the gold answer — brittle (a correct answer formatted as "$42.00" vs "42" can be marked wrong) but objective. For code, pass@k is the honest metric: generate k independent samples, run each against hidden unit tests, and count the problem solved if any sample passes. The unbiased estimator from n ≥ k samples with c correct is:

pass@k = 1 − ( _n−cC_k / _nC_k )

pass@1 measures single-shot reliability; pass@100 measures whether the model can solve it given many tries — a much higher number, and the right one to watch when a downstream verifier (a test suite, a math checker) can pick the winning sample. That gap between pass@1 and pass@k is exactly the lever that verifiable-reward RL exploits to turn "can sometimes solve it" into "reliably solves it" — see the reinforcement learning track on verifiers.

3 · The eval crisis, part one: contamination

Here is the failure that quietly invalidates a large fraction of published numbers. A benchmark measures skill only if the model has never seen the test items during training. But the training set is the entire scraped web (lesson 13), and benchmarks live on the web — in papers, GitHub, Hugging Face, blog posts, leaderboard dumps. So the test answers leak into pretraining, and the model can score high by memorization, not skill. A model that has memorized the GSM8K answer key looks like a brilliant mathematician and is a brilliant lookup table.

Contamination is insidious because it inflates scores in exactly the direction you were hoping to see, so it never trips your suspicion. The defenses, all imperfect:

Decontamination

Before training, scrub from the corpus any document that overlaps a known test set. This is the same decontamination step from lesson 13 — but it only works for benchmarks you knew about at training time, never for the one published next month.

n-gram overlap

Flag a test item as contaminated if a long n-gram (e.g. a 13-gram, or ≥50% of its tokens) appears verbatim in training. Cheap and standard, but blind to paraphrase — a reworded question slips straight through.

Canary strings

Embed a unique random GUID in the benchmark files (BIG-bench popularized this). If the model can reproduce the canary, the benchmark was in its training data — a tripwire that proves leakage after the fact.

Held-out / fresh tests

Build private test sets, or time-gate them (only items created after the model's training cutoff). The only contamination-proof option — and the reason new benchmarks appear constantly as old ones rot.

The brutal asymmetry: a tiny leak buys a huge fake gain. Memorizing even a few percent of the test set lifts the reported score far more than a real capability improvement that took a 10× compute increase to earn. The true held-out skill — performance on items genuinely never seen — can be flat while the reported number marches upward. That gap, reported-minus-true, is pure illusion, and you cannot see it from the reported number alone. The widget below makes the gap visible by holding a clean held-out partition aside and watching the two diverge.

4 · The eval crisis, part two: it barely reproduces

Even with zero contamination, the same model on the same benchmark gives different scores in different hands. The score is a function of dozens of unstated choices, and there is no canonical answer to most of them.

prompt formatThe exact wording, the answer delimiter, the order of options, even a trailing space change tokenization and shift accuracy by several points. Models are absurdly sensitive to the surface form of a question they "know."

few-shot vs zero-shotPrepend k worked examples (k-shot) and scores usually rise — the examples teach the format as much as the task. MMLU is conventionally 5-shot; report a zero-shot number against a 5-shot leaderboard and you look far worse for no real reason.

normalization & extractionLength-normalize the MC log-likelihoods or not? Which regex pulls the answer out of a generation? Each choice silently moves the number.

harnessEleutherAI lm-eval-harness vs HELM vs a lab's in-house code implement all of the above differently. The same checkpoint can read several points apart across harnesses — so cross-paper comparisons are often meaningless unless every knob matches.

The practical upshot: a leaderboard number is only meaningful relative to other numbers produced by the identical harness, prompt, and shot count. The headline "Model X scores 71.2 on MMLU" is nearly content-free without the protocol. This is why serious comparisons pin a single harness and re-run every model through it rather than copying numbers from papers — and why a 1-point benchmark delta should never decide anything.

5 · Eval during training, and judging the open-ended

You do not wait until the end to measure. During a run you track a basket of metrics every few thousand steps: held-out perplexity (the dense, low-variance early-warning signal), plus a handful of cheap multiple-choice benchmarks (HellaSwag, a slice of MMLU) for downstream signal. The basket matters because no single number is trustworthy — perplexity can keep falling while a benchmark plateaus (the model is compressing better without getting more capable), and a single benchmark is too noisy and too gameable to steer by. You watch the set move together, and you watch it against the scaling-law trajectory from lesson 12 to catch a run that has silently gone off the rails.

Multiple-choice and exact-match cover tasks with a known answer. But the things you ultimately ship a model for — "write a helpful reply," "summarize this without lying," "is this answer better than that one" — have no gold string to match. For these, the dominant method is LLM-as-judge: prompt a strong model (or a panel) to score or rank candidate outputs against a rubric. It correlates surprisingly well with human preference and scales to millions of comparisons that humans could never grade. It also inherits the judge's biases — a documented tilt toward longer answers, toward its own family's style, toward whichever candidate is presented first — so you randomize positions, calibrate against human labels, and never treat the judge as ground truth. This same machinery — a model (or a verifier) producing a reward signal over open-ended generations — is exactly what preference and reinforcement learning consume; the deeper treatment lives in the reinforcement learning track on verifiers and judges.

6 · Goodhart: the benchmark dies when it becomes a target

Goodhart's law: when a measure becomes a target, it ceases to be a good measure. A benchmark is a cheap proxy for a capability. The moment the field optimizes for the benchmark — training on its train split, tuning prompts to it, or worse, leaking its test split — the proxy decouples from the capability it stood for. MMLU was a sharp instrument in 2021; after years of being a headline target it is saturated and contaminated enough that a high score proves little. This is not cheating by a few bad actors; it is the structural fate of every public benchmark, because everyone is incentivized to climb it and climbing it is easier than improving the underlying capability.

The defenses follow directly: keep test sets private or fresh (held out from optimization), rotate benchmarks before they saturate, prefer verifiable tasks where the answer is checkable rather than judged (a unit test cannot be charmed), and never report a single number — report a basket, with the protocol, and treat a single saturated benchmark as expired. The instrument you build in this lesson is real and necessary; the honest version of it knows it is always one leak and one optimization cycle away from lying to you.

7 · Drive the contamination inflator

The widget below is the eval crisis in one knob. The slider is the fraction of the test set that leaked into training. The model scores leaked items by memorization (≈100% on the part it has seen) and by real skill (a fixed, modest rate) on the rest. The reported score is what a naive evaluation prints — it averages over the whole test set, memorized and not. The true held-out score is what the model would get on a clean partition it never saw: flat, because leaking test data teaches the model nothing transferable. Drag the leak up from zero and watch the two diverge — the growing gap is fake skill. Note how violently a small leak (5–10%) lifts the reported number. The second readout pairs perplexity against accuracy to show their loose coupling: nudge "real capability" and perplexity barely twitches while accuracy moves, or vice-versa.

Contamination inflator — how a small leak buys big fake skill

Slide % of test set seen in training and watch the reported benchmark score climb while the true held-out score stays flat — the gap between them is memorization, not capability. A 10% leak alone can add double-digit points. The real capability slider sets the model's genuine skill (the only thing that moves the true held-out curve and the bulk of perplexity); notice perplexity and accuracy are only loosely linked. The knob that "breaks" trust: any nonzero leak makes the reported number a lie.

reported score (naive eval) true held-out score current leak

% test set seen in training: 0% real capability: 45%

Reported score

—

True held-out

—

Fake skill (gap)

—

Held-out perplexity

—

The lesson of the widget is the lesson of the whole crisis: the reported number is not the model's skill plus noise — it is the model's skill plus a contamination term you cannot see and a protocol term you rarely control. An instrument with two hidden additive biases is not an instrument you trust blindly; it is one you cross-check, decontaminate, freshen, and read as a basket.

Failure modes & checklist

Failure modes

Comparing perplexity across tokenizers. Quoting a lower PPL for a 256k-vocab model as if it beat a 50k-vocab one. Signal: a model with a bigger vocab "wins" on PPL but loses on every downstream benchmark — the per-token loss was never comparable; normalize to bits-per-byte.
Trusting a leaderboard number without the protocol. Copying "MMLU 71.2" from a paper and comparing to your zero-shot, un-normalized run. Signal: your re-run of the same checkpoint lands several points off the published figure — different harness, shots, or normalization.
Undetected contamination. A new benchmark scrubbed only against old test sets; the model memorized the new one off the web. Signal: a suspiciously sharp jump on one benchmark with no movement on siblings, or the model reproduces the benchmark's canary string verbatim.
Steering a run by one benchmark. Tuning data mix to push a single MMLU slice. Signal: that one number climbs while the rest of the basket and held-out PPL stagnate — you optimized the proxy, not the capability (Goodhart).
Treating LLM-as-judge as ground truth. Ranking systems by a judge without position-randomization or human calibration. Signal: the longer or first-presented answer wins systematically; rankings flip when you swap option order.

Checklist

Report perplexity as bits-per-byte when tokenizers differ; otherwise pin the tokenizer and say so.
Pin the protocol — harness, prompt, shot count, normalization — and re-run every model through the identical pipeline before comparing.
Decontaminate against every known test set (lesson 13), embed canaries, run n-gram overlap, and prefer fresh or private held-out sets.
Track a basket over training steps — held-out PPL plus several cheap benchmarks — and watch it against the scaling-law trajectory, never a single number.
Use verifiable tasks where possible (tests, math checkers); for open-ended outputs use a calibrated, position-randomized LLM-judge and treat it as a proxy, not truth.
Treat a 1-point benchmark delta as noise. Decide on robust, multi-metric gaps that survive a protocol change.

Checkpoint

Try it

Open the contamination inflator. Set real capability to 45% and leak to 0 — reported and true held-out agree at ≈45%, the honest reading. Now drag the leak to just 10% and read the reported score: it jumps well into the 50s while the true held-out stays pinned at 45 — that ~5-point gap is fake skill bought by memorizing one item in ten. Push the leak to 100% and the reported number races toward 100% while true held-out never moves. Then return the leak to 0 and sweep real capability: confirm that the true held-out curve and the bulk of perplexity respond to that slider, not the leak. In one sentence, explain why a model that scores 95% with a 30% leak is worse than one that scores 60% with no leak — and which number you would have to compute to know.

Where this points next

You now have an instrument: perplexity for dense intrinsic signal, benchmarks scored by log-likelihood or generation for downstream signal, contamination defenses and a pinned protocol to keep them honest, a basket tracked over training, and an LLM-judge for the open-ended. Point it at the base model you have built across lessons 0–13 and it scores well — but every one of those numbers was measured on short inputs. There is a ceiling your instrument never tested: the model was trained at a short context window, because the activation budget of lesson 05 forced it, and it is strong only inside that window. The moment you ask it to read a long document, reason over a whole codebase, or run as a multi-step agent, you blow past its training length — and lesson 03's RoPE, the positional scheme that worked beautifully in-window, degrades sharply once positions run past anything it saw in training. A model that is excellent on a paragraph and useless on a book has not earned the use cases that matter. So before alignment, before anything else, the next forced step is to extend the window — keep the capability your instrument confirmed and make it hold over 10–100× more tokens. That is lesson 15, Long context — extending the window.

Takeaway

You cannot improve, compare, or trust what you cannot measure — so before any claim in parts I–III means anything, you need an instrument. Perplexity = exp(loss) is the cheap, dense, intrinsic ruler (the effective branching factor over next tokens); it is the exact quantity scaling laws predict, but it is only comparable within a fixed tokenizer (else use bits-per-byte) and only loosely coupled to usefulness — a model can compress text better without getting smarter. Benchmarks measure capability directly, scored two ways: multiple choice by the argmax of per-option log-likelihoods (MMLU, HellaSwag — watch length normalization), and generation by exact-match or pass@k (GSM8K, HumanEval). But the instrument is half-broken by the eval crisis: contamination (leaked test data inflates scores by memorization — a tiny leak buys huge fake skill; defend with decontamination, canaries, n-gram overlap, fresh held-out sets), brutal prompt/format/shot sensitivity and harness irreproducibility (a number is meaningless without its protocol), and Goodhart (a benchmark dies the moment it becomes a target). So you eval a basket over training steps, lean on verifiable tasks, and use a calibrated LLM-judge for the open-ended. And the punchline that forces the next lesson: every one of those numbers was measured on short inputs, so a well-measured base model carries an untested ceiling — it was trained at a short context window and is strong only inside it, which is why extending the window comes next.

Interview prompts

What is perplexity and why is it not comparable across tokenizers? (§1 — PPL = exp(per-token cross-entropy), the effective branching factor over next tokens; it is per token, and a bigger-vocab model packs more text per token so its per-token loss looks lower at equal real quality — normalize to bits-per-byte to compare.)
How is a multiple-choice benchmark like MMLU scored for a base model? (§2 — not by parsing a generated letter; you compute the summed (often length-normalized) log-likelihood of each option's text given the prompt and take the argmax — the least-surprising option — purely a likelihood computation, no sampling.)
What does pass@k measure and why does it differ from pass@1? (§2 — generate n samples, count solved if any of k passes hidden tests; pass@1 is single-shot reliability, pass@k is whether the model can solve it given tries — the gap is what a verifier-driven RL loop exploits.)
Why does a small training-set leak inflate a benchmark so much, and how do you defend? (§3 — memorized items score ≈100% vs a modest real-skill rate, so even a few percent leak adds big fake points the reported average can't distinguish from skill; defend with decontamination, n-gram overlap, canary strings, and fresh/private held-out sets.)
Two papers report different MMLU scores for the same checkpoint — why? (§4 — prompt wording, zero- vs few-shot, option-order, length-normalization, and harness (lm-eval vs HELM vs in-house) all shift the number by points; a benchmark figure is only comparable within an identical pinned protocol.)
Why track a basket of metrics during training instead of one? (§5 — perplexity is dense/low-variance but loosely coupled to usefulness and can fall while a benchmark plateaus; a single benchmark is noisy and gameable; you watch the set move together against the scaling-law trajectory to catch a run going off the rails.)
State Goodhart's law for benchmarks and its consequence. (§6 — when a measure becomes a target it stops measuring; optimizing for a public benchmark (tuning to it, leaking it) decouples it from the capability, so every public benchmark eventually saturates/contaminates — keep tests private/fresh, prefer verifiable tasks, report baskets.)
Your instrument scores the base model well — what has it not tested, and what forces the next lesson? (Where this points next — every number was measured on short inputs, but the model was trained at a short context window (activation budget, lesson 05) and is strong only inside it; long documents, codebases, and agents need 10–100× more context, where RoPE (lesson 03) degrades past its training length — so the next step is extending the window, lesson 15.)
Your instrument says the base model knows the answer but it won't act like an assistant — what's missing and what fixes it? (A separate gap the instrument exposes — it maximizes p(text), not helpful behavior, so it completes rather than follows; the fix is alignment, SFT (lesson 16) trains on instruction/response pairs to install the behavior — but that comes after the context window is extended.)