Part IV — Data & evaluation

Data — the pipeline that decides quality

Lesson 11 handed you a number: train on D* ≈ 20 tokens per parameter, and your fixed budget buys the lowest loss the scaling law allows. But that law was fit on good tokens. Point the same compute at raw web text and you get a model that babbles, repeats itself, and scores worse on every benchmark — same FLOPs, far worse model. D* is a quantity target; nothing in parts I–III says where the tokens come from or how good they are. Quality is the hidden multiplier on your entire budget, and it is manufactured, not found. This lesson is the factory.

The previous step left this broken

Lesson 11's IsoFLOP sweep tells you to feed the model D* ≈ 20·N tokens — for a 7B model that is ≈140B tokens; for a frontier run, trillions. It silently assumes those tokens are clean, diverse, non-repeated, and not the test set. They are not. The only place trillions of tokens exist is the open web, and the open web is mostly boilerplate, spam, near-duplicates, adult content, and — fatally — copies of the very benchmarks you will grade yourself on. Compute-optimal D spent on garbage is compute wasted: the loss floor E you actually hit is set by the data, not the law. Nothing so far turns raw bytes into D* tokens worth training on.

Linear position

Forced by: compute-optimal D* is worthless on garbage tokens — quality, not quantity, sets the loss you actually reach, and quality must be engineered.
New idea: data is a pipeline, not a corpus — source → extract → filter → dedup → decontaminate → mix — a funnel that turns petabytes of raw crawl into a few percent of usable, weighted, dedduped tokens, and most real-world quality gains live in this pipeline rather than in the architecture.
Forces next: every stage here claims to improve the model — "this filter is better," "this mix helps." The only way to know, and the only way to compare two runs at all, is to measure. That forces evaluation, lesson 14.

The plan

Eight moves. (1) Sources and the brutal scale gap (raw web ≫ usable). (2) Extraction: HTML → text, boilerplate removal. (3) Filtering: language ID, Gopher heuristics, classifier quality, perplexity — the funnel down to a few percent. (4) Deduplication: exact and fuzzy (MinHash/LSH), and why dups act like extra epochs. (5) Decontamination: cutting the eval set out (forward-ref lesson 14). (6) Mixing & weighting domains (DoReMi), curriculum, quality annealing, and data-constrained scaling (≈4 epochs is fine). (7) The data wall and synthetic data: spend compute to manufacture tokens when human text runs out. (8) Train the tokenizer on the final mix (callback lesson 1). Then drive the funnel widget and watch yield vs the D target.

1 · Sources, and the scale you start from

Pretraining data comes from a few buckets, each with a different quality-per-token and a different size.

Common Crawl

The web, monthly. Stored as WARC (raw HTTP responses, incl. HTML) and the derived WET (rough plaintext). One monthly snapshot ≈ a few hundred TB compressed; the full archive is petabytes. Vast, free, and mostly junk.

Code

GitHub-scale source (e.g. The Stack, ≈hundreds of GB to a few TB after license filtering). Punches above its weight: code teaches reasoning and structure, not just syntax.

Books & reference

Long-form, high-quality prose and curated reference (encyclopedic text, papers). Small in bytes, high in quality-per-token — usually heavily up-weighted in the mix.

Curated web

Pre-cleaned public corpora (C4, RefinedWeb, FineWeb, Dolma) — Common Crawl already run through somebody's pipeline. Most teams start here rather than from raw WARC.

The headline fact is the gap between raw and usable. A single Common Crawl snapshot is on the order of 10¹⁴–10¹⁵ bytes; after the full pipeline below, a published web corpus like RefinedWeb or FineWeb keeps only a few percent of the raw input as trainable tokens — FineWeb distilled ≈96 monthly crawls down to ≈15T tokens, with the great majority of every snapshot discarded. Plan from the top of the funnel: if you need D* = 1T clean tokens and your pipeline yields 4%, you must ingest ≈25T tokens of raw text. Filtering is not a tweak at the end; it sizes the whole ingestion job.

2 · Extraction — HTML is not text

A WARC record is an HTTP response: headers, then an HTML document wrapped in navigation bars, cookie banners, ad slots, sidebars, comment sections, and footers. The model should learn from the article, not the chrome. Extraction is the step that pulls the main content out of the markup and throws the rest away.

The naive route — strip every tag — leaves you with menu text, "Accept all cookies," and 40 copies of the site's footer per page. Real pipelines use boilerplate removal: tools like trafilatura, jusText, or resiliparse score each DOM block by text-density and link-density (a block that is 90% links is navigation; a block of flowing sentences is content) and keep only the main body. Whether you extract from the rich WARC/HTML (better, what FineWeb does) or the lossy pre-extracted WET (cheaper, what C4 used) measurably changes downstream quality. Extraction is unglamorous and it is where a surprising amount of the final quality is won or lost.

3 · Filtering — the funnel

After extraction you have a flood of plaintext documents of wildly varying value. Filtering is a cascade of cheap-to-expensive classifiers, each dropping a slice. Order matters: run the cheap, high-yield cuts first so the expensive ones see fewer documents.

language IDA fastText-style classifier tags each document's language and gives a confidence. Keep your target language(s) above a threshold (e.g. English > 0.65). This alone removes a large fraction of the web and most encoding garbage. Set the bar too high and you delete code-switched or technical text.

heuristic rules (Gopher)Hand-written quality signals from DeepMind's Gopher: drop documents outside 50–100,000 words; mean word length outside 3–10 chars; >90% of lines starting with a bullet; >30% lines ending in "…"; too few stop-words (real prose has many); excessive symbol-to-word or duplicate-line ratios. Cheap, interpretable, and they kill obvious spam and machine-generated lists.

classifier qualityTrain a binary classifier on "good vs random": positives = a reference corpus you trust (e.g. reference/encyclopedic/curated text), negatives = random Common Crawl. Score every document; keep the high-scoring tail. GPT-3 and the Pile used exactly this good-vs-random framing. Newer pipelines (FineWeb-Edu) instead score "educational value" with a small LLM-labeled classifier — a big quality win.

perplexity filterRun a small reference language model over each document and drop the ones it finds extremely surprising (very high perplexity) — usually gibberish, OCR noise, or broken encoding — and sometimes the suspiciously low-perplexity ones too (boilerplate, repeated templates). KenLM-style models make this cheap at corpus scale.

Stack these and the survival rate is brutal. A representative web funnel: start at 100% extracted text → language ID keeps ≈45% → Gopher rules keep ≈60% of that (≈27%) → quality classifier keeps ≈40% of that (≈11%) → and dedup (next) removes tens of percent more — often 20–50% depending on the corpus and the similarity threshold — leaving a few percent. The funnel is the central image of this lesson: raw → a few percent survive. Every threshold is a knob, and every knob trades quality against quantity — push too hard and you starve D*; too soft and you train on junk that raises your loss floor.

4 · Deduplication — duplicates are stealth epochs

The web is full of copies: mirror sites, quoted articles, license boilerplate, SEO spam that reprints the same paragraph a thousand times. Deduplication removes them, and it matters more than it looks.

The reason is mechanical. If a passage appears k times in your corpus, the model sees it k times per epoch — a duplicate is an unrequested extra epoch on that one passage. That has three costs: it wastes compute (you paid FLOPs to re-learn what you already knew), it biases the distribution toward whatever happens to be over-replicated, and it drives verbatim memorization — models reproduce duplicated training strings far more readily, which is both a privacy/copyright liability and a sign the model is storing text instead of generalizing. Deduplicated training measurably lowers held-out loss for the same token budget.

	Exact dedup	Fuzzy dedup (MinHash / LSH)
Catches	Byte-identical documents or paragraphs (hash and compare)	Near-duplicates: same article with a changed header, reordered ad, one edited line
Method	Hash each document / each overlapping n-gram chunk; collisions = dups	Shingle into n-grams → MinHash signature → LSH buckets candidates with high Jaccard overlap → verify
Cost	Cheap, one pass	More expensive, but LSH avoids the O(docs²) all-pairs comparison
Misses	Anything not byte-identical — most real web dups	Threshold-dependent: too aggressive merges distinct docs that share a quote

The fuzzy machinery is worth knowing in one breath: cut each document into overlapping word shingles (say 5-grams); a MinHash signature is the set of minimum hash values over those shingles, and the probability two documents share a given MinHash equals their Jaccard similarity — so a short signature estimates overlap without comparing full texts. LSH (locality-sensitive hashing) then bands those signatures so only documents likely to be similar land in the same bucket, turning an impossible all-pairs problem into a near-linear one. Real pipelines run exact dedup first (cheap), then MinHash/LSH fuzzy dedup at a Jaccard threshold around 0.8.

5 · Decontamination — cut out the answer key

One special kind of duplicate is catastrophic: copies of your evaluation data sitting in the training set. Benchmarks like MMLU, GSM8K, and HumanEval are published on the open web, so Common Crawl contains them. Train on them and your model memorizes the answer key; its benchmark scores climb while its real ability does not. That is contamination, and the fix is decontamination: before training, remove any training document that overlaps a known eval example — typically by n-gram overlap (drop a doc sharing, say, a 13-gram with any test item) or by hashing eval examples and filtering matches.

This is a forward reference you should hold onto: decontamination is a data step, but the reason it matters and how you detect a leak (held-out vs reported gaps, canary strings, n-gram overlap audits) is the subject of lesson 14 — Evaluation. The two lessons are a pair: data is where you prevent contamination; evaluation is where you measure whether you succeeded. Do it imperfectly and every number you report afterward is suspect.

6 · Mixing, curriculum, and annealing

You now have clean, deduplicated, decontaminated pools — web, code, books, math, multilingual. The last decision is the mixture: what fraction of training tokens comes from each domain. This is one of the highest-leverage knobs in the whole stack and it is mostly set by intuition-plus-ablation. Naively weighting by raw size drowns the small high-quality pools (books, math) under web text; you almost always up-weight the good-but-small domains.

DoReMi (Domain Reweighting with Minimax Optimization) makes this principled: train a small proxy model, use group-distributionally-robust optimization to find the domain weights that minimize worst-case excess loss across domains, then train the big model with those weights. It found mixtures that beat hand-tuned ones and sped up training at fixed quality — concrete evidence that the mix, not just the architecture, moves the loss.

Two refinements stack on top of the static mix:

Curriculum

Order the data instead of shuffling uniformly — e.g. easier/cleaner text early, harder/specialized text later. Effects are real but smaller and finickier than the mix itself.

Quality annealing

The big modern win: in the final phase of training, while the LR cosine-decays toward zero, shift the mix sharply toward the highest-quality data — curated text, textbooks, math, code, even some instruction-like data. The last tokens land while the model is most plastic, so they punch far above their count. Llama-3 and others credit a high-quality anneal for a real bump.

And what if a domain runs out of tokens before you hit D*? You repeat it — and data-constrained scaling (Muennighoff et al., 2023) says that is fine within limits: repeating data for up to ≈4 epochs is nearly as good as fresh unique tokens; beyond that, returns decay fast and extra epochs approach worthless. So the practical recipe when unique data is scarce is: dedup hard, then allow up to ≈4 passes on the high-quality pools rather than letting D fall short of the scaling-law target from lesson 12.

7 · Synthetic data and the data wall

The §6 epochs trick assumes you can at least repeat what you have. But step back to the whole web: the supply of high-quality human text is finite. After extraction, filtering, and dedup, the usable pool is on the order of ≈10–20T tokens of decent web text — and that is roughly all there is. Now look at the demand side from lesson 12: D* ≈ 20·N for the largest models already pushes into the tens of trillions. The two numbers are colliding. This is the data wall: the compute-optimal token count for a frontier model is approaching the total stock of good human text, so you can no longer assume fresh unique tokens are there for the taking.

The first response is the one §6 already gave: repeat. Data-constrained scaling says up to ≈4 epochs on the high-quality pool is roughly as good as the same volume of fresh tokens, so dedup hard and reuse before you undershoot D*. That buys a factor of a few, not a factor of a hundred. The second response is to manufacture tokens: synthetic data — text generated by a model rather than scraped. The common recipes: distillation-for-pretraining (sample from a stronger model and train on its outputs — see the distillation track); self-instruct (bootstrap instruction/response pairs from the model itself); textbook-style generation (the Phi line: prompt a strong model to write clean, pedagogical, "textbook-quality" explanations); and rephrasing/augmenting web text (rewrite messy crawl into cleaner, denser variants instead of inventing facts from scratch).

Put it back in budget terms — the spine of this whole course. Synthetic data spends compute (the FLOPs to generate it) to buy data you cannot otherwise obtain. It is a way of converting the one resource you have (compute) into the one you've run out of (good tokens), so it shifts the bottleneck rather than removing it: you are still paying, just on the other side of the ledger.

State the risks honestly, because synthetic data is not free quality. Model collapse: train enough on a model's own outputs and the corpus's distribution narrows — tails get clipped, diversity erodes, and successive generations drift toward a blander mean. Factual drift: a generator's mistakes and hallucinations become training labels, so errors compound instead of averaging out. And the quiet one — synthetic data can smuggle in eval contamination: if the generator was itself trained on benchmark text, it can regurgitate test items into your "fresh" synthetic corpus, re-poisoning the decontamination you did in §5 (callback to lesson 14). The practical rule mirrors the epochs rule: synthetic and rephrased data help as a fraction of a mostly-human corpus; let it dominate and the distribution you are modeling stops being the real world.

8 · The tokenizer is trained on this mix

Recall from lesson 1 that BPE is trained: its merges are learned from a corpus, and they decide your compression ratio (bytes per token) and therefore how many tokens a given text becomes. The corpus you train the tokenizer on should be the final data mix — not raw web, not a single domain. Train it on English-only web and code tokenizes terribly (every brace and indent costs tokens); train it on the actual blend and the merges match what the model will actually see, so your effective sequence length and your D budget come out as planned. The pipeline closes the loop: the data mix you engineer here is the same corpus that defines the tokens lesson 1 was counting.

The under-glamorized truth

Architecture papers get the citations; data work gets the results. Holding the model fixed, the difference between a careless corpus and a well-engineered one (extraction, aggressive dedup, a good quality classifier, a tuned mix, a quality anneal) is the difference between a mediocre and a strong model on the same compute. This is why frontier labs treat their data pipeline as the moat and publish architecture freely. Building this end to end — extraction, filtering, dedup, the funnel — is CS336 Assignment 4.

9 · Drive the funnel

The widget is the funnel from §3–§6 as a set of knobs. You start with raw Common Crawl in TB; each stage's slider sets how aggressive it is, and the bar shrinks to the tokens that survive it. The bottom KPI compares your final usable tokens against the D target from lesson 12 (≈20 tokens per parameter for your chosen N). The tension is the whole point: crank every filter to maximum and quality rises but the yield falls below D* — you have starved the run and must either ingest more raw data, allow more epochs (≤4), or shrink N. Loosen everything and you clear D* easily but the surviving tokens are junk, raising the loss floor the scaling law cannot fix. There is a sweet spot, and it moves with how much raw crawl you ingest.

Data funnel — raw crawl to usable tokens vs the D target

Each slider is one stage's aggressiveness; the bar shows tokens surviving it. The goal is to clear the D target (≈20 tokens/param for your N) with the cleanest tokens you can. Over-filter → yield drops below D* (run starved). Under-filter → you clear D* but a "junk fraction" warning fires (loss floor rises). Raise raw ingest to buy headroom. Repeat factor lets you re-use survivors up to 4 epochs.

raw ingest (TB CC text): 200 target model N:

language strictness: 55 quality threshold: 55 dedup aggressiveness: 60 decontam: on epochs (repeat): 1

Usable tokens (×epochs)

—

D target (≈20·N)

—

Overall yield

—

Verdict

—

Notice the asymmetry: the quality slider and the yield fight each other, but the decontam toggle is nearly free in tokens (it removes a sliver) and yet flipping it off is the single most damaging thing you can do — it doesn't starve D, it poisons every measurement you take afterward. That is exactly why the next lesson exists.

Failure modes & checklist

Failure modes

Filtering to quality, starving D. Cranking thresholds so the surviving corpus is pristine but smaller than D*, forcing far more than 4 epochs. Signal: training loss undershoots then plateaus/overfits early; held-out loss diverges from train.
Skipping fuzzy dedup. Running only exact dedup and assuming you're clean. Signal: the model emits verbatim training passages on prompting; memorization metrics and duplicated-string recall spike.
Forgetting decontamination. Benchmarks in Common Crawl leak into training. Signal: reported scores jump but a fresh held-out variant of the same task doesn't move (the gap is fake — see lesson 14).
Size-weighted mixing. Mixing domains by raw byte count, drowning books/math/code under web. Signal: weak reasoning and code despite a "large" corpus; ablations show up-weighting the small pools helps.
Tokenizer trained on the wrong corpus. BPE fit on English web, then training on a code-heavy mix. Signal: bytes/token far worse on real data, effective D shrinks, sequences blow past the planned length.

Checklist

Size the funnel from the top. If yield is ≈4% and you need D* clean tokens, ingest ≈25× that in raw text.
Cheap filters first. Language ID and Gopher rules before classifier and perplexity, so expensive scorers see fewer docs.
Exact then fuzzy dedup. Hash dedup, then MinHash/LSH at Jaccard ≈0.8; dedup before you decide epochs.
Decontaminate against every eval you'll report. N-gram overlap removal; log what was cut so lesson 14's numbers are trustworthy.
Tune the mix, then anneal. Up-weight small high-quality pools (DoReMi or ablation); shift sharply to the best data in the final LR-decay phase.
Repeat ≤ 4 epochs. If unique data is short of D*, re-use the best pools up to ≈4 passes before adding lower-quality tokens.
Train the tokenizer on the final mix. Not raw web, not one domain — the exact blend the model will see.

Checkpoint

Try it

Open the funnel and pick N = 7B, so the D target is ≈140B tokens. Start with 200 TB raw ingest and all sliders mid-range. Now crank quality threshold and dedup to maximum: watch usable tokens drop below the target — the verdict flips to "starved." Fix it two ways and compare: first raise raw ingest until you clear D*; then instead drop raw back and lift epochs to 4. In one sentence each: which fix is "free" and which costs you (a) more ingestion/extraction compute and (b) the diminishing returns past ≈4 epochs? Finally, toggle decontam off and note that the token count barely changes — then explain, in terms of lesson 14, why that "barely changes" is the most dangerous knob on the panel.

Where this points next

You have built the factory: raw crawl in, a weighted, deduplicated, decontaminated, tokenizer-matched corpus out, sized to hit D*. But every stage made a claim — this filter is better, this mix helps, this anneal bumps quality, this dedup lowered the loss floor. So far you have taken those claims on faith. The whole field runs on claims like these, and the only thing that separates a real improvement from a confident story is a number on a held-out set. You cannot improve, compare, or even trust any of it without honest measurement — and, as the decontam toggle just showed, measurement itself can be quietly broken by the data. That is the subject of lesson 14 — Evaluation: measuring what you can't see.

Takeaway

Scaling laws (lesson 12) tell you how many tokens — D* ≈ 20·N — but assume they're good; on raw web text the same compute buys a far worse model, so data is the hidden multiplier on the entire 6ND budget. Quality is manufactured by a pipeline, not found: source → extract → filter → dedup → decontaminate → mix. You start from petabytes of Common Crawl WARC and extract clean text out of HTML boilerplate; a cascade of filters (language ID, Gopher heuristics, a good-vs-random or educational-value classifier, perplexity) keeps only a few percent — the funnel. Deduplication (exact, then fuzzy MinHash/LSH) matters because a duplicate is a stealth extra epoch that wastes compute, skews the distribution, and drives verbatim memorization. Decontamination cuts the eval set out so your scores measure skill, not a memorized answer key (lesson 14). The mixture is high-leverage — up-weight small high-quality pools (DoReMi), order with curriculum, and especially anneal toward the best data as the LR decays — and when unique tokens run short, repeating up to ≈4 epochs is nearly free. The tokenizer is then trained on this final mix, closing the loop to lesson 1. The under-glamorized truth: data work dominates quality, which is why it's the moat (and CS336 Assignment 4). But every step here only claims to help — the next lesson is how you actually know.

Interview prompts

Why is compute-optimal D* from scaling laws not enough? (§intro — it's a quantity target fit on clean data; the same FLOPs on raw web give a worse model, because the loss floor you reach is set by data quality, which the law assumes away.)
Walk the data pipeline in order and say what each stage removes. (§1–6 — source (CC WARC/code/books) → extract (boilerplate removal) → filter (lang ID, Gopher rules, quality classifier, perplexity) → dedup (exact + fuzzy) → decontaminate (eval overlap) → mix/weight/anneal; raw → a few % survive.)
Why does deduplication lower loss for a fixed token budget? (§4 — a passage seen k times is k unrequested epochs on it: wastes compute, biases the distribution, and drives verbatim memorization; deduped corpora measurably reduce held-out loss.)
Explain MinHash + LSH for fuzzy dedup. (§4 — shingle docs into n-grams; MinHash signatures whose collision probability equals Jaccard similarity estimate overlap cheaply; LSH bands signatures so only likely-similar docs share a bucket, avoiding O(docs²); verify above a Jaccard threshold ≈0.8.)
What is contamination and why is decontamination a data-side fix? (§5 — eval benchmarks live on the open web and leak into Common Crawl; train on them and the model memorizes the answer key, inflating scores without skill; you remove training docs overlapping eval items by n-gram match before training — lesson 14 measures the leak.)
How do you set domain weights, and what is DoReMi? (§6 — don't weight by raw size (it drowns small high-quality pools); DoReMi trains a small proxy with group-DRO to find weights minimizing worst-case domain loss, then trains the big model with them; quality annealing shifts the mix to the best data during final LR decay.)
If unique data runs short of D*, what do you do? (§6 — repeat the high-quality pools; data-constrained scaling shows ≈4 epochs is nearly as good as fresh tokens, with sharp diminishing returns beyond that, so dedup hard then allow ≤4 passes rather than undershoot D.)
What is the data wall, and how does synthetic data address it? (§7 — usable human web text is finite (≈10–20T tokens), while compute-optimal D* ≈ 20·N for the largest models approaches that ceiling. Responses: repeat (≤4 epochs, §6) and synthetic data — distillation-for-pretraining, self-instruct, textbook-style (Phi), rephrasing web text. In budget terms it spends compute to buy tokens you can't otherwise get. Risks: model collapse / distribution narrowing, factual drift, and smuggled-in eval contamination (lesson 14).)
Why must the tokenizer be trained on the final mix? (§8 — BPE merges and compression depend on the corpus; fitting on the wrong distribution (English web while training on code-heavy data) tanks bytes/token, shrinks effective D, and breaks the planned sequence length — callback to lesson 1.)