Part VI — Synthesis

The whole machine — and what scale changes

Lesson 14 closed the toolkit: LoRA gave us a way to adapt the model for pennies. But every piece we built — the tokenizer, the embeddings, attention, the block, the loss, the loop, the fine-tuning heads — was built at toy scale, on one short story, in a 124M-parameter model. This finale does two things. First it replays the entire chain, from a single raw character to a chat assistant, with every box labeled by the lesson that built it. Then it names exactly what changes when you scale that toy toward a frontier model — and points you at the two sibling tracks that live there.

What the last step left broken

Nothing in the architecture is broken — that is the point. Across fourteen lessons we assembled a machine that genuinely works: it tokenizes text, predicts the next token, trains, generates, classifies, and follows instructions. The only thing left unaddressed is size. A real assistant is a thousand times larger, trained on a trillion times more text, on hardware and with algorithms we never touched. The last wall is not a missing component; it is the gap between a model you can run in a notebook and one that took a data center a month. This lesson names that gap precisely so you know it is a matter of degree, not of kind.

Linear position

Forced by: we built and trained every piece at toy scale (124M params, one short story). The natural question a working toy raises is: what actually changes on the way to a frontier model — and is any of it a new idea, or just more of the same?
New idea: replay the whole chain end to end as one pipeline, then separate the fixed part (the architecture and its one function) from the scaled part (data × compute × systems). The scaled part obeys a simple law — compute C ≈ 6ND — and everything expensive about a frontier model is a consequence of pushing N and D up that curve.
Forces next: nothing here — this is the last link. The chain now points outward: the scale, systems, and preference-optimization material lives in the sibling tracks (CS336 and Mini GPT), and this lesson's job is to hand you off to them, then back to the series map.

The plan

Three moves. (1) Replay the whole machine: one pipeline, sixteen boxes, from a raw character to a fine-tuned assistant, each box the lesson that built it. (2) Name what scale changes and where it lives — the compute law C ≈ 6ND, the systems that make large N and D tractable, the modern architecture swaps, and preference optimization beyond SFT — with a one-line cross-reference for each. (3) State what you can now do: read any choice in a frontier model and name the constraint that forced it. Then turn the scale dial yourself and watch the cost of the fixed machine explode.

1 · The whole machine, replayed

Here is everything, in order, as a single forward path. Start with a character; end with an assistant that answers a question. Read each line as "the tensor coming in, transformed into the tensor going out," and note that every transform is one you built and understood — nothing is a black box anymore.

Char → assistant, box by box

Text becomes numbers. A raw string of characters is compressed by byte-pair encoding into a sequence of integer ids, never out-of-vocabulary, with a fixed vocabulary V = 50,257 (01). Each id indexes a row of a learnable embedding table, and a position vector is added so order is not lost, giving a tensor of shape (T, d) (02). The training target is free: slide a window, and the next token is the label (03).

Attention, in three forced steps. Each token builds a context vector by pulling from the others — first as a bare softmax over dot-product similarity, no parameters (04); then with three learned projections W_q, W_k, W_v and the derived 1/√d_k scale that keeps the softmax alive (05); then with a causal mask so a token cannot read the future it must predict, run in parallel across h heads for h views at fixed cost (06).

The GPT. Wrap attention in a block: a feed-forward MLP (d→4d→d) does the per-position work, LayerNorm and residual shortcuts make the block deep-stackable (07). Stack L of them, add a final norm and an LM head (d→V), and run the autoregressive loop for real — logits → pick → append → repeat (08).

Pretraining. A random model has no notion of wrong; cross-entropy = −log p(true next token) is the single number to minimize, with perplexity as its readable twin (09). Lower it over a corpus with the AdamW loop, watching for divergence and overfitting (10). Control the sampling with temperature and top-k, and — because pretraining from scratch costs millions — load OpenAI's published GPT-2 weights straight into our matching architecture (11).

Fine-tuning. Swap the LM head for a two-class head reading the last token to get a spam classifier (12); or wrap examples in a prompt template and mask the loss to the response to turn the completer into an instruction-following assistant (13); or, instead of updating all 124M parameters, learn a thin low-rank adapter ΔW = A·B per task (14). The output is a model that answers.

Sixteen lessons, one unbroken argument. The diagram in the widget below draws exactly this pipeline as clickable boxes, and every arrow between them is a wall that the next box was forced to knock down. If you can narrate that path — character in, assistant out, and why each box has to be there — you understand a GPT.

2 · What scale changes, and where it lives

Now put our 124M toy next to GPT-3's 175B. Almost nothing about the architecture differs: GPT-3 is the same stack of pre-norm transformer blocks, the same attention, the same LM head, the same next-token objective. What differs is four things, and each has a home in a sibling track. This is the key idea of the finale: scale is not a new mechanism, it is the same machine pushed along a curve.

2.1 · More data, and the compute law C ≈ 6ND

Training compute is astonishingly predictable. A forward pass through a model with N parameters costs about 2N floating-point operations per token (each parameter is one multiply-add, and a multiply-add is two flops); the backward pass costs about twice the forward. So training on D tokens costs

C ≈ 6 · N · D FLOPs.

That single relation is the spine of the whole scaling story, and it forces a question: given a fixed compute budget C, how should you split it between a bigger model (N) and more data (D)? The Chinchilla result answers it: for compute-optimal training, scale N and D together, roughly 20 tokens per parameter. Our 124M toy would want ~2.5 billion tokens; a 175B model wants trillions. The dial in the widget makes this concrete — pick a model size and watch D ≈ 20N and C ≈ 6ND climb.

A caveat you should carry

GPT-3 itself predates Chinchilla and was not compute-optimal — it trained a 175B model on only ~300 billion tokens, far under the ~3.5-trillion-token Chinchilla ideal. That mismatch is exactly why the finding mattered: the same compute, re-split toward data, buys a stronger model. The widget shows the Chinchilla-optimal D so you see the target, not the historical accident.

This is the entire subject of CS336 · 12 Scaling laws, and the compute-budget framing runs through that whole track — its 05 Accounting lesson derives the 6ND figure carefully.

2.2 · The systems that make large N and D tractable

Push C to 10^{23} FLOPs and you cannot run it in a notebook. Three systems problems appear, none of which changes what the model computes — only how fast, and on what.

Hardware & kernels

The math runs on GPUs, whose throughput is bounded by memory bandwidth as much as by arithmetic. The T×T attention score matrix we built naively is quadratic in sequence length and never fits in fast memory at scale; FlashAttention computes it in tiles without ever materializing it. → CS336 · 06 GPUs, 07 Kernels.

Parallelism

One model no longer fits on one device. Split the batch across GPUs (data parallel), split each layer's matrices across GPUs (tensor/model parallel), or split the stack of layers across GPUs (pipeline parallel) — and combine all three. → CS336 · 08 Data parallel, 09 Model parallel.

Running the job

A month-long run on thousands of GPUs needs checkpointing, failure recovery, a data pipeline that streams trillions of tokens, and mixed precision to fit. The engineering is the training. → CS336 · 11 Running the job, 13 Data.

2.3 · Modern architecture swaps

A 2020-era GPT-2 and a current open model differ in a handful of well-motivated component swaps — each a small, local upgrade to a box you already built, not a redesign:

Our box (lesson)	Frontier swap	Why
LayerNorm (07)	RMSNorm	Drops the mean-centering and bias — cheaper, and empirically just as stable.
Learned position (02)	RoPE	Rotary embeddings encode relative position inside attention, and extrapolate to longer contexts.
GELU MLP (07)	SwiGLU	A gated activation that is a little more expressive per parameter.
Multi-head (06)	GQA	Grouped-query attention shares keys/values across heads, shrinking the memory read at inference.

Every one of these is a drop-in replacement for a component whose purpose you now understand, which is exactly why they are easy to read. They are collected in CS336 · 03 The modern recipe.

2.4 · Preference optimization, beyond SFT

Our instruction fine-tuning (13) taught the model the shape of a helpful answer by imitation. Frontier assistants go one step further: they are tuned against human (or model) preferences — RLHF optimizes a reward model with reinforcement learning, while DPO and GRPO reach a similar place more directly from preference pairs. This is what makes a model not just fluent but helpful, harmless, and honest. → CS336 · 17 Preferences, and Mini GPT's hands-on tour: 05 DPO and 06 RLVR. Serving that model to users at low latency — KV-caching, batching, quantization — and stretching its context window are their own subjects: CS336 · 18 Inference and 15 Long context.

3 · What you can now do

The payoff of building the machine from first principles is a specific, transferable skill: you can look at any design decision in a modern LLM and name the constraint that forced it. That is the whole point of reading the series as a chain of walls rather than a bag of tricks.

"Why RMSNorm not LayerNorm?"

Because at scale, shaving the mean-subtraction and bias off every normalization saves real compute for no measured loss in stability — a systems constraint acting on the box from lesson 07.

"Why does context length cost so much?"

Because the attention score matrix from lesson 06 is T×T — quadratic in sequence length — so doubling context quadruples that term, which is why FlashAttention and long-context tricks exist.

"Why train on more data instead of a bigger model?"

Because C ≈ 6ND is fixed by your budget, and Chinchilla says the optimal split is ~20 tokens per parameter — often more data, not more parameters, is where the marginal FLOP should go.

And underneath every one of those answers is the same sentence we opened the series with, fourteen lessons ago and still true at 175 billion parameters: an LLM is one function — it maps a sequence of tokens to a distribution over the next token. The tokenizer, attention, the block, the loss, the loop, the fine-tuning, and every frontier-scale elaboration on top exist for no other reason than to make that one function accurate, trainable at depth, and steerable. The function never changed. Only the scale did.

4 · The whole chain, and the scale dial

The widget has two halves. The top is the machine you built: the full pipeline as boxes, grouped by part, each labeled with its lesson. Click any box to read the one-line recap of what it does and which wall forced it. The bottom is the scale dial: step it through the real GPT-2 sizes and on to GPT-3, and watch the four numbers that define a training run — parameters N, the Chinchilla-optimal data D ≈ 20N, the training compute C ≈ 6ND, and a rough dollar cost. The architecture at the top does not change as you turn the dial. That is the entire thesis of the lesson made mechanical: the machine is fixed; scale is data × compute × systems.

The whole chain + scale dial

Proves the finale's claim: the pipeline (top) is identical from 124M to 175B — only N, D ≈ 20N, and C ≈ 6ND (bottom) move. Click a box for its recap; drag the dial and watch the compute and cost of the same machine explode.

Model scale: GPT-2 small (124M)

N · parameters

124M

D · tokens (≈20N)

2.5B

C · train FLOPs (6ND)

1.8e18

rough cost

<$1k

Click any box above to see what it does and which wall forced it. Drag the scale dial to watch N, D, and C move while the machine stays fixed.

Where to go next

This is the end of the from-scratch build. Everything past this point — training that machine at frontier scale, and shaping it into a genuinely helpful assistant — lives in the two sibling tracks. Here is the map, keyed to the questions this lesson raised.

Scale laws & the compute budgetHow to split a fixed C ≈ 6ND between model and data. CS336 · 12 Scaling laws · 05 Accounting.
GPUs, kernels & FlashAttentionThe T×T memory wall and how the hardware really runs. CS336 · 06 GPUs · 07 Kernels.
Parallelism & running the jobWhen one model no longer fits on one device. CS336 · 08 Data parallel · 09 Model parallel · 11 Running the job.
The modern architecture recipeRMSNorm, RoPE, SwiGLU, GQA — the swaps on the boxes you built. CS336 · 03 The modern recipe.
Preference optimization & post-trainingRLHF, DPO, GRPO — beyond imitation. CS336 · 17 Preferences · Mini GPT (a full hands-on post-training tour on a model whose architecture you just built): 05 DPO · 06 RLVR.
Long context & servingStretching the window and serving fast. CS336 · 15 Long context · 18 Inference.

Takeaway

You have built the whole machine: character → BPE ids (01) → embeddings + position (02) → sliding-window targets (03) → attention in three forced steps (04–06) → the transformer block (07) → a stacked GPT with an LM head and a generation loop (08) → cross-entropy (09) → the AdamW loop (10) → sampling and loaded GPT-2 weights (11) → a spam classifier (12), an instruction-following assistant (13), and a LoRA adapter (14). Scaling that machine to a frontier model changes four things and no more — more data governed by C ≈ 6ND and Chinchilla's ~20 tokens/param; the systems (GPUs, kernels, parallelism) that make large N and D tractable; a handful of architecture swaps (RMSNorm, RoPE, SwiGLU, GQA); and preference optimization (RLHF/DPO/GRPO) beyond SFT. None of them is a new kind of thing. The one function — a sequence of tokens in, a next-token distribution out — never changed, and now you can point at any part of any LLM and name the wall that forced it.

Interview prompts

Narrate the full path from a raw character to an instruction-following assistant, naming each stage. (§1 — BPE ids → embeddings+position → sliding-window targets → attention (no-weights → Q/K/V → causal multi-head) → block → stacked GPT + head + generate → cross-entropy → AdamW loop → sampling + loaded weights → classification / instruction / LoRA.)
State the training-compute law and derive its constant. (§2.1 — C ≈ 6ND; a forward pass is ~2N FLOPs/token (multiply-add = 2 flops per parameter), backward is ~2× forward, so forward+backward ≈ 6N per token times D tokens.)
What does Chinchilla say, and why did it matter for GPT-3? (§2.1 — compute-optimal training scales N and D together at ~20 tokens/param; GPT-3 (175B on ~300B tokens) was far under-trained on data, so re-splitting the same compute toward D buys a stronger model.)
Which parts of the architecture change from GPT-2 to a modern frontier model? (§2.3 — LayerNorm→RMSNorm, learned position→RoPE, GELU MLP→SwiGLU, multi-head→GQA; all local swaps to existing boxes, not a redesign.)
Why is long context expensive, in terms of a box you built? (§2.2/§3 — the attention score matrix is T×T, quadratic in sequence length; doubling context quadruples that term, which is why FlashAttention and long-context methods exist.)
How does a frontier assistant go beyond the instruction fine-tuning of lesson 13? (§2.4 — preference optimization: RLHF trains against a reward model with RL, while DPO/GRPO reach a similar objective directly from preference pairs, aligning the model to human preferences rather than pure imitation.)
What is the one sentence that stays true from lesson 00 to a 175B model? (§3 — an LLM is one function mapping a token sequence to a next-token distribution; every component, at any scale, exists to make it accurate, trainable at depth, and steerable.)

Companion tracks: CS336 — Language Modeling from Scratch takes this same machine to frontier scale (GPUs, kernels, parallelism, scaling laws, serving); Mini GPT is the compact post-training tour (SFT, CoT, DPO, RLVR) on a model whose architecture you built here.