Part VI — Synthesis

The whole budget — bytes → served assistant

Nineteen lessons each repaired the wall the one before it hit. You built a tokenizer, a Transformer, a modern recipe, an optimizer, a memory ledger, a kernel stack, a parallelism mesh, a way to run the job for weeks, a scaling law, a data pipeline, an eval harness, a long-context extension, an alignment pipeline, and an inference server. Every one of them was right in isolation — and that is the trap. The point of the course is that they are not nineteen choices; they are one decision chain, and each link is forced by the budget that flows through the previous one. This lesson walks a single concrete budget end to end, from raw bytes to a served assistant, and shows every earlier decision as the consequence it always was.

The previous step left this broken

Lesson 18 closed the build: you can now serve a trained model cheaply with paged KV, GQA, continuous batching, speculative decoding, and quantization. Nothing downstream is broken anymore — the model trains, aligns, and serves. What is still "broken" is your mental model: you have learned each stage as a self-contained module, with its own knobs and its own failure modes, and that is not how a frontier run is reasoned about. A real lab does not optimize the tokenizer, then the optimizer, then the data, independently. It picks one budget and propagates it through the whole chain, because every knob downstream is pinned by the choices upstream. Until you can run that propagation in your head, you can recite the stack but not spend it.

Linear position

Forced by: the nineteen stages only matter as a connected decision chain under a fixed budget — learned in isolation they are trivia, not engineering.
New idea: walk one concrete N-GPU, T-day budget end to end and show every earlier decision as a forced consequence of the one before it; the meta-lesson is that the frontier is a sequence of efficiency choices, and "from scratch" means you can now reason about each link from the budget alone.
Forces next: nothing inside this track — the chain is complete. What it forces is outward: the parts this course deliberately did not cover (multimodality, agents, deeper RL, deeper serving, compression) and the parts nobody has solved (data, evaluation, alignment).

The plan

Four moves. (1) Walk the whole chain as one lesson-flow, each link tagged with the lesson that owns it and shown as forced by the link above. (2) Restate the three motifs that recurred through all nineteen lessons — the 6ND budget, memory-bound vs compute-bound, params vs active-FLOPs. (3) Map what this course did not cover and point to where in the library it lives. (4) Be honest about the frontiers nobody has closed. Then drive the capstone widget: one GPU-hour budget flowing left→right through scaling, data, training, alignment, and serving, with the final readout being the biggest sensible model you can train and the rate you can serve it at.

1 · The chain, start to finish

Begin with the only thing you are actually handed: a cluster and a calendar. Say 128 H100s for 5 days. At ≈990 bf16 TFLOP/s peak per GPU and a realistic ≈45% MFU (lesson 06, 05), that is a budget of

C ≈ 128 × (990×10¹²) × 0.45 × (5 × 86,400 s) ≈ 2.5 × 10²² FLOPs

That single number C is the root of the tree. Everything below is the answer to "how do I spend C to buy the most capable served model?" Read the chain top to bottom; each link names the lesson that owns it, and each is forced by the one above.

0 · the budget128 GPUs × 5 days × peak × MFU ⇒ C ≈ 2.5×10²² FLOPs. The currency is GPU-hours; every choice below spends this.owns: 00 orientation · 05 accounting

1 · how big, how longForced by: one budget, no do-overs. Chinchilla on C=6ND says ≈20 tok/param — so ≈14B params on ≈280B tokens. But you will serve it, so over-train: shift to a smaller N on more D to cut the per-token 2N you pay forever.owns: 12 scaling laws · 18 inference (serving correction)

2 · that many quality tokensForced by: D* is a count, not a guarantee. ~280B+ good tokens means a pipeline — source → extract → filter → dedup → decontaminate → mix — because raw CommonCrawl at the same C buys a far worse model.owns: 13 data

3 · what the tokens feedForced by: tokens are integer ids (BPE), and they need a parallel differentiable next-token predictor. Decoder-only Transformer, residual stream, causal attention, weight-tied head.owns: 01 tokenization · 02 transformer

4 · the recipe that survives scaleForced by: the 2019 block is unstable and wasteful at 14B. Swap in pre-norm + RMSNorm, RoPE, SwiGLU, GQA, no-bias — each buys stability or efficiency at ≈zero cost.owns: 03 modern recipe

5 · made to fit on the hardwareForced by: 14B's training state (≈16 bytes/param ⇒ ≈220 GB) overflows one 80 GB GPU. FSDP shards optimizer/grads/params across the DP ranks; TP splits each matmul within a node; PP splits layers across nodes — a 3D mesh on the H100s you measured in 06.owns: 08 data-parallel · 09 model-parallel · 06 GPUs

6 · stepped without divergingForced by: a fitted model still has to descend the loss. AdamW + warmup→cosine, grad-clip 1.0, bf16 compute with fp32 master weights, FlashAttention so the T×T scores never touch HBM.owns: 04 training · 07 kernels

6′ · (optional) parameters without the FLOPsForced by: capability tracks params but you pay FLOPs per active param. If chosen, replace the MLP with E experts + top-k routing — total params grow E×, FLOPs/token stay flat — at the cost of all-to-all comm and load-balancing.owns: 10 MoE

7 · run for weeks without losing the budgetForced by: a fitted, stepping model still has to survive a multi-week wall-clock run on thousands of chips. Async checkpoint/resume, fault tolerance for the node that will die, and a loss-spike playbook — one un-recovered crash burns days of the fixed C.owns: 11 running the job

8 · measured, or you are blindForced by: every swap above claims a gain. Perplexity + downstream benchmarks tracked over steps, with decontamination and prompt-robustness, or you can't compare runs at all.owns: 14 evaluation

9 · stretched to a long contextForced by: the base model was trained at a short window, but real use wants tens of thousands of tokens. Rescale RoPE (PI/NTK/YaRN) and continue-train on long sequences to extend the window before you align — cheaper than pretraining long from scratch.owns: 15 long context

10 · turned from completer to assistantForced by: the base model maximizes p(text); asked a question it may ask more questions. SFT — same loss, masked to response tokens, wrapped in a chat template — teaches the behavior.owns: 16 SFT

11 · pushed past the demonstrationsForced by: SFT can only imitate; it can't say A is better than B. Optimize a preference/reward under a KL anchor — RLHF (reward model + PPO), DPO (closed-form, no RM), or GRPO/RLVR (verifier reward, group baseline).owns: 17 preferences

12 · served cheaply or it's uselessForced by: a trained model is worthless if each token costs too much. Prefill is compute-bound, decode is memory-bound and gated by the KV cache: paged KV + GQA + continuous batching + speculative decoding + INT8/INT4 quant.owns: 18 inference

Read it once more as a single sentence and the whole course collapses into one breath: a fixed compute budget picks how big a model and how many tokens; that demands a quality data pipeline; the tokens feed a Transformer built on the modern recipe; it is fit onto the hardware by a parallelism mesh and stepped by AdamW + FlashAttention, optionally sparsified with MoE; it is driven through a weeks-long run without losing the budget to a crash, measured, stretched to a long context, taught to be an assistant by SFT, sharpened by preference optimization, and finally served cheaply. Not one link is free. Pull on any one and the others move — train a bigger model and serving cost rises, so you over-train a smaller one; choose MoE and memory and comm blow up even though FLOPs don't; loosen the data filter and you need more tokens to hit the same loss. The chain is the model.

2 · The three motifs, restated

Three quantities recurred in every part of the course. They are the load-bearing intuitions; if you keep only three things, keep these.

The 6ND budget

Training compute is C ≈ 6·N·D FLOPs — forward ≈2N/token, backward ≈4N/token (00, derived in 05). It is the root of the tree: scaling laws split it into N and D (12), the data pipeline must supply that D (13), and the runtime is 6ND / (GPUs · peak · MFU) (05). Inference adds 2N per served token, forever — which is why you bend the split away from compute-optimal (18).

Memory-bound vs compute-bound

Arithmetic intensity — FLOPs per byte moved — decides whether the tensor cores or the HBM bandwidth is the bottleneck (06). Big GEMMs are compute-bound and cheap; norms, softmax, and especially decode are memory-bound and starve the chip. It explains kernel fusion and FlashAttention (07), the cost of all-reduce/all-gather (08–09), and why decode — re-loading every weight per token — is the whole serving cost (18).

Params vs active-FLOPs

Capability scales with total parameters; you pay FLOPs only for the active ones. In a dense model they are equal, so the budget caps both. MoE breaks the coupling — total params grow E× while active params (hence FLOPs/token) stay flat (10) — and inference re-uses the same decoupling: a model is big to store but you pay 2N_active per token (18).

3 · What this course did not cover — and where it lives

This was "language modeling from scratch," and it stopped exactly at the edges of that phrase. Four large neighbors were deliberately left out, each a track of its own. Cross-link them, do not re-derive them — that was the rule the whole way through.

MultimodalImages, audio, video — diffusion, VAEs/latent space, DiT, CLIP, the continuous-generation stack. The text Transformer is one modality; the rest live in generative_continuous.

Agents & toolsTool use, planning, multi-step reasoning loops, memory, and orchestration that turn a served model into a system that acts. See agentic_systems.

Deeper RLLesson 17 was the map; the territory — PPO mechanics, value functions, GRPO/RLVR at depth, reward modeling, the reasoning-model recipe — is reinforcement_learning.

Deeper servingLesson 18 sketched the levers; the production engines that implement them — paged attention, continuous batching, prefix caching, scheduling — are vLLM and SGLang.

CompressionQuantization beyond the sketch, plus distillation and the draft models that power speculative decoding (18). The full treatment is distillation.

The compact tourIf you want the whole pretrain → SFT → CoT → DPO → RLVR arc in six short lessons on a toy model, that is the sibling track, gpt_mini — the tour to this course's full build.

4 · The honest frontiers

The chain above is a solved engineering problem — given a budget, a competent team can execute it. What is not solved are the three things that actually separate frontier models, and a from-scratch understanding means knowing exactly where the certainty ends.

Data is the moat. Two labs with the same architecture, the same budget, and the same parallelism produce very different models, and the difference is overwhelmingly the data — its mix, its quality filters, its dedup, its late-stage annealing (13). Yet the best recipes are unpublished, the gains are hard to attribute, and "what makes a token good" has no theory. The most consequential lesson in the course (13) is also the least scientific.
Evaluation is unsolved. You cannot improve what you cannot measure, and we cannot reliably measure capability (14). Benchmarks contaminate, saturate, and are gamed; the moment a benchmark becomes a target it dies (Goodhart). LLM-as-judge inherits the judge's flaws. There is no trusted, contamination-proof, general measure of "how good is this model," which means every reported number — including your own — deserves suspicion.
Alignment is open. RLHF/DPO/GRPO optimize a proxy for what we want under a KL leash (17); they do not give us what we want, and the gap (reward hacking, sycophancy, the proxy diverging from human intent as capability rises) is an active, unresolved research problem. We can make a model an assistant; we cannot yet guarantee it is the assistant we meant.

That is the honest end state of "from scratch": the build is a chain you can now reason about link by link, and the open problems are no longer mysterious — they are located. You know which lesson owns each one, and that is the whole point of having built it yourself.

5 · The capstone: spend a budget end to end

The widget below is the entire course as one dashboard. A compute budget (GPU-count × days) enters on the left and flows through five stages, each a knob you have now earned: scaling (Chinchilla split, biased by how much you'll serve), data (do you have enough quality tokens for D*?), train (does the state fit on the mesh, and does it finish in budget?), align (a small post-training tax), and serve (decode throughput). It reuses the exact formulas from the lessons: C=6ND (05/12), the ≈20 tok/param optimum (12), ≈16 bytes/param of training state (05), and the weight-bandwidth-bound decode estimate (18). The final readout is the payoff of the whole course: the biggest sensible model this budget buys, and the rate you can serve it at. Push the serving slider up and watch the optimal model shrink — over-training is the budget defending itself against the 2N-per-token tax.

End-to-end budget flow — bytes → served assistant

One budget flows left→right through five stages. Each stage's output pins the next. Watch the chain react: more serving demand ⇒ smaller, over-trained model; MoE ⇒ params explode but FLOPs stay flat; too few quality tokens ⇒ the data stage goes red and starves D*; a model too big for the mesh ⇒ the train stage goes red. The final readout is the whole point of the course.

budget

GPUs (H100): 128 days: 5 MFU %: 45

scaling (12,18)

serving load 1× the train tokens MoE total/active: dense

data (13)

raw tokens available (T): 5.0 quality yield %: 8

train (05,08,09)

GPUs / node: 8 checkpointing

align + serve (17,18)

align tax %: 5 weight precision: bf16

budget → split N,D → check tokens → check fit → tax + serve →

Budget C (FLOPs)

—

N* (active params)

—

N total (incl. MoE)

—

D* (train tokens)

—

tokens / param

—

data: have enough?

—

train state / GPU

—

fits the mesh?

—

The payoff

—

Failure modes & checklist

Failure modes

Optimizing stages in isolation. Tuning the tokenizer, then the optimizer, then the data as separate projects, missing that each pins the next. Signal: a "great" data pipeline that produces fewer tokens than D* needs, or a model sized with no thought for serving cost.
Compute-optimal when you'll serve at scale. Training the Chinchilla-optimal giant, then paying its 2N/token forever in production. Signal: training was efficient but the inference bill dwarfs it within weeks.
Trusting a benchmark number. Reporting (or believing) a score without checking contamination, prompt format, and harness. Signal: a leap on a public benchmark that doesn't transfer to held-out or real use.
Treating alignment as solved. Cranking the reward and assuming "more reward = better assistant." Signal: high reward-model scores with sycophantic, hacked, or degenerate outputs (β too small, lesson 17).

Checklist

Start from C. Write the budget in FLOPs first; derive every downstream knob from it, never the reverse.
Price serving before sizing. Estimate lifetime inference tokens and bend the N/D split toward over-training accordingly.
Confirm the tokens exist. Check that the data pipeline's yield ≥ D* before committing the run.
Check the fit on paper. 16 bytes/param ÷ shard degree vs 80 GB, plus activations — before requesting the cluster.
Locate every open problem. Know which lesson owns data, eval, and alignment, and stay suspicious of all three.

Checkpoint

Try it

Open the widget at the defaults (128 GPUs, 5 days, 45% MFU). Read off N*, D*, and tokens/param — confirm the split lands near the Chinchilla optimum, then bent above 20 by the serving slider. Now do three sweeps and predict each before you drag. (1) Push "serving load" from 3× to 50×: which way does N* move, and why does tokens/param climb? (2) Drop "raw tokens available" to 0.5T at 8% yield: the data stage should go red — what is the gap to D*, and what two knobs close it? (3) Switch MoE to ×16 and watch N total versus N* active diverge while C is unchanged — explain in one sentence why FLOPs didn't move. Finally, flip precision to INT4 and read the served tokens/s jump: name the bottleneck that quantization just relieved.

Where this points next

There is no next lesson — this is the end of the chain, and the wall it leaves you at is the edge of the course itself. You can now reason about any single decision in a modern LLM from the budget down, which is exactly what "from scratch" was supposed to buy. The walls beyond this one are other people's tracks: the model that sees and hears (generative_continuous), the model that acts (agentic_systems), the model that reasons via deep RL (reinforcement_learning), and the engines that serve it at scale (vLLM / SGLang). Or go back to lesson 00 and re-read the budget thesis — it should now read like the table of contents it always was.

Takeaway

A modern LLM is not a bag of tricks; it is one decision chain under a fixed compute budget C ≈ 6ND, and every stage you built was the forced answer to the wall the previous one hit. The budget picks how big a model and how many tokens (scaling laws, 12), corrected for the 2N-per-token you'll pay serving it (18); that demands a quality data pipeline to supply D* (13); the tokens are BPE ids (01) feeding a decoder-only Transformer (02) on the modern recipe (03); it is fit onto known hardware (06) by an FSDP+TP+PP mesh (08–09) and stepped by AdamW + bf16 + FlashAttention (04, 07), optionally sparsified with MoE to grow params without FLOPs (10); it is driven through a weeks-long run that survives crashes and loss spikes without burning the budget (11), measured (14), stretched to a long context with rescaled RoPE and continued training (15), taught to be an assistant by SFT (16), sharpened by preference optimization under a KL anchor (17), and finally served cheaply with paged KV, GQA, speculative decoding, and quantization (18). Three motifs carried the whole way: the 6ND budget, memory-bound vs compute-bound, and capability-tracks-params-but-you-pay-active-FLOPs. The course did not cover multimodality, agents, deep RL, deep serving, or compression — but you now know exactly where each lives and which lesson owns the open problems (data, eval, alignment). That located certainty — knowing the chain link by link and knowing precisely where it ends — is what building it from scratch was for.

Interview prompts

You're handed 1,024 H100s for 30 days. Walk the chain from that to a served assistant. (§1 — budget C≈6ND in FLOPs → Chinchilla split for N,D bent toward over-training by serving cost → quality data pipeline for D* → BPE → Transformer + modern recipe → FSDP/TP/PP fit + AdamW/bf16/FlashAttention → optional MoE → eval → SFT → preference optimization → paged-KV/GQA/spec-decode/quant serving.)
Why does the optimal model get smaller as you plan to serve it more? (§1, 12, 18 — Chinchilla minimizes loss per training FLOP; serving adds 2N per token forever, so total lifetime FLOPs = 6ND + 2N·D_inf is minimized by a smaller N trained on more D — over-training.)
State the three recurring motifs and one place each shows up. (§2 — 6ND budget (split by scaling laws); memory-bound vs compute-bound (FlashAttention, decode); params vs active-FLOPs (MoE decoupling, served-token cost).)
MoE grows total params 8× at the same compute budget — why don't training FLOPs grow? (§2, 10 — C≈6ND counts FLOPs over active params; top-k routing keeps active params (hence FLOPs/token) ≈flat while total params grow E×; the cost moves to memory and all-to-all comm, not FLOPs.)
Your data pipeline yields fewer tokens than D* — what are your options? (§3 widget, 13 — loosen filters / add sources to raise yield, repeat data up to ~4 epochs (data-constrained scaling), or shrink N (raise tokens/param less, smaller D* target); each trades quality or capability.)
Which parts of the LLM stack did this course deliberately not cover, and where do they live? (§3 — multimodal → generative_continuous; agents/tools → agentic_systems; deep RL → reinforcement_learning; deep serving → vllm/sglang; compression → distillation.)
Name the three open frontiers and why each is unsolved. (§4 — data (the moat, no theory of token quality, recipes unpublished); evaluation (contamination, saturation, Goodhart, no trusted general measure); alignment (we optimize a proxy under a KL leash, not the true objective — reward hacking, sycophancy).)