Generative models · build one by hand

Build a Large Language Model — From Scratch

A linearized, first-principles build of a GPT: from a single character to a model that writes text, classifies spam, and follows instructions. Every piece — the tokenizer, embeddings, attention, the transformer block, the training loop, fine-tuning — is derived as the forced answer to one question, not asserted because "GPT has it."

This is the gentle, hands-on sibling of the systems-heavy CS336 — Language Modeling from Scratch. CS336's spine is a fixed compute budget spent at frontier scale (GPUs, kernels, parallelism, scaling laws, serving). This track stays at the workbench: we build every matrix by hand, watch the machine generate text, and only at the end name what changes at scale — handing you off to CS336 and the six-lesson Mini GPT post-training tour when you're ready.

The one idea the whole track turns on

An LLM is one function: it eats a sequence of tokens and returns a probability distribution over the next token. That is the entire job. Everything you are about to build is a forced move to make that one function accurate, trainable at depth, and steerable. Read the series as a single derivation: each lesson is a wall the previous one hits, and the next idea is whatever must appear so the chain doesn't break.

Who this is for

You can read Python and you know roughly what a neural network and gradient descent are. You want to understand a GPT well enough to reason about — or rebuild — any single part of it. No prior NLP or systems background assumed; tokenization, attention, and the training loop are all built up from the first character. By the end you should be able to point at any component of a modern LLM and say which breakage forced it into existence.

The spine — one forced chain

Read the diagram left to right, top row then bottom. Each box is a wall; the arrow into the next box is the idea it forces. Nothing is here "because LLMs have it" — remove any link and the chain breaks.

The lessons

The one function every LLM computes

The whole map in one sentence: an LLM turns a token sequence into a next-token distribution, and generation is just "predict → append → repeat." The three stages (build → pretrain → fine-tune) and the wall→fix chain of all 16 lessons. Interactive: run the autoregressive loop and watch a toy model write.

Orientation

Tokenization — from text to integer ids

A model eats integers, not text. Why not words (too many, still miss new ones) or characters (sequences too long) — subwords are the Goldilocks. Byte-pair encoding: greedily merge the most frequent pair to a fixed vocab, so nothing is ever out-of-vocabulary. Interactive: step BPE merges and watch the sequence shrink.

Part I · Text becomes numbers

Embeddings & positions — ids become meaning

An integer id is an arbitrary label; a net would read id-magnitude as meaning. Turn each id into a learnable vector (a lookup table trained by backprop), then add a position signal — because attention is a set operation and "dog bites man" must not equal "man bites dog." Interactive: watch the same token at two positions diverge.

Part I · Text becomes numbers

The training signal — a sliding window

Training needs (input → target) pairs and nobody labeled anything. The trick: the next token is the label. Slide a window of size context over the token stream; each window's shift-by-one is its target. This self-supervision is why LLMs can be trained on the whole internet. Interactive: slide the window, tune the stride, count the free labels.

Part I · Text becomes numbers

Attention, the idea — weighting by relevance

To predict token t the model must combine tokens ≤ t, and a fixed average is too rigid. Let each token build a context vector = a weighted sum of the others, with weights = softmax of dot-product similarity. No parameters yet — just the shape of the idea. Interactive: pick a query, watch scores → weights → blended context.

Part II · Attention

Self-attention with trainable weights (Q, K, V)

Part II · Attention

Causal & multi-head attention

A token predicting the future must not see the future: mask the upper triangle to −∞ before softmax so those weights are exactly zero. Then run h heads in parallel — h similarity "views" for almost free — plus dropout for regularization. Interactive: flip the causal mask and slide the head count on a live attention heatmap.

Part II · Attention

The transformer block — norm, GELU, residuals

Attention only mixes information across positions; it does no per-position transform, and a naive deep stack won't train. Wrap it: a feed-forward MLP (d→4d→d, GELU) does the heavy lifting; LayerNorm + residual shortcuts keep activations and gradients sane at depth. Interactive: strip out norm/residuals and watch the deep stack explode or vanish.

Part III · The GPT

Assembling GPT & generating text

Stack L blocks, cap with a final norm and an LM head back to the vocabulary, and run the autoregressive loop from lesson 00 for real. Count the parameters (N ≈ 12Ld² + Vd ≈ 124M) and see why a fresh model outputs gibberish — the architecture is right, the weights are random. Interactive: dial d, L to watch N move, then generate.

Part III · The GPT

The loss — cross-entropy & perplexity

A random model has no notion of "wrong." Define the one number to minimize: cross-entropy = −log p(true next token), which is maximum likelihood in disguise. Perplexity = e^{loss} is its human-readable twin — an "effective branching factor" that falls from V (random) to single digits. Interactive: drag probability onto the right token and watch loss and perplexity drop.

Part IV · Pretraining

The training loop — AdamW, batches, curves

One number on one batch isn't training. Lower it over a corpus: forward → loss → backprop → AdamW step, over batches and epochs. The learning rate is the master knob (too high diverges, too low crawls), and a tiny corpus will overfit — train loss falls while validation turns up. Interactive: drive a toy run to convergence, divergence, or overfitting.

Part IV · Pretraining

Decoding & loading pretrained weights

Greedy decoding repeats itself; real pretraining costs millions. Control randomness with temperature (sharpen/flatten the distribution) and top-k (cut the garbage tail), then stand on OpenAI's shoulders: our architecture is GPT-2's, so we load its published weights and get a strong base model for free. Interactive: reshape the distribution with temperature and k.

Part IV · Pretraining

Classification fine-tuning — a spam detector

A base model completes text; often you want a decision. Freeze most of it, replace the LM head with a tiny two-class head, and read the last token — the only one that has attended to the whole message. A few epochs turn GPT-2 into a spam classifier. Interactive: swap the head and see why last-token beats first or mean.

Part V · Fine-tuning

Instruction fine-tuning — completer to assistant

A base model has knowledge but not the behavior of answering. Wrap examples in a fixed prompt template, feed it (instruction → response) pairs, and mask the loss to the response so it learns to answer, not to parrot the question. Same loss, same loop as pretraining. Interactive: toggle the response-only loss mask and see parrot vs assistant.

Part V · Fine-tuning

LoRA — parameter-efficient fine-tuning

Full fine-tuning updates all ~124M parameters and stores a whole new model per task — mostly redundant. The change needed to adapt is low-rank: freeze W, learn a thin ΔW = A·B (rank r ≪ d), and train ~50× fewer parameters. One tiny adapter per task. Interactive: slide the rank and watch trainable parameters collapse.

Part V · Fine-tuning

The whole machine — and what scale changes

Replay the entire chain from one character to a chat assistant, each box labeled with its lesson. Then name exactly what changes when you scale our 124M toy toward a frontier model — data and the C ≈ 6ND compute law, GPUs and kernels, parallelism, modern architecture swaps, preference optimization — and where each lives in the library. Interactive: turn the scale dial from 124M to 175B.

Part VI · Synthesis

How to use this

Read it linearly. Each lesson is the answer to the wall the last one hit. Attention (04) only makes sense as the fix for the rigid average of lesson 03; instruction tuning (13) only as the thing classification (12) can't do.
Touch every knob. Each widget has a setting that breaks — attention that freezes without the √d_k scale, a deep stack that explodes without residuals, a run that diverges, a model that overfits. Find it. The failure is the lesson.
Follow the cross-links. Where a sibling track goes deeper — scale, kernels, parallelism, preference optimization — this track links out instead of repeating.

Companion tracks

This is the from-scratch, build-it-by-hand sibling of CS336 — Language Modeling from Scratch (the systems-and-scale course) and Mini GPT (the compact post-training tour). Where this track ends — scale laws, GPUs, kernels, parallelism, RLHF/DPO, long context, serving — those two pick up. The finale (lesson 15) hands you off explicitly.