Orientation

The one function every LLM computes

Before we build a single piece, here is the whole machine in one sentence — and the promise that every part of it is forced, not arbitrary. This lesson states the one function a language model computes, the three stages that turn that function into ChatGPT, and the chain of walls that the next fifteen lessons climb, each one the answer to the last.

The premise

An LLM looks like magic and is often taught like a bag of tricks — "it has attention, and layer norm, and positional encodings, and…". That list is unmemorable and unmotivating. The cure is to notice that a GPT does exactly one thing, and then to derive every part as the forced consequence of doing that one thing well. Get the seed right and the rest of the series is a single unbroken argument.

Linear position

Forced by: the goal of building a machine that continues text — writes, answers, classifies — the way ChatGPT does.
New idea: model it as one function, p(next token | tokens so far), applied over and over (autoregression). Text generation is just "predict the next token, append it, repeat." Training, architecture, and fine-tuning are all in service of that single conditional distribution.
Forces next: that function eats a sequence of tokens and scores a fixed vocabulary — but we start with raw text, and neural nets consume numbers, not characters. Lesson 01 must turn text into a sequence of integer ids.

The plan

Four moves. (1) State the one function and show that generation is a loop around it. (2) Explain what "large" buys — parameters, data, and behaviors nobody programmed. (3) Lay out the three stages: build the architecture, pretrain a foundation model, fine-tune it into an assistant or classifier. (4) Preview the forced chain — all sixteen lessons as a sequence of walls and the fixes they demand. Then drive the autoregressive loop yourself.

1 · The one function

Strip away every buzzword and a GPT is a function with a very specific type signature. It takes a sequence of tokens — pieces of text — and returns a probability distribution over what the next token should be:

model : (token₁, token₂, …, token_t) ↦ p(token_t+1 | token₁ … token_t)

That output is a vector of length V (the vocabulary size — 50,257 for GPT-2), one probability per possible next token, all summing to 1. The model does not "know" it is writing an essay or answering a question. At every step it only ranks: given everything so far, how likely is each token to come next?

To generate text you wrap that single call in a loop. Feed in a prompt, read off the distribution, pick one token, append it to the input, and call the function again on the longer sequence. Because each new token becomes part of the next input, the model is called autoregressive — it regresses on its own output:

tokens = encode("The cat")
repeat:
    dist   = model(tokens)          # length-V probability vector
    next   = pick(dist)             # argmax, or sample
    tokens = tokens + [next]        # append and go again
    stop if next == end-of-text

This is the entire interface of a language model. Everything else in this series exists to make that p(next | context) accurate (it puts mass on genuinely likely tokens), trainable at depth (we can learn it from data without the training collapsing), and steerable (we can bend it toward answering questions or flagging spam). Hold onto the three words — accurate, trainable, steerable — they name the three halves of the build.

Why "next token" is enough

Predicting the next token sounds too simple to produce reasoning, translation, or code. But to predict the next token well across all of human text, a model has no choice but to internalize grammar, facts, arithmetic, and the shape of an argument — because those are what determine which token actually comes next. The simplicity of the objective is exactly why it scales: the difficulty is smuggled entirely into the data.

2 · What "large" buys

The "large" in large language model is doing real work. Three quantities grow together:

Parameters

The learnable numbers. The model we build has ≈ 124 million; GPT-3 has 175 billion. More parameters = more capacity to store patterns. Nearly all of them live in the stacked transformer blocks (lesson 08).

Data

The text it learns from. GPT-3 saw hundreds of billions of tokens. Because the label is just "the next token," this data needs no human annotation — the text labels itself (lesson 03).

Behaviors

What emerges. A model trained only to predict the next token turns out to translate, summarize, and answer — capabilities nobody programmed. These emerge from scale, not from task-specific code.

The uncomfortable, wonderful fact is that we will not add a "translation module" or a "reasoning module" anywhere in this series. We build one next-token predictor and make it big and well-trained; the abilities come along for free. That is why understanding this one architecture deeply is worth more than memorizing a dozen specialized ones.

3 · The three stages

Turning the one function into a useful assistant happens in three stages, and this series is organized around them. Stage one builds the machine; stage two pretrains it into a general foundation model; stage three fine-tunes that foundation into something that does your task.

Stage 1 · Build

The architecture & the data pipeline

Turn text into numbers, build attention, and assemble the GPT that can run a forward pass. Lessons 01–08. Output: an untrained model with the right shape.

Stage 2 · Pretrain

The foundation model

Define the loss, run the training loop on raw text, and learn (or load) the weights that make next-token prediction accurate. Lessons 09–11. Output: a base model that completes text.

Stage 3 · Fine-tune

Classifier or assistant

Bend the base model to a task: a spam classifier, an instruction-following assistant, and a cheap adapter (LoRA) to do it efficiently. Lessons 12–14. Output: a model that does your job.

Notice the dependency: you cannot fine-tune what you have not pretrained, and you cannot pretrain what you have not built. The stages are a strict order, which is why the lessons are too. Pretraining from scratch is genuinely expensive — GPT-3's pretraining reportedly cost millions of dollars — so in stage two we will pretrain a tiny model for understanding and then load real published weights to skip the bill (lesson 11).

4 · The forced chain

Here is the whole series as one argument. Read the "wall" column top to bottom: each lesson introduces exactly the idea needed to knock down the previous lesson's wall, and in doing so raises the next one. This table is the spine — if you ever lose the thread, come back to it.

#	The wall	The forced fix
01	The function eats tokens; we have raw text and nets eat numbers.	Tokenization → integer ids (BPE).
02	An id is an arbitrary label; magnitude isn't meaning, and order is lost.	Learnable embeddings + positional signal.
03	We have inputs but no labeled targets to train on.	Sliding window — the next token is the label.
04	Predicting token t needs a flexible mix of tokens ≤ t; a fixed average is too rigid.	Attention — weight by relevance.
05	Raw similarity has no knobs and can't be learned.	Trainable Q, K, V + the √d_k scale.
06	A token can see the future it must predict; one "view" is limiting.	Causal mask + multi-head attention.
07	Attention only mixes; it does no per-position work, and depth won't train.	MLP + LayerNorm + residual = the block.
08	One block is shallow and outputs vectors, not text.	Stack L blocks + LM head + generate.
09	A random model has no notion of "wrong."	Cross-entropy loss + perplexity.
10	One loss on one batch isn't training.	The AdamW training loop.
11	Greedy output repeats; pretraining from scratch is unaffordable.	Temperature/top-k + load GPT-2 weights.
12	A base model completes text but won't make a decision.	Classification fine-tuning (swap the head).
13	Classification is one task; we want general instructions.	Instruction fine-tuning (template + mask).
14	Full fine-tuning is expensive and duplicative.	LoRA — low-rank adapters.
15	We built at toy scale; what changes at frontier scale?	Synthesis + handoff to the scale tracks.

5 · See the loop run

The widget below is the whole thesis in miniature. It holds a toy model over an eight-word vocabulary — just a fixed table of "how much does each word like to follow each other word." Press Step to run one forward pass: the model scores every vocabulary token, softmax turns the scores into a distribution (the bars), one token is chosen, and it is appended to the sequence. Press it again and the new last token drives the next distribution. That loop — predict, pick, append — is text generation. The temperature slider previews lesson 11: low temperature always takes the top bar (greedy, repetitive); high temperature flattens the bars so the choice gets adventurous.

What this series ISA first-principles build of a working GPT: attention derived in three steps, a full training loop, and fine-tuning into a real classifier and assistant. You will understand every matrix.

What it is NOTA scale/systems course. GPUs, kernels, parallelism, scaling laws, RLHF, and serving live in the sibling tracks. We link out; we don't re-teach.

What you'll leave withThe ability to look at any component of a modern LLM and name the exact breakage that forced it into existence — the point of linear thinking.

Checkpoint

Try it

Set the temperature to 0.00 and press Generate. The model always takes the top bar, so it falls into a short repeating cycle — the classic failure of greedy decoding. Now set temperature to ≈ 1.20 and reset and generate again: the sequence varies because flatter bars let lower-ranked tokens win. You have just previewed the entire content of lesson 11 (decoding) with the architecture standing in for a black box. Ask yourself: which of the three words — accurate, trainable, steerable — does the temperature knob touch?

Where this points next

The loop above cheated in one place: it assumed the input was already a sequence of vocabulary tokens with a ready-made scoring table. Real text is a string of characters, and our model will be a stack of matrix multiplies that only speak numbers. So the very first wall is translation: how do we turn "The cat sat" into a sequence of integers that the function can consume — reversibly, and with a vocabulary that never runs out of room for a word it has never seen? That is lesson 01, Tokenization — from text to integer ids.

Takeaway

An LLM is one function: p(next token | tokens so far). Text generation is that function in a loop — predict, pick, append, repeat (autoregression). "Large" means many parameters trained on vast self-labeled text, from which unprogrammed abilities emerge. The build has three ordered stages — build the architecture (01–08), pretrain a foundation model (09–11), fine-tune it into a classifier or assistant (12–14) — and every component we add is a forced move to make that one distribution accurate, trainable, and steerable. Keep the forced-chain table close; it is the map for everything that follows.

Interview prompts

What is the exact input/output type of a GPT, in one sentence? (§1 — a sequence of tokens in, a probability distribution over the next token out; length V, summing to 1.)
What does "autoregressive" mean and why does it matter? (§1 — the model's own outputs are fed back as inputs; generation is a loop of predict → append, so each token is conditioned on all previous ones.)
How can "just predict the next token" yield translation or reasoning? (§1 — to predict the next token well across all text, the model must internalize grammar, facts, and argument structure; the difficulty lives in the data, not the objective.)
Why is the training data effectively free of labels? (§2 — the label is the next token, which is already present in the text; this self-supervision is what lets the data scale to the whole internet.)
Name the three stages and their dependency order. (§3 — build → pretrain → fine-tune; each requires the previous, which is why the lessons are strictly ordered.)
Which three properties must the next-token function have, and which half of the build serves each? (§1 — accurate (pretraining), trainable at depth (architecture), steerable (fine-tuning).)
Why do we load pretrained GPT-2 weights instead of pretraining from scratch? (§3/§11 — pretraining a real model costs millions; our architecture matches GPT-2's exactly, so we can drop in published weights for free.)

Companion reads: cs336 · 00 The efficiency game frames the same field around a compute budget; gpt_mini is the compact post-training tour of a model whose architecture we build here.