Part IV — Pretraining

The loss — cross-entropy & perplexity

Lesson 08 assembled the full GPT and ran the generation loop — but with random weights it emits gibberish, and it has no way of knowing so. It ranks every token, yet nothing tells it that its ranking is wrong. This lesson defines the single scalar that measures wrongness — cross-entropy, the negative log-probability of the true next token — and its readable twin, perplexity. That one number is the entire target of pretraining; everything after this lesson exists to drive it down.

1 · The target is one-hot; the model outputs a distribution

At each position the situation is a small classification problem with V classes. The model reads the tokens so far and produces a length-V distribution p — its guess for the next token. The truth is a single token id, the one that actually appears next in the text (this is the free label from lesson 03: the next token is the answer). We can write that truth as a one-hot vector y — all zeros except a 1 in the slot of the correct token.

So we have two distributions over the same V slots: the model's spread-out p, and the truth's razor-thin spike y. A good model puts its mass where the spike is. The natural way to score "how well does p agree with y" is cross-entropy:

Because y is one-hot, every term drops out except the one where y_i = 1 — the correct token. The whole sum collapses to a single, memorable quantity:

That is the entire loss for one position. It only ever looks at the probability the model assigned to the token that actually came next — the other 50,256 numbers are irrelevant to the score (they matter only indirectly, because probabilities sum to 1, so putting mass on the right token means taking it away from the wrong ones). If the model is certain and right — p(correct) = 1 — then −log 1 = 0, a perfect loss. If it is certain and wrong — p(correct) → 0 — then −log p → +∞. The loss lives on [0, ∞) and it is minimized precisely when the model is confident about the truth.

2 · Why the logarithm — from a product to a sum

The choice of −log is not cosmetic. Consider the whole point of a language model, the autoregressive factorization from lesson 00: the probability the model assigns to an entire sequence is the product of its per-token conditional probabilities.

"Train the model" means "make the true corpus as likely as possible under the model" — this is maximum likelihood. But a product of thousands of numbers each below 1 underflows to zero in floating point almost immediately, and products are awkward to differentiate. Take the logarithm and the product becomes a sum, which is numerically stable and has clean, additive gradients:

Maximizing that sum is the same as maximizing the product (log is monotonically increasing), and by convention we minimize rather than maximize, so we flip the sign. Divide by the number of tokens to get a per-token average, and you have arrived — exactly — at mean cross-entropy:

These two ideas — "cross-entropy" and "negative log-likelihood" — are the same number seen from two directions. Cross-entropy comes from information theory (bits to encode the truth under the model's code); negative log-likelihood comes from statistics (how improbable the data is under the model). In practice the terms are used interchangeably, and PyTorch's cross_entropy computes exactly this quantity in one call. Concretely it fuses three steps you could do by hand — softmax the logits into probabilities, pick out the probability of each target token, then take the negative mean of the logs — into a single, numerically-stable operation.

3 · Perplexity — the loss made readable

A loss of 10.79 is correct but unintuitive. Perplexity fixes that by undoing the logarithm: exponentiate the loss.

The payoff is an interpretation in the model's own currency — tokens. Perplexity is the effective branching factor: roughly how many equally-likely options the model feels it is choosing among at each step. A perplexity of 20 means the model is about as uncertain as if it had to guess uniformly among 20 tokens. Lower is better; a perfect model has perplexity 1 (no uncertainty — it always knows the next token).

The clean anchor is a model that has learned nothing. If it spreads its mass uniformly over all V tokens, then p(correct) = 1/V, so the loss is −log(1/V) = log V and the perplexity is e^{log V} = V. Every value below V is progress away from pure ignorance.

4 · The batch loss — shapes and the one scalar

Training does not happen one position at a time; it happens over a batch of sequences. The LM head produces logits of shape (B, T, V) — for each of B sequences and each of T positions, a length-V score vector. The targets are the inputs shifted one step left (lesson 03), so they have shape (B, T) — one correct token id per position. The loss is the mean cross-entropy over every one of the B·T positions:

# logits : (B, T, V)   raw scores from the LM head
# targets: (B, T)      the true next-token id at each position

logits_flat  = logits.reshape(B*T, V)   # one row per position
targets_flat = targets.reshape(B*T)     # one correct id per position

loss = cross_entropy(logits_flat, targets_flat)   # a single scalar
#   = mean over all B*T positions of  -log softmax(logits)[correct]

Read the flattening as the point: cross-entropy does not care about the sequence structure, only about a bag of (distribution, correct-id) pairs, so we collapse the batch and time axes together into B·T independent classification problems and average their losses. The result is one number. That scalar is the entire training objective. Backpropagation (lesson 10) will push gradients from this single value back into every weight in the network — the embeddings, all L attention blocks, the MLPs, the LM head — nudging each one to make the true next token a little more probable next time.

5 · Watch the log punish overconfidence

The widget below is a single next-token distribution over an eight-token vocabulary, with one token marked correct (green outline). The slider concentrates or spreads the probability mass: drag it right and the distribution sharpens toward the token you point it at; drag it left and it flattens toward uniform. Point the mass at the correct token and watch the loss dive toward 0 and perplexity toward 1. Point it confidently at a wrong token and the loss explodes — that is the log's steep wall near p(correct) = 0. The dashed lines mark the random-guess baseline (loss = ln V, ppl = V); anything below them is a model that has learned something. The small inset plots the loss curve −log p so you can see the shape of the punishment directly.

Failure modes & checklist

Checkpoint

Where this points next

You now have the one number that defines success — and a way to read it as an effective branching factor. But a loss computed on a single batch is just a snapshot; it tells you how wrong the model is right now, not how to make it less wrong. To actually learn, we have to compute this loss batch after batch, take its gradient, step every parameter a little downhill, and repeat — thousands of times, across the whole corpus, while watching a validation split to catch the model memorizing instead of generalizing. And we must do all of that without the optimization exploding. That machinery — the optimizer (AdamW), the epochs, the learning rate as the master knob, and the train/validation loss curves — is lesson 10, The training loop — AdamW, batches, curves.

Interview prompts

• Write the cross-entropy loss for one next-token prediction and explain why it collapses to a single term. (§1 — H(y,p) = −Σ y_i log p_i; the target y is one-hot, so all terms vanish except the correct token, leaving −log p(correct).)
• Why do we take the logarithm of the probabilities at all? (§2 — the sequence probability is a product of per-token terms; log turns it into a numerically-stable sum with additive gradients, and mean CE equals the corpus's negative log-likelihood.)
• What is the relationship between cross-entropy and maximum likelihood? (§2 — minimizing mean cross-entropy is exactly maximizing the log-likelihood of the true corpus; they are the same objective up to sign and averaging.)
• Define perplexity and give the random-guess baseline. (§3 — perplexity = e^loss, the effective branching factor; a uniform model over V tokens has p(correct)=1/V, so loss = ln V and ppl = V — ≈ 50,257 for GPT-2.)
• An untrained GPT-2 shows a loss near 10.8. Bug or expected? (§3 — expected: ln 50{,}257 ≈ 10.82, the loss of a model that spreads mass uniformly over the vocabulary.)
• What are the shapes of the logits and targets, and how do they combine into the loss? (§4 — logits (B,T,V), targets (B,T); flatten to (B·T,V) and (B·T) and take the mean cross-entropy over all B·T positions → one scalar.)
• Why pass logits rather than probabilities to a cross-entropy function? (§4/failure modes — the op softmaxes internally for numerical stability; pre-softmaxing applies it twice and corrupts both the value and the gradient.)

Companion reads: cs336 · 02 The Transformer frames this same loss as the negative log-likelihood objective; cs336 · 14 Evaluation puts perplexity alongside downstream benchmarks and its limits.