Part V — Alignment & inference

Preferences & rewards — RLHF, DPO, GRPO

SFT (lesson 16) taught the base model to imitate a demonstrated answer: it copies the response tokens it was shown and stops. But imitation has a ceiling. A demonstrator who writes a 7-out-of-10 answer trains a 7-out-of-10 model — the loss has no vocabulary for "this answer is better than that one," only "reproduce this exact one." This lesson installs the missing vocabulary: a comparison signal. We optimize the model to produce answers people (or a verifier) prefer, while a KL anchor stops it from wandering off into gibberish to game that signal. Three methods — RLHF, DPO, GRPO — are three points on one map.

1 · Why a comparison, not a demonstration

Ask a person to write the ideal answer to a hard prompt and you get something slow, expensive, and noisy — two equally-skilled annotators produce different "ideal" answers, and an SFT loss treats every word of each as gospel. Ask the same person instead to compare two model outputs and say which is better, and three things change at once. The judgment is faster (seconds, not minutes). It is more reliable — people agree far more on "B is better than A" than on "here is the one true answer." And critically, the comparison can rank outputs the annotator could never have produced themselves: a domain expert who cannot write a flawless proof can still reliably tell which of two proofs is cleaner.

So we collect data of the form (x, y_w, y_l) — a prompt x, a winning response y_w, a losing response y_l, both sampled from the current model. The supervision is a single bit: y_w ≻ y_l. Everything in this lesson is a way to turn that bit into a gradient that makes the model produce more y_w-like outputs and fewer y_l-like ones — without forgetting how to write fluent, on-distribution text.

2 · RLHF — reward model, then PPO under a KL anchor

The original recipe (InstructGPT, the basis of ChatGPT) is a three-stage pipeline. SFT (lesson 16) gives the starting policy π_ref. Then:

The reward model is Bradley-Terry. The Bradley-Terry model says the probability that y_w beats y_l is a sigmoid of their reward gap:

Fit r_φ by maximizing the log-likelihood of the observed comparisons — i.e. minimize −log σ(r_φ(x, y_w) − r_φ(x, y_l)). Note this only ever uses reward differences: the reward is identified up to an additive constant, which is fine because the optimizer only cares about gaps. The output is a learned, differentiable proxy for "what humans prefer," dense enough to score any response the policy invents.

Then PPO optimizes the policy against that reward — minus a KL leash. The objective is:

The first term pushes the policy toward high-reward outputs. The second term — the KL anchor — is the entire reason this works. KL(π‖π_ref) measures how far the new policy has wandered from the SFT model; β is the price you charge per unit of wandering. Drop the anchor (β → 0) and the policy is free to chase the reward model anywhere, and it will: the reward model is an imperfect proxy trained on π_ref-like text, so far out of distribution it assigns garbage high scores. The policy discovers those exploits — repetitive sycophantic filler, degenerate token loops, a fixed phrase the RM happens to love — and reward-hacks them. Output reward soars; actual quality and fluency collapse. The KL term makes every step away from fluent, on-distribution text cost reward, so the policy only moves where the gain genuinely outweighs the drift. The sweet spot is an intermediate β: enough leash to improve, enough anchor to stay coherent. (PPO is the optimizer that maximizes this safely with clipped updates and a value network as a baseline; the mechanics live in reinforcement_learning/16 and reinforcement_learning/14.)

3 · DPO — the reward model was never needed

DPO's insight is that the KL-regularized objective in §2 has a closed-form optimal policy, and that closed form lets you skip the reward model and the rollouts entirely. Here is the derivation in three steps.

Step 1 — the optimum is a Gibbs reweighting of π_ref. Maximizing E[r] − β·KL(π‖π_ref) over all distributions π (a constrained optimization solvable with one Lagrange multiplier for normalization) has a unique solution:

This is intuitive: start from the reference, and tilt it by exp(reward/β) — up-weight high-reward responses, down-weight low-reward ones, with β setting how aggressively. Z(x) = Σ_y π_ref(y|x)·exp(r(x,y)/β) is the partition function that renormalizes it back to a distribution. Z(x) is intractable — it sums over all possible responses — which is normally why you can't use this formula directly. DPO's trick is to never need it.

Step 2 — invert it: the reward is a log-ratio. Solve the equation above for r:

Read this carefully: any reward function corresponds to some optimal policy, so we can express the reward of the policy we want purely in terms of that policy's log-probabilities relative to π_ref. The reward model is implicit — it is the policy, viewed through the lens β·log(π/π_ref).

Step 3 — the partition function cancels across a pair. Substitute this reward into the Bradley-Terry loss from §2. Bradley-Terry only ever uses the difference r(x, y_w) − r(x, y_l) — and the β·log Z(x) term is identical for both responses to the same prompt, so it cancels. What's left has no Z, no reward model, no rollouts — just a loss over the policy's own log-probs on the preference pair:

This is a plain pairwise classification loss: push the policy to assign relatively higher probability (vs π_ref) to the winner than to the loser. You compute four forward passes — π_θ and the frozen π_ref, each on y_w and y_l — and step the gradient. No reward model to train, no sampling loop, no value net, no PPO clipping. The KL anchor is built into the loss through the π_ref ratios and the β that scales them — it is the same trade-off as §2, now compiled into a single supervised objective. This is why DPO became the default for offline preference tuning; the line-by-line view is in gpt_mini/05.

4 · GRPO / RLVR — when a program can grade the answer

For math and code, you don't need humans or a learned reward model: a verifier grades the answer exactly and for free. Run the unit tests; check the final numeric answer against the key. This is RLVR — RL from Verifiable Rewards — and the reward is a clean, un-hackable bit: r = 1 if correct, 0 otherwise. Because the signal is honest, you can push hard on it, and the model learns to reason — to spend more tokens deriving the answer — because longer correct chains earn reward. This is the engine behind the reasoning models (DeepSeek-R1 and kin).

GRPO is the optimizer that makes this cheap. Vanilla policy-gradient (REINFORCE) has high variance; PPO tames it with a learned value network that predicts the expected reward — a baseline to subtract. But a value net is a second large model to train and store. GRPO (Group Relative Policy Optimization) deletes it with one idea: for each prompt, sample a group of K responses (say K = 8), grade them all, and use the group's mean reward as the baseline. Each response's advantage is its reward minus the group mean (often divided by the group's standard deviation):

Responses that beat the group average get reinforced; below-average ones get suppressed. The mean is a free, unbiased baseline — no value network needed, removing one of PPO's four models; and because RLVR also swaps the learned reward model for a program verifier, GRPO/RLVR runs on just two (policy + frozen reference), halving PPO's model count and memory. A KL term to π_ref still rides along to keep the policy from collapsing into a single high-reward rut. GRPO is online (it samples each step) and so pays rollout cost, but it buys the unbounded ceiling of verifiable reward — the model can become far better at math than any demonstration it was shown. The full PPO→GRPO derivation and the reasoning-model story are in gpt_mini/06; the RL-side depth is in reinforcement_learning/14.

5 · The map — reward source × optimizer

All three methods optimize a comparison/reward signal under a KL anchor; they differ on where the reward comes from and how you optimize it. That is the whole map.

In practice the modern recipe stacks them: SFT → DPO (cheap broad alignment from preference data) → GRPO/RLVR (push reasoning where a verifier exists). RLHF-with-PPO is still used where the reward is genuinely a fuzzy human judgment and online iteration pays off, but DPO has eaten much of its offline territory and GRPO owns the reasoning frontier. Every one of them is the same sentence: climb a comparison signal, on a KL leash.

Safety and red-teaming

Everything above optimizes one axis: helpfulness / quality — answers people prefer, correctness a verifier confirms. Safety is a distinct axis, and a high preference score does not imply it. Safety means refusing requests that are genuinely harmful, staying robust when a user actively tries to jailbreak the model with adversarial or role-play prompts, and not emitting harmful or biased generations even when the prompt invites them. A model can be maximally helpful on the preference data it saw and still hand over dangerous instructions the moment someone phrases the ask cleverly — because "be helpful" and "be safe" pull in opposite directions exactly on the prompts that matter most.

The machinery is the same comparison-and-reward engine, pointed at this second axis. You collect safety-specific preference data (a refusal preferred over a compliant-but-harmful completion) and fold a safety reward into the RLHF objective alongside the helpfulness reward. The hard prompts to train on come from red-teaming: systematically hunting for inputs that make the model fail — human red-teamers probing by hand, plus automated attackers that generate adversarial prompts at scale — and feeding every discovered failure back as new preference pairs. To collect enough of this signal without an army of labelers, Constitutional AI / RLAIF has the model critique and revise its own outputs against a written set of principles (a "constitution"), turning those AI judgments into preference data — scaling preference collection with far fewer human labels. The KL anchor matters here too: β·KL(π‖π_ref) keeps the policy near a reference whose safety behavior you trust, so chasing reward cannot quietly erode the refusals and guardrails the SFT model already had (this is the "forgetting π_ref" failure in the table above, now read on the safety axis).

Be honest about the limits: safety is an open problem, not a solved checkbox. Jailbreaks keep being found against models that passed extensive red-teaming, automated attacks co-evolve with defenses, and — most fundamentally — aligning a model to preferences is not the same as making it safe: the reward is still a proxy, and a determined adversary optimizes against the gaps you did not anticipate. This track only sketches the shape of the problem; the dedicated RL machinery for reward modeling, PPO, and the safety/alignment pipeline lives in reinforcement_learning.

6 · Drive the KL–reward trade-off

The widget below makes the §2 objective tactile. The slider is β, the KL coefficient — the price of drifting from π_ref. As you lower β, the policy is allowed to chase the reward harder: the achieved reward rises. But the divergence from π_ref rises with it, and past a point the reward you're chasing is the reward model's blind spots, not real quality — so a reward-hacking / fluency-loss indicator climbs and a sample of the policy's output degrades into gibberish. Crank β too high and the opposite failure: the policy is leashed so tightly it never moves off π_ref — zero reward-hacking, but zero improvement either. The usable region is the middle. Push the slider to its far-left, β → 0 end: reward maxed, output collapsed.

The shape of the white curve is the whole lesson in one picture: optimizing a proxy reward without a leash does not maximize the thing you actually want. The KL anchor is not a regularization afterthought — it is the mechanism that keeps "more reward" pointed at "better model" instead of at the reward model's failure modes.

Failure modes & checklist

Checkpoint

Where this points next

The model is now done in every sense this course set out to deliver: a base model trained compute-optimally on quality data (parts I–IV), turned into an instruction-follower by SFT (14), and tuned to human preferences and verifiable correctness here (15). It is also, right now, useless — because a frontier-scale model that takes seconds and dollars to emit each token is a model nobody can deploy. The cost structure of generation is completely different from training: a single forward pass per token, re-reading every weight and a growing KV cache from HBM each step, memory-bandwidth-bound rather than compute-bound. Serving the model you paid to build is its own engineering problem, and it is the last link in the chain: lesson 18, Inference — serving the model you paid for.

Interview prompts

• Why use a comparison signal instead of more SFT demonstrations? (§1 — humans rank faster and more reliably than they write, comparisons can rank outputs the labeler couldn't produce, and a relative signal lets the model exceed its demonstrator — none of which SFT's imitation loss can express.)
• What does the KL term in the RLHF objective actually buy you? (§2 — it prices drift from π_ref; without it the policy reward-hacks the imperfect reward model out of distribution — RM score soars while fluency and true quality collapse. The sweet spot is an intermediate β.)
• Write the Bradley-Terry reward-model loss and explain why it only uses reward differences. (§2 — −log σ(r(x,y_w) − r(x,y_l)); the reward is identified only up to an additive constant, which is irrelevant because the optimizer acts on gaps.)
• Derive DPO: why is there no reward model and no rollouts? (§3 — the KL-regularized optimum is π* ∝ π_ref·exp(r/β); invert to r = β·log(π/π_ref) + β·log Z; substitute into Bradley-Terry, where log Z cancels across the pair, leaving a pairwise classification loss over policy log-ratios.)
• How does GRPO eliminate PPO's value network? (§4 — sample a group of K responses per prompt, grade them, and use the group mean reward as the baseline; advantage = reward minus group mean (over std). The mean is a free unbiased baseline, deleting the critic model.)
• When do you reach for RLVR over RLHF, and what's the catch? (§4–5 — when correctness is verifiable (math, code) the reward is exact, free, and un-hackable, so it can exceed human quality; the catch is a gameable verifier — guard against shortcut exploits and test on held-out problems.)
• Why is DPO offline a weakness, and how do you mitigate it? (§3 — fixed pairs go stale as the policy moves, so it can mis-rank current outputs; re-sample preference pairs from the current policy or use an online variant, at the cost of paying rollout compute again.)