The LLM era (上) — PPO, DPO, GRPO: the map

Lesson 10 ended at PPO's clipped surrogate. Here is the surprise that organizes the next three lessons: training a large language model with RL is PPO, unchanged. The "actions" are tokens, the "policy" is the LLM, and a full generated response is a trajectory. Once you see the mapping, PPO, DPO, and GRPO are just three answers to three different complaints about the same loop.

The hook: nothing new, just renamed

For ten lessons we built one machine. An MDP (lesson 01): states s, actions a, rewards r, a policy π_θ(a|s). A policy gradient (lessons 03, 07): ∇J = 𝔼[A · ∇log π_θ]. An advantage A = Q − V to cut variance (lesson 08). An importance ratio ρ = π_θ/π_old to reuse old rollouts (lesson 09). And a trust region to keep each step safe — TRPO's hard KL constraint, then PPO's cheap clip (lesson 10).

The claim of this lesson is that the entire "RLHF / reasoning-RL" revolution in LLMs reuses exactly this machinery. There is no new optimization theory. What changed is the shape of the MDP: the action set is the vocabulary, the trajectory is a sentence, and the reward arrives once, at the end.

The mapping: MDP-RL → token-RL

Read this table as a dictionary. Every row is something you already know from the left column, renamed for the right.

So a response is a trajectory of length T through a known-dynamics MDP, and the policy-gradient objective from lesson 01 is, verbatim:

and its gradient factorizes over tokens, because log π_θ(y|x) = Σ_t log π_θ(y_t | x, y_<t):

That is lesson 07's policy-gradient theorem, with a per-token advantage A_t. Everything below is about (a) how to keep the step safe and (b) where A_t comes from.

PPO, re-derived as the cheap TRPO — for tokens

Recall the chain from the last two lessons. Importance sampling (lesson 09) lets us reuse a batch of rollouts sampled from π_old for several gradient steps, by reweighting with the per-token ratio

giving the surrogate objective L(θ) = 𝔼[ ρ_t · A_t ]. But lesson 09 also showed the danger: when π_θ drifts from π_old, ρ explodes on rare tokens and the estimate's variance blows up. TRPO (lesson 10) fixed this with a hard constraint, 𝔼[KL(π_old ‖ π_θ)] ≤ δ, solved with a natural gradient and Fisher-vector products — correct, but expensive.

PPO replaces the hard trust region with a first-order trick: clip the ratio so the surrogate stops rewarding drift past [1−ε, 1+ε], and take the pessimistic min:

The min is asymmetric protection. When the update would push the policy further past the trust region in the reward-increasing direction, the clipped branch is flatter, the min selects it, and its gradient w.r.t. θ is zero — the step is capped. In the corrective direction (snapping a rare-token excursion back toward π_old), the unclipped branch wins and the gradient flows. The clip is TRPO's trust region, implemented as one min and one clamp instead of a constrained optimization. The widget below is exactly this function; play with it until the asymmetry is obvious.

The KL-to-reference anchor

The clip keeps π_θ near π_old — the policy from a few gradient steps ago. But in the LLM era there is a second, separate anchor that classical RL did not need: a penalty pulling π_θ toward a frozen reference π_ref, almost always the SFT model we started from.

Why both? They have different jobs. The clip (vs π_old) is a numerical guardrail on the importance ratio — it keeps the off-policy estimate valid step to step. The KL-to-π_ref is a semantic guardrail across the whole run: it stops the policy from wandering far from a fluent, general model just to chase reward — the failure mode called reward hacking, where the policy finds degenerate high-reward text the reward function never meant to endorse. The coefficient β trades these off — and as the widget shows for ε, the wrong setting breaks learning at either extreme. We unpack this anchor's two jobs fully in lesson 13 (RLHF).

Interactive · the PPO clip visualizer

This plots the two branches of L^CLIP as a function of the ratio ρ, for a fixed advantage. The dashed blue line is the unclipped surrogate ρ·A; the solid orange line is the clipped objective min(ρ·A, clip(ρ,1−ε,1+ε)·A) that PPO actually optimizes. Flip the sign of A, and drag ε. Where the orange line goes flat, the gradient is zero — the update is capped.

// The two branches of the PPO objective, for advantage A and clip ε.
function unclipped(rho, A){ return rho * A; }                 // ρ·A
function clipped(rho, A, eps){
  const c = Math.max(1 - eps, Math.min(1 + eps, rho));        // clamp ρ
  return c * A;                                               // clip(ρ)·A
}
// PPO optimizes the pessimistic min of the two.
function ppoObjective(rho, A, eps){
  return Math.min(unclipped(rho, A), clipped(rho, A, eps));
}
// The gradient w.r.t. θ is zero wherever ppoObjective is flat in ρ:
//   A>0 and ρ>1+ε   → clipped branch wins, flat → capped
//   A<0 and ρ<1−ε   → clipped branch wins, flat → capped
// elsewhere the unclipped branch passes the gradient through.

The family: one loop, three complaints

PPO works, and it is still the reference algorithm. But it carries three burdens, and the two most influential LLM-RL methods each delete one of them. Here is the whole map on one page; lessons 12–13 derive the pieces.

The shape of the family is: PPO is the full actor–critic with a trust region (the reunion from lesson 03, scaled). GRPO removes the critic by computing the baseline from a group of samples instead of learning it — pure policy-gradient with a clever Monte-Carlo baseline. DPO removes the actor–critic loop entirely by exploiting a closed-form solution: the optimal KL-regularized policy is a known function of the reward, so the reward can be folded into the policy and fit with a single classification-style loss on preference pairs. Three points on the value↔policy fork that has organized this whole course.

Where this leaves us

We have the map. PPO is TRPO's clip, applied to a token-MDP with terminal reward and a KL anchor to the SFT model. DPO and GRPO are each one deletion from PPO, motivated by one concrete cost. Lesson 12 (下) earns both deletions: it derives DPO's closed form from the KL-regularized optimum and the Bradley–Terry likelihood (watching Z(x) cancel), and derives GRPO's group baseline from the policy-gradient theorem's freedom to subtract any function of the state. Lesson 13 then wires PPO into the full RLHF recipe (SFT → reward model → PPO with the KL anchor).