Knowledge Distillation, From First Principles

A small model can learn more from a big model's answer than from the right answer. Lessons 01–02 build the foundations on language models; lessons 03–06 carry the framework to diffusion models — how to turn a 1000-step image teacher into a 1–4-step student — every claim derived, every trade-off made honest.

Three knobs organize the whole field: what the student matches (the teacher's outputs, its internal features, or the sequences it generates), how the gap is measured (temperature, forward vs reverse KL), and where the training data comes from (a fixed set, the teacher's generations, or the student's own rollouts). A concrete method is one choice per knob — Hinton KD is response × forward-KL × fixed-set; MiniLLM is sequences × reverse-KL × on-policy.

Who this is for

You know cross-entropy, softmax, and what an autoregressive language model is. By the end you can: derive the T² gradient correction in a few lines; explain why reverse KL makes a student commit and forward KL makes it hedge; say why sequence-level KD works through a black-box API but word-level KD does not; and lay out a distill-then-prune-then-quantize deployment stack. The diffusion arc (03–06) additionally assumes the score / PF-ODE picture from the generative series (lessons 16–17) — it picks up where that series' "distillation" row left off.

The one picture

A hard label is a single point; a teacher is a whole distribution. Distillation moves the teacher's function into a smaller student — and the three knobs are the degrees of freedom in how you do it.

Part 1 · Foundations & language models

Distillation foundations

The core idea and dark knowledge; soft targets and temperature, with the T² gradient correction derived from scratch and the high-T logit-matching limit; the four lenses on why it works and the capacity-gap paradox; and the WHAT knob — response, feature, and relational distillation, plus born-again / self-distillation.

Distilling generative models & the frontier

Word-level vs sequence-level KD and the exposure bias both inherit; the HOW knob — forward vs reverse KL, mode-covering vs mode-seeking; the WHERE knob — on-policy distillation, GKD, MiniLLM, and distillation as RL with the teacher as a dense reward; how it composes with quantization and pruning; and the frontier — reasoning traces, black-box style-vs-substance, and weak-to-strong.

Part 2 · Distilling diffusion models (lessons 03–06)

A diffusion teacher isn't a distribution over tokens — it's a multi-step solver that needs ~1000 (or ~25) network calls per image. The goal is a student that needs 1–4. The three knobs survive the move, but each changes meaning: WHAT becomes "match the trajectory / the distribution," HOW falls back to L2-in-sample-space (no density to KL) before re-earning a distributional loss, and WHERE keeps its off-policy/on-policy axis. Built on the generative series' score/PF-ODE foundations.

The setup & progressive distillation

Why a diffusion teacher (a sampler, not a distribution) breaks the lesson-02 picture; the three knobs re-read for diffusion; the smoothness wall as the thing distillation crosses. Progressive distillation: train a student to replace two teacher DDIM steps with one, then promote-and-halve down to 4–8 steps. Why halving (not 1-step-from-scratch), and why v-prediction.

Consistency models

Map every point on a PF-ODE trajectory to its origin, pinned by a boundary condition and an EMA-stabilized "adjacent points must agree" loss — one call from noise to image. Consistency distillation (rides a teacher) vs consistency training (no teacher); why one step still averages and 2–4 (jump, re-noise) rounds win; LCM/LCM-LoRA and CTM.

Distribution matching & adversarial

Why L2-on-a-trajectory blurs (the optimum is the mean of a one-to-many target). Match the output distribution using only the score (a gradient, no density): SDS (teacher score as a force), DMD (s_real − s_fake, the fake-score net as a GAN discriminator in disguise), and ADD/Turbo/LADD (an explicit GAN for realism). The GAN diffusion dethroned, re-hired for speed.

Rectified flow & the decision map

The other lever: straighten the path instead of leaping across its curvature. Rectified flow + reflow re-pair (noise, sample) endpoints until trajectories are straight enough that one Euler step is exact (InstaFlow, and why SD3 chose it). Then the whole field on two axes, a decision table, and the three knobs revisited end-to-end.

Where this connects

Distillation is SFT on a teacher's outputs — so the gpt_mini SFT lesson is the prerequisite in spirit, and on-policy distillation borrows its machinery from the RL track. Few-step diffusion samplers are distilled too (the generative series), and the speculative-decoding draft model is a distilled student (the vLLM / SGLang serving tracks).