Rectified flow & the decision map
Lessons 03–05 amortize the integral of a curved teacher ODE. A different idea: make the ODE straight in the first place, so one Euler step is already exact. Then the whole diffusion-distillation field on one map.
The other lever: change the path, not just the solver
Every method so far accepts the teacher’s curved probability-flow trajectory (generative lesson 16) and works hard to leap across its bends — progressive distillation strides across two steps, consistency models jump to the origin, DMD matches the distribution. But the difficulty was the curvature: a straight trajectory needs no leaping, because Euler’s method is exact on a straight line. So instead of distilling a curved path into few steps, re-train the path to be straight.
Rectified flow and the reflow operation
Rectified flow (Liu et al. 2022) defines the generative process as flow matching along the straight-line interpolation between noise and data (generative lesson 05): xt = (1−t)·x0 + t·ε, with the network regressing the constant velocity ε − x0. But individual sample paths still bend, because many straight noise→data lines cross, and the learned velocity at a crossing is the average of the conflicting directions — a curved effective trajectory.
The fix is reflow, and it is a distillation operation. Generate paired data with the current model: sample noise z, run the full sampler to get its image x = \text{sampler}(z). Now re-train a new flow on these (z, x) pairs — but pairing each noise with the specific image it actually produces. Because the endpoints are now coupled, the straight lines between them stop crossing, so the new model’s trajectories are straighter. Repeat (2-reflow, 3-reflow) and the paths approach perfectly straight, at which point one Euler step reproduces the multi-step output. InstaFlow reflows Stable Diffusion to a genuinely one-step generator.
Picture noise points on the left, data points on the right, joined by elastic threads. Flow matching pulls every point along the local average of all threads passing nearby — where threads cross, that average bends, so a sample follows a curved route. Reflow re-pairs each noise point with the exact data point its own route reached, then re-lays the threads. Re-paired threads cross far less, so the new local-average field is nearly straight. Do it again and the threads are parallel — now a single straight snip from any noise point lands on its data point. You distilled by removing the curvature, not by jumping across it.
Reflow is why SD3 (generative lesson 22) chose a rectified-flow objective: it starts from a path that is already much straighter than DDPM’s, so it is cheaper to distill toward few-step, and the v-like velocity target is well-behaved at both endpoints. Straightness is a head start on distillation.
The whole field on two axes
Every diffusion distiller is one choice on each of two axes: what target the student matches, and whether the path is curved or straightened.
| method | WHAT it matches | HOW (loss) | WHERE / path | steps |
|---|---|---|---|---|
| Progressive distillation | teacher’s 2-step output (a trajectory point) | L2 (v-pred) | teacher DDIM, curved; serial halving | 4–8 |
| Consistency (CD/CT) · LCM | the trajectory’s origin x₀ from anywhere | L2 / LPIPS + EMA target | curved PF-ODE; one run | 1–4 |
| CTM | any-t-to-any-s jump | L2 + adversarial option | curved; one run | 1–∞ |
| SDS / VSD | the output distribution (teacher score as force) | score gradient | distribution, no fixed path | opt. loop |
| DMD / DMD2 | the output distribution (sreal − sfake) | KL via score difference (+GAN) | distribution | 1–4 |
| ADD / Turbo / LADD | realism + manifold | GAN + score distillation | distribution | 1–4 |
| Rectified flow / reflow | its own (noise, sample) pairs | flow-matching L2 | straightens the path | 1–4 |
The decision map
| If you… | reach for | why |
|---|---|---|
| want a quick, robust 4–8-step win with minimal risk | progressive distillation | simplest, stable, just L2 against your own teacher |
| need 2–4 steps on a Stable Diffusion checkpoint, portable | LCM / LCM-LoRA | one training run, ships as an adapter (generative lesson 23), guidance folded in |
| want the best 1-step quality and can afford adversarial training | DMD2 or ADD/Turbo | distribution matching + GAN beat L2’s blur ceiling |
| are designing the base model from scratch | rectified flow + reflow (SD3-style) | start straight so few-step is cheap downstream |
| train without any teacher | consistency training (iCT) | builds the few-step sampler from the forward process alone |
| have no logits, only the teacher’s images | reflow / black-box pairs | needs only (noise, generated image) pairs |
Back to the three knobs
The framework from lessons 01–02 survived the whole journey, re-read for a teacher that is a sampler rather than a token distribution:
- WHAT moved from “token distribution” to a ladder: a trajectory point (progressive) → the trajectory’s origin (consistency) → the whole output distribution (DMD/ADD) → its own paired endpoints (reflow). Climbing the ladder buys fewer steps and higher quality at rising training complexity.
- HOW went from KL (which needs a density) to L2 in sample space (no density needed), then earned back a distributional objective through the score (DMD) and a discriminator (ADD) — the GAN that lesson 21 said diffusion had dethroned, re-hired for speed.
- WHERE kept its lesson-02 meaning exactly: off-policy on the teacher’s trajectories (CD, progressive) vs. on the student’s own current samples (CT, DMD’s sfake, reflow). On-policy still costs wall-clock and still buys robustness.
Interactive · curved teacher vs. reflowed straight path
Left state: the teacher’s sample trajectories bend (crossing straight-line couplings averaged into a curve), so a 1-step Euler leap (dashed) misses the data. Hit reflow: the endpoints get re-paired to what each path actually reached, the trajectories straighten, and the same 1-step leap now lands on the moons. The endpoint-gap KPI is the payoff of straightening.