Consistency models

Progressive distillation halves the steps one slow round at a time. Consistency models change the target: learn a function that jumps from anywhere on a trajectory straight to its origin — trained in a single run.

The limitation we’re escaping

Progressive distillation (lesson 03) is serial: ~10 dependent training rounds to reach 1–2 steps, and the 1-step result is still soft. The reason is that each round only ever learns to double a stride — it never learns the destination directly. Consistency models (Song et al. 2023) ask for the destination outright, but constrain the problem so that asking is learnable.

The self-consistency idea

Recall the probability-flow ODE (generative lesson 16): every noisy point x_t sits on exactly one deterministic trajectory that flows down to a clean sample x₀ at t=0. Define a network f_θ(x_t, t) whose job is to output that endpoint — the origin of whatever trajectory x_t is on. Two requirements pin it down:

Boundary condition. At t=0 there is nothing to denoise: f_θ(x₀, 0) = x₀. This is hard-wired by construction (a skip-connection parameterization f_θ = c_skip(t)·x + c_out(t)·F_θ with c_skip(0)=1, c_out(0)=0), not learned.
Self-consistency. Any two points on the same trajectory must map to the same output. If x_t and x_t′ are both on the trajectory through x₀, then f_θ(x_t,t) = f_θ(x_t′,t′) = x₀.

is the model

A function satisfying both is a one-step sampler: feed it x_T∼N(0,I) and it returns f_θ(x_T,T) ≈ x₀ in a single call.

The loss: adjacent points must agree

You can’t enforce “maps to x₀” directly — you don’t know x₀ for a given trajectory. But you can enforce that two adjacent points on the trajectory agree, and let the boundary condition anchor the chain. Take a sample, noise it to a step t_n+1, take one solver step back to t_n, and require the network’s outputs at the two points to match:

L_CD = 𝔼 d( f_θ(x_{t_n+1}, t_n+1), f_θ⁻(x̂_{t_n}, t_n) )

where x̂_{t_n} is one ODE step from x_{t_n+1} (using the teacher’s score), d is a distance (L2 or LPIPS), and θ⁻ is an EMA copy of the weights — a frozen target that updates slowly, exactly like the target network in deep RL, to stop the “match yourself” objective from collapsing to a trivial constant.

The intuition: a bucket brigade to the origin

You can’t teach everyone the address of the destination directly. So you teach each person only to hand the bucket to the next person down the line and agree on what’s in it. The person standing at the destination (the boundary condition, t=0) knows the truth. Because every adjacent pair agrees, and the chain is anchored at the end that knows, the agreement propagates all the way up: everyone ends up reporting the true destination. Consistency training is that propagation, run by gradient descent over all adjacent pairs at once.

Two ways to get the trajectory: distillation vs. training

	Consistency Distillation (CD)	Consistency Training (CT / iCT)
where x̂_{t_n} comes from	one step of a pretrained teacher’s ODE solver	an unbiased one-step estimate using the known forward noise — no teacher at all
needs a teacher?	yes (a trained diffusion model)	no — trains from scratch
quality	higher (rides a good teacher)	improved CT (iCT, Song & Dhariwal 2023) closed most of the gap

This is the diffusion echo of lesson 02’s WHERE knob: CD is off-policy on the teacher’s trajectories (white-box, needs the teacher); CT generates its own targets from the forward process (no teacher). Same axis, new setting.

One step isn’t free: multistep consistency

A single jump from pure noise asks the network to resolve all the ambiguity at once, and the L2/LPIPS target still pulls toward an average — so 1-step consistency samples are good but not teacher-quality. The fix is cheap: multistep sampling. Jump to x̂₀, re-noise to an intermediate t, jump again. 2–4 alternations of (jump, re-noise) recover most of the quality, which is why “few-step” (not strictly one-step) is the production sweet spot — the same shape as lesson 17’s step-count curve, but starting from a far better one-step point.

LCM and CTM: the variants you’ll meet

LCM (Latent Consistency Model). Consistency distillation run in Stable Diffusion’s latent space (generative lesson 19), with classifier-free guidance (generative lesson 18) folded into the distillation so the student bakes in a fixed guidance scale — no double network call at inference. This is the method behind 2–4-step Stable Diffusion. LCM-LoRA packages the distillation as a LoRA (generative lesson 23), so the “make it few-step” capability becomes a portable adapter you can drop onto any SD checkpoint.
CTM (Consistency Trajectory Models). Generalize from “always jump to t=0” to “jump from any t to any s < t.” That subsumes both consistency models (jump to 0) and ODE solvers (small jumps), and lets you trade steps for quality smoothly along one trained model.

Interactive · self-consistency vs. a raw one-step regressor

Two one-step maps from noise to the two-moons data. The consistency map (oracle) sends each noise point down its PF-ODE trajectory to the true origin — the cloud lands on the moons. The naive x̂₀-at-T regressor predicts the conditional mean from pure noise — the L2-optimal but blurry average, which collapses toward the data center. Then add re-noise rounds and watch the consistency sampler sharpen toward the data as 1→2→4 steps.

Takeaway

A consistency model is a network forced to map every point on a PF-ODE trajectory to that trajectory’s origin — pinned by a hard boundary condition at t=0 and an EMA-stabilized “adjacent points must agree” loss, so one network call jumps from noise to image. Consistency distillation rides a teacher’s ODE (off-policy, white-box); consistency training needs no teacher. One step is good-not-great because the L2 target still averages, so 2–4 (jump, re-noise) rounds are the real sweet spot. LCM puts all this in SD’s latent space with guidance folded in; CTM generalizes the jump to any-to-any. Next: stop matching trajectories at all — match the distribution.