Fast & deterministic sampling

DDPM needs ~1000 steps (lesson 4). DDIM, DPM-Solver, and distillation cut that to 50, 10, and 1 — and every one of them is the same probability-flow ODE (lesson 16), integrated more cleverly or amortized into the weights.

The cost we are attacking

Lesson 4’s ancestral sampler is Euler–Maruyama on the reverse SDE: one network call per step, ~1000 steps, because the stochastic term forces a tiny step size. Lesson 16 handed us the escape: the deterministic probability-flow ODE

dx/dt = −½ β(t) [ x − ε_θ(x,t)/√(1−ᾱ_t) ]

has the same marginals but no noise term, so its step size is limited only by integration error, not by stochasticity. Fast sampling is the numerical-ODE question: integrate this in as few network calls as possible. The interview taxonomy (“probabilistic vs. numerical samplers”) is really “how good is your ODE solver, and did you exploit the ODE’s structure.”

DDIM: the non-Markovian re-derivation

DDIM (Song et al. 2021) was derived before the SDE picture, by a different route worth knowing because it’s the one interviewers expect. The intuition first: the trained network never learned anything about how the noise was added step by step — it only ever saw single snapshots x_t at a known noise level and learned to name the noise in them. So we are free to swap out the step-by-step story for a faster one, as long as each individual snapshot still has the same statistics the network was trained on. Concretely: the DDPM training loss (lesson 3) depends only on the marginals q(x_t | x₀) — the per-level snapshot distributions — never on the joint chain being Markovian. So you are free to invent a different forward process with the same marginals and a non-Markovian joint, and reuse the trained ε_θ unchanged. DDIM picks the family that makes the reverse step

x_t−1 = √ᾱ_t−1 · x̂₀(x_t) + √(1−ᾱ_t−1 − σ_t²) · ε_θ(x_t,t) + σ_t z, x̂₀ = (x_t − √(1−ᾱ_t) ε_θ) / √ᾱ_t

Read this in three pieces: predict the clean image x̂₀ (Tweedie, lesson 16), re-noise it to level t−1 using the same predicted direction, plus an optional fresh-noise term σ_t z. The single knob σ_t interpolates a whole family:

σ_t choice (often written via η)	What you get
σ_t = √β̃_t (η = 1)	exactly DDPM ancestral sampling (lesson 4) — fully stochastic. (β̃_t is the posterior variance from lesson 4, so its square root is the std σ_t.)
σ_t = 0 (η = 0)	DDIM: deterministic. Same x_T ⇒ same image, every time
0 < σ_t < √β̃_t	a partial-noise interpolation; trades reproducibility for self-correction

Two consequences make DDIM the workhorse. (1) Determinism — with η=0 the map from latent x_T to image is a function, so you can interpolate in latent space, invert real images (DDIM inversion) for editing, and reproduce results. (2) Step-skipping — because the update only references the marginal levels ᾱ_t, ᾱ_t−1, you can run it on a subsequence of timesteps (1000 → 50) with no retraining. And the η=0 update is precisely a first-order discretization of the PF-ODE — DDIM is the ODE’s Euler step, in disguise.

Intuition · linear unpacking

Claim: DDIM reuses the exact same trained network, takes far fewer steps with no retraining, and turns out to be the plain Euler step of the lesson-16 ODE.

What the net actually knows. It was trained on isolated noisy snapshots, each tagged with its noise level — never on “what came one step before.” So it has no opinion about the path between levels; it only knows how to read the noise out of a single level.
So we can rewrite the path. Any sampler whose snapshots match those trained noise levels is fair game. DDIM keeps the levels, throws away the requirement that you visit them one tiny step at a time, and shuts off the injected noise (σ_t=0). Same net, new route.
Why skipping is free. Each DDIM update mentions only the two levels it’s jumping between, ᾱ_t and ᾱ_t−1 — nothing about the levels in the gap. Drop 950 of the 1000 levels and the formula still type-checks, because it never referenced them.
Why it’s Euler. “Predict the clean image, then re-noise it to the next level” is, after you write it in the ODE’s coordinates, exactly one step of the crudest ODE solver. Nothing new was invented; the 2021 derivation just reached the same place by a different road.

Central point. DDIM is not a new model and barely a new idea — it is the same denoiser, run on fewer of the noise levels it already understands, which happens to be the simplest way to integrate the probability-flow ODE.

DPM-Solver: exploit the ODE’s shape

Plain Euler treats dx/dt = f(x,t) as a black box. But the PF-ODE is semilinear: a linear-in-x part plus a nonlinear part that only enters through the network. Change variables to the log-SNR λ_t = log(√ᾱ_t/√(1−ᾱ_t)) and the linear part integrates exactly (variation-of-constants), leaving an integral of ε_θ against an exponential kernel:

x_t = (√ᾱ_t/√ᾱ_s) x_s − √ᾱ_t ∫_{λ_s}^λ_t e^−λ ε_θ(x_λ, λ) dλ

Now you only have to approximate the integral of the network output, which is smooth, with a Taylor expansion in λ. First order recovers DDIM. Second/third order (DPM-Solver-2/3, Lu et al. 2022) use one or two extra evaluations to cancel the leading error terms — an exponential integrator, the same idea as “ETD” in stiff-ODE numerics. This is why the DPM family hits indistinguishable-from-1000-step quality in 10–20 calls.

Intuition · linear unpacking

Claim: by solving the part of the ODE we already know by hand, DPM-Solver spends every expensive network call on the only part that’s actually hard — so a handful of calls suffices.

The ODE has two parts. One part is a simple shrink-and-stretch of x itself; the other part comes only through the network ε_θ. Plain Euler ignores the split and approximates the whole thing at once, wasting effort re-discovering the simple part numerically.
Do the easy part exactly. The shrink-and-stretch part is a textbook integral — variation-of-constants solves it with pen and paper, no network calls. After that, the only thing left to approximate is the integral of the network’s output against a known kernel.
The leftover is smooth. The network’s output, as you slide along the log-SNR axis λ, changes gently. A smooth quantity is exactly what a low-order Taylor expansion nails with very few sample points — one point is DDIM, two or three points cancel the next error terms.
Hence few calls. Each extra network call buys a whole order of accuracy, because it’s spent on a smooth target rather than on the curvature the exact part already absorbed. Ten to twenty calls land where naïve Euler needed a thousand.

Central point. The speedup isn’t a better network or a cleverer guess — it’s refusing to spend network calls on the half of the ODE you can integrate by hand.

The whole sampler zoo, demystified

Euler / Heun (k-diffusion): generic ODE solvers on the PF-ODE; Heun is 2nd-order (predictor–corrector).
PLMS / PNDM: linear-multistep — reuse the last few ε evaluations (Adams–Bashforth) instead of fresh ones, so higher order at ~one call per step.
DPM-Solver / DPM-Solver++ / DPM-2 / DPM-adaptive: exponential integrators of the semilinear ODE; ++ stabilizes the high-guidance regime; adaptive picks step size from a local error estimate.
UniPC: a unified predictor–corrector that squeezes another step or two out.

They differ in which numerical-analysis trick they apply to one fixed ODE. None of them retrain or change the model.

The wall every solver hits

An order-p solver has error O(Δt^{p+1}) per step — but only while the true trajectory is smooth enough that the Taylor expansion holds. As you push below ~10 steps, Δt is so large that the curvature of the path dominates and no fixed-order solver helps. You have left the regime where “better integration” pays. To go to 1–4 steps you must change the target itself: distillation.

Distillation: amortize the integral into the weights

If the expensive thing is integrating a curved ODE, train a network to jump along it.

Method	Idea	Steps
Progressive distillation (Salimans & Ho 2022)	train a student to match two teacher steps in one. Halve, repeat: 1024→512→…→4.	4–8
Consistency models (Song et al. 2023)	train f_θ(x_t,t) ≈ x₀ for every point on a PF-ODE trajectory, enforcing that all points on one trajectory map to the same origin (self-consistency). One call jumps straight to x₀.	1–4
LCM (Latent Consistency Model, 2023)	consistency distillation applied in Stable Diffusion’s latent space (lesson 19), with CFG (lesson 18) folded in — the few-step sampler interviewers cite as “LCM.”	2–4
Adversarial distillation (SDXL-Turbo / ADD, LADD)	add a GAN loss (lesson 21) so the few-step student’s output looks real, not just close-in-L2.	1–2

The intuition for consistency

Picture the PF-ODE’s streamlines as rivers all draining to the data sea. An ODE solver canoes down one river step by step. A consistency model has memorized the delta from any point to the river’s mouth — drop it anywhere on any river and it teleports to the mouth in one paddle. You pay for that at training time (it must see the whole flow) and lose some fidelity (one jump can’t correct itself), which is why 2–4 steps usually beats 1.

Interactive · the step-count cliff

Same two-moons target, oracle model. Three samplers race across step budgets K: DDPM-ancestral (stochastic), DDIM (deterministic, η=0), and a 2nd-order solver (Heun-style on the PF-ODE). Watch DDPM shred the modes at low K while DDIM and the 2nd-order solver stay coherent — then watch even those degrade past the smoothness wall.

So which sampler do I use?

Budget	Pick	Why
50–100 calls, want best quality	DDIM or DPM-Solver++ (2M)	deterministic, reproducible, near-ceiling quality
10–20 calls	DPM-Solver++ / UniPC	exponential integrator earns its order here
1–4 calls, production latency	LCM / Turbo (distilled)	past the solver wall — only amortization works
need to edit / invert real images	DDIM (η=0)	deterministic map is invertible
model samples better stochastically	ancestral / η>0	self-correction hides score error (lesson 16 caveat)

Punchline

There are not many samplers — there is one probability-flow ODE and a ladder of ways to integrate it: Euler (DDIM) → multistep (PLMS) → exponential integrators (DPM-Solver) → and, once curvature beats every solver, distillation (consistency / LCM / Turbo) that bakes the integral into the weights. “Acceleration” is “move up that ladder.”