Distribution matching & adversarial distillation

Trajectory methods (lessons 03–04) chase one teacher path with L2 — which averages, hence blur. The frontier drops the path entirely and matches the output distribution, using the teacher’s score as a gradient and a GAN for realism.

Why L2-on-a-trajectory hits a ceiling

Progressive and consistency distillation both regress the student toward a specific teacher output under L2 (or LPIPS). But the noise→image map is one-to-many, and the L2-optimal answer to a one-to-many target is the mean of the valid answers — which is not a valid answer. That is the residual softness at one step. To beat it you must stop asking “match this particular sample” and start asking “make your distribution equal to the data distribution.” The catch from lesson 03 returns: there is no tractable density over images. The escape is that we never needed the density — we need its gradient, the score, and the teacher already estimates that (generative lesson 16: s = −ε_θ/√(1−ᾱ_t)).

Score distillation (SDS): the teacher score as a gradient

Score Distillation Sampling (Poole et al., DreamFusion 2022) was invented to optimize a 3D scene from a 2D diffusion model, but it is the seed of every distribution-matching distiller. The idea: take the student’s output, noise it to level t, ask the frozen teacher “which way is more realistic?” — that direction is ε_θ(x_t,t) − ε — and push the student that way:

∇_φ L_SDS = 𝔼_t,ε [ w(t) ( ε_θ(x_t, t) − ε ) ∂x/∂φ ]

No backprop through the teacher — just its output as a force field pointing toward the data manifold. SDS works but is notoriously over-smooth and over-saturated, because comparing against the fixed Gaussian noise ε is a high-variance, mode-seeking signal. The fix is to compare against a second, learned score instead of raw noise.

DMD: the difference of two scores

Distribution Matching Distillation (Yin et al. 2024) makes the objective honest. Minimize the KL between the student’s output distribution p_fake and the data distribution p_real, at every noise level. Its gradient is a difference of two scores:

∇_φ KL(p_fake ‖ p_real) = 𝔼 [ −( s_real(x_t,t) − s_fake(x_t,t) ) ∂x/∂φ ]

s_real = the frozen teacher — the score of the data distribution.
s_fake = a second diffusion model, trained online on the student’s current outputs — the score of whatever the student produces right now.

Where the two scores disagree, the student is in the wrong place; the difference points from “where my samples are” toward “where real data is.” When the student’s distribution matches the data, the two scores coincide and the gradient vanishes. DMD2 drops the regression term and adds a GAN loss, reaching one-step generation competitive with the full teacher.

The intuition: two referees, one chasing the other

SDS has one referee (the frozen teacher) shouting “more realistic, that way!” against a fixed baseline of pure noise — loud but crude, so the student over-corrects into a bland safe zone. DMD hires a second referee who studies the student’s actual current output and learns its score. Now the signal is “here is where you are (fake referee) and here is where real is (teacher) — move along the difference.” As the student improves, the fake referee re-learns to track it, so the gap always points at the remaining error and shrinks to zero exactly when the distributions match. This adversarial-but-cooperative chase is why DMD covers modes that SDS flattens.

You have seen s_fake before

The online s_fake network playing “model the generator’s current output” is the diffusion analog of a GAN discriminator (generative lesson 21): an auxiliary network, retrained every step, whose only job is to characterize the fake distribution so the generator can be pushed off it. The variational-score-distillation (VSD / ProlificDreamer) lineage is the same trick used to fix SDS’s blur in 3D.

The adversarial route: ADD, SDXL-Turbo, LADD

The bluntest way to kill “L2 average blur” is to add a real discriminator and demand the few-step output look real, not just close. Adversarial Diffusion Distillation (ADD, Sauer et al. 2023 — SDXL-Turbo) combines two losses on a 1–4-step student:

A score-distillation term (the teacher pulls toward the data manifold), for mode coverage and prompt alignment.
A GAN term (a discriminator on real vs. student images), for high-frequency realism — the part L2 can never supply.

LADD (Latent ADD, used for SD3-Turbo) runs the adversarial game in latent space (generative lesson 19) instead of pixels, which is cheaper and higher-resolution. This is the diffusion-distillation echo of lesson 02’s “style transfers before substance”: the GAN term is exactly what makes one-step output look finished, while the score term does the slower work of getting the content right.

The trade you are buying

Distribution-matching and adversarial methods reach the best one/few-step quality, but they re-introduce everything diffusion was happy to leave behind in lesson 21: an auxiliary network trained online (s_fake or a discriminator), a min-max dynamic that can be unstable, and the GAN’s mode-coverage worries. You traded the teacher’s slow-but-stable sampling for a fast student bought with adversarial training. The diffusion field came full circle: it beat GANs (lesson 21), then re-hired them to go fast.

Interactive · score difference repairs a collapsed student

The student starts as a too-tight blob (mode-collapsed) sitting between the two data moons. The arrows show the DMD gradient at each student point: s_real − s_fake, the real-teacher score minus the score of the student’s own current cloud. Step the optimization and watch the blob get pushed apart and out onto both moons — the gradient vanishes only when the student covers the data. Toggle to SDS (compare against fixed noise instead of s_fake) and watch it under-spread and stall.

Takeaway

L2-on-a-trajectory averages, so it blurs; the cure is to match the output distribution, which needs only the score (a gradient), not a density. SDS uses the frozen teacher score against a noise baseline — works but over-smooths. DMD subtracts a second, online “fake-score” network (a GAN discriminator in disguise) so the gradient always points from where the student is to where the data is, vanishing at a match. ADD / Turbo / LADD add an explicit GAN term for the high-frequency realism L2 can’t give, hitting 1–4-step quality — at the cost of re-adopting adversarial training. Diffusion beat GANs, then re-hired them for speed.