Guidance: classifier, classifier-free, and the score picture

Conditioning is “add a vector to the score” (lesson 16). Once you see that, classifier guidance, classifier-free guidance, the w·(cond−uncond) formula, negative prompts, and offset noise all fall out of one Bayes split.

What “conditional generation” means in score language

To generate x given a condition y (a class label, a text embedding), you sample from p_t(x | y) instead of p_t(x). Lesson 16 said samplers only ever need the score of the target. So the entire question is: what is ∇_x log p_t(x | y)? Apply Bayes’ rule, p(x|y) ∝ p(x) p(y|x), then take ∇_x log — the data-evidence term p(y) has no x and drops:

∇_x log p_t(x | y) = ∇_x log p_t(x) + ∇_x log p_t(y | x)

The whole lesson in one line

The conditional score = the unconditional score (what your diffusion model already learned) plus the gradient of a classifier that scores how well x matches y. Steering is vector addition in score space. The two guidance methods differ only in how they get that second term.

Intuition · linear unpacking

Claim: adding a classifier’s gradient to the score nudges generation toward samples the classifier likes.

The score is a compass. A score ∇_x log p points in the direction that makes the current sample more probable. The sampler just keeps stepping along it. So whatever you add to the score, you add to where the sampler walks.
Bayes splits the target. “Looks like data and matches y” factors into “looks like data” (p(x)) times “matches y” (p(y|x)). Taking the log turns the product into a sum, and the gradient of a sum is the sum of the gradients — two compasses, added tip to tail.
The second compass points at y. ∇_x log p(y|x) answers “which way should I move x so the classifier is more sure it’s y?” Following it walks x uphill on the classifier’s confidence.
The data term vanishes from the math, not the walk. p(y) has no x in it, so its gradient is zero and it drops out — it’s a constant that can’t steer anything.

Central point. Conditioning is just bolting a second “move toward y” arrow onto the “move toward realistic data” arrow the model already had.

Classifier guidance: build the second term explicitly

Dhariwal & Nichol (2021) did the literal thing: train a separate classifier p_φ(y | x_t, t) on noised images (it has to read x_t at every noise level), and add its input-gradient to the diffusion score at sampling time, scaled by a knob w:

ε̂(x_t, t, y) = ε_θ(x_t, t) − w · √(1−ᾱ_t) · ∇_{x_t} log p_φ(y | x_t)

(The −√(1−ᾱ_t) converts “add to score” into “subtract from ε,” lesson 16.) The properties interviewers list, now with the reason attached:

No retraining of the diffusion model — the classifier is bolted on at sampling time.
But you must train a noise-robust classifier, and you can only condition on whatever it classifies. “Classify N classes ⇒ control N classes” — arbitrary text is out of reach.
w > 1 over-weights the classifier: sharper class adherence, less diversity. This previews the universal guidance trade-off below.

Classifier-free guidance: get the second term for free

Ho & Salimans (2022) noticed you never needed a separate classifier at all — the model can grade itself. The idea: the classifier gradient is just “how much does turning the condition on change the model’s prediction?” If the model predicts one thing when it knows y and another when it doesn’t, the difference between those two predictions already points toward y — that difference is the classifier gradient, no classifier needed. Concretely, rearrange the same Bayes identity to express the awkward term using only diffusion scores:

∇_x log p_t(y | x) = ∇_x log p_t(x | y) − ∇_x log p_t(x)

The right side is “(conditional score) − (unconditional score).” If one network can produce both — by training it with the condition present, and 10% of the time dropping the condition to a learned NULL token so it also learns the unconditional model — then you implicitly own the classifier gradient. Substitute back into the guided score and rearrange into the form everyone memorizes:

ε̂ = ε_θ(x_t, ∅) + w · ( ε_θ(x_t, y) − ε_θ(x_t, ∅) )

Intuition · linear unpacking

Claim: CFG is “start at the unconditional guess, then step toward the conditional one — and w says how far.”

Two predictions for the same noisy image. Ask the one network twice: once told the prompt (ε_θ(x, y)), once told nothing (ε_θ(x, ∅)). Both are guesses at the noise to remove; they differ because the prompt changed the answer.
Their difference is the “prompt direction.” ε_θ(x, y) − ε_θ(x, ∅) is exactly the part of the prediction that exists only because of y. It’s an arrow pointing from the generic guess toward the on-prompt guess.
w is how hard you push along that arrow. Start at the unconditional guess and add w copies of the difference. w=0: ignore the prompt entirely. w=1: land exactly on the honest conditional guess. w>1: overshoot past it — exaggerate whatever made the prompt-aware guess different.
Push away from the baseline, toward the condition. The same step both pulls toward y and repels from the generic, prompt-less prediction — that repulsion is what sharpens identity, and overdoing it (large w) is what distorts.

Central point. CFG never builds a classifier; it reads the prompt’s effect off the gap between the model’s own two answers, and w dials how aggressively to amplify that gap.

This is the five lines from lesson 9 — now you know it’s the Bayes split with the classifier replaced by a difference of the model’s own conditional and unconditional predictions. Read the knob:

w	ε̂ equals	Behaviour
0	ε_θ(x,∅)	unconditional — ignores y
1	ε_θ(x,y)	plain conditional sampling, no extra push
> 1	extrapolation past the conditional	amplifies y-aligned direction: sharper prompt adherence, lower diversity, risk of over-saturation

The interview contrast is now precise: classifier guidance needs an extra noised classifier and is limited to its label set; classifier-free guidance needs the diffusion model retrained with condition-dropout but then conditions on anything you can embed (a CLIP text vector, lesson 20). That generality is why every text-to-image model uses CFG.

Why w > 1 sharpens and distorts

CFG doesn’t sample from p(x|y). It samples from a tilted distribution ∝ p(x) p(y|x)^w. Raising p(y|x) to a power w>1 sharpens the “does this look like y” landscape — it concentrates mass on the most prototypical, high-likelihood-of-y samples. You get crisper class identity and stronger prompt following, but you pay in diversity (the tails are crushed) and, at large w, in artifacts (over-contrast, blown highlights) because you’re extrapolating the score outside where it was trained. Reported sweet spots: w ≈ 1.5–4 class-conditional ImageNet; w ≈ 5–9 text-to-image. DPM-Solver++ exists partly to stay stable in this high-w regime (lesson 17).

Negative prompts: the NULL slot is programmable

Nothing forces the second term to be the unconditional model. Swap ∅ for a second condition y⁻ (“blurry, extra fingers, watermark”):

ε̂ = ε_θ(x, y⁻) + w · ( ε_θ(x, y⁺) − ε_θ(x, y⁻) )

Now guidance pushes toward y⁺ and away from y⁻ along the same difference vector. “Negative prompts” are not a separate feature — they are the baseline argument of CFG, set to something you want to repel from. Same two network calls, no retraining.

Offset noise: a training/inference mismatch that guidance can’t fix

The offset-noise question lives here because it’s about the conditioning the schedule silently imposes. Recall x_t = √ᾱ_t x₀ + √(1−ᾱ_t) ε. Even at the last step t=T, √ᾱ_T is small but not zero on a high-dimensional image, so a faint shadow of x₀’s low-frequency content — its overall brightness/color mean — survives into x_T. The model learns to rely on that leaked mean. At inference you start from pure N(0,I), whose per-image mean is ≈ 0, so the model can only ever produce images with mean luminance near the dataset average — it physically cannot make a truly black or truly white scene. (This is the SD v1/v2 “everything is mid-gray” complaint.)

Offset noise (Guttenberg 2023) fixes it by adding, during fine-tuning, a per-image shared offset to the noise: ε = ε_iid + γ · c where c is one scalar (or per-channel) value broadcast over all pixels. This deliberately corrupts the low-frequency mean during training, forcing the model to predict the brightness from the prompt rather than copy it from the leaked signal. After offset-noise fine-tuning, “a solid black background” actually renders black.

The deeper lesson

Offset noise is a band-aid for the linear/VP schedule not reaching zero terminal SNR on images. The principled fix is to rescale the schedule to zero terminal SNR (Lin et al. 2024) so √ᾱ_T=0 and there is no leak to exploit — the same disease the cosine schedule (lesson 9) and v-prediction were partly treating. Different patches, one root cause: train and sample must see the same endpoint distribution.

Interactive · the guidance dial

A 2D conditional toy: data is two labelled clusters (class A, class B). We sample class A with the oracle conditional and unconditional scores and apply CFG at scale w. At w=1 you get the honest class-A distribution; crank w up and watch the cloud contract toward the class-A prototype and pull away from the decision boundary — sharper identity, collapsing diversity. Past ~6 it overshoots outside the data.

Punchline

Conditioning splits the score by Bayes into unconditional + classifier. Classifier guidance builds the classifier; classifier-free guidance fakes it with (cond − uncond) and a 10% dropout at training, giving the ε̂ = ε_∅ + w(ε_y − ε_∅) dial. Negative prompts reprogram the baseline; offset noise repairs the brightness leak that no amount of guidance can. One Bayes split, the whole steering toolkit.