The score-function & SDE view

The “deeper theory” lesson 7 kept deferring. One object — the score ∇_x log p_t(x) — turns DDPM, the deterministic samplers, and guidance into corollaries of a single picture.

Where we are · start of Part C

Part A built unconditional continuous diffusion and flow matching. Part B took the discrete / multimodal fork. Part C returns to continuous diffusion and builds the actual production text-to-image stack (the Stable Diffusion lineage): deterministic samplers, guidance, the VAE latent space, CLIP conditioning, evaluation, and the adapter ecosystem. It branches off Part A, not Part B — so the hooks below point back to lessons 03–07.

The unfinished business from lesson 7

Lesson 7 said DDPM and flow matching are the same object, and that “the deeper theory — score matching, SDE limits — lives elsewhere.” This is elsewhere. The payoff is concrete: by the end of this lesson, the 1000-step ancestral sampler (lesson 4), the deterministic DDIM sampler (lesson 17), and classifier-free guidance (lesson 18) are all one formula evaluated three ways.

The bridge is a single quantity, the score of the noised data distribution at time t:

s(x, t) ≝ ∇_x log p_t(x)

Forget the gradient symbol for a moment and read it as a question the model can answer at any point in space: which way is the data? The score is a vector field — one arrow at every x — and each arrow points in the direction that most steeply increases the log-density of the noised data. Stand anywhere in a fog of noise and the arrow points uphill on the probability landscape, i.e. back toward where the real data lives. The symbol ∇_x log p_t just makes “steepest uphill direction of the log-density” precise.

Intuition · linear unpacking

Claim: the score is a compass that, from anywhere, points toward higher data density — and that compass is the only thing you need to walk noise back into data.

Density is a landscape. p_t(x) is tall where noised data piles up and flat where it almost never lands. Take its log so the tall parts don’t dwarf everything else, and you have a hilly surface over x.
The gradient is “which way is up.” ∇_x of any landscape is the arrow pointing in the steepest-uphill direction. So ∇_x log p_t at a point says: from here, this is the fastest way toward more-probable data.
Uphill means toward the data. The peaks of the landscape are the data. Following the arrows is therefore a recipe for drifting out of empty noise-space and into the regions where real samples sit.

Central point. Knowing the score everywhere is knowing, from every point, the direction home — which is exactly the knowledge a sampler needs to undo the noising.

Why ε-prediction is score estimation

Here is the load-bearing identity. Recall the DDPM marginal (Eq. ⋆, lesson 2): x_t = √ᾱ_t x₀ + √(1−ᾱ_t) ε, so conditioned on x₀, x_t ∼ N(√ᾱ_t x₀, (1−ᾱ_t) I). The log-density of a Gaussian has a gradient you can write by hand:

∇_{x_t} log p(x_t | x₀) = − (x_t − √ᾱ_t x₀) / (1−ᾱ_t) = − ε / √(1−ᾱ_t)

The last step used ε = (x_t − √ᾱ_t x₀) / √(1−ᾱ_t). Taking the expectation over x₀ | x_t (a fact about Gaussian mixtures, “Tweedie’s formula”) turns the conditional score into the marginal score:

s(x_t, t) = ∇_{x_t} log p_t(x_t) = − 𝔼[ε | x_t] / √(1−ᾱ_t) = − ε_θ(x_t, t) / √(1−ᾱ_t)

The reframing

Your ε-network was never “predicting noise” in some ad-hoc sense. It was estimating the score of the noised data distribution, up to the fixed scalar −1/√(1−ᾱ_t). Denoising-score-matching (Vincent 2011) and the DDPM loss (lesson 3) are the same regression. Everything in lesson 3 was secretly score matching.

The forward process is an SDE

The DDPM chain (1000 discrete steps) is the Euler discretization of a continuous-time stochastic differential equation (Song et al. 2021). For the variance-preserving (VP) schedule from lesson 2, the forward SDE is

dx = −½ β(t) x dt + √β(t) dw

where dw is Brownian motion (the continuous limit of “add a little fresh Gaussian noise each step”). The drift −½β x shrinks the signal; the diffusion term √β dw injects noise. Run it from t=0 (data) to t=1 (pure noise). This is lesson 2’s forward chain with the step size taken to zero.

Two ways to run it backward

Running the forward SDE is easy because its drift −½β x just shrinks whatever you hand it — you never need to know where the data is. Reversing it is harder, and this is the crux: to walk backward from noise toward data you must know, at each point, which way the data lies. Anderson’s 1982 theorem makes this exact. It gives the time-reversal of any SDE in closed form, and for the VP-SDE the reverse is itself an SDE whose drift is the old drift plus a correction — and that correction is precisely the score:

dx = [ −½ β(t) x − β(t) ∇_x log p_t(x) ] dt + √β(t) dw̄

The −½β x piece is the forward drift run in reverse; the −β ∇_x log p_t piece is the new ingredient that aims each step uphill toward the data. Plug in s = −ε_θ/√(1−ᾱ_t) and discretize: you recover exactly the ancestral sampler of lesson 4, noise term and all. So DDPM sampling is “Euler–Maruyama on the reverse SDE.”

Intuition · linear unpacking

Claim: you can’t reverse a diffusion by running its arrow backward — you need the score, because the forward process throws away the one thing reversal requires.

Forward is blind on purpose. Going forward you only ever shrink the signal and sprinkle in fresh noise. That rule works the same everywhere; it never has to ask where the data was.
Backward is not blind. A blob of noise could have come from many different clean images. To undo a step you must lean toward the data that was likely to produce what you’re holding — you need a sense of direction the forward process never used.
The score supplies that direction. The missing “which way was the data” is exactly the uphill arrow ∇_x log p_t. Anderson’s theorem says: take the forward drift, flip it, and add this score term — that and nothing else is the true reverse.

Central point. Reversing diffusion is the forward rule run backward with one extra nudge, and that nudge is the score. Estimating the score (your ε-net) is the whole job.

But there is a second object with the same marginals p_t at every time — the probability-flow ODE:

dx/dt = −½ β(t) [ x + ∇_x log p_t(x) ] = −½ β(t) [ x − ε_θ(x,t)/√(1−ᾱ_t) ]

It is deterministic — no dw. Here is the surprising part: this clean, jitter-free ODE visits the same distribution p_t at every time as the noisy reverse SDE. The reason is that a distribution only cares about how mass flows on average, not about whether each particle wiggles. The SDE moves mass with a noisy drift; the ODE moves the same mass with a drift adjusted to absorb the noise’s average effect. Anderson’s reverse SDE and this ODE therefore push the same probability mass along the same sequence of distributions; they differ only in whether individual trajectories are stochastic or deterministic. That single fact is the seed of everything in lesson 17.

Intuition · linear unpacking

Claim: a smooth deterministic flow and a jittery random flow can trace out the exact same distributions at every instant, because the histogram doesn’t see the jitter.

What a marginal actually tracks. p_t is just “how much mass is where at time t” — a histogram. It is indifferent to which individual particle is which, or to how wiggly any one path was.
Two ways to move a histogram. A histogram changes only through net inflow and outflow at each location. The SDE produces that flow with a random drift; the ODE produces the identical flow with a velocity that has the noise’s average smoothing already folded in (the half-score term).
Same flow ⇒ same sequence of histograms. If two processes inject and remove mass at the same rate everywhere, their histograms evolve identically — even though one is built from trembling paths and the other from glassy streamlines.

Central point. Matching marginals is matching mass-flow, not matching trajectories — so you may swap the noisy paths for clean deterministic ones and still hit every p_t on the nose.

The intuition: river vs. weather

Picture probability mass as water flowing from the noise cloud back to the data. The reverse SDE is the actual weather over that river — every water molecule takes a jittery, random path, but the bulk flow (the histogram of where molecules are at time t) matches p_t. The probability-flow ODE is the smooth average current: follow it and you also reproduce p_t at every instant, but each particle now glides on a clean, deterministic streamline.

Stochastic sampling can self-correct (noise lets a particle that drifted into a low-density region get kicked back); deterministic sampling is reproducible and lets you take big steps with a good integrator. Lesson 17 trades between exactly these.

The exact same field flow-matching learned

Lesson 5’s flow matching learned a velocity field v(x,t) and integrated an ODE. Lesson 7 gave the ε↔v conversion. The probability-flow ODE above is that velocity field, written in score coordinates. The three quantities are one object in three currencies:

Currency	What the net outputs	Convert to score s = ∇log p_t
noise ε	the Gaussian that was added	s = −ε / √(1−ᾱ_t)
clean x₀ (Tweedie)	𝔼[x₀ \| x_t]	s = (√ᾱ_t x̂₀ − x_t) / (1−ᾱ_t)
velocity v	dx/dt along the PF-ODE	affine in (x, s); see lesson 7’s formula

Pick the currency that is numerically best behaved at the t you care about (lesson 3’s scale argument, lesson 9’s table). The underlying vector field does not change.

Interactive · reverse SDE vs. probability-flow ODE

Both samplers below start from the same noise cloud and target the same two-moons data, using the oracle score. The SDE path (stochastic) and the ODE path (deterministic) end at the same distribution but along visibly different individual trajectories — that is the payoff: matching marginals does not mean matching paths. One subtlety worth getting right: turning the SDE noise down (η → 0) removes the jitter, but it does not turn the SDE into the ODE. The deterministic limit of the reverse SDE still carries the full score term, whereas the probability-flow ODE carries exactly half of it — and that factor of ½ is precisely what lets the noise-free ODE reproduce the same marginals.

Why this picture is worth the algebra

Samplers (lesson 17). DDIM = a particular discretization of the PF-ODE. DPM-Solver = a smarter integrator of the same ODE. All "fast sampling" is "integrate the PF-ODE in fewer steps."
Guidance (lesson 18). Conditioning on y means sampling with the score of p_t(x | y). Bayes splits that into the unconditional score plus ∇_x log p(y | x) — you steer by adding a vector to the score. Trivial in this picture; mysterious without it.
Likelihood. The PF-ODE is a continuous normalizing flow, so you can compute exact log-likelihoods via the instantaneous change-of-variables. The stochastic sampler cannot.

The honest caveat

The marginals match only if the score is exact. With a learned ε_θ, the SDE and ODE samplers give subtly different sample distributions, because score error compounds differently under noise (the SDE’s self-correction can paper over errors the ODE bakes in). This is why some models sample best stochastically and others deterministically — an empirical knob, not a theorem.

Punchline

Your noise-predictor is a score estimator. The forward chain is an SDE; it has a stochastic reverse (= DDPM ancestral sampling) and a deterministic probability-flow ODE (= DDIM / flow-matching territory) that share every marginal. Conditioning is “add a vector to the score.” Three lessons’ worth of machinery are corollaries of s = −ε_θ/√(1−ᾱ_t).