generative_continuous / 02 · DDPM forward chain lesson 2 / 15

DDPM — the forward chain

A fixed noising process, designed so we never have to simulate it at training time.

The design move

We want a path of distributions from pdata back to a tractable prior. The trick: instead of designing the path directly, design a noising process that does it for us. Pick a Markov chain

q(xt | xt−1) = N( xt ;  √(1 − βt) · xt−1,  βt · I )

with small per-step variance βt > 0, run for T steps. The chain progressively adds Gaussian noise and shrinks the signal. The induced marginals q(xt) are a path that takes pdata (at t = 0) to a near-Gaussian (at t = T).

Why the odd-looking √(1 − βt) factor in front of xt−1? Because we want a known destination. We are going to start generation from pure N(0, I) noise, so the chain had better land there no matter how many steps we take. If we just kept piling on noise — q(xt | xt−1) = N(xt−1, βt I) — the spread would grow without bound: after T steps we’d sit at N(0, Tβ I), a target that moves as we change T. The fix is to shrink the old signal by a hair at each step before adding the noise. That shrink factor is exactly tuned so the shrinkage in variance cancels the variance we add. This is what makes the chain variance-preserving: if Var(xt−1) = I, then Var(xt) = (1 − βt) · I + βt · I = I. The total spread never budges, so the terminal distribution stays at unit variance regardless of T.

Intuition · linear unpacking

Claim: the √(1 − βt) factor isn’t cosmetic — it’s what pins the end of the chain to a fixed N(0, I) we can sample from for free.

  1. We need a known starting line for generation. Sampling later means drawing a fresh xT from a prior and denoising. That prior has to be something trivial to sample — standard Gaussian noise. So the forward chain’s endpoint must be N(0, I).
  2. Plain noise-adding overshoots. Each step contributes its own variance βt. If nothing pushes back, those variances stack up and the cloud keeps spreading — the endpoint depends on T and isn’t unit-variance.
  3. So shrink a little before adding. Multiplying the old sample by √(1 − βt) deflates its variance by a factor (1 − βt). Adding noise of variance βt tops it right back up: (1 − βt) + βt = 1.
  4. The accounting closes every step. Start at variance 1, and you stay at variance 1 forever — the loss from shrinking is precisely the gain from the noise, by design.

Central point. The shrink factor is the thermostat that holds total variance at 1, guaranteeing the chain always ends at the same easy-to-sample distribution — very close to N(0, I) (the leftover signal √ᾱT is tiny but not exactly zero) — which is the only reason we can later begin generation from scratch.

VP vs. VE
This is the variance-preserving (VP) schedule. The other family, variance-exploding (VE, used in NCSN / EDM), drops the shrink factor and lets variance grow. VE’s terminal distribution is N(0, σ2max I) with big σmax. Both work; VP is what the DDPM paper used and what diffusion.py implements.

The miracle: a closed-form marginal

Naively, to get a sample xt from q(xt | x0) we’d simulate t Gaussian steps — that’s t network-free passes, but we’d need to do it freshly for every training example at every step. For Gaussians this is unnecessary.

Here is the reason it’s unnecessary. Each step does the same two simple things: scale the current point a little, then add independent Gaussian noise. Chaining many such steps is still just “scale and add noise” — a sum of Gaussians is Gaussian, and scaling a Gaussian keeps it Gaussian. So all the intermediate randomness collapses into a single equivalent Gaussian that depends only on x0 and how far we’ve travelled. We just need to track the running product of the per-step scale factors. Let αt = 1 − βt be the per-step survival factor and ᾱt = ∏s ≤ t αs the cumulative one — the fraction of the original signal’s variance still alive at step t. Then:

q(xt | x0) = N( xt ;  √ᾱt · x0,  (1 − ᾱt) · I )

Equivalently, with ε ∼ N(0, I):

xt = √ᾱt · x0 + √(1 − ᾱt) · ε    [Eq. ★]

One Gaussian sample, no simulation, jumps straight to any t. This is the line that makes diffusion practical — without it, training would mean simulating T steps for every minibatch element.

Intuition · linear unpacking

Claim: Eq. ★ lets you teleport from clean data x0 to any noise level xt in one shot, instead of walking t steps.

  1. The naive picture is a long walk. To reach step t you’d apply the one-step rule t times — t separate noise draws and rescalings, redone for every training example.
  2. But the whole walk is one Gaussian. Every step only scales and adds noise, so the combined effect of all t steps is itself just “keep a fraction of x0, add one lump of noise.” Nothing about the intermediate points matters to the endpoint.
  3. How much signal survives: √ᾱt, the running product of the per-step shrink factors. As t grows, ᾱt drifts toward 0 and the original picture fades out.
  4. How much noise replaces it: √(1 − ᾱt) — the leftover, set so signal-variance plus noise-variance stays at 1 (the same variance-preserving bookkeeping as before). One draw ε ∼ N(0, I), scaled and added, reproduces the exact distribution the t-step walk would have given.

Central point. Because the chain is linear-Gaussian, the entire forward process at any timestep is a single weighted blend — √ᾱt of the data plus √(1 − ᾱt) of fresh noise — so training can sample any noise level in O(1), not O(t).

Show the derivation (induction on t)

Base case t = 1: x1 = √α1 · x0 + √β1 · ε1. Conditional on x0, this is Gaussian with mean √α1 · x0 = √ᾱ1 · x0 and variance β1 · I = (1 − ᾱ1) · I. Matches the claim.

Inductive step: assume xt−1 = √ᾱt−1 · x0 + √(1 − ᾱt−1) · η with η ∼ N(0, I). Then

xt = √αt · xt−1 + √βt · εt = √αt ( √ᾱt−1 · x0 + √(1 − ᾱt−1) · η ) + √βt · εt

= √ᾱt · x0 + ( √(αt(1 − ᾱt−1)) · η + √βt · εt )

The bracketed term is a sum of independent Gaussians with total variance αt(1 − ᾱt−1) + βt = αt − αt·ᾱt−1 + (1 − αt) = 1 − ᾱt. So xt = √ᾱt · x0 + √(1 − ᾱt) · ε as claimed. ▪

The schedule, visualized

The two things you care about across the chain are the signal coefficient √ᾱt (how much of x0 remains in xt) and the noise coefficient √(1 − ᾱt). A good schedule keeps both above zero across most of the chain — if signal collapses to zero too early, mid-range timesteps are pure noise and the net has nothing to denoise.

Schedule explorer — what does T and (β₁, β_T) buy you?
DDPM’s default is T=1000, β₁=1e-4, β_T=0.02 linear. Drag the knobs; watch where ᾱ_t hits zero. For images (high-D data) the linear schedule collapses signal too early — cosine fixes that (lesson 9).
ᾱ at T/4
—
ᾱ at T/2
—
ᾱ at 3T/4
—
ᾱ at T
—

Interactive · drive the chain on the two moons

Below is Eq. ★ applied to actual two-moons samples. Pull the t slider: as ᾱt drops, the two crescents wash out into a Gaussian. The opposite direction is what the network will learn to do.

Forward jump x_t = √ᾱ_t · x_0 + √(1 − ᾱ_t) · ε
Same N=2000 points throughout; the only thing changing is t. Notice the orange dots stay on the same scale across t — that’s the variance-preserving property.
√ᾱ_t (signal)
1.000
√(1−ᾱ_t) (noise)
0.000
empirical Var (avg axis)
1.000

The path as a heatmap

Above we visualized one slice of the path at a chosen t. Here’s the whole path at once: project the moons to their x-coordinate, bin the histogram, and stack the histograms left-to-right as columns of a heatmap. Each column is q(xt) (marginalized over y) at one timestep. Reading left-to-right is the forward process; reading right-to-left is what the reverse model has to learn.

Density evolution — q(xt(x)) over t
Left edge = t = 0 (the two moons projected to one dim — you can see the two humps). Right edge = t = T (a single Gaussian hump centered at 0). The transition between them is the path.

Try the cosine schedule: with linear-β at high T the right side of the heatmap pins to pure noise too fast — most of the heatmap is wasted “already-Gaussian” columns. Cosine spreads the transition over more of the path.

Three claims to take away

  1. Eq. ★ is the entire forward process. Code-wise that’s the four lines of DDPM.q_sample: index ᾱ at the chosen t, multiply x0 by its square root, add √(1 − ᾱt) · noise.
  2. Training picks t uniformly. t ~ U{0, …, T−1} in DDPM.loss. We don’t weight by anything — that’s a deliberate choice (the “L_simple” choice) we’ll defend in lesson 3.
  3. The forward process is not learned. It’s a fixed schedule baked in as buffers (register_buffer in diffusion.py). Only the denoiser’s weights are trained.
Punchline
We have built a path: q(xt | x0) = N(√ᾱt x0, (1 − ᾱt) I). It costs nothing to sample any point on it. Next lesson: invert the path. Given xt, what is xt−1 — and what should the net learn to predict to make that reversal work?