Did it work? Evaluation & the GAN contrast
Every lesson so far has said “better samples” without measuring. Here is how IS and FID actually quantify it — and what they miss. Then the model diffusion dethroned: GANs, mode collapse, and why the trade was worth it.
The measurement problem
We never have pdata in closed form — only samples (lesson 1). So “is the generated distribution close to the real one” can’t be a likelihood for most models (and for GANs there is none). The field’s answer: push both real and fake images through a frozen Inception-v3 classifier and compare them in feature space, where “looks like a real image” becomes a measurable statistic. Two metrics do this differently.
Inception Score (IS): sharp and varied, from one network
IS uses only generated images and the classifier’s label distribution p(y|x). It rewards two things at once:
- Sharpness — each image should be confidently one class: p(y|x) low-entropy (peaked).
- Diversity — across all images, every class should appear: the marginal p(y) = 𝔼x p(y|x) high-entropy (uniform).
Both at once = each conditional far from the marginal = high KL between them, averaged and exponentiated:
- It never looks at real data — only at the classifier’s opinions about fakes. It cannot tell you the fakes resemble this dataset, only that they trip the classifier crisply across classes.
- Intra-class mode collapse is invisible. Generate one perfect image per ImageNet class, copied 10× each: peaked p(y|x), uniform p(y) ⇒ great IS, zero within-class diversity.
- Tied to ImageNet classes — meaningless on data unlike ImageNet, and blind to overfitting (memorized training images score perfectly).
Fréchet Inception Distance (FID): compare to the real data
FID fixes IS’s biggest hole — it compares fakes against reals. The whole idea is simple even though the formula looks heavy: run both sets of images through Inception, and each image becomes a point in a 2048-dimensional “what-it-looks-like” space. The real images form one cloud of points; the fakes form another. Summarize each cloud by where its center sits and how it’s spread out — i.e. fit one Gaussian blob to each — then ask: how far apart are these two blobs? The closed-form answer is the Fréchet (Wasserstein-2) distance between two Gaussians:
The formula is just “distance between two blobs,” in two parts. First term: how far apart are the two centers (do fakes have the right average content)? Second term: how differently are the two clouds spread out (do fakes have the right diversity)? Mode collapse shrinks the fake cloud — its spread Σg drops — which inflates the trace term, so unlike IS, FID catches collapse. This is why FID is the field standard and why the diffusion lessons’ “better samples” ultimately means “lower FID at matched compute.”
Claim: the scary FID formula is just the distance between two clouds of points in feature space — one cloud of real images, one of fakes.
- Images become points. Inception turns each image into 2048 numbers. “Looks like a real photo” is now a location in that space, and a whole dataset is a cloud of locations.
- Summarize each cloud by two facts. Where is it centered (the mean μ), and how fat / which way is it stretched (the covariance Σ)? That’s all a Gaussian blob is — a center plus a spread.
- Compare the two blobs, piece by piece. The ‖μr − μg‖² term measures the gap between the two centers; the trace term measures the gap between the two spreads. Both zero only when the fake blob sits exactly where the real one does and is shaped the same.
- Why collapse shows up. If the generator only makes a few looks, its cloud is tiny and tight. Even if its center is right, its spread no longer matches the real cloud’s — the trace term grows and FID rises.
Central point. Lower FID = the two point-clouds sit in the same place and are shaped the same way; the matrix algebra is only the bookkeeping for “same center, same spread.”
- Gaussian assumption — real Inception features aren’t actually Gaussian; FID only matches first two moments.
- No spatial reasoning — it scores feature presence, not arrangement. A face with the mouth above the eyes can still score well if the right features exist.
- Overfitting blind — copying the training set gives a near-zero FID. (Hence precision/recall metrics and CLIP-score as complements.)
- Sample-size biased — FID drifts with the number of samples; comparisons must fix N (typically 50k).
For text-to-image specifically you also want CLIP-score — cosine similarity between the prompt’s CLIP-text embedding and the image’s CLIP-image embedding (lesson 20) — because FID measures realism, not prompt adherence. Production reports usually plot the FID–CLIP trade-off as you sweep guidance scale (lesson 18).
The model diffusion beat: GANs
A GAN (Goodfellow 2014) is two networks in a game: a generator G maps noise z∼N(0,I) to an image; a discriminator D tries to tell real from fake. The single line below is just “these two fight” written as math: D wants to push the expression up (score reals near 1, fakes near 0), G wants to push it down (fool D into scoring its fakes near 1). The maxD says “D plays first and best”; the minG says “G then tries to undo that.”
No likelihood, no slow sampling — one forward pass through G makes an image. That speed was the appeal. The cost is the contrast with everything in Part A:
Claim: the GAN game is unstable because G chases a moving target, while diffusion just regresses against a fixed answer key.
- Diffusion has a fixed grader. Its loss ‖ε − εθ‖² compares the model’s guess to the actual noise that was added — a known, unchanging right answer. “Better” always means the same thing, so the target never wobbles.
- A GAN has no fixed answer key. G’s only feedback is D’s opinion — and D is itself being trained. The thing G is graded against keeps changing every step.
- Two players chasing each other. G learns to beat today’s D; D then adapts to beat that G. Neither sits still, so the loss can oscillate forever instead of settling — like two people leaning to counter each other and toppling.
- And nothing demands coverage. The objective only asks G to fool D. If one image reliably fools it, there is no term saying “also make the rest of the data” — the seed of mode collapse.
Central point. Stable regression against a known target (diffusion) is a far calmer thing to optimize than a tug-of-war against an opponent who keeps moving the goalposts (GAN).
| GAN | Diffusion | |
|---|---|---|
| Training | adversarial minimax — unstable, oscillates, needs careful balancing | plain regression (lesson 3) — stable, just MSE |
| Sampling | 1 forward pass — fast | many steps (lesson 17) — slower, but distillable |
| Diversity | prone to mode collapse | covers modes (the loss touches all data) |
| Likelihood | none | ELBO / exact via PF-ODE (lesson 16) |
Mode collapse: the GAN failure diffusion avoids by construction
Mode collapse is the generator dumping all its mass on a few outputs that reliably fool D, abandoning whole regions of pdata. The mechanism: G only ever receives gradient through D; if it finds one image D can’t reject, the minimax has no term forcing coverage of other modes — nothing in the objective says “also generate the rest.” Diffusion can’t do this: its loss is 𝔼x₀∼data‖ε−εθ‖² over the real data, so every mode contributes gradient every step.
The standard list of fixes, with the lever each pulls:
| Fix | Lever |
|---|---|
| WGAN (Wasserstein distance) | replace JS divergence (which gives vanishing gradient when supports don’t overlap) with Earth-Mover distance — smoother signal |
| WGAN-GP (gradient penalty) | enforce the 1-Lipschitz constraint WGAN needs, without weight clipping |
| Minibatch discrimination / std | let D see batch-level variety, so a collapsed batch is detectable |
| Unrolled GAN | let G optimize against D’s future steps, so it can’t exploit a momentary D |
| Experience replay, multiple Gs, one-sided label smoothing | various ways to stop D from being exploited or over-confident |
Interactive · FID and mode collapse on a toy
The target is a ring of 8 Gaussian modes (the classic mode-collapse benchmark). Dial a generator from “full coverage” to “collapsed onto 2 modes” and watch the metrics: the toy-FID (computed in raw 2D here, same formula) rises as coverage drops because the covariance term Tr(Σr+Σg−2(ΣrΣg)½) blows up — FID sees the collapse. A naive “mean distance only” score (mimicking IS’s blind spot) barely moves.