Generative vision

生成式视觉

Lesson 16 added time but stayed in analysis: every model so far reads pixels and emits a label, a box, a track, or an embedding. Now we reverse the arrow. A generative model takes a label, a caption, or pure noise and produces pixels. This is not classification run backward — it is a genuinely harder problem, and the reasons it is hard explain every architecture in the family, from the blur of a VAE to the clean samples of diffusion.

第 16 课加入了时间，但仍停留在分析层面：迄今为止的每个模型都是读入像素、输出一个标签、边界框、轨迹或嵌入向量。现在我们把箭头反过来。生成模型接受一个标签、一句描述或纯噪声，然后产出像素。这不是把分类反着跑一遍——它是一个实实在在更难的问题，而它之所以难的原因，解释了这一族里的每一种架构，从 VAE 的模糊到扩散模型干净利落的采样。

The plan计划

Five moves. (1) Why modeling p(x) differs from p(y|x) — and why "many valid outputs" is the central difficulty. (2) Autoencoder → VAE: the reparameterization trick, and the correct reason VAE samples blur. (3) GANs: the minimax game and mode collapse. (4) Diffusion done properly: forward noising, learning to predict ε, DDPM vs DDIM, latent diffusion, classifier-free guidance, ControlNet, and the frontier (flow matching, consistency distillation). (5) FID — what two distributions it compares — and safety. 五步走。(1) 为什么建模 p(x) 不同于 p(y|x)——以及为什么"存在许多有效的输出"是核心难点。(2) 自编码器 → VAE：重参数化技巧，以及 VAE 采样会模糊的正确原因。(3) GAN：极小极大博弈与模式坍缩。(4) 把扩散模型讲透：前向加噪、学习预测 ε、DDPM 与 DDIM、潜在扩散、无分类器引导、ControlNet，以及前沿方向（流匹配、一致性蒸馏）。(5) FID——它比较的是哪两个分布——以及安全性。

1 · Generation is not classification backward

1 · 生成不是把分类反过来

A classifier estimates p(y|x): given an image, a distribution over a handful of labels. The output space is tiny and the answer is usually nearly unique — one image, one correct class. A generative model estimates p(x) (or, conditioned, p(x|c) for a caption or class c): a distribution over the entire space of images. For a 256×256 RGB image that space has 256·256·3 ≈ 196,000 dimensions — one coordinate per pixel-channel — so the cube of all possible such images holds 256¹⁹⁶⁶⁰⁸ points, an unimaginably large number. The model must place probability mass on the vanishingly thin manifold — a low-dimensional curved surface — of plausible images sitting inside that enormous cube, and be able to sample from it.

分类器估计的是 p(y|x)：给定一张图像，输出在少数几个标签上的分布。输出空间极小，而答案通常近乎唯一——一张图像，一个正确的类别。生成模型估计的是 p(x)（或者带条件的 p(x|c)，其中 c 是一句描述或一个类别）：一个覆盖整个图像空间的分布。对一张 256×256 的 RGB 图像来说，这个空间有 256·256·3 ≈ 196,000 个维度——每个像素通道对应一个坐标——所以所有可能图像构成的超立方体里有 256¹⁹⁶⁶⁰⁸ 个点，一个大到无法想象的数字。模型必须把概率质量放在这个巨大立方体内部那层薄得几乎没有厚度的流形（manifold，一个低维的弯曲曲面）上——也就是看起来合理的图像所在之处——并且能从中采样。

The defining property is multimodality: for one condition, many distinct outputs are all correct. "A dog on grass" describes millions of genuinely different images — breeds, poses, lighting, framing. A classifier maps many inputs to one label (a many-to-one collapse, which is easy); a generator must map one condition to many outputs (a one-to-many expansion, which is hard). This single fact — that p(x|c) has many separated modes (distinct clusters of valid images: a dog facing left is one mode, facing right another, each a separate peak of probability) — is the thread running through everything below. It is why a model that "averages" its options produces mush, why GANs can drop modes, and why we needed a process that can commit to one sharp sample.

其决定性的性质是多模态性（multimodality）：对同一个条件，许多截然不同的输出都是正确的。"草地上的一只狗"描述了数以百万计确实各不相同的图像——不同的品种、姿态、光照、构图。分类器把许多输入映射到一个标签（一种多对一的坍缩，这很容易）；而生成器必须把一个条件映射到许多输出（一种一对多的展开，这很难）。这一个事实——p(x|c) 拥有许多彼此分离的模式（mode，即一簇簇不同的有效图像：朝左的狗是一个模式，朝右的又是另一个，每一个都是概率的一个独立峰值）——是贯穿下文一切内容的线索。它解释了为什么一个对各个选项做"平均"的模型会产出一团糊、为什么 GAN 会丢掉模式，以及为什么我们需要一个能够果断地选定一个清晰样本的过程。

2 · Autoencoder → VAE, and why VAEs blur

2 · 自编码器 → VAE，以及 VAE 为何模糊

Start with the autoencoder: an encoder f compresses an image to a low-dimensional code z, a decoder g reconstructs it, and you train to minimize reconstruction error ‖x − g(f(x))‖². The bottleneck forces the network to keep only what matters. But a plain autoencoder is not generative: there is no rule for which codes z are valid, so sampling a random z and decoding gives garbage. The latent space has holes.

先从自编码器（autoencoder）说起：编码器 f 把一张图像压缩成一个低维编码 z，解码器 g 把它重建出来，训练目标是最小化重建误差 ‖x − g(f(x))‖²。这个瓶颈迫使网络只保留重要的东西。但普通的自编码器不是生成式的：没有任何规则规定哪些编码 z 是有效的，所以随机采一个 z 再解码只会得到一团垃圾。潜在空间里满是空洞。

The variational autoencoder (VAE) fixes this by making the encoder output a distribution instead of a point — a Gaussian q(z|x) = 𝒩(μ(x), σ²(x)) — and pulling that distribution toward a fixed prior p(z) = 𝒩(0, I). Now the latent space is dense and samplable: draw z ∼ 𝒩(0, I), decode, get a plausible image. We would like to maximize the data likelihood log p(x), but it is intractable to compute; the training objective is instead the evidence lower bound (ELBO) — a tractable lower bound on that likelihood (hence the name), so pushing the ELBO up pushes the true likelihood up. It splits into two readable terms:

变分自编码器（VAE）修复了这一点：它让编码器输出一个分布而不是一个点——一个高斯分布 q(z|x) = 𝒩(μ(x), σ²(x))——并把这个分布往一个固定的先验 p(z) = 𝒩(0, I) 上拉。现在潜在空间是稠密且可采样的：采一个 z ∼ 𝒩(0, I)，解码，就能得到一张看起来合理的图像。我们希望最大化数据似然 log p(x)，但它无法直接计算；于是训练目标改用证据下界（ELBO）——它是那个似然的一个可计算的下界（名字由此而来），所以把 ELBO 往上推就等于把真实似然往上推。它拆成两个易读的项：

ℒ = 𝔼_q(z|x)[ log p(x|z) ] − D_KL( q(z|x) ‖ p(z) ) .

The first term is reconstruction (decode z, score how well it matches x); the second is the Kullback–Leibler divergence (KL — a measure of how far one distribution is from another) that regularizes the encoder's posterior toward the unit-Gaussian prior so the latent space has no holes.

第一项是重建（把 z 解码，评估它与 x 匹配得有多好）；第二项是 Kullback–Leibler 散度（KL——衡量一个分布离另一个分布有多远的度量），它把编码器的后验往单位高斯先验上正则化，使潜在空间没有空洞。

The reparameterization trick. There is a problem: we need to backpropagate through the sampling step z ∼ 𝒩(μ, σ²), but sampling is not differentiable — you cannot take a gradient through a coin flip. The trick moves the randomness outside the network. Instead of sampling z directly, sample a fixed noise ε ∼ 𝒩(0, I) and compute

重参数化技巧。这里有个问题：我们需要对采样步骤 z ∼ 𝒩(μ, σ²) 做反向传播，但采样是不可微的——你没法对一次抛硬币求梯度。这个技巧把随机性挪到了网络外面。不直接采样 z，而是采一个固定的噪声 ε ∼ 𝒩(0, I)，然后计算

z = μ(x) + σ(x) ⊙ ε .

Now z is a deterministic, differentiable function of the network outputs μ, σ, with all the randomness parked in ε, which carries no parameters. Gradients flow through μ and σ cleanly; the sampling is just an input. That one re-expression is what makes the VAE trainable end to end by ordinary backprop.

现在 z 是网络输出 μ, σ 的一个确定性、可微函数，所有随机性都停放在 ε 里，而 ε 不携带任何参数。梯度干净利落地流经 μ 和 σ；采样只是一个输入而已。正是这一次重新表达，使得 VAE 能用普通的反向传播端到端地训练。

Why VAE samples blur — the real reasonVAE 采样为何模糊——真正的原因

The common explanation, "the KL term dominates the likelihood," is wrong as a general account. The actual culprit is the reconstruction term itself. The standard decoder models each pixel as an independent Gaussian, so log p(x|z) reduces to a per-pixel L2 (squared-error) loss. And the L2-optimal prediction under uncertainty is the mean of all valid outputs. Recall §1: given a code, many sharp images are plausible (a dog could face left or right). L2 reconstruction does not pick one — it averages them, and the average of two sharp images is a blurry one. This is mean-of-modes blur, baked into the per-pixel Gaussian likelihood. A secondary contributor is the limited/approximate posterior (a diagonal Gaussian q(z|x) can't capture the true latent structure), which leaves residual uncertainty that the mean-seeking decoder again blurs over. Fixes attack the likelihood: a perceptual or adversarial reconstruction loss (VQ-GAN), or richer decoders. 常见的解释"KL 项压过了似然"作为一般性说法是错的。真正的元凶是重建项本身。标准解码器把每个像素建模成一个独立的高斯，于是 log p(x|z) 退化成逐像素的 L2（平方误差）损失。而在不确定性下，L2 最优的预测是所有有效输出的均值。回想第 1 节：给定一个编码，许多清晰的图像都合理（狗可能朝左也可能朝右）。L2 重建不会从中挑一个——它把它们平均掉，而两张清晰图像的平均是一张模糊的。这就是模式均值模糊（mean-of-modes blur），它被烙进了逐像素高斯似然里。次要的成因是受限的/近似的后验（对角高斯 q(z|x) 无法刻画真实的潜在结构），它留下的残余不确定性又被追求均值的解码器抹糊了。修复手段都是从似然入手：换用感知或对抗式的重建损失（VQ-GAN），或者更强大的解码器。

3 · GANs: a minimax game, and mode collapse

3 · GAN：极小极大博弈与模式坍缩

If averaging modes causes blur, what if we train a loss that punishes blur directly? That is the GAN (generative adversarial network) idea. Two networks compete. A generator G maps noise z to an image; a discriminator D tries to tell real images from G's fakes. They play a minimax game with value

如果对模式求平均会导致模糊，那么如果我们训练一个直接惩罚模糊的损失会怎样？这就是 GAN（生成对抗网络）的思路。两个网络相互竞争。一个生成器 G 把噪声 z 映射成一张图像；一个判别器 D 试图把真实图像和 G 造的假图区分开。它们进行一场极小极大博弈，其价值函数为

min_G max_D 𝔼_x∼data[ log D(x) ] + 𝔼_z[ log(1 − D(G(z))) ] .

D maximizes its accuracy at spotting fakes; G minimizes that, i.e. learns to fool D. At the ideal equilibrium D cannot do better than 50/50 and G's outputs are indistinguishable from real. There is no averaging loss anywhere — a blurry sample is an easy sample for D to flag, so the adversarial pressure pushes G toward sharp, realistic detail. That is why GANs (StyleGAN) produced the crispest faces of their era.

D 最大化自己识别假图的准确率；G 则最小化它，也就是学着去骗过 D。在理想的均衡处，D 做不到比五五开更好，而 G 的输出与真图无法区分。这里没有任何求平均的损失——一张模糊的样本对 D 来说是很容易识破的样本，所以对抗压力会把 G 推向清晰、逼真的细节。这就是为什么 GAN（StyleGAN）造出了那个年代最锐利的人脸。

The characteristic failure is mode collapse, and it is the dark twin of §1's multimodality. G only needs to fool D; it is never rewarded for coverage. So it can discover one or a few outputs that reliably fool D and emit only those — generating gorgeous but nearly identical faces, ignoring whole modes of the real distribution. Combined with unstable, oscillating training (two networks chasing a moving target, no single loss to monitor), GANs are powerful but fiddly. We want sharpness and stable mode coverage. Diffusion delivers both.

它标志性的失败是模式坍缩（mode collapse），它是第 1 节多模态性的阴暗孪生。G 只需要骗过 D；它从不因覆盖度而得到奖励。于是它可能发现一个或少数几个能可靠骗过 D 的输出，然后只产出这些——生成一批漂亮却几乎一模一样的人脸，无视真实分布里一整块一整块的模式。再加上训练不稳定、来回震荡（两个网络在追一个移动的目标，没有单一的损失可供监控），GAN 强大却难伺候。我们想要的是清晰加上稳定的模式覆盖。扩散模型两者都能给。

4 · Diffusion: destroy structure slowly, then learn to undo it

4 · 扩散：缓慢地摧毁结构，再学着把它撤销

Diffusion turns generation into a sequence of small, easy denoising steps. It has two processes.

扩散把生成变成一连串微小而容易的去噪步骤。它有两个过程。

The forward process is fixed and requires no learning. Start from a real image x₀ and add a little Gaussian noise repeatedly over T steps (T is often a few hundred to a thousand), each step governed by a variance schedule β_t:

前向过程是固定的，无需学习。从一张真实图像 x₀ 出发，在 T 步里反复加入少量高斯噪声（T 通常是几百到一千），每一步都由一个方差调度 β_t 控制：

x_t = √(1 − β_t) · x_t−1 + √β_t · ε , ε ∼ 𝒩(0, I) .

By the end, x_T is indistinguishable from pure Gaussian noise — all structure destroyed. A convenient identity lets us jump to any step directly: with α_t = 1 − β_t and ᾱ_t = ∏_s≤t α_s,

到最后，x_T 与纯高斯噪声无法区分——结构全部被摧毁。一个方便的恒等式让我们能直接跳到任意一步：令 α_t = 1 − β_t 且 ᾱ_t = ∏_s≤t α_s，

x_t = √ᾱ_t · x₀ + √(1 − ᾱ_t) · ε .

(Drag the slider in the widget below to watch x_t degrade from signal to noise as t runs 0→T — that is this exact equation.)

（拖动下面小部件里的滑块，看着 x_t 随 t 从 0→T 由信号退化成噪声——那就是这个方程本身。）

The reverse process is what we learn. If we could undo each noising step, we could start from pure noise x_T ∼ 𝒩(0,I) and walk back to a clean image x₀ — i.e. sample. The key realization: the forward equation tells us exactly which noise ε was added to make x_t from x₀. So we train a network ε_θ(x_t, t) to predict that noise, with a dead-simple loss:

反向过程才是我们要学的东西。如果我们能把每一步加噪撤销，就可以从纯噪声 x_T ∼ 𝒩(0,I) 出发、一路走回一张干净的图像 x₀——也就是采样。关键的洞察是：前向方程精确地告诉了我们，是哪个噪声 ε 把 x₀ 加成了 x_t。于是我们训练一个网络 ε_θ(x_t, t) 去预测那个噪声，损失简单到不能再简单：

ℒ = 𝔼_{x₀, t, ε} ‖ ε − ε_θ(x_t, t) ‖² .

Take a real image, pick a random step t, noise it with a known ε, ask the network to recover ε, and penalize the error. This is an L2 loss — but notice it does not suffer the mean-of-modes blur of the VAE, because it predicts the noise at a single small step, not the whole image in one shot; the multimodal choice is spread across hundreds of tiny, locally near-unimodal steps.

取一张真实图像，随机挑一个步数 t，用一个已知的 ε 给它加噪，让网络把 ε 还原出来，并惩罚误差。这确实是一个 L2 损失——但注意它并不会像 VAE 那样陷入模式均值模糊，因为它预测的是单个微小步骤上的噪声，而不是一步到位地预测整张图像；多模态的选择被分摊到了几百个微小、局部近乎单模态的步骤上。

At sampling time, start from noise x_T and take one reverse step at a time. The DDPM update is the forward identity solved for x_t−1, using the network's noise prediction ε_θ:

采样时，从噪声 x_T 出发，一次走一个反向步骤。DDPM 的更新式就是把前向恒等式解出 x_t−1，并用上网络的噪声预测 ε_θ：

x_t−1 = (1/√α_t) ( x_t − ((1−α_t) / √(1−ᾱ_t)) · ε_θ(x_t, t) ) + σ_t z , z ∼ 𝒩(0, I) .

In words: predict the noise ε_θ(x_t, t) currently in the image, subtract a scaled portion of it to remove a little of the corruption, rescale by 1/√α_t, and then add back a small dose of fresh noise σ_t z — which keeps the reverse process stochastic so it can reach different modes — except at the very last step (t=1), where you drop the σ_t z term and return the clean x₀. Iterating this from t=T down to 1 climbs back from noise to a sharp x₀.

用大白话说：预测图像里当前含有的噪声 ε_θ(x_t, t)，减去其中一个按比例缩放的部分以去掉一点点污染，再乘以 1/√α_t 重新缩放，然后加回一小剂新鲜噪声 σ_t z——它让反向过程保持随机性，从而能够到达不同的模式——只有在最后一步（t=1）例外，那时你去掉 σ_t z 这一项、直接返回干净的 x₀。从 t=T 一直迭代到 1，就能从噪声一路爬回一张清晰的 x₀。

DDPM vs DDIM. The original sampler, DDPM (denoising diffusion probabilistic model), runs the full stochastic reverse chain — high quality but slow (hundreds of network calls per image). DDIM (denoising diffusion implicit model) reformulates the reverse process as deterministic (no re-injected noise), which lets you skip steps — sample in 20–50 calls instead of 1000, and makes the mapping noise→image reproducible (useful for editing and interpolation). Same trained network; different sampler.

DDPM 与 DDIM。最初的采样器 DDPM（去噪扩散概率模型，denoising diffusion probabilistic model）跑的是完整的随机反向链——质量高但慢（每张图要几百次网络调用）。DDIM（去噪扩散隐式模型，denoising diffusion implicit model）把反向过程重新表述为确定性的（不再注入噪声），这让你可以跳步——用 20–50 次调用而不是 1000 次就能采样，并且使噪声→图像的映射可复现（对编辑和插值很有用）。同一个训练好的网络，不同的采样器。

Latent diffusion (Stable Diffusion). Diffusing in raw 512×512 pixel space is expensive — every reverse step is a full-resolution forward pass. Latent diffusion first trains a VAE/VQ autoencoder (§2) to compress images into a small latent grid (e.g. 64×64), then runs the entire diffusion process in that latent space, decoding to pixels only at the end. The diffusion model never touches full-resolution pixels, cutting compute by the spatial compression factor squared. This is the architecture behind Stable Diffusion: VAE encoder → latent diffusion U-Net → VAE decoder. (The U-Net here is the encoder-decoder with skip connections from lesson 10 — it ingests the noisy latent and outputs the predicted noise ε_θ at the same resolution.)

潜在扩散（Stable Diffusion）。在原始的 512×512 像素空间里做扩散很昂贵——每个反向步骤都是一次全分辨率的前向传播。潜在扩散先训练一个 VAE/VQ 自编码器（第 2 节）把图像压缩成一个小的潜在网格（例如 64×64），然后在那个潜在空间里跑完整个扩散过程，只在最后才解码回像素。扩散模型从不接触全分辨率像素，把算力开销削减了空间压缩因子的平方倍。这就是 Stable Diffusion 背后的架构：VAE 编码器 → 潜在扩散 U-Net → VAE 解码器。（这里的 U-Net 就是第 10 课那个带跳连的编码器-解码器——它吃进带噪的潜在表示，输出同分辨率的预测噪声 ε_θ。）

Conditioning and guidance. Text-to-image needs p(x|c), not p(x). The caption is encoded (a CLIP/T5 text encoder from lesson 15) and injected into the denoiser via cross-attention — lesson 13's attention mechanism run across two modalities, where the text tokens supply the keys and values and the image features supply the queries, so each image region looks up the words most relevant to it. To make the model obey the prompt strongly, classifier-free guidance (CFG) is the standard lever: during training, randomly drop the condition so the same network learns both conditional ε_θ(x_t, c) and unconditional ε_θ(x_t) prediction. At sampling, extrapolate away from the unconditional toward the conditional with a guidance scale w:

条件与引导。文生图需要的是 p(x|c) 而不是 p(x)。文字描述被编码（第 15 课的 CLIP/T5 文本编码器），并通过交叉注意力（cross-attention）注入去噪器——也就是第 13 课的注意力机制跨两种模态运行，其中文本 token 提供键和值、图像特征提供查询，于是每个图像区域都去检索与自己最相关的词。为了让模型强烈地服从提示词，无分类器引导（CFG）是标准的调节手段：训练时随机丢弃条件，让同一个网络既学会有条件的预测 ε_θ(x_t, c)，也学会无条件的预测 ε_θ(x_t)。采样时，用一个引导强度 w 从无条件向有条件的方向做外推：

ε̂ = ε_θ(x_t) + w · ( ε_θ(x_t, c) − ε_θ(x_t) ) .

Read the equation as extrapolation: it starts at the unconditional estimate and steps w× along the direction (conditional − unconditional). Worked numbers at the Stable Diffusion default w = 7.5: w = 1 recovers plain conditional sampling ε̂ = ε_θ(x_t, c) (no guidance — the bracket is added back exactly once); and if the conditional estimate sits +0.4 past the unconditional along some noise coordinate, then w = 7.5 pushes that coordinate 7.5 · 0.4 = +3.0 — a 7.5× exaggeration of the "what the prompt wants" direction far beyond where the conditional model alone would land. Higher w means tighter prompt adherence but less diversity and more saturation/artifacts — a direct knob on the §1 fidelity-vs-diversity trade. ControlNet adds spatial conditioning: a parallel copy of the encoder ingests an edge map, depth map, or pose skeleton and injects it into the frozen base model, so you can say "this layout, that prompt" — controllable generation without retraining the whole network.

把这个方程读成外推：它从无条件估计出发，沿着（有条件 − 无条件）这个方向走 w 倍的步长。用 Stable Diffusion 默认值 w = 7.5 算几个具体数：w = 1 恢复成普通的有条件采样 ε̂ = ε_θ(x_t, c)（没有引导——括号里的项恰好被加回一次）；而如果有条件估计在某个噪声坐标上比无条件多出 +0.4，那么 w = 7.5 会把那个坐标推到 7.5 · 0.4 = +3.0——把"提示词想要什么"这个方向放大了 7.5 倍，远远超出有条件模型单独会落到的位置。w 越高意味着越贴合提示词，但多样性越低、越容易过饱和/出现伪影——这是直接调节第 1 节保真度-多样性权衡的旋钮。ControlNet 加入了空间条件：一份并行的编码器副本吃进边缘图、深度图或姿态骨架，再把它注入到冻结的基础模型里，于是你可以说"就用这个布局、配那句提示词"——在不重训整个网络的情况下实现可控生成。

The frontier. Flow matching / rectified flow replace the noising chain with a learned straight-line transport from noise to data, training the network to predict a velocity field; the straighter paths sample in far fewer steps and underlie recent systems (SD3, Flux). Consistency models / distillation attack speed directly — distill a slow multi-step diffusion teacher into a student that maps noise to image in 1–4 steps, approaching real-time generation. Both target diffusion's one real weakness: iterative sampling is slow.

前沿方向。流匹配 / 修正流（flow matching / rectified flow）用一条学习出来的、从噪声到数据的直线传输取代加噪链，训练网络去预测一个速度场；更直的路径能用少得多的步数采样，也支撑起了近期的一些系统（SD3、Flux）。一致性模型 / 蒸馏（consistency models / distillation）则直接向速度开刀——把一个慢速的多步扩散教师蒸馏成一个学生，让它用 1–4 步就把噪声映射成图像，逼近实时生成。两者瞄准的都是扩散唯一真正的弱点：迭代采样太慢。

Family族	Core mechanism核心机制	Strength优势	Characteristic failure标志性失败
Autoencoder自编码器	compress + reconstruct (L2)压缩 + 重建（L2）	simple, learns codes简单，能学到编码	not generative; holey latent space不是生成式；潜在空间有空洞
VAE	probabilistic encoder + ELBO, reparam. trick概率编码器 + ELBO、重参数化技巧	samplable latent, stable潜在空间可采样，稳定	mean-of-modes blur (per-pixel L2)模式均值模糊（逐像素 L2）
GAN	generator vs discriminator minimax生成器与判别器的极小极大	sharp, fast 1-step sampling清晰，单步采样快	mode collapse, unstable training模式坍缩，训练不稳定
Diffusion (DDPM/DDIM)扩散（DDPM/DDIM）	learn to predict added noise ε学习预测加入的噪声 ε	sharp and diverse, stable既清晰又多样，稳定	slow iterative sampling迭代采样慢
Latent diffusion潜在扩散	diffuse in a VAE latent在 VAE 潜在空间里做扩散	cheap high-res (Stable Diffusion)廉价的高分辨率（Stable Diffusion）	VAE decoder artifacts / lost fine detailVAE 解码器伪影 / 丢失细节

5 · Evaluating generators — FID — and safety

5 · 评估生成器——FID——以及安全性

You cannot just look at loss: a low diffusion loss does not mean good images, and there is no single ground-truth image to compare against (§1). The standard automatic metric is FID — Fréchet Inception Distance. It compares two distributions: the set of real images and the set of generated images. Both are pushed through a fixed pretrained Inception network to get feature vectors; each set's features are summarized as a Gaussian (mean μ, covariance Σ); FID is the Fréchet distance between those two Gaussians:

你不能只看损失：扩散损失低并不代表图像好，而且根本没有单一的真值图像可供比较（第 1 节）。标准的自动指标是 FID——Fréchet Inception 距离（Fréchet Inception Distance）。它比较的是两个分布：真实图像的集合与生成图像的集合。两者都被送过一个固定的预训练 Inception 网络以得到特征向量；每个集合的特征都被概括成一个高斯（均值 μ、协方差 Σ）；FID 就是这两个高斯之间的 Fréchet 距离：

FID = ‖μ_r − μ_g‖² + Tr( Σ_r + Σ_g − 2(Σ_r Σ_g)^1/2 ) .

Lower is better (0 = identical feature distributions). FID rewards both fidelity (samples look like real features) and diversity (the spread Σ_g matches real spread) — crucially, a mode-collapsed GAN has too-small Σ_g and is penalized, which a per-sample quality score would miss. But FID is incomplete: it is blind to prompt adherence (use CLIPScore — image-text similarity from lesson 15), insensitive to a small number of bad samples, and dependent on sample count and the Inception feature space. Production evaluation layers on human preference, prompt-following rates, artifact/NSFW rates, and — for synthetic augmentation data — the only honest test: does generated data improve performance on real held-out validation? Impressive-looking samples that don't move the real metric are worthless for that use.

越低越好（0 = 特征分布完全相同）。FID 同时奖励保真度（样本看起来像真实的特征）和多样性（散布 Σ_g 与真实的散布相匹配）——关键在于，一个发生模式坍缩的 GAN 会有过小的 Σ_g 从而被惩罚，而这是逐样本的质量评分会漏掉的。但 FID 并不完整：它对提示词的贴合程度视而不见（要用 CLIPScore——第 15 课的图文相似度），对少量坏样本不敏感，且依赖于样本数量和 Inception 的特征空间。生产环境的评估还要叠加人类偏好、提示词遵循率、伪影/NSFW 比率，而对于合成的增强数据，唯一诚实的检验是：生成的数据能否提升在真实留出验证集上的表现？那些看起来惊艳、却撼动不了真实指标的样本，对这个用途而言毫无价值。

Safety is not optional for generators对生成器而言，安全性不是可选项

Analysis models can be wrong; generative models can be harmful. The risk surface is new: non-consensual imagery and deepfakes, memorization and regurgitation of training images (copyright and privacy leakage), demographic bias amplified into every sample, and CSAM. Standard mitigations: prompt and output safety classifiers, dataset filtering, concept erasure / unlearning, invisible watermarking and provenance (C2PA) for attribution, and rate/identity controls. Treat safety as part of the system spec — which is exactly the systems mindset lesson 19 formalizes. 分析型模型可能会出错；生成型模型则可能造成危害。它的风险面是全新的：未经同意的图像与深度伪造、对训练图像的记忆与逐字吐出（版权与隐私泄露）、被放大进每一个样本的人群偏见，以及儿童性虐待材料（CSAM）。标准的缓解手段有：提示词与输出的安全分类器、数据集过滤、概念擦除 / 遗忘（unlearning）、用于溯源归属的隐形水印与出处标准（C2PA），以及速率/身份管控。把安全性当作系统规格的一部分——这正是第 19 课要形式化的那种系统思维。

Interactive · the forward-noising process

互动 · 前向加噪过程

Drag t from 0 to T and watch a toy 1-D signal degrade to Gaussian noise via the closed-form forward equation x_t = √ᾱ_t x₀ + √(1−ᾱ_t) ε. Early on, the clean structure dominates; past the midpoint, noise drowns it; at t=T the signal is gone — pure 𝒩(0,I). This destruction is the easy half. Diffusion's entire job is to train a network that runs this slider backward, predicting the ε at each step. KPIs report the two mixing coefficients so you can see the signal/noise balance tip.

把 t 从 0 拖到 T，看着一个玩具级的一维信号通过闭式前向方程 x_t = √ᾱ_t x₀ + √(1−ᾱ_t) ε 退化成高斯噪声。一开始，干净的结构占主导；过了中点，噪声把它淹没；到 t=T 时信号已荡然无存——纯粹的 𝒩(0,I)。这场摧毁是容易的那一半。扩散模型的全部工作，就是训练一个网络把这个滑块反着跑，在每一步预测那个 ε。KPI 显示两个混合系数，让你看到信号/噪声的天平如何倾斜。

Forward diffusion — drag t from 0 (clean) to T (pure noise)

前向扩散——把 t 从 0（干净）拖到 T（纯噪声）

Blue = signal contribution √ᾱ_t·x₀; gray = the noisy result x_t. As t rises, ᾱ_t → 0, the blue collapses, and x_t becomes indistinguishable from Gaussian noise. Reverse this and you have a generator. This widget shows only the forward (noising) process — the easy half; the reverse process (the actual generation) needs a trained network and is not interactive here. Note the widget defines ᾱ_t directly via a cosine schedule (rather than from per-step β_t as in the prose), so don't expect the code's ᾱ values to match a specific β_t chain.

蓝色 = 信号贡献 √ᾱ_t·x₀；灰色 = 加噪后的结果 x_t。随着 t 上升，ᾱ_t → 0，蓝色坍缩，x_t 变得与高斯噪声无法区分。把它反过来，你就有了一个生成器。这个小部件只展示前向（加噪）过程——容易的那一半；反向过程（真正的生成）需要一个训练好的网络，此处不可交互。注意本部件通过一个余弦调度直接定义 ᾱ_t（而不是像正文那样由逐步的 β_t 推出），所以不要指望代码里的 ᾱ 值能与某个特定的 β_t 链对应上。

step t: 步数 t: 0 / 1000

step t

步数 t

√ᾱ_t (signal)

√ᾱ_t（信号）

1.000

√(1−ᾱ_t) (noise)

√(1−ᾱ_t)（噪声）

0.000

state

状态

clean signal

Show the core JS (≈14 lines)查看核心代码（约 14 行）

// cosine schedule for cumulative ᾱ_t (not per-step β_t) over T steps
const T = 1000;
const abar = [];                       // ᾱ_t, decreasing 1 → ~0
for (let t=0;t<=T;t++){
  const f = Math.cos(((t/T)+0.008)/1.008 * Math.PI/2)**2;  // smooth decay
  abar.push(f);
}
// forward sample at step t:  x_t = √ᾱ_t · x0 + √(1−ᾱ_t) · ε
function forward(x0, eps, t){
  const s = Math.sqrt(abar[t]), n = Math.sqrt(1 - abar[t]);
  return x0.map((v,i)=> s*v + n*eps[i]);   // signal shrinks, noise grows
}

Where this points next

这将指向何处

We closed the loop on what vision models can do: read pixels (01–16) and now write them (17) — and generation is also the modern engine of image restoration. Lesson 18 (Image restoration and computational photography) turns the generative priors from this lesson toward inverse problems: super-resolution, denoising, deblurring, inpainting, HDR, and the camera ISP pipeline — recovering a clean image from a degraded measurement. Then the capstone, lesson 19 (Evaluation, deployment, and CV system design), asks how any of it becomes a system: slice-aware evaluation that survives a new camera, compression to hit a latency budget, the batching-vs-p99 serving math, drift monitoring, and privacy — closing the arc back to lesson 01. (The diffusion machinery here also lifts into 3D: the companion 3D Vision track uses it for text-to-3D via score distillation.)

我们把视觉模型能做什么这件事闭环了：读入像素（01–16），如今又能写出像素（17）——而生成也是现代图像复原的引擎。第 18 课（图像复原与计算摄影）把本课的生成先验转向反问题：超分辨率、去噪、去模糊、图像修复、HDR，以及相机的 ISP 流水线——从一次退化的测量中恢复出一张干净的图像。接着的收官之作，第 19 课（评估、部署与 CV 系统设计），追问这一切如何变成一个系统：能挺过换一台新相机的分片评估、为满足延迟预算而做的压缩、批处理与 p99 之间的服务权衡算术、漂移监控，以及隐私——把整条弧线绕回到第 01 课。（这里的扩散机器也能抬进三维：配套的 3D Vision 课程用它通过分数蒸馏来做文本到三维。）

Takeaway要点

Generation models p(x) (or p(x|c)), a vast multimodal space — not classification backward — and that one-to-many structure explains the whole family. VAEs use a probabilistic encoder trained via the ELBO and the reparameterization trick z = μ + σ⊙ε; they blur because the per-pixel Gaussian/L2 reconstruction returns the mean of valid modes (plus a limited posterior) — not because "KL dominates." GANs play a generator-vs-discriminator minimax that yields sharpness but risks mode collapse. Diffusion fixes the forward noising chain x_t=√ᾱ_tx₀+√(1−ᾱ_t)ε and learns a network to predict ε; DDIM skips steps, latent diffusion (Stable Diffusion) runs it cheaply in a VAE latent, classifier-free guidance and ControlNet steer it, and flow matching / consistency distillation push toward few-step sampling. Evaluate with FID (Fréchet distance between real and generated Inception-feature Gaussians — rewards fidelity and diversity), plus human and prompt-adherence metrics, and treat safety as part of the spec. 生成建模的是 p(x)（或 p(x|c)），一个庞大的多模态空间——不是把分类反过来——而这种一对多的结构解释了整个族。VAE 使用一个通过 ELBO 训练的概率编码器和重参数化技巧 z = μ + σ⊙ε；它之所以模糊，是因为逐像素的高斯/L2 重建返回的是有效模式的均值（再加上一个受限的后验）——而不是因为"KL 压过一切"。GAN 玩的是生成器对判别器的极小极大博弈，带来清晰度但有模式坍缩的风险。扩散固定了前向加噪链 x_t=√ᾱ_tx₀+√(1−ᾱ_t)ε，并学习一个网络去预测 ε；DDIM 可跳步，潜在扩散（Stable Diffusion）在 VAE 潜在空间里廉价地跑它，无分类器引导和 ControlNet 为它掌舵，流匹配 / 一致性蒸馏则推动它走向少步采样。用 FID（真实与生成的 Inception 特征高斯之间的 Fréchet 距离——同时奖励保真度和多样性）来评估，再加上人类偏好与提示词遵循等指标，并把安全性当作规格的一部分。

Interview prompts

面试题

Why is generation harder than classification, even though both are "just" learning a distribution? (§1: p(x|c) is high-dim and one-to-many multimodal vs one-to-one p(y|x).)为什么生成比分类更难，尽管两者都"不过是"在学习一个分布？(§1：p(x|c) 是高维且一对多的多模态，而 p(y|x) 是一对一的。)
What is the reparameterization trick and what problem does it solve? (§2: z=μ+σ⊙ε moves randomness outside the net so gradients flow through μ,σ.)什么是重参数化技巧，它解决了什么问题？(§2：z=μ+σ⊙ε 把随机性挪到网络外面，从而让梯度能流经 μ,σ。)
Why do VAEs produce blurrier images than GANs? (§2: per-pixel Gaussian/L2 reconstruction = mean of valid modes; plus limited posterior — not "KL dominates.")为什么 VAE 产出的图像比 GAN 更模糊？(§2：逐像素的高斯/L2 重建 = 有效模式的均值；再加上受限的后验——而不是"KL 压过一切"。)
What does diffusion training actually optimize? (§4: predict the noise ε added at a random step, simple L2 on ε.)扩散训练实际上在优化什么？(§4：预测在某个随机步骤加入的噪声 ε，就是对 ε 做简单的 L2。)
DDPM vs DDIM, and what is latent diffusion? (§4: stochastic full chain vs deterministic step-skipping sampler; diffuse in a VAE latent for cheap high-res.)DDPM 与 DDIM 有何区别，什么是潜在扩散？(§4：随机的完整链 vs 确定性、可跳步的采样器；在 VAE 潜在空间里做扩散以廉价地得到高分辨率。)
What is classifier-free guidance and what does turning it up cost? (§4: extrapolate conditional from unconditional; higher w = more adherence, less diversity, more artifacts.)什么是无分类器引导，把它调高的代价是什么？(§4：从无条件向有条件外推；w 越高 = 越贴合提示词、多样性越低、伪影越多。)
What two distributions does FID compare and why is it incomplete? (§5: real vs generated Inception-feature Gaussians; blind to prompt adherence, sensitive to sample count.)FID 比较的是哪两个分布，它为什么不完整？(§5：真实与生成的 Inception 特征高斯；对提示词贴合程度视而不见，且对样本数量敏感。)
When can synthetic data hurt a vision model? (§5: only real held-out gains count; impressive samples that don't move the real metric, or that amplify bias, hurt.)合成数据什么时候会损害一个视觉模型？(§5：只有真实留出集上的收益才算数；那些撼动不了真实指标、或放大偏见的惊艳样本，都是有害的。)