Image restoration and computational photography

图像复原与计算摄影

Lessons 01–16 learned to read pixels and lesson 17 learned to write them from scratch. But a huge slice of real computer vision is neither classification nor generation from a blank canvas — it is recovering a clean image from a degraded measurement: a noisy low-light frame, a motion-blurred grab, a low-resolution thumbnail, a photo with a scratch through it. This lesson frames that task as an inverse problem, walks the family (denoising, super-resolution, deblurring, inpainting), shows how the generative priors of lesson 17 became the modern engine, and covers the camera ISP and computational-photography pipeline that already does much of this on-device — the hook into deployment (lesson 19).

第 01–16 课学会了读像素，第 17 课学会了从零开始写像素。但真实计算机视觉中很大一块既不是分类，也不是从一块白布上生成——而是从退化的测量中复原出一张干净图像：一帧带噪的弱光画面、一张运动模糊的抓拍、一张低分辨率缩略图、一张被划痕划过的照片。本课把这个任务表述为一个反问题，梳理它的家族（去噪、超分辨率、去模糊、图像修复），展示第 17 课的生成式先验如何成为现代引擎，并讲解相机 ISP 与计算摄影流水线——它们已经在端侧完成了其中很大一部分工作，这正是通往部署（第 19 课）的引子。

1 · Restoration as an inverse problem

1 · 把复原看作反问题

Every restoration task shares one skeleton. The clean image you want, x, was passed through some degradation before it reached you, and what you actually observe is the corrupted result y. Model the degradation as a (linear) operator A plus additive noise n:

每个复原任务都共享同一副骨架。你想要的那张干净图像 x，在到达你手里之前经过了某种退化，而你实际观测到的是被破坏后的结果 y。把退化建模为一个（线性）算子 A 加上加性噪声 n：

This is the forward model: it says how a clean scene becomes a measurement. Read it as a recipe you can run forward easily — take the true image, apply A, sprinkle noise — and your job is to run it backward, recovering the cause x from the effect y. The single matrix A is what specializes the skeleton into a task:

这就是前向模型：它描述了一个干净场景如何变成一次测量。把它当作一份可以轻松正向执行的配方——取真实图像，施加 A，撒上噪声——而你的工作是反向执行它，从结果 y 中恢复出成因 x。正是那一个矩阵 A，把这副骨架特化成一个具体任务：

Why this is hard: the inverse is ill-posed. A well-posed problem (Hadamard) has a solution, a unique one, that depends stably on the data. Restoration fails all three. A is either non-invertible — a downsampling or a mask literally discards information, so infinitely many clean images x produce the exact same y — or ill-conditioned — a blur is formally invertible but its inverse divides by the kernel's near-zero high-frequency response, so the tiniest noise n gets amplified into enormous garbage. Naively "solving" x = A⁻¹ y either has no unique answer or explodes. You cannot recover what the measurement destroyed from the measurement alone; you need to inject knowledge about what real images look like. That knowledge is a prior.

为什么这很难：逆过程是病态的。一个适定（Hadamard 意义下）的问题有解、解唯一，且解稳定地依赖于数据。复原三条全都不满足。A 要么不可逆——下采样或掩码实实在在地丢弃了信息，于是无穷多张干净图像 x 会产生完全相同的 y——要么病态（ill-conditioned）——一个模糊在形式上可逆，但它的逆要除以卷积核接近零的高频响应，于是最微小的噪声 n 都会被放大成巨量的垃圾。天真地"求解" x = A⁻¹ y 要么没有唯一答案，要么直接爆炸。你无法只凭测量本身去恢复测量所摧毁的东西；你需要注入关于真实图像长什么样的知识。这份知识就是先验。

The classical way to write this is a two-term optimization. Find the image that both explains the measurement and looks like a real image:

经典写法是一个两项的优化问题。找一张图像，它既能解释测量，又看起来像一张真实图像：

The first term is data fidelity: run your candidate x back through the known forward operator A and penalize how far the result strays from the actual observation y. The second term is the regularizer R(x) — a prior that scores how plausible x is on its own, independent of the data. Classic choices: a smoothness penalty ‖∇x‖² (Tikhonov — punishes any variation, so it blurs), total variation ‖∇x‖₁ (punishes total gradient magnitude but tolerates a few large jumps, so it smooths flat regions while keeping edges), or a sparsity penalty on wavelet/dictionary coefficients (natural images are sparse in the right basis).

第一项是数据保真：把你的候选 x 通过已知的前向算子 A 送回去，并惩罚结果偏离实际观测 y 的程度。第二项是正则项 R(x)——一个先验，它独立于数据、单凭 x 本身给它的可信程度打分。经典选择有：平滑性惩罚 ‖∇x‖²（Tikhonov——惩罚任何变化，因此会模糊）、总变差（total variation） ‖∇x‖₁（惩罚梯度幅值之和，但容忍少数几个大跳变，因此在平滑平坦区域的同时保留边缘），或对小波/字典系数的稀疏性惩罚（自然图像在恰当的基下是稀疏的）。

The weight λ ≥ 0 is the whole game. It trades the two terms off:

权重 λ ≥ 0 才是全部关键。它在两项之间做权衡：

Notice that this is the same inverse-problem shape as the rest of vision. Recognition recovers a semantic cause (a label) from a lossy pixel measurement; stereo recovers depth from a pair of projections; here we recover a clean image from a corrupted one. Every one is "invert a lossy forward map, regularized by a prior." What changes across the field is only where the prior comes from — hand-written smoothness here, a learned generative model in §3.

请注意，这与视觉中其余部分是同一种反问题结构。识别是从一次有损的像素测量中恢复语义成因（一个标签）；立体视觉是从一对投影中恢复深度；这里我们是从一张被破坏的图像中恢复出干净图像。每一个都是"求逆一个有损的前向映射，并用先验加以正则化"。整个领域里唯一变化的，只是先验从何而来——这里是手工写的平滑性，第 3 节则是一个学习得到的生成式模型。

2 · The task family

2 · 任务家族

Four canonical tasks, each an instance of §1 with a different A. For each: what A is, the classical method, the learned method, and the key difficulty.

四个典型任务，每一个都是第 1 节在不同 A 下的一个实例。对每个任务，我们给出：A 是什么、经典方法、学习型方法，以及关键难点。

Denoising — A = I

去噪——A = I

The purest case: y = x + n, no geometric distortion, just noise. The noise model matters and follows the physics of the sensor (lesson 01):

最纯粹的情形：y = x + n，没有几何畸变，只有噪声。噪声模型很重要，它遵循传感器的物理特性（第 01 课）：

• Gaussian — additive, signal-independent read noise; the default assumption, n ∼ 𝒩(0, σ²).高斯噪声——加性、与信号无关的读出噪声；默认假设，n ∼ 𝒩(0, σ²)。
• Poisson / shot noise — photon arrival is a counting process, so noise variance equals the signal. This dominates in low light where photon counts are tiny, making the per-pixel SNR terrible — the reason dark photos are grainy.泊松 / 散粒噪声——光子到达是一个计数过程，因此噪声方差等于信号本身。它在弱光下占主导，此时光子计数极小，使得每像素信噪比糟糕透顶——这正是暗照片颗粒感强的原因。
• Salt-and-pepper — dead/hot pixels or transmission errors flip scattered pixels to pure black or white; a median filter kills it because outliers do not survive a median.椒盐噪声——坏点/热点像素或传输错误把零散的像素翻成纯黑或纯白；中值滤波能除掉它，因为离群值挺不过一次取中值。

Classical: the bilateral filter averages neighbors but weights them by both spatial distance and intensity similarity, so it blurs within a smooth region but not across an edge (edge-preserving). Non-local means (NLM) generalizes this — it averages pixels whose surrounding patches look alike, even if they are far apart, exploiting the self-similarity of natural images. BM3D (block-matching + 3D filtering) stacks similar patches into a 3D group and filters them jointly in a transform domain; it was the classical state of the art for years. Learned: DnCNN made the key move of residual learning — instead of predicting the clean image, predict the noise n̂ = f(y) and subtract it, x̂ = y − n̂. The noise is a simpler, more homogeneous target than the whole image, which makes training easier and faster. Difficulty: real texture and noise are both high-frequency (lesson 02's frequency view), so a denoiser that is too aggressive erases fine detail along with the grain — exactly the trade-off of §1.

经典方法：双边滤波（bilateral filter）对邻域求平均，但按空间距离和强度相似度两者来加权，因此它在平滑区域内部模糊、却不跨越边缘（保边）。非局部均值（NLM）把这一思路推广——它对那些周围图块看起来相似的像素求平均，哪怕它们相距很远，从而利用了自然图像的自相似性。BM3D（块匹配 + 三维滤波）把相似图块堆叠成一个三维组，在变换域中对它们联合滤波；它多年来一直是经典方法的最先进水平。学习型方法：DnCNN 走出了残差学习这关键一步——不去预测干净图像，而是预测噪声 n̂ = f(y) 并把它减掉，x̂ = y − n̂。噪声比整张图像是一个更简单、更均匀的目标，这让训练更容易也更快。难点：真实纹理和噪声都是高频的（第 02 课的频域视角），所以一个过于激进的去噪器会连同颗粒一起抹掉精细细节——这正是第 1 节的那个权衡。

Super-resolution — A = blur + downsample

超分辨率——A = 模糊 + 下采样

You have a low-resolution image and want a high-resolution one; the forward model first blurs then decimates (the correct blur-then-downsample of lesson 02). Classical: bicubic interpolation — fit a smooth surface through the known samples. It is a fixed low-pass reconstruction, so it never invents detail; it just makes a bigger, smoother version. Learned: SRCNN was the first CNN to beat bicubic; ESRGAN added an adversarial loss (lesson 17) to synthesize photo-realistic texture. Difficulty: super-resolution is the most ill-posed member of the family. Each low-res pixel is an average of many high-res pixels, so an enormous set of distinct high-res images all downsample to the same input. The missing high frequencies are not recoverable from the data — they must be hallucinated from a prior about what plausible detail looks like. That is precisely why super-resolution lives or dies by the perceptual-vs-fidelity tension of §5: a model can produce a sharp, convincing texture that is not the real one.

你有一张低分辨率图像，想要一张高分辨率图像；前向模型先模糊、再抽取（就是第 02 课那个正确的"先模糊再下采样"）。经典方法：双三次插值——用一个光滑曲面拟合已知的采样点。它是一种固定的低通重建，因此从不发明细节；它只是做出一个更大、更平滑的版本。学习型方法：SRCNN 是第一个超越双三次插值的 CNN；ESRGAN 加入了对抗损失（第 17 课）来合成逼真的纹理。难点：超分辨率是这个家族中最病态的成员。每个低分辨率像素都是许多高分辨率像素的平均，因此有一个庞大集合的不同高分辨率图像会下采样到同一个输入。缺失的高频从数据本身是无法恢复的——它们必须依据"可信细节长什么样"的先验被臆造（hallucinate）出来。这正是为什么超分辨率的成败系于第 5 节的"感知 vs 保真"张力：一个模型可以生成锐利、令人信服、却不是真实那一个的纹理。

Deblurring — A = convolution with a PSF

去模糊——A = 与 PSF 做卷积

Motion blur (camera or subject moved during exposure) and defocus blur are both convolutions of the sharp image with a point-spread function (PSF) — the image a single point of light smears into (lesson 02). Non-blind deconvolution assumes the kernel k is known and inverts it — e.g. the Wiener filter, which divides in the frequency domain but tempers the division by the noise level to avoid blow-up, or Richardson–Lucy iteration. Blind deconvolution is the realistic and much harder case: you must estimate the kernel and the sharp image jointly from the blurry image alone — a chicken-and-egg problem, since a sharp-looking result and a blurry-looking result can both be consistent with the data. Learned: CNNs regress the sharp image (and often the kernel) directly. Difficulty: deconvolution is the ill-conditioned end of §1 — inverting the blur amplifies exactly the high-frequency noise that the blur suppressed, so deblurring without a strong prior turns a soft photo into a noisy one.

运动模糊（曝光期间相机或被摄物移动了）和失焦模糊，都是清晰图像与一个点扩散函数（PSF）的卷积——PSF 就是单个光点被抹开后成的像（第 02 课）。非盲反卷积假设卷积核 k 已知并对它求逆——例如维纳滤波器（Wiener filter），它在频域中做除法，但用噪声水平来缓和这个除法以避免爆炸，或者用 Richardson–Lucy 迭代。盲反卷积才是现实且难得多的情形：你必须仅凭那张模糊图像，同时估计出卷积核和清晰图像——这是个先有鸡还是先有蛋的问题，因为一个看起来清晰的结果和一个看起来模糊的结果都可能与数据一致。学习型方法：CNN 直接回归清晰图像（往往还有卷积核）。难点：反卷积是第 1 节里病态的那一端——对模糊求逆恰恰会放大被模糊压制掉的高频噪声，所以没有强先验的去模糊，会把一张柔和的照片变成一张带噪的照片。

Inpainting — A = a mask

图像修复——A = 一个掩码

A region of the image is missing (a scratch, an occluding object to remove, a hole from a moved object) and A = M is a binary mask that keeps the known pixels and zeroes the rest. Small holes can be filled by diffusing surrounding color inward or by PatchMatch (copy the best-matching patch from elsewhere in the image). But filling a large or structured hole — the space where a removed person stood, the missing half of a face — needs a semantic prior: an understanding of what plausibly belongs there, not just what is nearby. That is exactly what a large generative model supplies, which is why modern inpainting ("generative fill") is a direct application of §3.

图像的某块区域缺失了（一道划痕、一个要移除的遮挡物、一处物体被移走后留下的空洞），而 A = M 是一个二值掩码，它保留已知像素、把其余置零。小的空洞可以靠把周围颜色向内扩散来填补，或者用 PatchMatch（从图像别处复制最匹配的图块）。但填补一个大的或有结构的空洞——一个被移除的人所站的位置、一张脸缺失的半边——则需要一个语义先验：对那里"应该有什么"的理解，而不仅仅是附近有什么。这恰恰是一个大型生成式模型所提供的，也正是现代图像修复（"生成式填充"）成为第 3 节的直接应用的原因。

2.5 · Interactive · denoising and the fidelity trade-off

2.5 · 互动 · 去噪与保真权衡

This is the heart of the lesson made tangible. We build a clean 1-D signal x with sharp step edges and a low-amplitude sinusoidal texture, corrupt it with Gaussian noise (a σ slider — this is the forward model y = x + n), and restore x̂ by smoothing with an edge-aware total-variation step whose strength is λ (the second slider — the regularizer of §1). Watch three curves: the clean x, the noisy y, and the restored x̂. The KPIs report input PSNR (how bad the measurement is), output PSNR (how good the restoration is), and λ.

这是本课的核心，被做成了可触可感的形式。我们构造一个干净的一维信号 x，它带有锐利的阶跃边缘和一段低幅值的正弦纹理，用高斯噪声把它破坏（一个 σ 滑块——这就是前向模型 y = x + n），再用一个保边的总变差步骤做平滑来复原 x̂，其强度就是 λ（第二个滑块——第 1 节的正则项）。观察三条曲线：干净的 x、带噪的 y，以及复原的 x̂。KPI 报告输入 PSNR（测量有多糟）、输出 PSNR（复原有多好）以及 λ。

// Forward model: y = x + Gaussian noise (σ)
function degrade(x, eps, sigma){
  return x.map((v,i) => v + sigma*eps[i]);   // eps ~ N(0,1), fixed per resample
}

// Restore x̂ by iterated edge-aware (TV-style) smoothing.
// lam sets iteration count → smoothing strength (the §1 regularizer weight).
function restore(y, lam){
  let x = y.slice();
  const iters = Math.round(lam);              // more iters = stronger prior
  for (let it=0; it<iters; it++){
    const nx = x.slice();
    for (let i=1;i<x.length-1;i++){
      const dL = x[i-1]-x[i], dR = x[i+1]-x[i];
      // edge-aware weights: big jumps (edges) get pulled less → edges survive longer
      const wL = 1/(1 + (dL*dL)/400), wR = 1/(1 + (dR*dR)/400);
      nx[i] = x[i] + 0.25*(wL*dL + wR*dR);    // one Jacobi step toward neighbors
    }
    x = nx;
  }
  return x;
}

// PSNR with MAX = peak-to-peak range of the clean signal
function psnr(a, b, MAX){
  let mse = 0;
  for (let i=0;i<a.length;i++){ const d=a[i]-b[i]; mse += d*d; }
  mse /= a.length;
  return 10 * Math.log10((MAX*MAX)/mse);
}

Play with it and you have felt §1 and previewed §5 at once. At λ=0 the restored curve is the noisy curve — output PSNR equals input PSNR. As you raise λ, noise averages out and output PSNR climbs above the input; keep going and the step edges start to slump into ramps, texture flattens, and output PSNR turns over and drops. The peak is the sweet spot. Turn up σ and the whole curve shifts: a noisier input needs a larger λ at its peak — more noise, lean harder on the prior. "Jump to best λ" sweeps every λ and lands on the maximizing one, so you can see where it moves.

玩一玩它，你就同时既感受到了第 1 节、又预览了第 5 节。在 λ=0 时，复原曲线就是那条带噪曲线——输出 PSNR 等于输入 PSNR。当你调大 λ，噪声被平均掉，输出 PSNR 爬升到输入之上；继续往上，阶跃边缘开始塌陷成斜坡，纹理被抹平，输出 PSNR 见顶回落。那个峰值就是甜点。调大 σ，整条曲线就会平移：越嘈杂的输入，其峰值处需要越大的 λ——噪声越多，就越要依靠先验。"跳到最佳 λ"会扫遍每一个 λ 并落在使其最大化的那个上，让你看清它是怎么移动的。

3 · The modern engine: generative priors

3 · 现代引擎：生成式先验

The classical objective of §1 wrote the prior R(x) by hand — smoothness, total variation, sparsity. These are weak, generic statements about images; they know nothing about faces, text, or textures. The revolution of the last few years is to replace that hand-written term with a learned generative model of natural images — the very models of lesson 17. This is cleanest in the Bayesian view.

第 1 节的经典目标是手工写出先验 R(x) 的——平滑性、总变差、稀疏性。这些都是关于图像的、微弱而笼统的陈述；它们对人脸、文字或纹理一无所知。过去这几年的革命，是用一个学习得到的自然图像生成式模型去替换那个手写项——正是第 17 课的那些模型。从贝叶斯视角看，这一点最为清晰。

Restoration is posterior inference: given the measurement y, find (or sample) the clean image x. By Bayes' rule the posterior factors into a likelihood and a prior:

复原就是后验推断：给定测量 y，去寻找（或采样）干净图像 x。由贝叶斯法则，后验分解为一个似然和一个先验：

Take the negative log and this is §1's objective: the likelihood p(y|x) — how probable the measurement is given a candidate clean image — is fixed by the known degradation A and the noise model (for Gaussian noise, −log p(y|x) ∝ ‖Ax − y‖², exactly the data-fidelity term), and the prior p(x) plays the role of λ R(x). So the two terms are a likelihood and a prior, and maximizing the posterior is the MAP estimate. The only question is what to use for p(x).

取负对数，这就是第 1 节的目标：似然 p(y|x)——给定一张候选干净图像时该测量有多大概率——由已知的退化 A 和噪声模型确定（对高斯噪声，−log p(y|x) ∝ ‖Ax − y‖²，恰好就是数据保真项），而先验 p(x) 扮演了 λ R(x) 的角色。所以那两项就是一个似然和一个先验，而最大化后验就是 MAP（最大后验）估计。唯一的问题是：p(x) 用什么。

Diffusion models are a spectacular p(x). A pretrained diffusion model (lesson 17) is exactly a learned prior over natural images — it knows the manifold of plausible pictures. The trick is to guide the reverse diffusion with the measurement: at each denoising step, take the model's ordinary generative update (which pulls the sample toward the image manifold, the prior), then nudge it to also stay consistent with the data — enforce A x̂ ≈ y via a gradient step on ‖A x̂ − y‖² or by re-projecting the known pixels. The reverse process still lands on the manifold of real images (so the output looks real, not blurry), but now on the slice of that manifold that explains your measurement.

扩散模型是一个绝佳的 p(x)。一个预训练的扩散模型（第 17 课）恰恰就是一个学习得到的自然图像先验——它了解可信图片所在的流形。诀窍在于用测量去引导反向扩散：在每一个去噪步上，取模型平常的生成式更新（它把样本拉向图像流形，也就是先验），再推它一把、让它同时与数据保持一致——通过在 ‖A x̂ − y‖² 上做一步梯度、或通过把已知像素重新投影回去，来强制 A x̂ ≈ y。反向过程仍然落在真实图像的流形上（所以输出看起来真实、而非模糊），但如今是落在该流形中那一片能解释你的测量的切片上。

The payoff is enormous: because the likelihood is the only task-specific part and it is just the known A, one pretrained diffusion prior solves denoising, super-resolution, deblurring, and inpainting with no task-specific training — you swap in the A for the task and keep the same prior. Change the mask and it inpaints; change to blur+downsample and it super-resolves. (This is the same "one prior, many likelihoods" structure the whole field keeps rediscovering.) For real-world photos where the degradation is unknown and compound, Real-ESRGAN trains on a rich simulated degradation pipeline, and diffusion-based super-resolution models use the diffusion prior for restoration in the wild.

回报是巨大的：因为似然是唯一与任务相关的部分，而它只不过是已知的 A，所以一个预训练的扩散先验就能无需任何任务专属训练地解决去噪、超分辨率、去模糊和图像修复——你只需为任务换上对应的 A，先验保持不变。换个掩码，它就做图像修复；换成模糊+下采样，它就做超分辨率。（这正是整个领域一再重新发现的那种"一个先验、多种似然"的结构。）对于退化未知且复合的真实照片，Real-ESRGAN 在一条丰富的模拟退化流水线上训练，而基于扩散的超分辨率模型则用扩散先验来做真实场景（in the wild）中的复原。

4 · Computational photography and the ISP

4 · 计算摄影与 ISP

Here is the fact that reframes everything above: the image you get from a camera is already the output of a restoration pipeline, not raw light. A phone photo has been demosaicked, denoised, white-balanced, tone-mapped, sharpened, and gamma-encoded before you ever see it. That pipeline is the Image Signal Processor (ISP), and it runs on-device in real time.

下面这个事实重新定义了以上一切：你从相机得到的图像本身已经是一条复原流水线的输出，而不是原始光。一张手机照片在你看到它之前，已经过了去马赛克、去噪、白平衡、色调映射、锐化和伽马编码。这条流水线就是图像信号处理器（ISP），它在端侧实时运行。

The sensor does not measure RGB. Each photosite sits under one color filter in a Bayer mosaic (a repeating 2×2 pattern of R, G, G, B), so the RAW frame has one color per pixel. The ISP then, roughly in order:

传感器并不测量 RGB。每个感光点都位于拜耳阵列（Bayer mosaic）中的一个滤色片之下（一个由 R、G、G、B 组成的、不断重复的 2×2 图案），所以 RAW 帧里每个像素只有一种颜色。随后 ISP 大致按以下顺序处理：

Read left to right: demosaic reconstructs a full RGB triple at every pixel by interpolating the missing two channels from neighbors (lesson 01) — itself an inverse problem; white balance rescales the channels so a neutral surface reads gray under the scene's illuminant; denoise is exactly §2; tone-mapping compresses a high-dynamic-range capture into what a display can show; sharpening is lesson 02's high-boost filter; and finally gamma / sRGB encoding applies the perceptual curve L ≈ V^2.2 from lesson 01. Every stage is a restoration or enhancement operator, tuned and fused rather than run in isolation.

从左到右读：去马赛克通过从邻域插值出缺失的两个通道，在每个像素上重建出完整的 RGB 三元组（第 01 课）——它本身就是一个反问题；白平衡重新缩放各通道，使中性表面在场景光源下读起来是灰色的；去噪恰恰就是第 2 节；色调映射把一次高动态范围的拍摄压缩到显示器所能呈现的范围；锐化就是第 02 课的高频增强滤波器；最后伽马 / sRGB 编码施加第 01 课那条感知曲线 L ≈ V^2.2。每一级都是一个复原或增强算子，它们是经过调校并融合在一起的，而不是各自孤立运行的。

Then computational photography fuses multiple frames. The biggest quality wins on modern phones come not from a better single-shot pipeline but from capturing a burst and combining frames:

接着，计算摄影会融合多帧。现代手机上最大的画质提升，并非来自更好的单帧流水线，而是来自拍摄一个连拍序列（burst）并把多帧组合起来：

• HDR — capture a bracket of exposures (dark, medium, bright), merge them so highlights come from the short exposure and shadows from the long one, then tone-map the wide-range result into a displayable image. Recovers detail a single exposure would clip or crush.HDR——拍摄一组包围曝光（暗、中、亮），将它们合并，使高光来自短曝光、阴影来自长曝光，再把这个宽范围结果做色调映射成一张可显示的图像。它恢复出单次曝光会被削顶或压死的细节。
• Burst / low-light — the noise model of §2 says shot noise is random per frame. Align many frames and average them: the signal reinforces while zero-mean noise cancels, so averaging N frames cuts noise by √N. This is how Night Sight-style modes turn a near-black scene into a clean photo — restoration by fusion, not by a stronger single-frame denoiser.连拍 / 弱光——第 2 节的噪声模型指出散粒噪声在每帧上都是随机的。把许多帧对齐并求平均：信号相互增强，而零均值噪声相互抵消，所以对 N 帧求平均能把噪声降到原来的 √N 分之一。这正是夜视（Night Sight）之类的模式把一个近乎全黑的场景变成一张干净照片的方式——靠融合来复原，而不是靠一个更强的单帧去噪器。
• Multi-frame super-resolution — tiny hand tremor shifts each burst frame by a sub-pixel amount, so the frames sample the scene on slightly different grids. Merging them recovers real detail beyond one frame's resolution — genuine information, not hallucinated, because the extra samples actually exist.多帧超分辨率——手的微小抖动会把每一帧连拍平移亚像素级的量，于是这些帧在略有不同的网格上对场景采样。把它们合并起来，就能恢复出超过单帧分辨率的真实细节——这是货真价实的信息，而非臆造，因为那些额外的采样点确实存在。
• Burst deblurring — pick or combine the sharpest frames of a burst to beat motion blur.连拍去模糊——从连拍中挑选或组合最清晰的那些帧，以战胜运动模糊。

5 · Evaluation and the perceptual–distortion trade-off

5 · 评估与感知–失真权衡

How do you score a restored image? There are two incompatible families of metric, and the tension between them is the deepest idea in the field.

你如何给一张复原后的图像打分？这里有两个互不相容的度量家族，而它们之间的张力是这个领域里最深刻的思想。

Distortion metrics measure pixel-level fidelity to the ground truth. PSNR (peak signal-to-noise ratio) is a monotone function of mean squared error:

失真度量衡量的是相对于真值的像素级保真度。PSNR（峰值信噪比）是均方误差的一个单调函数：

where MAX is the peak signal value (255 for 8-bit, or the peak-to-peak range). It is in decibels, higher is better, and it is exactly what the widget reports. SSIM (structural similarity) improves on raw MSE by comparing local luminance, contrast, and structure, correlating better with perceived quality — but it is still fundamentally a fidelity measure. Perceptual metrics instead measure whether the output looks like a real image. LPIPS compares deep features (distance in a pretrained network's feature space, which tracks human judgments of similarity far better than pixels); NIQE is no-reference — it scores naturalness without any ground truth, useful when there is no clean image to compare against.

其中 MAX 是峰值信号值（8 位下为 255，或取峰峰值范围）。它以分贝为单位，越高越好，也正是小部件所报告的量。SSIM（结构相似性）通过比较局部的亮度、对比度和结构，改进了原始 MSE，与感知质量的相关性更好——但它本质上仍是一个保真度量。感知度量则衡量输出是否看起来像一张真实图像。LPIPS 比较深度特征（在一个预训练网络的特征空间中的距离，它对相似性的人类判断的刻画远胜于像素）；NIQE 是无参考的——它不需要任何真值就能给自然度打分，在没有干净图像可供对比时很有用。

The Blau–Michaeli result: you cannot maximize both. There is a fundamental perception–distortion trade-off: no algorithm can simultaneously achieve the best possible distortion (fidelity) and the best possible perceptual quality. Improving one past a point necessarily costs the other. The reason ties straight back to §1 and lesson 17's mean-of-modes blur:

Blau–Michaeli 的结论：两者无法同时最大化。存在一个根本性的感知–失真权衡：没有任何算法能同时达到可能的最优失真（保真度）和可能的最优感知质量。把其中一个改进过某一点，必然要以另一个为代价。其原因直接联系回第 1 节，以及第 17 课那种"取众模态之均值"式的模糊：

An L2-trained super-resolver scores high PSNR but looks soft: faced with many valid high-res completions, MSE is minimized by their average, and the average of many sharp textures is a blur (the identical mechanism that blurs VAE samples in lesson 17). A GAN or diffusion restorer produces crisp, convincing detail — and scores lower PSNR, because it commits to one specific plausible texture that differs pixel-for-pixel from the truth. Worse, it can hallucinate detail that was never in the scene: invent freckles, sharpen a blur into the wrong license-plate digits, synthesize a texture the prior finds likely but the measurement never supported.

一个用 L2 训练的超分辨率器 PSNR 得分很高，但看起来偏软：面对许多有效的高分辨率补全方式，MSE 由它们的平均来最小化，而许多锐利纹理的平均就是一团模糊（这与第 17 课中让 VAE 样本变模糊的机制一模一样）。一个 GAN 或扩散复原器会生成清晰、令人信服的细节——却拿到更低的 PSNR，因为它押定了某一个特定的、可信的纹理，而它与真值逐像素地不同。更糟的是，它可能臆造出场景里从不存在的细节：凭空造出雀斑、把一团模糊"锐化"成错误的车牌数字、合成一种先验觉得可信但测量从未支持的纹理。

Where this points next

这将指向何处

We can now recover a clean image from a degraded one — framing every task (denoise, super-resolve, deblur, inpaint) as one inverse problem y = A x + n regularized by a prior, using lesson 17's generative models as that prior, and running much of it inside the camera ISP. But §4's punchline was a deployment fact: this restoration ships on-device, in milliseconds, on a power budget, and §5's punchline was an evaluation fact: the right metric depends on the stakes and can disagree with what looks good. Both point at the same unanswered question — how does any vision capability become a system? Lesson 19 (Evaluation, deployment, and CV system design) is the capstone: slice-aware evaluation that survives a new camera or lighting condition, compression and quantization to hit a latency budget, the batching-versus-p99-latency serving math, drift monitoring, and privacy — closing the arc back to lesson 01's pixels and preprocessing-parity rule.

现在我们能从一张退化的图像中恢复出干净图像了——把每个任务（去噪、超分辨率、去模糊、图像修复）都表述为一个由先验正则化的反问题 y = A x + n，用第 17 课的生成式模型充当那个先验，并把其中很大一部分放在相机 ISP 内部运行。但第 4 节的落点是一个部署事实：这份复原是在端侧、在毫秒之内、在功耗预算之下落地的；第 5 节的落点则是一个评估事实：正确的度量取决于利害轻重，并且可能与"看起来好"相左。两者都指向同一个悬而未决的问题——任何一项视觉能力究竟如何变成一个系统？第 19 课（评估、部署与 CV 系统设计）是收官之作：能挺过新相机或新光照条件的分片感知（slice-aware）评估、为满足延迟预算而做的压缩与量化、批处理与 p99 延迟之间的服务权衡计算、漂移监控，以及隐私——把这条弧线一路收回到第 01 课的像素和"预处理一致性"规则。

Interview prompts

面试题

• Write the forward model for restoration and say what A is for denoising, deblurring, super-resolution, and inpainting. (§1, §2 — y = Ax + n; identity, convolution/PSF, blur+downsample, binary mask.)写出复原的前向模型，并说出去噪、去模糊、超分辨率和图像修复各自的 A 是什么。(§1、§2 —— y = Ax + n；恒等、卷积/PSF、模糊+下采样、二值掩码。)
• Why is image restoration ill-posed, and what fixes it? (§1 — A is non-invertible or ill-conditioned so the naive inverse is non-unique or noise-amplifying; a prior/regularizer R(x) weighted by λ resolves it.)图像复原为什么是病态的，靠什么解决？(§1 —— A 不可逆或病态，因此天真的逆要么不唯一、要么放大噪声；用一个由 λ 加权的先验/正则项 R(x) 来解决。)
• What does the λ in ‖Ax−y‖² + λR(x) control, and why is there a sweet spot? (§1, §2.5 — fidelity vs prior; too small leaks noise, too large over-smooths edges; output PSNR peaks then falls.)‖Ax−y‖² + λR(x) 中的 λ 控制什么，为什么存在一个甜点？(§1、§2.5 —— 保真 vs 先验；太小会漏噪声，太大会把边缘过度平滑；输出 PSNR 先见顶再回落。)
• How does a pretrained diffusion model become a universal restoration solver? (§3 — sample the posterior p(x|y) ∝ p(y|x)p(x); the likelihood is the known A and the prior is the diffusion model, guided by data-consistency steps — swap A per task.)一个预训练的扩散模型是如何变成通用复原求解器的？(§3 —— 采样后验 p(x|y) ∝ p(y|x)p(x)；似然是已知的 A、先验是扩散模型，由数据一致性步骤引导——每个任务只需替换 A。)
• Why does an L2-trained super-resolver look blurry while a GAN/diffusion one looks sharp but can be wrong? (§5 — MSE returns the mean of many valid completions (a blur), the perceptual model commits to one sharp sample that may hallucinate detail; the perception–distortion trade-off.)为什么用 L2 训练的超分辨率器看起来模糊，而 GAN/扩散的看起来锐利却可能是错的？(§5 —— MSE 返回许多有效补全的均值（一团模糊），感知模型押定某一个锐利样本、可能臆造出细节；这就是感知–失真权衡。)
• The image from a phone camera is already processed — name the ISP stages and one multi-frame technique. (§4 — demosaic → white balance → denoise → tone-map → sharpen → gamma; HDR bracketing or burst low-light averaging (√N noise reduction).)手机相机出来的图像已经过处理——说出 ISP 各个阶段以及一种多帧技术。(§4 —— 去马赛克 → 白平衡 → 去噪 → 色调映射 → 锐化 → 伽马；HDR 包围曝光，或连拍弱光求平均（噪声降为 √N 分之一）。)