Filtering, convolution, edges, frequency

滤波、卷积、边缘与频率

Lesson 01 left us with a problem and a missing tool. Raw pixels are unstable to a shift or a lighting change, and we discovered that even sampling them honestly requires a low-pass blur we could not yet build — because that blur is a convolution. This lesson builds convolution from the arithmetic up: we slide a tiny weighted window over the grid, work one output pixel entirely by hand, and discover that edge detectors, blurs, and the anti-alias filter are all the same operation with different weights — and that a CNN's learned kernels are just this, with the weights set by gradient descent.

第 01 课给我们留下了一个问题和一个缺失的工具。原始像素对一次平移或一次光照变化都不稳定，而且我们发现，即便只是忠实地采样它们，也需要一个我们当时还造不出来的低通模糊——因为那个模糊就是一个卷积。本课从算术层面把卷积从头造出来：我们把一个小小的加权窗口滑过网格，完全用手算出一个输出像素，并发现边缘检测器、模糊和抗混叠滤波器其实都是同一个操作、只是权重不同——而 CNN 学到的卷积核也不过如此，只是权重由梯度下降来设定。

1 · The sliding weighted sum — worked by hand

1 · 滑动加权求和——手把手算一遍

A spatial filter replaces every pixel by a weighted combination of its neighbors. The weights live in a small array called the kernel (or filter), here 3×3. We place the kernel over a 3×3 neighborhood of the input, multiply each input pixel by the kernel weight sitting on top of it, and sum — a multiply-accumulate. That single number becomes the output pixel at the window's center. Then we slide the window by the stride and repeat. Formally, the operation libraries implement is:

空间滤波器把每个像素替换为其邻域的一个加权组合。权重存放在一个称为卷积核（或滤波器）的小数组里，这里是 3×3。我们把卷积核盖在输入的一个 3×3 邻域上，把每个输入像素乘以压在它上面的卷积核权重，再求和——这就是一次乘加。这一个数就成为窗口中心处的输出像素。然后我们按步幅滑动窗口并重复。形式上，各个库实现的操作是：

Let us actually compute one. Take this input patch and a Sobel-x kernel (a horizontal-gradient detector — we will justify its weights in section 3):

我们真的来算一个。取下面这个输入图块和一个 Sobel-x 卷积核（一个水平梯度检测器——我们会在第 3 节说明它的权重是怎么来的）：

Step by step. Sum the three row results: 70 + 140 + 70 = +280. The output at the center is a large positive number, signaling a strong dark-to-bright transition going left-to-right — which is exactly what is in the patch (10s on the left, 80s on the right). Now imagine sliding the same kernel onto a flat region where every input pixel is 10: each row gives (10·−1)+(10·0)+(10·+1) = 0, so the output is 0. That is the whole idea of an edge filter: it outputs zero on flat regions (the negative and positive weights cancel) and a large magnitude where intensity changes. The sign of the weights — negatives on the left, positives on the right — is what makes it a left-to-right difference, i.e. a discrete derivative.

一步一步来。把三行的结果相加：70 + 140 + 70 = +280。中心处的输出是一个很大的正数，标示出从左到右有一次强烈的暗到亮过渡——这恰恰就是图块里的情况（左边是 10、右边是 80）。现在设想把同一个卷积核滑到一个每个输入像素都是 10 的平坦区域上：每一行给出 (10·−1)+(10·0)+(10·+1) = 0，所以输出是 0。这就是边缘滤波器的全部思想：它在平坦区域输出零（负权重与正权重相互抵消），而在强度发生变化处输出很大的幅值。权重的符号——左负右正——正是让它成为一次从左到右的差分、也就是一个离散导数的原因。

Three knobs control the geometry of the sliding:

有三个旋钮控制着滑动的几何：

• Padding — what to assume beyond the border (zeros, reflection, replication). Without padding the output shrinks by (kernel − 1) on each axis; "same" padding keeps the size. Which padding you pick matters: zero-padding invents a black border that the kernel reads as real pixels, so a blur near the edge averages in fake darkness and the output edge darkens (and an edge filter sees a spurious step). Reflection padding mirrors the actual border pixels back, inventing no new intensity, so edge outputs stay faithful — the usual default for image filtering.填充（padding）——在边界之外假设有什么（零、反射、复制）。不做填充时，输出在每个轴上会缩小 (kernel − 1)；"same" 填充则保持尺寸。选哪种填充很重要：零填充凭空造出一圈黑色边框，卷积核会把它当成真实像素，于是靠近边缘处的模糊会把虚假的黑暗平均进来、让输出边缘变暗（而边缘滤波器则会看到一个虚假的阶跃）。反射填充把真实的边界像素镜像回去，不凭空造出新的强度，因此边缘处的输出保持忠实——这是图像滤波常用的默认方式。
• Stride — how far the window jumps each step. Stride 2 outputs every other position, halving resolution — a downsampling we revisit in section 4 and in lesson 06.步幅（stride）——窗口每一步跳多远。步幅 2 只输出每隔一个的位置，把分辨率减半——这是一种下采样，我们会在第 4 节和第 06 课再讨论。
• Dilation — spacing the kernel taps apart to see a wider area for the same number of weights (used heavily by segmentation's atrous convolutions in lesson 10).空洞/膨胀（dilation）——把卷积核的抽头间隔开来，用同样数量的权重看到更大的区域（第 10 课分割中的空洞卷积大量使用它）。

Output size for a square input W, kernel k, padding p, stride s, dilation d:

对一个方形输入 W、卷积核 k、填充 p、步幅 s、空洞 d 而言，输出尺寸为：

Plug in the everyday "same-size 3×3, stride 1" case (k=3, p=1, s=1, d=1): W_out = ⌊(W + 2 − 2 − 1)/1⌋ + 1 = W. Size preserved, as designed. Now turn the stride knob to 2 on an 8-wide input (W=8, k=3, p=1, s=2, d=1): W_out = ⌊(8 + 2 − 2 − 1)/2⌋ + 1 = ⌊7/2⌋ + 1 = ⌊3.5⌋ + 1 = 3 + 1 = 4 — the floor discards the half, and the resolution roughly halves, as a stride-2 downsample should.

代入日常的"同尺寸 3×3、步幅 1"情形（k=3, p=1, s=1, d=1）：W_out = ⌊(W + 2 − 2 − 1)/1⌋ + 1 = W。尺寸得以保持，正如设计的那样。现在把步幅旋钮拧到 2，作用在一个宽为 8 的输入上（W=8, k=3, p=1, s=2, d=1）：W_out = ⌊(8 + 2 − 2 − 1)/2⌋ + 1 = ⌊7/2⌋ + 1 = ⌊3.5⌋ + 1 = 3 + 1 = 4——向下取整丢掉了那个半，分辨率大致减半，正如一次步幅为 2 的下采样应有的表现。

2 · Correctness: this is cross-correlation, not convolution

2 · 正确性：这是互相关，不是卷积

The formula in section 1 — y[i,j] = ∑∑ x[i+u, j+v]·k[u,v] — is the operation PyTorch, TensorFlow, OpenCV, and friends all call "convolution." It is actually cross-correlation: the kernel is laid over the image as-is. True mathematical convolution flips the kernel in both axes first:

第 1 节里的公式——y[i,j] = ∑∑ x[i+u, j+v]·k[u,v]——就是 PyTorch、TensorFlow、OpenCV 及其同类都称之为"卷积"的操作。它实际上是互相关：卷积核原样盖在图像上。真正的数学卷积会先把卷积核在两个轴上都翻转：

The minus signs mean the kernel is reflected through its center (rotated 180°) before the multiply-accumulate. The flip is what makes true convolution commutative and associative — properties the frequency-domain theorem in section 4 relies on. So the two operations are related by a 180° rotation of the kernel:

这些负号意味着在做乘加之前，卷积核先绕其中心做了反射（旋转 180°）。正是这次翻转让真正的卷积具备交换律与结合律——这些性质正是第 4 节的频域定理所依赖的。所以这两个操作之间相差卷积核的一次 180° 旋转：

Why a CNN does not care. If the kernel weights are learned by gradient descent, the network is free to discover whichever orientation of the weights minimizes the loss. Whether the framework flips the kernel or not just relabels which learned pattern ends up where; the set of functions the layer can represent is identical. So "convolution" in deep learning is cross-correlation, and nobody flips anything — the distinction only bites when you (a) compare against a textbook formula or a classical filter, or (b) need the algebraic properties (commutativity) that the flip provides. For asymmetric hand-designed kernels like Sobel, getting the flip wrong flips the sign of your gradient (left-edges read as right-edges), which is a real bug in classical code.

为什么 CNN 对此并不在乎。如果卷积核权重是由梯度下降学出来的，那么网络可以自由地找到任何能最小化损失的权重朝向。框架翻不翻转卷积核，只是重新标注哪个学到的模式最终落在哪里；该层能表示的函数集合是完全一样的。所以深度学习里的"卷积"就是互相关，谁都不翻转任何东西——这个区别只有在你 (a) 与教科书公式或某个经典滤波器对照，或 (b) 需要翻转所提供的代数性质（交换律）时，才会咬你一口。对于像 Sobel 这样非对称的手工设计卷积核，把翻转弄错会翻转你梯度的符号（左边缘被读成右边缘），这在经典代码里是一个实打实的 bug。

2.5 · Interactive · the convolution explorer

2.5 · 交互 · 卷积探索器

This is the centerpiece of the track. Pick a kernel, hover the input grid to choose which 3×3 window to inspect, and read off the full multiply-accumulate that produces that one highlighted output pixel — the same arithmetic we did by hand above, on real numbers. The full output panel updates so you can see what each kernel does: identity copies, the blurs smear, Sobel lights up edges of one orientation, Laplacian rings around all edges, sharpen exaggerates them.

这是整门课的核心部件。选一个卷积核，把鼠标悬停在输入网格上以选择要查看哪个 3×3 窗口，然后读出产生那一个高亮输出像素的完整乘加过程——就是我们上面手算的那套算术，只是换成了真实数值。完整输出面板会随之更新，让你看到每个卷积核到底做了什么：identity 原样复制，各种模糊把图像抹开，Sobel 点亮某一朝向的边缘，拉普拉斯在所有边缘周围产生振铃，锐化则把它们夸大。

// img is an 8x8 array of 0..255; k is a 3x3 kernel (row-major)
function convAt(img, k, r, c){            // one output pixel = multiply-accumulate
  let acc = 0, terms = [];
  for (let u=-1; u<=1; u++){
    for (let v=-1; v<=1; v++){
      const ri = r+u, ci = c+v;
      const px = (ri>=0 && ri<8 && ci>=0 && ci<8) ? img[ri][ci] : 0; // zero-pad
      const w  = k[u+1][v+1];
      acc += px * w;                       // x[i+u,j+v] * k[u,v]  (cross-correlation)
      terms.push(px + '·' + w);
    }
  }
  return { value: acc, terms };            // value is the highlighted output pixel
}
const kernelSum = k.flat().reduce((a,b)=>a+b, 0);   // 1 → blur, 0 → edge filter

3 · The kernel zoo — low-pass vs high-pass, and separability

3 · 卷积核动物园——低通与高通，以及可分离性

Every linear filter is one of two characters, readable straight off the kernel:

每一个线性滤波器都属于两种性格之一，直接从卷积核上就能读出来：

The kernel sum is the giveaway. Sum 1 preserves overall brightness (a flat region maps to itself) → it is a smoothing / blurring low-pass filter, passing slow variations and attenuating fast ones. Sum 0 zeroes out flat regions and responds only to change → it is a derivative / high-pass filter, passing fast variations (edges, texture, noise) and killing slow ones. Sharpen is literally identity + λ·(high-pass) = original plus extra edge energy, which is why its center weight exceeds 1. Notice the trade-off baked in: blurring removes noise but also removes edges; edge filters find edges but amplify noise — because noise is high frequency. The Canny detector chains both (Gaussian smooth → Sobel gradient → thin to non-maximum edges → hysteresis threshold) to get clean edges despite noise.

卷积核之和是关键线索。和为 1 会保持整体亮度（平坦区域映射到自身）→ 它是一个平滑/模糊的低通滤波器，通过慢变化、衰减快变化。和为 0 会把平坦区域清零、只对变化作出响应 → 它是一个导数/高通滤波器，通过快变化（边缘、纹理、噪声）、抹掉慢变化。锐化实际上就是 identity + λ·(high-pass) = 原图加上额外的边缘能量，这正是它中心权重超过 1 的原因。注意其中内建的权衡：模糊去掉噪声，但也去掉了边缘；边缘滤波器找出边缘，却也放大了噪声——因为噪声本身就是高频。Canny 检测器把两者串起来（高斯平滑 → Sobel 梯度 → 细化为非极大值边缘 → 滞后阈值），从而在有噪声的情况下也能得到干净的边缘。

Separability — and why it is a big deal. Some 2D kernels factor into the outer product of two 1D kernels. The Gaussian is the canonical example:

可分离性——以及它为什么了不起。有些二维卷积核可以分解为两个一维卷积核的外积。高斯核就是最典型的例子：

When a kernel separates, you can convolve with the 1D kernel along rows, then along columns, and get the identical result — but for far fewer multiplies. A k×k 2D convolution costs k² multiply-accumulates per output pixel; two 1D passes cost 2k. For a 3×3 that is 9 → 6 (1.5×), but for an 11×11 Gaussian it is 121 → 22 — a 5.5× speedup, and the gap widens with kernel size. This is why big blurs are always done separably, and it foreshadows lesson 06's depthwise-separable convolutions, which apply the same factorization trick to learned kernels to slash the FLOPs of mobile CNNs.

当一个卷积核可分离时，你可以先沿行、再沿列分别用一维卷积核做卷积，得到完全相同的结果——但乘法次数少得多。一个 k×k 的二维卷积每个输出像素需要 k² 次乘加；两趟一维卷积只需 2k 次。对 3×3 来说是 9 → 6（1.5 倍），但对 11×11 的高斯核则是 121 → 22——5.5 倍加速，而且卷积核越大，差距越大。这正是大尺寸模糊总是用可分离方式来做的原因，它也为第 06 课的深度可分离卷积埋下伏笔——后者把同样的分解技巧用在学出来的卷积核上，从而大幅削减移动端 CNN 的 FLOPs。

4 · The frequency view — closing the loop on aliasing

4 · 频域视角——为混叠收尾

There is a second way to see filtering, and it makes lesson 01's aliasing precise. Any image can be written as a sum of sinusoidal patterns of different frequencies (its Fourier spectrum). Low frequencies are slow changes — sky gradients, broad surfaces, smooth shading. High frequencies are rapid changes — edges, fine texture, noise, JPEG block artifacts. The convolution theorem states that convolving in the spatial domain is the same as multiplying spectra in the frequency domain:

看待滤波还有第二种方式，它把第 01 课的混叠讲精确了。任何图像都可以写成不同频率的正弦图案之和（它的傅里叶谱）。低频是慢变化——天空的渐变、大片表面、平滑的明暗过渡。高频是快变化——边缘、精细纹理、噪声、JPEG 块伪影。卷积定理指出，在空间域做卷积等价于在频域把谱相乘：

where X and K are the Fourier transforms of the image x and the kernel k — each tells you how much energy that signal carries at every frequency.

其中 X 和 K 分别是图像 x 与卷积核 k 的傅里叶变换——各自告诉你该信号在每个频率上携带了多少能量。

So a filter is just a frequency mask. A Gaussian's spectrum is itself a smooth bump concentrated at low frequencies → multiplying by it attenuates the high frequencies → it is a low-pass filter (hence "blur removes detail"). A Sobel/Laplacian spectrum is small at low frequencies and large at high → it passes the fast stuff → high-pass. The names in section 3 were not metaphors; they are literally what the kernel does to the spectrum. The intuition is a width trade: a wide Gaussian in space is a narrow bump in frequency, so a wide blur kernel keeps only the slowest variations and discards everything fast; conversely a derivative kernel's frequency response grows with frequency, so it ignores flat regions and lights up where the signal changes quickest.

所以一个滤波器不过是一个频率掩码。高斯核的谱本身就是一个集中在低频的平滑凸包 → 乘以它会衰减高频 → 它是一个低通滤波器（这就是"模糊去掉细节"的由来）。Sobel/拉普拉斯的谱在低频处很小、在高频处很大 → 它通过快变化的东西 → 高通。第 3 节里的这些名字并不是比喻；它们就是卷积核对谱做的事情。直觉在于一个宽度上的权衡：空间中越宽的高斯，在频率上就是越窄的凸包，所以一个宽的模糊核只保留最慢的变化、丢掉一切快变化；反过来，导数核的频率响应随频率增大而增大，所以它忽略平坦区域、在信号变化最快处点亮。

Now recall lesson 01: downsampling drops the sample rate f_s, dragging the Nyquist limit f_s/2 down. Any frequency content above the new Nyquist aliases — folds down into a false low-frequency pattern. The fix falls right out of the frequency view: apply a low-pass filter to delete everything above the new Nyquist, then decimate. The low-pass filter is exactly a Gaussian blur. So the correct downsampling operation is:

现在回想第 01 课：下采样会降低采样率 f_s，把奈奎斯特限 f_s/2 往下拉。任何高于新奈奎斯特限的频率内容都会混叠——折叠成一个虚假的低频图案。修复办法直接从频域视角掉出来：先施加一个低通滤波器，删掉一切高于新奈奎斯特限的内容，然后再抽取。这个低通滤波器恰恰就是一个高斯模糊。所以正确的下采样操作是：

This is the "anti-alias filter" the lesson 01 widget toggled — now we can actually build it. It is also why strided convolutions and pooling in CNNs (lesson 06) are mild anti-aliasing built into the architecture, and why naive nearest-neighbor resize in a serving pipeline (lesson 01's preprocessing-parity rule) can introduce aliasing the model never saw in training.

这就是第 01 课的小部件所切换的那个"抗混叠滤波器"——现在我们真的能把它造出来了。它也解释了为什么 CNN 里的带步幅卷积和池化（第 06 课）是内建在架构中的温和抗混叠，以及为什么服务流水线里天真的最近邻缩放（第 01 课的预处理一致性规则）会引入模型在训练时从未见过的混叠。

Where this points next

这将指向何处

We can now extract local structure from an image — edges, blobs, smoothed regions — with a single sliding weighted sum, and we understand it from three angles: arithmetic, sign/derivative, and frequency. But one filter response is not enough to match a scene across two photographs. An edge map tells you where intensity changes, not which specific point in image A corresponds to the same physical point in image B. To stitch panoramas, track motion, or recover 3D, we need repeatable keypoints — locations we can re-find from a different viewpoint — and invariant descriptors that recognize them despite rotation, scale, and lighting. Lesson 03 builds those, starting from the question of which pixels are even localizable (corners win, edges lose), and shows that the same gradient operators we built here are the raw material for both the Harris corner score and the SIFT descriptor.

现在我们能用一个滑动加权求和从图像中提取局部结构——边缘、斑点、平滑区域——并且从三个角度理解了它：算术、符号/导数、频率。但单个滤波器响应还不足以在两张照片之间匹配同一个场景。一张边缘图告诉你强度在哪里变化，而不是图像 A 中哪个具体的点对应于图像 B 中同一个物理点。要拼接全景、跟踪运动或恢复三维，我们需要可重复的关键点——能从不同视角重新找到的位置——以及不变的描述子，让它们在旋转、尺度和光照变化下仍能被认出。第 03 课会把这些造出来，从"到底哪些像素是可定位的"这个问题出发（角点胜出，边缘落败），并说明我们这里造出来的同一批梯度算子，正是 Harris 角点得分和 SIFT 描述子共同的原材料。

Interview prompts

面试题

• Convolution vs cross-correlation — what do deep-learning frameworks actually compute? (§2 — cross-correlation; true convolution flips the kernel; learned kernels are indifferent because they can learn either orientation.)卷积与互相关——深度学习框架实际计算的是哪一个？(§2 —— 互相关；真正的卷积会翻转卷积核；学出来的卷积核对此无所谓，因为它可以学到任一朝向。)
• Why should you blur before downsampling? (§4 — decimation lowers Nyquist; content above it aliases into false low-frequency structure; a Gaussian low-pass deletes it first.)为什么下采样前应该先模糊？(§4 —— 抽取会降低奈奎斯特限；高于它的内容会混叠成虚假的低频结构；先用一个高斯低通把它删掉。)
• How do stride, padding, and dilation change output shape and receptive field? (§1 — W_out = ⌊(W+2p−d(k−1)−1)/s⌋+1; stride decimates, padding preserves size, dilation widens reach for the same weights.)步幅、填充和空洞如何改变输出形状与感受野？(§1 —— W_out = ⌊(W+2p−d(k−1)−1)/s⌋+1；步幅做抽取，填充保持尺寸，空洞在权重数不变的情况下扩大触及范围。)
• Why are edge detectors sensitive to noise? (§3, §4 — they are high-pass; noise is high frequency, so derivative filters amplify it — hence Canny smooths first.)为什么边缘检测器对噪声敏感？(§3、§4 —— 它们是高通；噪声是高频，所以导数滤波器会放大它——这正是 Canny 先做平滑的原因。)
• What does the kernel sum tell you? (§3 — sum 1 ⇒ low-pass/brightness-preserving; sum 0 ⇒ high-pass/derivative.)卷积核之和告诉你什么？(§3 —— 和为 1 ⇒ 低通/保持亮度；和为 0 ⇒ 高通/导数。)
• Why is a separable Gaussian faster? (§3 — k² multiplies become 2k via two 1D passes; e.g. 11×11 goes 121 → 22.)为什么可分离的高斯更快？(§3 —— 通过两趟一维卷积，k² 次乘法变成 2k 次；例如 11×11 从 121 → 22。)