Semantic, instance, and panoptic segmentation

语义、实例与全景分割

Lesson 09 localized objects with boxes and learned that a box is the coarsest possible answer to "where?" — it brackets an object but also swallows the background inside the rectangle. Some tasks (a tumor outline, a drivable-road mask, a green-screen matte) need a label at every pixel. That is a fundamentally harder output shape, and earning it forces us to confront a structural tension inside the CNN itself.

第 09 课用边界框定位物体，并让我们明白框是对"在哪里？"这个问题最粗糙的答案——它框住了一个物体，但也把矩形内的背景一起吞了进去。有些任务（勾勒肿瘤轮廓、可行驶道路掩码、绿幕抠像）需要在每一个像素上给出标签。这是一种本质上更难的输出形态，而要拿下它，就迫使我们直面 CNN 内部的一个结构性张力。

1 · Three tasks, three output shapes

1 · 三种任务，三种输出形态

"Segmentation" is three problems wearing one name. They differ in what a pixel's label is allowed to mean.

"分割"是三个共用一个名字的问题。它们的区别在于一个像素的标签被允许表示什么含义。

The vocabulary that makes panoptic precise: "things" are countable objects (people, cars) where instance identity matters; "stuff" is amorphous regions (sky, grass, road) where counting is meaningless. Semantic segmentation handles both but cannot separate two adjacent things; instance segmentation separates things but ignores stuff; panoptic unifies them — every pixel gets exactly one (class, id) pair with no overlaps and no gaps. Hold onto that "no overlaps, no gaps" constraint; it is what makes the panoptic metric in section 4 non-trivial.

让全景分割变精确的术语："things"是可数的物体（人、汽车），其实例身份很重要；"stuff"是无固定形态的区域（天空、草地、道路），对它计数毫无意义。语义分割两者都能处理，但无法区分两个相邻的 thing；实例分割能区分 things，却忽略 stuff；全景分割把二者统一起来——每个像素恰好获得一个 (类别, id) 对，既无重叠也无空隙。记住"无重叠、无空隙"这个约束；正是它让第 4 节的全景度量变得不平凡。

2 · Why dense prediction needs an encoder-decoder

2 · 为什么稠密预测需要编码器-解码器

Recall the receptive-field story from lesson 06: stacking strided convolutions and pooling grows each unit's receptive field, so deep layers "see" a large region and can answer what is this? But that same downsampling throws away spatial resolution. A 224×224 image after a typical backbone is a 7×7 feature grid — each cell summarizes a ~32×32 image patch. That grid is rich in semantics (it knows "dog") and useless for boundaries (it cannot tell you which of the 1024 pixels in that patch are dog vs grass).

回忆第 06 课关于感受野的故事：堆叠带步幅的卷积和池化会增大每个单元的感受野，于是深层"看"到一大片区域，能够回答这是什么？但同样的下采样也丢弃了空间分辨率。一张 224×224 的图像经过典型骨干网络后是一个 7×7 的特征网格——每个格子概括了一个约 32×32 的图像块。这个网格语义丰富（它知道"狗"），却对边界毫无用处（它无法告诉你那个块里的 1024 个像素中哪些是狗、哪些是草）。

This is the central tension of dense prediction, and it is worth stating as a law:

这是稠密预测的核心张力，值得当作一条定律来陈述：

2.5 · Interactive · downsample → upsample → skip

2.5 · 交互 · 下采样 → 上采样 → 跳跃连接

This is the lesson's central claim made tangible: downsampling averages a crisp boundary into blocky cells, bilinear upsampling can only smear those blocks back out (it cannot invent the lost edge), and a skip connection hands the decoder the original sharp boundary that downsampling destroyed. Raise the downsample factor and watch the boundary error climb; toggle the skip and watch it collapse.

这是把本课的核心论断变得可触摸：下采样把一条锐利的边界平均成块状的格子，双线性上采样只能把这些块重新涂抹开（它无法凭空造出丢失的边缘），而跳跃连接把下采样所摧毁的那条原始锐利边界交还给解码器。调大下采样倍率，看边界误差攀升；切换跳跃连接，看它骤然坍塌。

// downsample: average each f×f block → coarse cell (the "deep, low-res" map)
function downsample(mask, N, f){
  const M = N / f, coarse = [];
  for (let r=0;r<M;r++){ coarse[r]=[];
    for (let c=0;c<M;c++){ let s=0;
      for (let dr=0;dr<f;dr++) for (let dc=0;dc<f;dc++) s += mask[r*f+dr][c*f+dc];
      coarse[r][c] = s/(f*f);                 // fractional → boundary detail is gone
    }}
  return coarse;
}
// bilinear upsample back to N×N: smears blocks, cannot recreate the lost edge
function bilinear(coarse, M, N){ /* distance-weighted avg of 4 nearest coarse cells */ }
// skip ON → decoder uses the ORIGINAL fine mask's boundary, which the coarse map lost

So a segmentation network has two halves. The encoder (a classification backbone like a ResNet) downsamples, building semantics while shedding resolution. The decoder reverses that: it upsamples the coarse semantic map back to full image resolution, pulling in detail from shallow encoder layers as it goes. The Fully Convolutional Network (FCN, Long et al. 2015) was the first to make this explicit: replace the classifier's final fully-connected layers with convolutions and an upsampling stage, so the network maps an image to a same-shape label map.

所以一个分割网络有两半。编码器（像 ResNet 这样的分类骨干网络）做下采样，一边构建语义、一边丢弃分辨率。解码器则将其逆转：它把粗糙的语义图上采样回完整的图像分辨率，一路从浅层编码器特征中拉入细节。全卷积网络（FCN，Long 等，2015）第一个把这一点讲明白：用卷积和一个上采样阶段替换分类器最后的全连接层，让网络把一张图像映射到一张同形状的标签图。

3 · Building the decoder for real

3 · 动真格地搭建解码器

"Upsample" is the word that hides all the work. A 7×7 grid has 49 cells; the output needs ~50,000 pixels. Where do the in-between values come from? There are three mechanisms, and knowing the difference is a common interview probe.

"上采样"这个词把所有的活儿都藏了起来。一个 7×7 网格有 49 个格子；而输出需要约 50,000 个像素。中间那些值从哪儿来？有三种机制，弄清它们的区别是常见的面试考点。

3a · Transposed convolution (learned upsampling)

3a · 转置卷积（可学习的上采样）

A normal (strided) convolution maps many input pixels to one output. A transposed convolution — sometimes loosely called "deconvolution," though it does not invert anything — runs that mapping backward: it takes each input value, multiplies it by a learned kernel, and scatters the result into a larger output grid, summing where scatters overlap. With stride 2 it doubles spatial size. The kernel weights are learned, so the network learns how to upsample rather than using a fixed rule.

一次普通的（带步幅的）卷积把许多输入像素映射到一个输出。转置卷积——有时被含糊地称作"反卷积（deconvolution）"，尽管它并不还原任何东西——把这个映射反着跑：它取每个输入值，乘以一个可学习的卷积核，再把结果散布到一个更大的输出网格上，在散布重叠处求和。步幅为 2 时它把空间尺寸翻倍。卷积核权重是学出来的，所以网络学的是如何上采样，而不是套用一条固定规则。

The cost: where strided scatters overlap unevenly, you get the infamous checkerboard artifact — a grid of brighter/darker squares, because some output pixels receive more kernel contributions than others (e.g. a 3×3 kernel at stride 2, in 2-D, gives some output pixels 1 kernel tap and others up to 4). The fix is either a kernel size divisible by the stride, or the next option.

代价是：当带步幅的散布不均匀地重叠时，你会得到臭名昭著的棋盘格伪影——一片明暗相间的方格，因为有些输出像素接收到的卷积核贡献比别的多（例如二维下步幅为 2 的 3×3 卷积核，会让有些输出像素只接收 1 个核抽头、有些则多达 4 个）。修复办法要么让卷积核尺寸能被步幅整除，要么改用下一个方案。

A 1-D worked example makes the uneven overlap concrete. Take a length-2 input [a, b], a stride-2 transposed conv with a width-3 kernel [k₀, k₁, k₂]. Each input value is scattered into the output, centered on its strided position, then overlaps are summed. Input a sits at output position 0 and paints positions {−1, 0, 1}; input b sits at output position 2 and paints {1, 2, 3}:

一个一维的算例把不均匀的重叠讲具体。取一个长度为 2 的输入 [a, b]，一个步幅为 2、核宽为 3 的转置卷积 [k₀, k₁, k₂]。每个输入值都以其带步幅的位置为中心散布到输出上，然后重叠处求和。输入 a 位于输出位置 0，涂到位置 {−1, 0, 1}；输入 b 位于输出位置 2，涂到 {1, 2, 3}：

The positions that fall between two strided centers receive two kernel taps; the others receive only one. Even with a constant input (a = b) and constant kernel, the output alternates 1·k vs 2·k — a periodic bright/dark ripple with no signal behind it. In 2-D this overlap count is the product of the per-axis counts, so a stride-2 3×3 kernel gives output pixels 1, 2, or 4 taps → the familiar checkerboard tiling. A kernel width that is a multiple of the stride makes every output position receive the same tap count, which is why "kernel divisible by stride" removes the artifact.

落在两个带步幅中心之间的位置接收两个核抽头；其余位置只接收一个。即便输入恒定（a = b）、核也恒定，输出仍会在 1·k 与 2·k 之间交替——一圈周期性的明暗涟漪，其背后并无任何信号。在二维下，这个重叠计数是各轴计数的乘积，所以步幅为 2 的 3×3 卷积核会让输出像素获得 1、2 或 4 个抽头 → 就是那熟悉的棋盘格铺排。让核宽是步幅的整数倍，就能使每个输出位置接收到相同的抽头数，这正是"核能被步幅整除"消除该伪影的原因。

3b · Bilinear upsample + convolution (the modern default)

3b · 双线性上采样 + 卷积（现代默认做法）

Separate the two jobs. First resize the feature map with a fixed, parameter-free interpolation — bilinear: each new pixel is a distance-weighted average of its four nearest source pixels. This is smooth and artifact-free but blurry and learns nothing. Then apply an ordinary 3×3 convolution to sharpen and let the network add learnable structure on top of the smooth resize. "Resize-then-conv" avoids checkerboards and is the standard decoder upsampling step today.

把两件事分开。先用一种固定、无参数的插值——双线性——缩放特征图：每个新像素是它最近的四个源像素的距离加权平均。这平滑、无伪影，却模糊、且什么都学不到。然后施加一次普通的 3×3 卷积来锐化，让网络在平滑缩放之上叠加可学习的结构。"先缩放再卷积（resize-then-conv）"避开了棋盘格，是当今标准的解码器上采样步骤。

3c · Unpooling (remember where the max was)

3c · 反池化（记住最大值在哪里）

If the encoder used max-pooling, it threw away everything except the maximum in each window — and crucially, where that maximum was. Max-unpooling records those argmax positions on the way down and, on the way up, places each value back at its remembered location, zeroing the rest. This restores sharp boundaries cheaply (SegNet used it) because boundary pixels are exactly the high-response locations pooling kept.

如果编码器用了最大池化，它就丢弃了每个窗口里除最大值以外的一切——而关键在于，也丢弃了那个最大值在哪里。最大反池化（max-unpooling）在下行时记录这些 argmax 位置，在上行时把每个值放回它被记住的位置，其余置零。这以很低的代价恢复了锐利的边界（SegNet 用了它），因为边界像素恰恰就是池化所保留的那些高响应位置。

3d · U-Net skip connections — give the decoder back what the encoder discarded

3d · U-Net 跳跃连接——把编码器丢掉的东西还给解码器

Upsampling alone cannot invent detail it never had: a 7×7 semantic map blown up to full size is just a blurry colored blob. The U-Net insight (Ronneberger et al. 2015, originally for biomedical images) is to copy each encoder feature map across to the matching-resolution decoder stage and concatenate it. The decoder layer at resolution H/4 receives both the upsampled-semantic features (knows "this region is liver") and the original H/4 encoder features (knows "the edge is here"). It fuses class and boundary in one place. This is why U-Net dominates medical segmentation, where boundaries are the whole point and training sets are tiny.

仅靠上采样无法凭空造出它从未拥有的细节：把一个 7×7 的语义图放大到全尺寸，只是一团模糊的彩色斑块。U-Net 的洞见（Ronneberger 等，2015，最初用于生物医学图像）是把每一层编码器特征图复制到分辨率相匹配的解码器阶段并做拼接。分辨率为 H/4 的解码器层同时接收到上采样后的语义特征（知道"这块区域是肝脏"）和原始的 H/4 编码器特征（知道"边缘在这里"）。它在同一个地方融合了类别与边界。这正是 U-Net 在医学分割中占据主导地位的原因——那里边界就是全部意义所在，而训练集又非常小。

3e · Atrous / dilated convolution — keep resolution instead of recovering it

3e · 空洞/膨胀卷积——保住分辨率，而不是把它找回来

A different philosophy: what if we never downsampled so aggressively in the first place? A dilated (atrous, French for "with holes") convolution inserts gaps between kernel taps. A 3×3 kernel with dilation rate r=2 samples input pixels at offsets {−2, 0, +2} in each axis — it covers a 5×5 region but still uses only 9 weights. Tie this directly to lesson 06's receptive-field math: dilation multiplies the effective kernel extent by r, so receptive field grows without stride and without losing resolution.

另一种哲学：如果我们一开始就不那么激进地下采样呢？膨胀（atrous，法语意为"带孔"）卷积在卷积核抽头之间插入间隙。一个膨胀率 r=2 的 3×3 卷积核，在每个轴上以偏移 {−2, 0, +2} 采样输入像素——它覆盖了一个 5×5 的区域，却仍然只用 9 个权重。把这直接接到第 06 课的感受野公式上：膨胀把有效核尺寸乘以 r，于是感受野在不用步幅、不丢分辨率的情况下增长。

DeepLab stacks dilated convs and adds ASPP (Atrous Spatial Pyramid Pooling): several parallel dilated convs at different rates, capturing context at multiple scales without ever dropping below, say, H/8 resolution. The trade vs U-Net: dilation keeps a large feature map throughout (more memory, more compute) but needs less decoder; skips downsample then reconstruct (cheaper feature maps, more decoder machinery). Both reach full resolution; they just pay for it differently.

DeepLab 堆叠膨胀卷积，并加入 ASPP（空洞空间金字塔池化，Atrous Spatial Pyramid Pooling）：若干条不同膨胀率的并行膨胀卷积，在多个尺度上捕获上下文，同时分辨率始终不掉到比如 H/8 以下。与 U-Net 相比的权衡：膨胀卷积自始至终保持一张大的特征图（更多内存、更多计算），但需要的解码器更少；跳跃连接则是先下采样再重建（特征图更省，但解码器机构更多）。两者都能到达完整分辨率，只是付出的代价方式不同。

For instance segmentation the dominant pattern is different again: Mask R-CNN takes the lesson-09 detector (boxes + classes), and for each detected box adds a small fully-convolutional mask head that predicts a binary mask inside that box. Detection localizes; the mask head refines to pixels. That is why instance segmentation inherits all of detection's machinery (anchors/NMS or set prediction) plus a per-region decoder.

对实例分割，主流范式又不一样：Mask R-CNN 拿来第 09 课的检测器（边界框 + 类别），并为每个检测到的框加上一个小型的全卷积掩码头，在该框内预测一个二值掩码。检测负责定位；掩码头把结果细化到像素。这正是实例分割为何会继承检测的全套机构（锚框/NMS 或集合预测）再加上一个逐区域解码器的原因。

4 · Losses and metrics under class imbalance

4 · 类别不平衡下的损失与度量

The natural loss is per-pixel cross-entropy (lesson 08): treat each pixel as an independent classification. It works, but it has a fatal weakness on the tasks segmentation is often used for. Consider a 256×256 medical image where a tumor occupies 500 pixels out of 65,536 — 0.76% foreground. A model that predicts "background everywhere" gets 99.24% pixel accuracy and near-zero cross-entropy, while being clinically worthless. Cross-entropy averages over pixels, so the 65,000 easy background pixels drown out the 500 that matter.

最自然的损失是逐像素交叉熵（第 08 课）：把每个像素当作一次独立的分类。它能用，但在分割经常服务的那类任务上有一个致命弱点。设想一张 256×256 的医学图像，肿瘤在 65,536 个像素中只占 500 个——前景占 0.76%。一个"到处都预测背景"的模型能拿到 99.24% 的像素精度和近乎零的交叉熵，却在临床上毫无价值。交叉熵在像素上取平均，于是那 65,000 个容易的背景像素把真正重要的那 500 个淹没了。

Dice loss — optimize overlap, not per-pixel correctness

Dice 损失——优化重叠，而非逐像素的对错

The Dice coefficient measures overlap between predicted mask P and ground truth G directly, as a ratio that ignores the size of the background:

Dice 系数直接度量预测掩码 P 与真值 G 之间的重叠，是一个忽略背景大小的比值：

In a network we need this differentiable, so we use soft probabilities p_i ∈ [0,1] and one-hot targets g_i, with a small ε for numerical safety:

在网络中我们需要它可微，所以改用软概率 p_i ∈ [0,1] 和独热目标 g_i，并加上一个小的 ε 以保证数值安全：

Worked numbers, the imbalanced tumor. Take a toy 10-pixel image, 2 pixels are tumor (g = 1), 8 are background. A lazy model outputs p=0.1 everywhere.

算个具体数，不平衡的肿瘤。取一张玩具般的 10 像素图像，2 个像素是肿瘤（g = 1），8 个是背景。一个偷懒的模型到处都输出 p=0.1。

• Cross-entropy sees 8 pixels predicted ~right (−log 0.9 ≈ 0.105 each) and 2 wrong (−log 0.1 ≈ 2.30 each): mean ≈ (8·0.105 + 2·2.30)/10 ≈ 0.544 — and the gradient on those 2 pixels is diluted by the 8 confident ones.交叉熵看到 8 个像素预测得大致正确（每个 −log 0.9 ≈ 0.105）、2 个错误（每个 −log 0.1 ≈ 2.30）：均值 ≈ (8·0.105 + 2·2.30)/10 ≈ 0.544——而那 2 个像素上的梯度被 8 个自信的像素稀释了。
• Dice: intersection ∑ p·g = 2·(0.1) = 0.2; denominator ∑p + ∑g = (10·0.1) + 2 = 3. Dice = 2·0.2/3 ≈ 0.133, so Dice loss ≈ 0.867 — large, because the model is missing nearly all the overlap. Dice loss is computed over the foreground region as a whole, so it cannot be diluted by a sea of correct background. Push p on the two tumor pixels to 0.9 (background still 0.1): intersection ∑ p·g = 2·0.9 = 1.8; denominator ∑p + ∑g = (8·0.1 + 2·0.9) + 2 = 4.6; Dice = 2·1.8/4.6 ≈ 0.78 — the loss responds to exactly the pixels that matter. (Note what this 0.78 is: because it is computed from soft probabilities p_i ∈ [0,1], it is the differentiable training surrogate the optimizer descends, not the reported score; the evaluation metric first thresholds each prediction to a binary 0/1 mask and then computes the same Dice ratio on those hard masks.)Dice：交集 ∑ p·g = 2·(0.1) = 0.2；分母 ∑p + ∑g = (10·0.1) + 2 = 3。Dice = 2·0.2/3 ≈ 0.133，所以 Dice 损失 ≈ 0.867——很大，因为模型几乎错过了全部重叠。Dice 损失是把前景区域作为一个整体来计算的，所以它无法被一片正确的背景海洋稀释。把两个肿瘤像素上的 p 推到 0.9（背景仍为 0.1）：交集 ∑ p·g = 2·0.9 = 1.8；分母 ∑p + ∑g = (8·0.1 + 2·0.9) + 2 = 4.6；Dice = 2·1.8/4.6 ≈ 0.78——损失恰好对真正重要的那些像素作出响应。（注意这个 0.78 是什么：因为它是从软概率 p_i ∈ [0,1] 算出来的，它是优化器下降所用的可微训练代理，而不是最终报告的分数；评估指标会先把每个预测阈值化为二值 0/1 掩码，再在这些硬掩码上计算同样的 Dice 比值。）

That is the whole reason Dice (and its cousins: Tversky, which weights false positives vs false negatives asymmetrically; focal Dice; or a Dice+CE sum) wins on imbalanced segmentation. In practice people often combine Dice (overlap-aware, imbalance-robust) with cross-entropy (smoother gradients, calibrated per-pixel confidence) and get the best of both.

这就是 Dice（以及它的一众亲戚：非对称加权假正例与假负例的 Tversky；focal Dice；或 Dice+CE 之和）在不平衡分割上胜出的全部原因。实践中人们常把 Dice（关注重叠、对不平衡鲁棒）与交叉熵（梯度更平滑、逐像素置信度经过校准）组合起来，兼取两者之长。

The metrics: mIoU and Panoptic Quality

度量指标：mIoU 与全景质量

IoU (Intersection-over-Union, lesson 09) reused per-class: for class c, IoU = (pixels correctly labeled c) / (pixels predicted c OR truly c) = TP / (TP + FP + FN) in pixels. One line of arithmetic: a class with TP = 40 correctly-labeled pixels, FP = 10 over-predicted, and FN = 10 missed scores IoU = 40 / (40 + 10 + 10) = 40/60 ≈ 0.667. mIoU (mean IoU) averages this over all classes — and because it averages over classes, a rare class counts as much as a common one, which is exactly what you want and why mIoU is the standard semantic-segmentation metric.

IoU（交并比，第 09 课）按类复用：对类别 c，IoU =（被正确标为 c 的像素）/（预测为 c 或真实为 c 的像素）= 以像素计的 TP / (TP + FP + FN)。一行算术：某个类有 TP = 40 个正确标注的像素、FP = 10 个多预测的、FN = 10 个漏掉的，得分 IoU = 40 / (40 + 10 + 10) = 40/60 ≈ 0.667。mIoU（平均 IoU）在所有类别上对此取平均——而由于它是在类别上取平均，一个稀有类和一个常见类的权重一样重，这恰恰是你想要的，也是 mIoU 成为语义分割标准指标的原因。

Panoptic Quality (PQ) scores the unified task. You match each predicted segment to a ground-truth segment when their IoU > 0.5 (this threshold guarantees a unique match). Then:

全景质量（PQ）为统一任务打分。当一个预测片段与一个真值片段的 IoU > 0.5 时就将二者匹配（这个阈值保证了匹配是唯一的）。然后：

This factors cleanly into two interpretable parts: SQ (Segmentation Quality) = average IoU over matched segments — how good are the masks you got right? — and RQ (Recognition Quality) = |TP| / (|TP| + ½|FP| + ½|FN|), which is the F1 score over segments — did you find the right segments at all? A model can have high SQ (crisp masks) but low RQ (misses many objects), or vice versa, and PQ exposes which.

它干净地分解为两个可解释的部分：SQ（分割质量，Segmentation Quality）= 匹配片段上的平均 IoU——你做对的那些掩码有多好？——以及 RQ（识别质量，Recognition Quality）= |TP| / (|TP| + ½|FP| + ½|FN|)，也就是片段层面的 F1 分数——你究竟有没有找到正确的片段？一个模型可以有高 SQ（掩码锐利）却低 RQ（漏掉很多物体），或反之，而 PQ 会把是哪一种暴露出来。

Worked numbers. Suppose a scene gives 2 true-positive segments matched at IoU 0.8 and 0.6, plus 1 false positive (a predicted segment matching nothing) and 1 false negative (a real segment we missed). Then SQ = (0.8 + 0.6)/2 = 0.70 (average IoU over the matched segments) and RQ = |TP| / (|TP| + ½|FP| + ½|FN|) = 2 / (2 + 0.5 + 0.5) = 2/3 ≈ 0.667. So PQ = SQ × RQ = 0.70 × 0.667 ≈ 0.467. Check it against the direct one-line formula: PQ = (∑_TP IoU) / (|TP| + ½|FP| + ½|FN|) = (0.8 + 0.6) / (2 + 0.5 + 0.5) = 1.4 / 3 ≈ 0.467 — the two routes agree, which is exactly the PQ = SQ × RQ factorization made arithmetic.

算个具体数。假设一个场景给出 2 个真正例片段，匹配 IoU 分别为 0.8 和 0.6，外加 1 个假正例（一个匹配不到任何东西的预测片段）和 1 个假负例（一个我们漏掉的真实片段）。那么 SQ = (0.8 + 0.6)/2 = 0.70（匹配片段上的平均 IoU），RQ = |TP| / (|TP| + ½|FP| + ½|FN|) = 2 / (2 + 0.5 + 0.5) = 2/3 ≈ 0.667。于是 PQ = SQ × RQ = 0.70 × 0.667 ≈ 0.467。用那条直接的一行公式核对一下：PQ = (∑_TP IoU) / (|TP| + ½|FP| + ½|FN|) = (0.8 + 0.6) / (2 + 0.5 + 0.5) = 1.4 / 3 ≈ 0.467——两条路径结果一致，这正是 PQ = SQ × RQ 这个分解在算术上的体现。

5 · Promptable segmentation: SAM and SAM2

5 · 可提示分割：SAM 与 SAM2

Everything above assumes a fixed, closed set of classes baked in at training time. The Segment Anything Model (SAM, Meta 2023) reframes the task: instead of "label every pixel with one of N classes," it answers "given a prompt — a click, a box, a rough mask — return the mask of the thing being pointed at," for any object, class-agnostic. The architecture has three parts that map cleanly onto this lesson:

以上一切都假设有一组在训练时就固化好的、封闭的类别。Segment Anything Model（SAM，Meta 2023）重新定义了任务：它回答的不是"用 N 个类别之一给每个像素打标签"，而是"给定一个提示——一次点击、一个框、一个粗略掩码——返回被指向的那个东西的掩码"，面向任意物体、与类别无关。它的架构有三个部分，恰好能干净地对应到本课：

• A heavy image encoder (a ViT, lesson 13) run once per image to produce a dense embedding — the expensive part.一个笨重的图像编码器（一个 ViT，第 13 课），每张图像只跑一次，产生一个稠密嵌入——这是昂贵的部分。
• A lightweight prompt encoder that turns clicks/boxes/masks into embeddings.一个轻量的提示编码器，把点击/框/掩码转成嵌入。
• A fast mask decoder that fuses image + prompt embeddings (via cross-attention) and outputs masks in milliseconds.一个快速的掩码解码器，（通过交叉注意力）融合图像嵌入与提示嵌入，在毫秒级内输出掩码。

The split matters for product design: encode the image once, then interact in real time as the user clicks. SAM was trained on ~1B masks via a data engine (model proposes, humans correct, model retrains). SAM2 (2024) extends this to video by adding a memory bank so a single click propagates a mask across frames — which is precisely the identity-over-time problem the next lesson tackles. Promptable models also change the annotation workflow: they become fast interactive labelers and generic mask proposers, shifting the engineering question from "train a closed-set head" to "manage prompt ambiguity, mask-proposal quality, and how much human correction remains."

这种拆分对产品设计很重要：图像只编码一次，随后随着用户点击进行实时交互。SAM 通过一个数据引擎（模型提议、人工修正、模型再训练）在约 10 亿个掩码上训练。SAM2（2024）通过加入一个记忆库把它扩展到视频，使得单次点击就能把一个掩码在各帧之间传播——这正是下一课要攻克的"跨时间保持身份"的问题。可提示模型还改变了标注工作流：它们成为快速的交互式标注器和通用的掩码提议器，把工程问题从"训练一个封闭集的头"转变为"管理提示的歧义、掩码提议的质量，以及还需要多少人工修正"。

Where this points next

这将指向何处

A mask answers "which pixels?" beautifully, but it is still a static, per-frame, unstructured answer. It does not tell you the 17 joints of a person and how they articulate, and — critically — when SAM2 had to propagate a mask across video frames, it needed something masks alone do not provide: a notion of the same object over time. Lesson 11 turns to sparse structured outputs: keypoints and pose (predicting specific landmarks as heatmaps, decoded with soft-argmax), then tracking — detecting objects per frame and stitching them into persistent identities with motion models and data association. And it will end by noticing that associating identity by appearance is, at bottom, a distance-in-an-embedding-space problem — which is lesson 12.

掩码把"是哪些像素？"回答得很漂亮，但它仍然是一个静态、逐帧、无结构的答案。它不会告诉你一个人的 17 个关节以及它们如何联动，而且——关键在于——当 SAM2 不得不把一个掩码在视频各帧间传播时，它需要一样掩码本身并不提供的东西：跨时间的同一个物体这一概念。第 11 课转向稀疏结构化输出：关键点与姿态（把特定地标预测成热力图，用 soft-argmax 解码），然后是跟踪——逐帧检测物体，并用运动模型和数据关联把它们缝合成持续的身份。它最后会注意到，按外观来关联身份，本质上是一个"嵌入空间中的距离"问题——这就是第 12 课。

Interview prompts

面试题

• Semantic vs instance vs panoptic: what exactly does each pixel's label mean, and what is "stuff" vs "things"?语义、实例与全景：每个像素的标签究竟表示什么，"stuff"与"things"又各指什么？
• Why does dense prediction need an encoder-decoder rather than one feature map? Tie it to the depth/resolution trade.为什么稠密预测需要编码器-解码器，而不是单一特征图？请把它联系到深度/分辨率的权衡上。
• Name three ways to upsample in a decoder and the failure mode of each. When does a transposed conv produce checkerboards?说出解码器中三种上采样方式及各自的失效模式。转置卷积在什么情况下会产生棋盘格？
• Why do U-Nets use skip connections — what specific information do they carry that the decoder cannot reconstruct?U-Net 为什么使用跳跃连接——它们携带了哪种解码器无法重建的具体信息？
• What does a dilated convolution buy you, and how does dilation change the receptive-field formula?膨胀卷积能给你带来什么，膨胀又如何改变感受野公式？
• Derive why Dice loss beats cross-entropy on a 1%-foreground task. Work a small numeric example.推导为什么在前景占 1% 的任务上 Dice 损失胜过交叉熵。做一个小的数值算例。
• Define mIoU and PQ. What do SQ and RQ each tell you when PQ is low?定义 mIoU 和 PQ。当 PQ 偏低时，SQ 和 RQ 各自告诉你什么？
• How does SAM's prompt-encoder / mask-decoder split change an annotation product? What does SAM2 add for video?SAM 的提示编码器/掩码解码器拆分如何改变一款标注产品？SAM2 为视频增加了什么？