Object detection

目标检测

Lesson 08 named the whole image and asked whether to trust the number. But a single label cannot say where an object is or how many there are. Detection answers both: a class and a box per object. That one step forces three new first-principles tools — a way to measure box overlap (IoU), a way to deduplicate overlapping guesses (NMS), and a way to score a ranked list of boxes (mAP).

第 08 课为整幅图像命名，并追问是否该相信那个数字。但单个标签说不出物体在哪里，也说不出有几个。检测同时回答这两点：每个物体一个类别加一个框。这一步硬是逼出了三个新的第一性原理工具——一种度量框重叠程度的方法（IoU）、一种对重叠猜测去重的方法（NMS），以及一种为一串排好序的框打分的方法（mAP）。

1 · Boxes and IoU — measure overlap, by hand

1 · 边界框与 IoU——亲手度量重叠

A detection is a class score plus a box, usually (x₁, y₁, x₂, y₂) for the top-left and bottom-right corners. To say whether a predicted box matches a ground-truth box we need a single number for "how much do these overlap?" that is scale-free and symmetric. The answer is Intersection over Union (IoU) — the area they share divided by the area they jointly cover:

一次检测就是一个类别分数加一个框，通常用 (x₁, y₁, x₂, y₂) 表示左上角和右下角。要判断一个预测框是否与真值框匹配，我们需要一个既无关尺度又对称的单一数字来回答"它们重叠多少？"。答案就是交并比（Intersection over Union，IoU）——它们共享的面积除以它们共同覆盖的面积：

IoU shows up twice: at training time to decide which prediction is responsible for which target, and at evaluation time to decide whether a detection counts as correct. It is the spine of the whole lesson.

IoU 出现两次：在训练时用来决定哪个预测负责哪个目标，在评估时用来判断一次检测是否算正确。它是贯穿整节课的主心骨。

Detectors come in two shapes: two-stage (first propose a few hundred candidate regions, then classify and refine each) and one-stage (predict class and box densely at every location in one pass). Keep both in mind — the assignment and imbalance choices below split along this line.

检测器分为两种形态：两阶段（先提出几百个候选区域，再对每个区域分类并精修）与单阶段（一次前向就在每个位置稠密地预测类别和框）。把这两者都记在心里——下面关于分配和不平衡的取舍正是沿着这条线分开的。

2 · Assignment and box parameterization

2 · 分配与边界框参数化

The first hard problem is assignment: which of the model's many spatial predictions is responsible for a given ground-truth object? Two philosophies.

第一个难题是分配：在模型众多的空间预测中，究竟哪一个负责某个给定的真值物体？有两种理念。

Anchor-based detectors tile the image with reference boxes ("anchors") of several scales and aspect ratios at every feature location, then label each anchor positive / negative / ignore by its IoU with ground truth (e.g. ≥0.7 positive, ≤0.3 negative). The network learns to refine the nearest anchor. Anchor-free detectors drop the priors and predict directly from points or centers (FCOS, CenterNet), assigning by which locations fall inside an object with center sampling and per-level scale ranges. Anchor-free is the modern default — fewer hyperparameters, no anchor-tuning ritual.

基于锚框的检测器在每个特征位置上铺满若干尺度和长宽比的参考框（"锚框"），再按每个锚框与真值的 IoU 把它标为正 / 负 / 忽略（例如 ≥0.7 为正，≤0.3 为负）。网络学习去精修最接近的锚框。无锚框检测器丢掉这些先验，直接从点或中心进行预测（FCOS、CenterNet），通过哪些位置落在物体内部，配合中心采样和逐层的尺度范围来做分配。无锚框是如今的默认选择——超参数更少，也不用做调锚框那套仪式。

Why log-scale offsets? A head (the small task-specific output sub-network on top of the backbone) does not regress raw pixel coordinates; it predicts a shift from its anchor/point and a log ratio of size:

为什么用对数尺度偏移量？检测头（骨干网络之上那个小型、任务专用的输出子网络）并不回归原始的像素坐标；它预测相对其锚框/点的平移量以及尺寸的对数比值：

Two reasons. The center shift is normalized by anchor size so a 5-pixel error means the same thing for a tiny box and a huge one. And size uses log because scale is multiplicative: predicting log(w/w_a) makes doubling and halving a box symmetric (±log 2) and keeps width strictly positive after w = w_a·e^t_w. The target distribution becomes roughly zero-mean and scale-invariant — far easier to regress than raw coordinates spanning four orders of magnitude.

两个原因。中心平移量用锚框尺寸做了归一化，所以 5 个像素的误差对一个极小的框和一个极大的框意味着同一件事。尺寸则使用 log，因为尺度是乘性的：预测 log(w/w_a) 使得把框放大一倍和缩小一半是对称的（±log 2），并且经过 w = w_a·e^t_w 之后宽度严格为正。这样目标分布就变得大致零均值且尺度不变——远比回归跨越四个数量级的原始坐标容易。

3 · Imbalance, focal loss, and FPN

3 · 不平衡、focal loss 与 FPN

A dense one-stage detector evaluates tens of thousands of locations per image, of which a handful contain objects. This foreground/background imbalance (often 1000:1) means easy background examples, each contributing a small loss, collectively swamp the gradient and the model learns to predict "background" everywhere. Two-stage detectors dodge this with a proposal stage that pre-filters; one-stage detectors need a direct fix.

一个稠密的单阶段检测器每张图要评估上万个位置，其中只有寥寥几个含有物体。这种前景/背景不平衡（常常是 1000:1）意味着大量简单的背景样本，虽然每个贡献的损失都很小，却集体淹没了梯度，于是模型学会了处处都预测"背景"。两阶段检测器靠一个预先筛选的候选阶段来规避这个问题；单阶段检测器则需要一个直接的修复办法。

Focal loss is that fix. Take standard cross-entropy −log p_t (where p_t is the probability of the true class) and multiply it by a modulating factor that down-weights well-classified examples:

Focal loss 就是这个修复办法。取标准交叉熵 −log p_t（其中 p_t 是真实类别的概率），再乘上一个降低易分类样本权重的调制因子：

Mechanism: when an example is easy and confident (p_t = 0.99), (1−0.99)² = 0.0001 — its loss is scaled to almost nothing, so the 10,000 easy backgrounds stop dominating. A hard example (p_t = 0.3) keeps (0.7)² = 0.49 of its loss. So the easy example's loss is down-weighted by roughly 0.49/0.0001 ≈ 5000× relative to the hard one — versus equal weighting under plain cross-entropy. The gradient refocuses on the examples the model is actually getting wrong. It is the class-imbalance principle from lesson 07, sharpened into a loss. (Localization itself uses smooth-L1 on the offsets, or IoU-family losses — GIoU/DIoU/CIoU — that optimize overlap directly. Plain IoU loss fails when the two boxes do not overlap at all: IoU = 0 everywhere in the neighborhood, so the gradient is zero and the box gets no signal pointing it toward the target — it cannot tell "just missed" from "miles away." GIoU adds a penalty based on the smallest box enclosing both, which stays informative even under zero overlap and pulls the prediction toward the target.)

机制：当一个样本既简单又自信时（p_t = 0.99），(1−0.99)² = 0.0001——它的损失被缩放到几乎为零，于是那 10,000 个简单背景不再占主导。一个难样本（p_t = 0.3）则保留了 (0.7)² = 0.49 的损失。所以相对于难样本，简单样本的损失大约被降权了 0.49/0.0001 ≈ 5000×——而在普通交叉熵下二者是等权的。梯度于是重新聚焦到模型实际做错的样本上。这就是第 07 课的类别不平衡原理，被磨砺成了一个损失函数。（定位本身对偏移量用 smooth-L1，或用直接优化重叠的 IoU 系列损失——GIoU/DIoU/CIoU。当两个框完全不重叠时，朴素的 IoU 损失会失效：邻域内处处 IoU = 0，于是梯度为零，框得不到任何指向目标的信号——它分不清"差一点没中"和"差了十万八千里"。GIoU 加了一项基于包围两者的最小框的惩罚，即使在零重叠下也仍然有信息，能把预测拉向目标。）

Scale via FPN. Lesson 06 built the feature pyramid network: combine deep, semantic, low-resolution features with shallow, high-resolution ones via top-down + lateral connections. Detection uses it because objects appear at wildly different sizes — a detector predicts small objects from high-resolution early levels and large objects from coarse deep levels, each level responsible for its own scale band. Without FPN, small-object recall collapses.

用 FPN 处理尺度。第 06 课构建了特征金字塔网络：通过自顶向下 + 横向连接，把深层、语义强、低分辨率的特征与浅层、高分辨率的特征结合起来。检测之所以用它，是因为物体的尺寸差异极大——检测器从高分辨率的早期层预测小物体、从粗糙的深层预测大物体，每一层负责自己的那一段尺度。没有 FPN，小物体的召回率就会崩塌。

4 · NMS — collapse the duplicates (and DETR's escape)

4 · NMS——压掉重复的框（以及 DETR 的另辟蹊径）

Dense detectors fire many overlapping boxes on the same object — neighboring anchors all "see" the cat. Non-maximum suppression (NMS) is the mechanical post-process that keeps one per object. It is not optional cleanup; it is part of the model's output contract, and its threshold trades precision for recall.

稠密检测器会在同一个物体上打出许多重叠的框——相邻的锚框都"看见"了那只猫。非极大值抑制（Non-maximum suppression，NMS）就是那个机械式的后处理，为每个物体只保留一个框。它不是可有可无的收尾工作；它是模型输出契约的一部分，而它的阈值是在精确率和召回率之间做权衡。

Set τ too high and duplicates survive (false positives); too low and two genuinely adjacent objects get merged (missed detection) — the classic NMS failure in crowds. Soft-NMS decays neighbor scores instead of hard-deleting them to soften this. The widget below steps through this exact loop.

τ 设得太高，重复框就会存活（误检）；设得太低，两个确实相邻的物体就会被合并（漏检）——这正是 NMS 在密集人群里的经典失败。Soft-NMS 不硬删邻居，而是衰减它们的分数，以缓和这一点。下面的小部件会一步步走完这个循环。

DETR removes NMS entirely by reframing detection as set prediction. A transformer decodes a fixed set of N "object queries" into N (class, box) slots, and training uses bipartite matching (the Hungarian algorithm) to find the one-to-one assignment between predictions and ground-truth objects that minimizes total cost. The per-pair matching cost is a weighted sum of a classification term (does this query predict the right class?) and a box term (an L1 distance on coordinates plus a GIoU term on overlap). Because the assignment is strictly one-to-one — each ground-truth object is claimed by exactly one query — a second query that tries to cover the same object is matched to "no object" and penalized, so the model learns not to emit it; that is what removes duplicates without NMS. Because the loss is computed against a one-to-one matching, the model is directly penalized for emitting two boxes on one object — duplicates are trained away rather than filtered away, so no NMS is needed. The cost is slow convergence and a hunger for data and compute.

DETR 通过把检测重新表述为集合预测，彻底去掉了 NMS。一个 Transformer 把固定的一组 N 个"对象查询（object query）"解码成 N 个 (类别, 框) 槽位，训练时用二分图匹配（匈牙利算法）在预测与真值物体之间找到使总代价最小的一对一分配。每一对的匹配代价是一个分类项（这个查询预测的类别对不对？）与一个框项（坐标上的 L1 距离加上重叠上的 GIoU 项）的加权和。因为分配严格是一对一的——每个真值物体只被恰好一个查询认领——试图覆盖同一物体的第二个查询会被匹配到"无物体"并被惩罚，于是模型学会了不去输出它；这正是不用 NMS 就能去重的原因。由于损失是针对一对一匹配来计算的，模型会因在一个物体上输出两个框而被直接惩罚——重复项是被训练掉的，而不是被过滤掉的，所以不需要 NMS。代价是收敛慢，以及对数据和算力的贪婪需求。

function iou(a, b){
  const ix = Math.max(0, Math.min(a.x2,b.x2) - Math.max(a.x1,b.x1));
  const iy = Math.max(0, Math.min(a.y2,b.y2) - Math.max(a.y1,b.y1));
  const inter = ix*iy;
  const areaA = (a.x2-a.x1)*(a.y2-a.y1), areaB = (b.x2-b.x1)*(b.y2-b.y1);
  return inter / (areaA + areaB - inter);
}
function nms(boxes, tau){
  const order = boxes.slice().sort((p,q)=> q.score - p.score); // sort by score desc
  const keep = [];
  const alive = order.map(()=> true);
  for (let i=0;i<order.length;i++){
    if (!alive[i]) continue;
    keep.push(order[i]);                  // highest surviving score -> keep
    for (let j=i+1;j<order.length;j++){   // suppress its duplicates
      if (alive[j] && iou(order[i], order[j]) > tau) alive[j] = false;
    }
  }
  return keep;
}

5 · mAP — scoring a ranked list of boxes

5 · mAP——为一串排好序的框打分

A detector outputs many boxes with scores; how good is it? Accuracy is meaningless here (there is no fixed denominator of "examples"). Instead we ask, across a confidence ranking: of the boxes we call positive, what fraction are correct (precision), and of the real objects, what fraction did we find (recall)? Sweep the confidence threshold from high to low and you trace a precision-recall curve: at a high threshold you predict only your surest boxes (high precision, low recall); lowering it surfaces more true objects but also more false positives.

检测器输出许多带分数的框；它到底有多好？准确率在这里毫无意义（没有一个固定的"样本"分母）。我们转而沿着一个置信度排名来问：在我们判为正的框里，有多大比例是正确的（精确率），以及在真实物体里，我们找到了多大比例（召回率）？把置信度阈值从高扫到低，你就描出一条精确率-召回率曲线：阈值高时你只预测最有把握的框（高精确率、低召回率）；调低它会浮现更多真实物体，但也带来更多误检。

Average Precision (AP) is the area under that precision-recall curve for one class — a single number summarizing the whole precision/recall trade-off. It rewards ranking: putting true detections above false ones. A detection counts as a true positive only if its IoU with an unmatched ground-truth box exceeds a threshold. mAP ("mean AP") averages AP over classes.

平均精度（Average Precision，AP）是某一个类别那条精确率-召回率曲线下的面积——一个概括整个精确率/召回率取舍的单一数字。它奖励排序：把真检测排在假检测之上。一次检测只有当它与某个尚未被匹配的真值框的 IoU 超过阈值时，才算作真阳性。mAP（"mean AP"，平均 AP）把各类别的 AP 求平均。

Worked toy — compute the AP number. Two ground-truth objects; rank three predictions by confidence — TP, FP, TP. After box 1: precision 1/1 = 1.0, recall 1/2 = 0.5. After box 2 (a false positive): precision drops to 1/2 = 0.5, recall still 0.5. After box 3 (a true positive): precision 2/3 ≈ 0.67, recall 2/2 = 1.0. So the raw curve traces the points (R=0.5, P=1.0), (R=0.5, P=0.5), (R=1.0, P=0.67).

玩具算例——把 AP 这个数算出来。两个真值物体；把三个预测按置信度排序——TP、FP、TP。第 1 个框之后：精确率 1/1 = 1.0，召回率 1/2 = 0.5。第 2 个框（一个误检）之后：精确率降到 1/2 = 0.5，召回率仍是 0.5。第 3 个框（一个真阳性）之后：精确率 2/3 ≈ 0.67，召回率 2/2 = 1.0。于是原始曲线描出这些点 (R=0.5, P=1.0)、(R=0.5, P=0.5)、(R=1.0, P=0.67)。

You do not integrate that zig-zag directly. The AP definition includes an interpolation step: at each recall level, replace the precision with the maximum precision achieved at that recall or any higher recall (p_interp(r) = max_{r′ ≥ r} p(r′)). This monotone "max precision to the right" envelope is part of what AP means (VOC's all-point / 11-point AP); it discards the dip caused by the false positive. Here the envelope is flat at 1.0 for recall in [0, 0.5] (the only point at recall ≥ that level with the highest precision is the first one, P=1.0) and flat at 0.67 for recall in (0.5, 1.0] (the only point that far right is the last one, P=0.67). Integrating that step function:

你不会直接对那条锯齿线积分。AP 的定义里包含一个插值步骤：在每个召回率水平上，用该召回率或任何更高召回率处所达到的最大精确率去替换该处的精确率（p_interp(r) = max_{r′ ≥ r} p(r′)）。这条单调的"向右取最大精确率"的包络本身就是 AP 含义的一部分（VOC 的全点 / 11 点 AP）；它抹掉了误检造成的那个凹陷。这里，对召回率在 [0, 0.5] 内，包络平在 1.0（在召回率 ≥ 该水平的点中精确率最高的只有第一个点，P=1.0），对召回率在 (0.5, 1.0] 内平在 0.67（那么靠右的点只有最后一个，P=0.67）。对这个阶梯函数积分：

That is a concrete number you can check: the false positive in the middle cost the detector about 0.17 of a perfect AP of 1.0. AP near 1.0 would mean almost all your top-ranked boxes are correct before any false alarm appears.

这是一个你可以核对的具体数字：中间那个误检让检测器在满分 1.0 的 AP 上损失了约 0.17。AP 接近 1.0 就意味着在任何误报出现之前，你排名最靠前的框几乎全都正确。

The ranking emphasis has a practical bite: a model with good boxes can still post low AP if its confidence ordering is bad — true detections ranked below false positives drag the precision down at every recall level. Calibration (lesson 08) is not just cosmetic here; it moves the curve.

这种对排序的强调有其现实的杀伤力：一个框画得不错的模型，若置信度排序很糟，仍然会得到很低的 AP——真检测被排在误检之下，会在每个召回率水平上把精确率往下拽。置信度校准（第 08 课）在这里绝不只是锦上添花；它会移动整条曲线。

6 · The modern landscape (2024–2026)

6 · 现代格局（2024–2026）

Open-vocabulary detection is the biggest shift. Classic detectors have a frozen class list baked into the final layer. GLIP, OWL-ViT, and GroundingDINO instead align region features with text embeddings (the CLIP idea, lesson 15), so you detect "a red mug" or any phrase without retraining — grounding language to boxes. RT-DETR brought the DETR set-prediction formulation (no NMS) down to real-time latency, putting it head-to-head with the still-evolving YOLO line, which remains the practical default for edge and real-time deployment. The throughline: assignment, IoU, and the precision/recall scoring you learned here are unchanged — only the architecture around them moves.

开放词表检测是最大的转变。经典检测器在最后一层里烧死了一份固定的类别表。GLIP、OWL-ViT 和 GroundingDINO 转而把区域特征与文本嵌入对齐（CLIP 的思路，第 15 课），于是你无需重新训练就能检测"一个红色马克杯"或任意短语——把语言定位（grounding）到框上。RT-DETR 把 DETR 的集合预测表述（无 NMS）拉到了实时延迟，使它与仍在演进的 YOLO 系列正面交锋，而 YOLO 仍是边缘端和实时部署的实用默认选择。贯穿始终的一条线是：你在这里学到的分配、IoU 以及精确率/召回率打分并没有变——变的只是它们周围的架构。

Where this points next

这将指向何处

A box is a coarse answer: it says "an object is roughly here" but draws a rectangle around a curved, occluded, irregular thing. Many tasks need the boundary exactly — medical measurement, image editing, autonomous free-space. Lesson 10 pushes localization to its limit with segmentation: a class label at every pixel. That demands new machinery the box head never needed — encoder-decoder architectures, upsampling back to full resolution (transposed conv, bilinear+conv, unpooling), U-Net skip connections, and dense losses like Dice/IoU — plus the promptable frontier (SAM/SAM2) that echoes the open-vocabulary turn we just met. The detector's box head, in fact, becomes Mask R-CNN's first stage there.

框是一个粗糙的答案：它说"一个物体大致在这儿"，却给一个弯曲、被遮挡、不规则的东西画了一个矩形。许多任务需要精确的边界——医学测量、图像编辑、自动驾驶的可行驶空间。第 10 课用分割把定位推到极限：给每个像素一个类别标签。这需要框头从来不需要的新机器——编码器-解码器架构、把分辨率上采样回全尺寸（转置卷积、双线性+卷积、反池化）、U-Net 的跳跃连接，以及 Dice/IoU 这类稠密损失——外加可提示（promptable）的前沿（SAM/SAM2），它呼应着我们刚刚遇到的开放词表转向。事实上，检测器的框头在那里会变成 Mask R-CNN 的第一阶段。

Interview prompts

面试题

• Compute IoU for two given boxes. (intersection area / (areaA + areaB − intersection); see the worked 6/26 example.)给定两个框，计算 IoU。(交集面积 / (areaA + areaB − 交集)；见 6/26 的算例。)
• What problem does NMS solve and when does it fail? (removes duplicate boxes on one object via sort-keep-suppress; fails in crowds where τ too low merges distinct objects, too high keeps duplicates.)NMS 解决什么问题，它何时会失败？(通过"排序-保留-抑制"去掉同一物体上的重复框；在密集人群中失败——τ 太低会合并不同物体，太高会留下重复框。)
• Anchor-based vs anchor-free assignment? (IoU-thresholded reference boxes vs direct point/center prediction with scale ranges; anchor-free is the default.)基于锚框与无锚框的分配有何区别？(用 IoU 阈值筛选的参考框，对比配合尺度范围的直接点/中心预测；无锚框是默认。)
• Why regress log-scale box offsets? (anchor-normalized shifts are scale-invariant; log makes scaling symmetric and width positive.)为什么要回归对数尺度的框偏移量？(用锚框归一化的平移量尺度不变；对数使缩放对称且宽度为正。)
• Why does focal loss help dense detection? ((1−p_t)^γ down-weights easy/confident backgrounds so the 1000:1 imbalance stops dominating the gradient.)focal loss 为什么有助于稠密检测？((1−p_t)^γ 降低了简单/自信背景的权重，使 1000:1 的不平衡不再主导梯度。)
• Why does DETR not need NMS? (one-to-one bipartite matching penalizes duplicate boxes during training, so they are never emitted.)DETR 为什么不需要 NMS？(一对一的二分图匹配在训练时就惩罚重复框，所以它们根本不会被输出。)
• Why do detectors use feature pyramids? (objects span scales; predict small objects from high-res levels, large from deep levels — lesson 06's FPN.)检测器为什么使用特征金字塔？(物体跨越多种尺度；从高分辨率层预测小物体、从深层预测大物体——第 06 课的 FPN。)
• How is mAP different from accuracy, and VOC vs COCO? (AP integrates a precision-recall curve per class; VOC = AP@0.5, COCO = AP averaged over IoU .5:.95.)mAP 与准确率有何不同，VOC 与 COCO 又有何区别？(AP 对每个类别的一条精确率-召回率曲线积分；VOC = AP@0.5，COCO = 在 IoU .5:.95 上求平均的 AP。)
• Real-time detection on mobile — what levers? (anchor-free one-stage / modern YOLO, lower resolution, quantization, prune/distill, NMS tuning; consider RT-DETR.)移动端的实时检测——有哪些抓手？(无锚框单阶段 / 现代 YOLO、降低分辨率、量化、剪枝/蒸馏、调 NMS；可考虑 RT-DETR。)