Training vision models and annotation strategy

视觉模型的训练与标注策略

Lesson 06 built the backbone: locality and translation-equivariance as inductive bias, receptive fields, residual blocks, feature pyramids, and a pretrained representation you can transfer. But an architecture is inert. It is the data, the loss, and the optimization that turn weights into a model — and in practice that is where almost all of the win, and almost all of the failure, lives.

第 06 课搭好了骨干网络：作为归纳偏置的局部性与平移等变性、感受野、残差块、特征金字塔，以及一个可供你迁移的预训练表示。但架构本身是惰性的。真正把权重变成一个模型的，是数据、损失和优化——而在实践中，几乎所有的胜利、以及几乎所有的失败，都藏在这里。

1 · Splits and leakage — the number you trust is the number you can break

1 · 划分与泄漏——你能相信的数字，是你能把它打破的数字

Every claim a model makes rests on one assumption: the validation and test sets are drawn from the same distribution as deployment, and the model never saw them during training. Leakage is any path by which test information reaches the model. In tabular ML leakage is usually a stray feature; in vision it is almost always correlated images landing on both sides of the split.

一个模型做出的每一个论断都建立在一个假设之上：验证集和测试集与部署环境来自同一分布，且模型在训练时从未见过它们。泄漏是指测试信息触达模型的任何一条路径。在表格类 ML 里，泄漏通常是某个不该出现的特征；在视觉里，它几乎总是相关的图像同时落在划分的两侧。

Why is this the #1 CV failure? Because images are rarely independent. A random per-image split assumes independence that does not exist:

为什么这是计算机视觉的头号故障？因为图像很少是相互独立的。而随机的逐图像划分，假设了一种并不存在的独立性：

The discipline is one sentence: choose the split unit that matches what is independent in deployment. If two test images could not have been produced by the same patient, camera, or moment that any training image was, your split is honest. The SVG below is the timeline version of the most common silent killer — frames.

这条纪律就一句话：选择与部署中真正独立的东西相匹配的划分单位。如果任意一张测试图像都不可能由生成过某张训练图像的同一位患者、同一台相机或同一时刻所产生，那你的划分就是诚实的。下面这张 SVG 是那个最常见的隐形杀手——帧——的时间线版本。

2 · Annotation is a design decision (then you QA it)

2 · 标注是一个设计决策（然后你对它做质检）

Labels are not free truth; they are expensive measurements, each with its own cost and information content. The rule: choose the cheapest label that still preserves the information the task needs. A box cannot train a boundary-sensitive segmenter; a mask is wasted money if you only need to count.

标签不是免费的真相；它们是昂贵的测量，各自有其成本和信息含量。规则是：选择仍能保留任务所需信息的、最便宜的那种标签。一个边界框无法训练出对边界敏感的分割器；而如果你只需要计数，一张掩码就是白花钱。

A good pipeline defines an ontology (the class list and its boundaries), worked examples, edge cases, consensus rules, and relabel triggers. Two recurring traps: ontology drift — "vehicle" later splits into car / truck / bus / bike / scooter and your old labels become inconsistent — and uncalibrated active learning, where you ask the model for "uncertain" samples but its uncertainty is noise, so you label garbage. Synthetic data is validated only by improvement on a real held-out set, never by how good it looks. For safety-critical work (medical, autonomy), add multi-annotator consensus, an adjudication step for disagreements, and a gold-standard audit set the annotators never see.

一个好的流水线要定义一套本体（ontology）（类别清单及其边界）、若干范例、边缘案例、共识规则和重新标注的触发条件。两个反复出现的陷阱：本体漂移——"车辆"后来被拆分成轿车 / 卡车 / 公交 / 自行车 / 滑板车，你的旧标签就变得不一致了——以及未校准的主动学习，即你向模型索要"不确定"的样本，但它的不确定性只是噪声，于是你标了一堆垃圾。合成数据只能靠在一个真实的留出集上带来提升来验证，绝不能靠它看起来有多好。对于安全攸关的工作（医疗、自动驾驶），要加上多标注者共识、针对分歧的裁定环节，以及一个标注者永远看不到的金标准审计集。

3 · Augmentation must preserve the label

3 · 数据增强必须保标签

Augmentation is how you teach invariance cheaply: you simulate nuisance variation you expect at serving time so the model learns to ignore it. But there is exactly one rule that governs whether an augmentation helps or hurts.

数据增强是你廉价地教会模型不变性的方式：你模拟出服务时预期会遇到的各种干扰变化，让模型学会忽略它们。但只有唯一一条规则，决定了一次增强是帮忙还是帮倒忙。

For boxes and masks the label is geometric, so the augmentation must transform the label too. Flip the image → flip the box coordinates. Crop → clip and possibly drop boxes. This is the most common silent bug in a detection pipeline: the image is flipped but the box is not, and you have just trained on systematically wrong targets.

对于边界框和掩码，标签是几何的，所以增强必须同时变换标签。翻转图像 → 翻转框坐标。裁剪 → 裁剪框并可能丢弃框。这是检测流水线里最常见的隐形 bug：图像被翻转了、框却没翻，于是你刚刚就在系统性错误的目标上训练了模型。

RandAugment removes the hand-tuning: instead of choosing each transform's magnitude, you pick two numbers — N, how many ops to chain, and M, a single global magnitude — and it samples ops from a fixed menu. Far fewer knobs, near state-of-the-art.

RandAugment 免去了手工调参：你不再为每一个变换单独选幅度，而是只挑两个数——N，串联多少个操作，以及 M，一个统一的全局幅度——它便从一个固定的操作菜单里采样。旋钮少得多，效果却接近最先进（SOTA）。

Mixup and CutMix are different in kind: they blend two training examples, and crucially they blend the labels too, which is exactly the label-preservation rule applied to soft targets. Mixup takes a pixel-wise convex combination; CutMix pastes a rectangular patch of image B onto A and gives class B (the pasted patch's class) a weight equal to the patch's area fraction, so the class-A weight is λ = 1 − (patch area fraction). The blend weight is drawn from Beta(α,α), a distribution over the blend weight λ ∈ [0,1] — and the one thing to internalize is what α does: small α → λ piles up near 0 or 1, i.e. each sample is mostly one image (a gentle blend); α=1 is uniform on [0,1]; large α concentrates λ near 0.5 (aggressive half-and-half mixes).

Mixup 和 CutMix 是另一类东西：它们混合两个训练样本，而且关键在于它们也混合标签，这恰恰是把保标签规则施加到软目标上。Mixup 做逐像素的凸组合；CutMix 把图像 B 的一块矩形补丁贴到 A 上，并赋予类别 B（被贴补丁的类别）一个等于该补丁面积占比的权重，于是类别 A 的权重是 λ = 1 − (patch area fraction)。混合权重从 Beta(α,α) 中抽取，这是混合权重 λ ∈ [0,1] 上的一个分布——需要内化的一点是 α 的作用：小 α → λ 堆积在 0 或 1 附近，即每个样本基本上就是一张图（温和的混合）；α=1 时在 [0,1] 上均匀；大 α 则把 λ 集中在 0.5 附近（激进的一半对一半混合）。

Worked example: a CutMix patch covers 30% of a 224×224 image. Then λ = 1 − 0.30 = 0.70, so the target is 70% class-A, 30% class-B. The loss is the same convex combination of the two cross-entropies. Both regularize by forbidding the network from betting everything on one class for one image — they smooth the decision boundary and, like label smoothing (lesson 08), they fight overconfidence.

算个具体例子：一块 CutMix 补丁覆盖了一张 224×224 图像的 30%。那么 λ = 1 − 0.30 = 0.70，所以目标是 70% 类别 A、30% 类别 B。损失就是这两个交叉熵的同一个凸组合。两者都通过禁止网络为一张图把全部赌注押在一个类别上来实现正则化——它们平滑了决策边界，并且和标签平滑（第 08 课）一样，都在对抗过度自信。

4 · The recipe: normalization, init, precision, schedule, EMA

4 · 配方：归一化、初始化、精度、调度、EMA

Why ImageNet mean/std normalization? When you fine-tune a pretrained backbone, every weight in it was learned on inputs that had a specific per-channel statistic subtracted and divided out — the ImageNet means (0.485, 0.456, 0.406) and stds (0.229, 0.224, 0.225) in RGB. The first conv layer's weights are tuned to that input distribution. Feed it raw [0,1] pixels and you have shifted the input off the distribution the network expects; activations land in the wrong regime and fine-tuning fights to undo it. You normalize to match pretraining — train/serve normalization parity is part of the model contract (the capstone, lesson 19, closes back to exactly this). If you train from scratch, compute your own dataset's mean/std instead.

为什么用 ImageNet 的均值/标准差归一化？当你微调一个预训练骨干网络时，其中的每一个权重都是在减去并除以了某个特定的逐通道统计量的输入上学到的——即 RGB 下的 ImageNet 均值 (0.485, 0.456, 0.406) 和标准差 (0.229, 0.224, 0.225)。第一层卷积的权重是针对那个输入分布调好的。若给它喂原始的 [0,1] 像素，你就把输入挪离了网络所预期的分布；激活会落到错误的区间，微调只能费力地去纠正它。你做归一化就是为了匹配预训练——训练/服务归一化的一致性是模型契约的一部分（收官的第 19 课正好会回到这一点）。如果你从零开始训练，那就改为计算你自己数据集的均值/标准差。

Initialization. Before pretraining existed, the whole game was keeping activation variance stable across depth so signals neither vanish nor explode. Kaiming (He) init scales weights by √(2/fan_in) precisely to preserve variance through a ReLU, where fan_in is the number of inputs feeding each output unit. Worked: a 3×3 conv with 64 input channels has fan_in = 3·3·64 = 576, so the weight scale is √(2/576) ≈ √0.00347 ≈ 0.0589 — weights are drawn from a distribution with that standard deviation. With a pretrained backbone you initialize from the checkpoint and only the new head needs fresh init — but the principle is why deep nets train at all.

初始化。在预训练出现之前，整场游戏就是让激活的方差在深度方向上保持稳定，使信号既不消失也不爆炸。Kaiming（He）初始化把权重按 √(2/fan_in) 缩放，正是为了让方差穿过 ReLU 后得以保持，其中 fan_in 是喂给每个输出单元的输入数目。算一遍：一个输入通道为 64 的 3×3 卷积，其 fan_in = 3·3·64 = 576，所以权重尺度是 √(2/576) ≈ √0.00347 ≈ 0.0589——权重就从一个标准差为该值的分布中抽取。若使用预训练骨干网络，你从检查点初始化，只有新加的头需要全新初始化——但正是这条原理，才让深层网络得以训练起来。

Mixed precision. Store and compute most ops in 16-bit (FP16 or BF16) instead of FP32: roughly half the memory and 2–3× throughput on tensor-core GPUs. The catch is FP16's tiny dynamic range — small gradients underflow to zero. The fix is loss scaling: multiply the loss by a large constant before backward so gradients land in FP16's representable range, then unscale before the optimizer step. BF16 has FP32's exponent range and usually needs no loss scaling. Master weights are kept in FP32 for the update.

混合精度。把大多数运算以 16 位（FP16 或 BF16）而非 FP32 来存储和计算：显存大约减半，在带张量核心的 GPU 上吞吐提升 2–3 倍。麻烦在于 FP16 的动态范围极小——小梯度会下溢为零。解决办法是损失缩放（loss scaling）：在反向传播前把损失乘以一个大常数，使梯度落进 FP16 可表示的范围，然后在优化器更新前再反缩放回去。BF16 拥有和 FP32 一样的指数范围，通常无需损失缩放。用于更新的主权重（master weights）则以 FP32 保留。

LR warmup + schedule. At step 0 the weights are random (or the head is), the gradients are huge and ill-conditioned, and a full learning rate can blow up — especially with large batches and mixed precision. Warmup ramps the LR linearly from ~0 over the first few hundred to few thousand steps, then a schedule (cosine decay is the modern default) anneals it toward zero so the model can settle into a minimum. The shape:

学习率预热 + 调度。在第 0 步，权重是随机的（或头是随机的），梯度既大又病态，一上来就用满学习率可能会炸——尤其是在大批量和混合精度下。预热（warmup）在最初的几百到几千步里把学习率从约 0 线性拉起，随后一个调度（余弦衰减是当下的默认选择）把它退火到接近零，好让模型稳稳落进一个极小值。其形状如下：

4.5 · Interactive · LR warmup + cosine schedule

4.5 · 交互 · 学习率预热 + 余弦调度

The two-line formula above is the whole schedule, but the shape is what builds intuition. Set warmup length, total steps, and the peak LR; drag the step marker to read the learning rate off the exact piecewise branch (linear ramp, then cosine anneal) the optimizer would use at that moment.

上面那两行公式就是整个调度，但真正建立直觉的是它的形状。设定预热长度、总步数和峰值学习率；拖动步数标记，即可在优化器此刻会用到的那一段分段分支（先线性拉升、再余弦退火）上读出学习率。

// piecewise: linear warmup to peak, then cosine decay to ~0
function lrAt(t, tWarm, T, peak){
  if (t <= tWarm) return tWarm > 0 ? peak * (t / tWarm) : peak;   // linear ramp
  const prog = (t - tWarm) / (T - tWarm);                          // 0..1 after warmup
  return 0.5 * peak * (1 + Math.cos(Math.PI * Math.min(prog, 1))); // cosine anneal
}

Regularization. Weight decay (use AdamW, which decouples weight decay from the adaptive step — decay is applied directly to the weights, not pushed through the gradient/RMS denominator the way classic Adam-with-L2 does, so every parameter actually decays at the rate you set) penalizes large weights and is one of the highest-leverage knobs in modern recipes. Dropout and stochastic depth randomly drop units / whole residual blocks. Label smoothing (lesson 08) softens targets. Early stopping on validation. EMA (exponential moving average) keeps a shadow copy of the weights, θ̄ ← m·θ̄ + (1−m)·θ with m ≈ 0.999, and evaluates with θ̄; at m = 0.999 each step moves the averaged weights only 1 − m = 0.001 = 0.1% of the way toward the live weights, so θ̄ is a slow, heavily-smoothed trail that averages out the late-training noise of SGD and almost always gives a cleaner, better-generalizing model for free.

正则化。权重衰减（用 AdamW，它把权重衰减与自适应步长解耦——衰减直接施加到权重上，而不像经典的 Adam+L2 那样把它推过梯度/RMS 分母，因此每个参数都实实在在按你设定的速率衰减）会惩罚过大的权重，是现代配方里杠杆率最高的旋钮之一。Dropout 和随机深度（stochastic depth）随机丢弃单元 / 整个残差块。标签平滑（第 08 课）软化目标。在验证集上做早停。EMA（指数移动平均）维护一份权重的影子副本，θ̄ ← m·θ̄ + (1−m)·θ，取 m ≈ 0.999，并用 θ̄ 来评估；在 m = 0.999 时，每一步平均权重只朝当前权重移动 1 − m = 0.001 = 0.1%，所以 θ̄ 是一条缓慢、被大幅平滑的轨迹，它平均掉了 SGD 训练后期的噪声，几乎总能白白换来一个更干净、泛化更好的模型。

5 · Debugging: find the bug before blaming the backbone

5 · 调试：在归咎骨干网络之前先找到 bug

When training disappoints, the architecture is the least likely culprit. Run this checklist, cheapest first:

当训练不尽如人意时，架构是最不可能的元凶。按这份清单来做，从最省事的开始：

1. Overfit one tiny batch. Take 8 images and drive train loss to ~0. If you cannot, the bug is in the data/loss/optimizer wiring, not capacity.过拟合一个极小批量。取 8 张图，把训练损失压到约 0。如果做不到，bug 就出在数据/损失/优化器的接线上，而不是模型容量。
2. Visualize augmented inputs with labels overlaid. This catches flipped-image-but-not-flipped-box, wrong normalization, BGR↔RGB swaps.把增强后的输入连同叠加的标签一起可视化。这能抓出"图翻了框没翻"、归一化错误、BGR↔RGB 互换等问题。
3. Diff train vs val preprocessing. Any mismatch (resize, normalization, channel order) is the classic "great train, terrible val" cause — and it foreshadows train/serve skew.对比训练与验证的预处理。任何不匹配（缩放、归一化、通道顺序）都是经典的"训练很棒、验证很烂"的原因——而且它预示着训练/服务偏斜。
4. Check the class distribution and the loss curve. A flat loss often means LR too high (diverged) or too low (stuck), or warmup missing.检查类别分布和损失曲线。损失走平往往意味着学习率过高（发散）或过低（卡住），或缺了预热。
5. Compute metrics by slice. Per-class, per-size, per-camera. An average hides the failure that matters.按切片计算指标。分类别、分尺寸、分相机。一个平均值会掩盖掉真正要命的失败。

Class imbalance deserves its own note: re-weight the loss toward rare classes, or resample, but watch the trade — up-weighting the tail can wreck head-class precision and calibration (the very thing lesson 08 measures). It is the dense-detection imbalance problem in miniature, and focal loss (lesson 09) is its sharpest tool.

类别不平衡值得单独提一句：把损失朝稀有类别重新加权，或做重采样，但要留意其中的权衡——给长尾加大权重可能会毁掉头部类别的精确率和置信度校准（正是第 08 课要度量的东西）。这是稠密检测里那个不平衡问题的缩小版，而 focal loss（第 09 课）是对付它的最锋利的工具。

6 · Domain adaptation & sim2real — when train and serve distributions differ

6 · 域适应与 sim2real——当训练分布与服务分布不同时

Every recipe above assumes the validation set looks like deployment. Often it doesn't. You train on daytime dashcam footage and serve at night; you train on synthetic renders and serve on real photos; you label one hospital's scanner and deploy to another's. This is domain shift: the input distribution at serving time (the target domain) differs from the one you trained on (the source domain). The model's perspective makes the danger precise — gradient descent only ever minimized loss on the source distribution, so the network is constrained only where it saw data. Anything off-distribution at serve time is unconstrained: the features may be meaningless there, and the model has no idea it is guessing. Concretely, a classifier at 92% accuracy on its source test set can drop to 61% on a shifted target (night, new sensor) with no code change at all — the weights are simply being asked a question they were never trained on.

上面每一套配方都假设验证集长得像部署环境。而现实往往并非如此。你在白天的行车记录仪画面上训练，却在夜间服务；你在合成渲染图上训练，却在真实照片上服务；你标注了一家医院的扫描仪，却部署到另一家。这就是域偏移（domain shift）：服务时的输入分布（目标域）与你训练时的（源域）不同。从模型的视角看，这个危险变得精确无比——梯度下降从头到尾只在源分布上最小化过损失，所以网络只有在它见过数据的地方才受到约束。服务时任何离分布的东西都是不受约束的：特征在那里可能毫无意义，而模型却浑然不知自己在瞎猜。具体来说，一个在源测试集上有 92% 精度的分类器，在一个偏移了的目标（夜间、新传感器）上可能跌到 61%，而代码一行都没改——只是有人在拿一个它从未训练过的问题去问这些权重罢了。

Domain adaptation (DA) is the family of methods that close that gap, ideally with few or no target labels. Three families, from first principles:

域适应（DA）是一大类用来弥合这个差距的方法，理想情况下只需很少甚至完全不需要目标域标签。从第一性原理出发，有三大家族：

The unifying idea: domain shift is unconstrained territory, and each family adds a constraint there — adversarial alignment forces the feature space to ignore the domain, randomization makes the target a subset of the training distribution, and test-time adaptation re-fits the parts of the model (BN stats, decision boundary) that are most domain-specific. How much of the 92%→61% collapse above you recover is highly task- and shift-dependent — there is no universal number — but applied well these methods reclaim a substantial fraction of the gap: not free, and rarely the full distance, but often the difference between unusable and shippable.

统一的思想是：域偏移是一片不受约束的地带，而每个家族都在那里加上一条约束——对抗对齐迫使特征空间忽略域，随机化把目标域变成训练分布的一个子集，测试时适应则重新拟合模型中最具域特异性的部分（BN 统计量、决策边界）。上面那个 92%→61% 的崩塌你能挽回多少，高度依赖具体任务和偏移类型——没有一个通用的数字——但只要用得好，这些方法能夺回相当一部分差距：不是免费的，也很少能补齐全程，但往往就是"不可用"和"可上线"之间的差别。

Where this points next

这将指向何处

We now have a trained, well-regularized backbone and a recipe that does not lie to us. The simplest possible thing to ask of it is also the most fundamental: name the whole image. Lesson 08 takes the logits this backbone emits and earns the loss that trains them — deriving cross-entropy from softmax step by step — then confronts what a single label cannot tell you, and how to know whether to trust the number the classifier prints (calibration, ECE, temperature scaling, label smoothing). That trust question is why label smoothing and Mixup showed up here: they are the training-time half of a story lesson 08 finishes at evaluation time.

现在我们有了一个训练好、正则化良好的骨干网络，以及一套不会骗我们的配方。能对它提出的最简单的问题，也是最根本的：给整张图像命名。第 08 课取用这个骨干网络输出的 logit，并挣得训练它们所需的损失——一步步从 softmax 推导出交叉熵——然后直面单个标签所无法告诉你的东西，以及如何判断分类器打印出来的那个数字值不值得相信（置信度校准、ECE、温度缩放、标签平滑）。正是这个"信任"问题，才让标签平滑和 Mixup 在这里登场：它们是这个故事发生在训练时的那一半，而第 08 课会在评估时把它讲完。

Interview prompts

面试题

• How would you fine-tune an ImageNet-pretrained model on 5k labeled images? (normalize to pretraining stats, freeze→unfreeze, low LR with warmup+cosine, heavy augmentation, EMA, early stop on a leakage-free val split.)你会如何在 5k 张有标注的图像上微调一个 ImageNet 预训练模型？(按预训练统计量做归一化，先冻结→再解冻，用带预热+余弦的低学习率，重度增强，EMA，在无泄漏的验证划分上早停。)
• When can augmentation make a model worse? (when the transform changes the label — flips on text/laterality, crops that remove the object, color jitter when color is the class — or when box/mask labels are not transformed with the image.)数据增强什么时候会让模型变差？(当变换改变了标签时——对文本/左右侧位做翻转、把目标裁掉的裁剪、当颜色就是类别时的颜色抖动——或当框/掩码标签没有随图像一起变换时。)
• What is a leakage pattern unique to image/video datasets? (adjacent video frames or near-duplicates split per-image; same patient/camera across splits; augmenting before splitting.)图像/视频数据集特有的一种泄漏模式是什么？(把相邻视频帧或近重复样本按逐图像划分；同一患者/相机横跨多个划分；在划分之前就做增强。)
• How do you choose among image labels, boxes, masks, keypoints, tracks? (cheapest label that preserves the information the task needs.)你如何在图像级标签、框、掩码、关键点、轨迹之间做选择？(选能保留任务所需信息的最便宜的标签。)
• What annotation QA would you add for a safety-critical model? (multi-annotator consensus, adjudication, a held-out gold set, ontology versioning, relabel triggers.)对于一个安全攸关的模型，你会加上哪些标注质检？(多标注者共识、裁定环节、一个留出的金标准集、本体版本管理、重新标注触发条件。)
• Train accuracy high, val poor — first checks? (overfit one batch, diff train/val preprocessing, look for split leakage, visualize augmented labels, slice metrics.)训练精度高、验证很差——最先检查什么？(过拟合一个批量、对比训练/验证预处理、排查划分泄漏、可视化增强后的标签、按切片看指标。)
• Why normalize with ImageNet mean/std? (to match the input distribution the pretrained weights were tuned to — train/serve parity.)为什么用 ImageNet 的均值/标准差做归一化？(为了匹配预训练权重当初被调好的那个输入分布——训练/服务的一致性。)
• Why warmup and a schedule? (early random-weight gradients are huge/ill-conditioned; ramp up to avoid divergence, then anneal to settle into a minimum.)为什么要预热和调度？(早期随机权重的梯度又大又病态；先拉升以避免发散，再退火以稳稳落进一个极小值。)
• Your model is 92% on the test set but 61% in production at night — what's happening and what do you do? (domain shift: serving distribution is off the source the model was trained on, so it's unconstrained there. Options: collect/label some night data, adversarial domain-invariant features (DANN + gradient reversal), domain randomization if the source is synthetic, or test-time adaptation of BatchNorm stats / pseudo-labeling on the target.)你的模型在测试集上有 92%，但在夜间的生产环境里只有 61%——发生了什么，你会怎么做？(域偏移：服务分布偏离了模型训练所用的源，所以模型在那里不受约束。可选做法：采集/标注一些夜间数据、对抗式的域不变特征（DANN + 梯度反转）、若源是合成的则用域随机化，或在目标域上做 BatchNorm 统计量的测试时适应 / 伪标签。)