Computer Vision, from first principles
计算机视觉，从第一性原理出发
A linearized tour from image formation to production CV systems. The goal is coverage without memorization: understand what signal is in the pixels, what invariances the model needs, which task head follows, and which failure mode matters.
一条从图像成像到生产级计算机视觉系统的线性化路线。目标是覆盖全面而无需死记硬背：理解像素里有什么信号、模型需要哪些不变性、后面接什么任务头，以及哪种失效模式才重要。
Who this is for本课程适合谁
You know basic ML and neural networks, but CV interview topics feel scattered: cameras, CNNs, detection, segmentation, ViTs, CLIP, video, and deployment. This track puts those topics on one line so each lesson answers why the next one exists.
你已经掌握了基础的机器学习和神经网络，但计算机视觉的面试话题却显得零散：相机、CNN、检测、分割、ViT、CLIP、视频以及部署。这门课把这些话题串在同一条线上，让每一课都回答下一课为何存在。
The first-principles frame
第一性原理框架
A camera turns a 3D world into a 2D grid of noisy measurements. CV asks a model to recover useful structure from that grid while ignoring nuisance variation: translation, scale, viewpoint, illumination, occlusion, blur, compression, sensor noise, and dataset bias. Every interview question reduces to four things.
相机把一个三维世界变成一个带噪声的二维测量网格。计算机视觉要求模型从这个网格中恢复出有用的结构，同时忽略那些无关的干扰变化：平移、尺度、视角、光照、遮挡、模糊、压缩、传感器噪声以及数据集偏置。每一道面试题都可归结为四件事。
Signal
信号
Pixels, color, edges, geometry, motion
像素、颜色、边缘、几何、运动
Bias
偏置
Locality, equivariance, hierarchy, attention
局部性、等变性、层次结构、注意力
Task
任务
Classify, localize, segment, match, track, describe
分类、定位、分割、匹配、跟踪、描述
System
系统
Latency, robustness, monitoring, privacy, cost
延迟、鲁棒性、监控、隐私、成本
Why this order
为什么是这个顺序
The order starts with the measurement process, not with ResNet. Pixels and filtering explain why edges and local features are useful. Classical matching and camera geometry explain what deep networks eventually learn to approximate, and why 2D shortcuts fail when viewpoint changes. CNNs, training, and classification form the backbone layer. Detection, segmentation, pose, retrieval, transformers, self-supervision, multimodal models, video, generation, and deployment then become task-specific extensions of the same representation problem.
这个顺序从测量过程开始，而不是从 ResNet 开始。像素与滤波解释了为什么边缘和局部特征有用。经典匹配与相机几何解释了深度网络最终学着去逼近的是什么，以及为什么当视角改变时二维的捷径会失效。CNN、训练与分类构成了骨干层。检测、分割、姿态、检索、Transformer、自监督、多模态模型、视频、生成与部署，随后都成为同一个表示问题的、面向具体任务的扩展。
| Stage阶段 | Lessons课程 | What you should be able to do你应当能够做到的 |
|---|---|---|
| Measurement测量 | 01-02 | Reason from sampling, color, convolution, gradients, edges, and aliasing.从采样、颜色、卷积、梯度、边缘和混叠出发进行推理。 |
| Geometry几何 | 03-05 | Explain features, matching, camera projection, calibration, stereo, depth, and SLAM failure modes.解释特征、匹配、相机投影、标定、立体视觉、深度以及 SLAM 的失效模式。 |
| Backbones骨干网络 | 06-08 | Choose and train CNN classifiers with the right architecture, augmentation, loss, and metric.用合适的架构、数据增强、损失和评价指标来选择并训练 CNN 分类器。 |
| Spatial tasks空间任务 | 09-11 | Turn features into boxes, masks, keypoints, poses, and tracks.把特征转化为边界框、掩码、关键点、姿态和轨迹。 |
| Representations表示 | 12-15 | Build embedding, transformer, self-supervised, and vision-language answers from first principles.从第一性原理出发，构建关于嵌入、Transformer、自监督和视觉-语言的答案。 |
| Dynamics and synthesis动态与合成 | 16-17 | Handle temporal signals and generative image models.处理时序信号和生成式图像模型。 |
| Restoration复原 | 18 | Recover a clean image from a degraded one — denoise, super-resolve, deblur, inpaint — plus the camera ISP and computational photography.从退化图像中恢复出干净的图像——去噪、超分辨率、去模糊、图像修复——外加相机 ISP 和计算摄影。 |
| Production生产 | 19 | Evaluate, compress, serve, monitor, and design CV systems under real constraints.在真实约束下评估、压缩、部署、监控并设计计算机视觉系统。 |
The lessons
课程列表
01
Image representation, pixels, color, and coordinates
图像表示、像素、颜色与坐标
Pixels as samples, coordinate conventions, RGB vs HSV/YUV, dynamic range, quantization, noise, resizing, interpolation, and the tiny details that cause real bugs.
把像素看作采样、坐标约定、RGB 与 HSV/YUV、动态范围、量化、噪声、缩放、插值，以及那些引发真实 bug 的细枝末节。
02
Filtering, convolution, edges, and frequency intuition
滤波、卷积、边缘与频率直觉
Correlation vs convolution, Gaussian blur, Sobel/Laplacian/Canny, separability, padding, stride, dilation, anti-aliasing, and how hand-designed filters become learned kernels.
相关与卷积、高斯模糊、Sobel/Laplacian/Canny、可分离性、填充、步幅、空洞卷积、抗混叠，以及手工设计的滤波器如何变成可学习的卷积核。
03
Classical features, descriptors, and matching
经典特征、描述子与匹配
Corners, blobs, SIFT/ORB-style descriptors, nearest-neighbor matching, ratio tests, RANSAC, and why keypoint pipelines are still useful for debugging learned systems.
角点、斑点、SIFT/ORB 风格的描述子、最近邻匹配、比率测试、RANSAC，以及为什么关键点流水线在调试学习型系统时依然有用。
04
Cameras, projection, calibration, and distortion
相机、投影、标定与畸变
Pinhole projection, intrinsics/extrinsics, homogeneous coordinates, radial distortion, calibration, coordinate frames, and where 2D image assumptions break.
针孔投影、内参/外参、齐次坐标、径向畸变、标定、坐标系，以及二维图像假设在哪里失效。
05
Multi-view geometry, stereo, depth, and SLAM
多视图几何、立体视觉、深度与 SLAM
Homographies, epipolar geometry, essential/fundamental matrices, triangulation, depth estimation, point clouds, NeRF vs 3D Gaussian splatting, and SLAM failure cases.
单应矩阵、对极几何、本质/基础矩阵、三角化、深度估计、点云、NeRF 与 3D 高斯泼溅（Gaussian Splatting），以及 SLAM 的失效情形。
06
CNN foundations, receptive fields, and backbones
CNN 基础、感受野与骨干网络
Locality, translation equivariance, pooling, receptive field math, residual blocks, bottlenecks, depthwise separable convolution, feature pyramids, and transfer learning.
局部性、平移等变性、池化、感受野的计算、残差块、瓶颈结构、深度可分离卷积、特征金字塔以及迁移学习。
07
Training vision models and annotation strategy
训练视觉模型与标注策略
Data splits, annotation strategy, label QA, augmentation, normalization, initialization, regularization, class imbalance, transfer learning, fine-tuning, mixed precision, and debugging.
数据划分、标注策略、标签质检、数据增强、归一化、初始化、正则化、类别不平衡、迁移学习、微调、混合精度以及调试。
08
Classification and recognition
分类与识别
Softmax heads, top-k metrics, confusion matrices, long-tail recognition, calibration, open-set recognition, fine-grained classification, and when classification is the wrong task.
softmax 头、top-k 指标、混淆矩阵、长尾识别、置信度校准、开集识别、细粒度分类，以及什么时候分类是错误的任务选择。
09
Object detection
目标检测
Boxes, anchors, IoU, NMS, two-stage vs one-stage detectors, FPN, YOLO-style heads, DETR-style set prediction, open-vocabulary detection, localization losses, hard negatives, and mAP.
边界框、锚框、交并比（IoU）、非极大值抑制（NMS）、两阶段与单阶段检测器、特征金字塔（FPN）、YOLO 风格的检测头、DETR 风格的集合预测、开放词表检测、定位损失、难负样本以及 mAP。
10
Semantic, instance, and panoptic segmentation
语义、实例与全景分割
Dense prediction, encoder-decoder models, U-Net, FCN, DeepLab, Mask R-CNN, SAM/promptable segmentation, Dice/IoU losses, boundary quality, class imbalance, and mask evaluation.
稠密预测、编码器-解码器模型、U-Net、FCN、DeepLab、Mask R-CNN、SAM/可提示分割、Dice/IoU 损失、边界质量、类别不平衡以及掩码评估。
11
Keypoints, pose, and tracking primitives
关键点、姿态与跟踪基元
Heatmaps, coordinate regression, PCK/OKS, tracking-by-detection, data association, Kalman filters, re-ID, multi-object tracking metrics, and the bridge to video.
热力图、坐标回归、PCK/OKS、检测式跟踪、数据关联、卡尔曼滤波、重识别、多目标跟踪指标，以及通往视频的桥梁。
12
Metric learning and visual retrieval
度量学习与视觉检索
Embedding spaces, contrastive/triplet/proxy losses, hard negatives, face/product retrieval, duplicate detection, ANN search, thresholds, and retrieval evaluation.
嵌入空间、对比/三元组/代理损失、难负样本、人脸/商品检索、重复检测、近似最近邻检索、阈值以及检索评估。
13
Vision transformers and hybrid architectures
视觉 Transformer 与混合架构
Patch embeddings, positional encodings, class tokens, self-attention complexity, FlashAttention and windowed attention, DeiT/Swin intuition, CNN vs ViT trade-offs, data regimes, and deployment costs.
图块嵌入、位置编码、类别 token、自注意力的复杂度、FlashAttention 与窗口注意力、DeiT/Swin 的直觉、CNN 与 ViT 的取舍、数据规模区间以及部署成本。
14
Self-supervised visual representation learning
自监督视觉表示学习
Pretext tasks, contrastive learning, BYOL/SimSiam collapse avoidance, MAE, DINO/DINOv2-style distillation, transfer, linear probing, and evaluation protocols.
前置任务、对比学习、BYOL/SimSiam 的坍缩规避、MAE、DINO/DINOv2 风格的蒸馏、迁移、线性探针以及评估协议。
15
Multimodal vision-language models
多模态视觉-语言模型
CLIP, image-text contrastive learning, captioning, VQA, OCR/document understanding, grounding, vision-encoder-to-LLM (LLaVA-style), visual instruction tuning, and retrieval-augmented vision.
CLIP、图文对比学习、图像描述、视觉问答（VQA）、OCR/文档理解、视觉定位（grounding）、视觉编码器接 LLM（LLaVA 风格）、视觉指令微调以及检索增强视觉。
16
Video and temporal vision
视频与时序视觉
Frame sampling, optical flow, 3D CNNs, temporal attention, action recognition, video detection, streaming constraints, latency budgets, and temporal metrics.
帧采样、光流、3D CNN、时序注意力、动作识别、视频检测、流式约束、延迟预算以及时序指标。
17
Generative vision models
生成式视觉模型
Autoencoders, VAEs, GANs, diffusion (DDPM/DDIM), latent diffusion, classifier-free guidance, conditioning, controllability, evaluation, safety, and why generation is not classification backward.
自编码器、VAE、GAN、扩散模型（DDPM/DDIM）、潜在扩散、无分类器引导、条件控制、可控性、评估、安全，以及为什么生成并不是把分类反过来做。
18
Image restoration and computational photography
图像复原与计算摄影
Restoration as an inverse problem (y = Ax + n): denoising, super-resolution, deblurring, and inpainting; classical regularizers vs learned and diffusion priors; the camera ISP and computational photography (HDR, burst); and the perceptual–distortion trade-off.
把复原看作一个反问题（y = Ax + n）：去噪、超分辨率、去模糊与图像修复；经典正则项与学习型及扩散先验的对比；相机 ISP 与计算摄影（HDR、连拍）；以及感知–失真之间的取舍。
19
Evaluation, deployment, and CV system design
评估、部署与计算机视觉系统设计
Leakage, robustness, slice evaluation, fairness, quantization (PTQ/QAT), pruning, distillation, edge serving, monitoring drift, human review, privacy, and end-to-end design prompts.
数据泄漏、鲁棒性、分片评估、公平性、量化（PTQ/QAT）、剪枝、蒸馏、边缘部署、漂移监控、人工复核、隐私，以及端到端的系统设计题。
How to use this
如何使用本课程
- Read it linearly once. Detection assumes feature pyramids. Feature pyramids assume receptive fields. Receptive fields assume convolution.先线性地通读一遍。检测以特征金字塔为前提。特征金字塔以感受野为前提。感受野以卷积为前提。
- Answer every question in task form. "Use ResNet" is weak. "This is a low-latency detector with small objects, so I need high-resolution features and an FPN" is the right level.用任务的语言回答每一道题。"用 ResNet"太弱了。"这是一个低延迟、带小目标的检测器，所以我需要高分辨率特征和一个 FPN"才是恰当的层次。
- Practice failure modes. CV systems fail on lighting, occlusion, viewpoint, domain shift, duplicates, class imbalance, annotation noise, and thresholds that do not match the business cost.练习各种失效模式。计算机视觉系统会在光照、遮挡、视角、域偏移、重复样本、类别不平衡、标注噪声，以及与业务成本不匹配的阈值上翻车。
- Connect model and system constraints. A 60 FPS camera pipeline, offline product matching, radiology triage, and document OCR need different trade-offs even when they share a backbone.把模型与系统约束联系起来。一条 60 FPS 的相机流水线、离线商品匹配、放射影像分诊和文档 OCR，即便共用同一个骨干网络，也需要不同的取舍。
Companion material配套材料
Continue into the companion 3D Vision track — the from-scratch 3D half of the field: SE(3) & rotations, depth and point clouds, registration, meshes and SDFs, the rasterization pipeline, NeRF, Gaussian Splatting, 3D detection, and generative 3D. Use deep_learning for backprop, optimizers, normalization, attention, and scaling laws. Use traditional_ml for calibration, imbalance, and evaluation basics. Use gpu_kernels when the interview shifts from model design to GPU throughput.
继续进入配套的 3D Vision 课程——这一领域从零构建的三维那一半：SE(3) 与旋转、深度与点云、配准、网格与 SDF、光栅化流水线、NeRF、高斯泼溅（Gaussian Splatting）、3D 检测以及生成式 3D。反向传播、优化器、归一化、注意力和缩放律请用 deep_learning。置信度校准、不平衡和评估基础请用 traditional_ml。当面试从模型设计转向 GPU 吞吐时，请用 gpu_kernels。