Video and temporal models

视频与时序模型

Lesson 15 closed our tour of single still images: CLIP aligned one frozen frame to text, and the VLM stack (vision encoder → projector → LLM) could caption or answer questions about a photograph. But the photograph never moved. Lesson 15 closed on exactly that promise — static-image representations are done, and a video is just images plus motion — so this lesson makes good on it by adding the time axis. The real world is a stream — an action is not a pixel pattern, it is a pixel pattern changing over time — and that one extra dimension brings a new free variable that quietly decides everything: which frames the model even gets to see.

第 15 课结束了我们对单张静止图像的巡礼：CLIP 把一帧冻结的画面与文本对齐，而 VLM 技术栈（视觉编码器 → 投影器 → LLM）能够对一张照片做图像描述或回答问题。但照片从不运动。第 15 课正是以这个承诺收尾的——静态图像的表示已经讲完，而视频不过是图像加上运动——所以本课就通过加上时间轴来兑现它。真实世界是一条数据流——一个动作不是一种像素模式，而是一种随时间变化的像素模式——而这多出来的一个维度带来了一个悄悄决定一切的新自由变量：模型究竟能看到哪些帧。

What we are standing on. This lesson borrows tools earned earlier rather than rebuilding them: lesson 02's spatial gradients and the Nyquist/aliasing sampling limit (now applied to time, not space), lesson 06's convolution and its FLOP accounting and receptive field, lesson 07's train/serve split discipline (which a video makes far easier to violate), lesson 09's IoU and mAP, and lesson 13's self-attention with its O(n²) cost. Keep those five in reach; the new material is just the time axis layered on top.

我们所站立的基础。本课借用此前已经掌握的工具，而不是重新搭建它们：第 02 课的空间梯度与奈奎斯特/混叠采样限（现在应用于时间，而非空间）、第 06 课的卷积及其 FLOP 计算和感受野、第 07 课的训练/服务划分纪律（视频让它更容易被违反）、第 09 课的 IoU 与 mAP，以及第 13 课带有 O(n²) 开销的自注意力。把这五样东西放在手边；新的内容只是叠加在其上的时间轴而已。

The plan计划

Four moves. (1) Treat frame sampling — rate, stride, clip length — as a modeling decision, not a loader detail, and count the data it throws away. (2) Pin down what optical flow is numerically: the brightness-constancy assumption and the one scalar equation it yields. (3) Compare 3D conv, two-stream, and temporal attention with like-for-like clip FLOPs, then place recognition inside the modern landscape of video foundation encoders, predictive models, and action-conditioned world models. (4) Put it in production: streaming latency budgets, causal vs offline models, and temporal metrics. 四步走。(1) 把帧采样——帧率、步幅、片段长度——当成建模决策，而不是数据加载器细节，并算清它丢掉了多少信息。(2) 从数值上钉死光流是什么：亮度恒定假设，以及它导出的标量方程。(3) 以相同片段为基准比较 3D 卷积、双流与时序注意力的 FLOP，再把识别模型放进视频基础编码器、预测模型和动作条件世界模型的现代版图中。(4) 把它落到生产中：流式延迟预算、因果与离线模型，以及时序指标。

1 · Frame sampling is a modeling choice

1 · 帧采样是一个建模决策

A one-minute clip at 30 frames per second (fps) and 224×224×3 is 60 · 30 · 224 · 224 · 3 ≈ 2.7 × 10⁸ values — about 1.1 GB as float32 (4 bytes each; ~270 MB even stored as 8-bit bytes) before a single layer runs. You cannot feed it whole. So every video model begins with a sampling policy: how many frames, at what spacing, covering how much wall-clock time. That policy is not plumbing; it sets the ceiling on what the model can possibly learn, because information you do not sample, you cannot recover.

一段一分钟、每秒 30 帧（fps）、分辨率为 224×224×3 的片段是 60 · 30 · 224 · 224 · 3 ≈ 2.7 × 10⁸ 个数值——以 float32 计约 1.1 GB（每个 4 字节；即便以 8 位字节存储也有约 270 MB），而这还是在任何一层运行之前。你无法把它整个喂进去。所以每个视频模型都从一个采样策略开始：取多少帧、以什么间隔、覆盖多少真实时长。这个策略不是水管工程；它设定了模型可能学到的上限，因为你没有采样的信息，就无法复原。

Three knobs, with concrete consequences:

三个旋钮，各有具体后果：

Frame rate / stride. If you keep every s-th frame from a 30 fps source, your effective rate is 30/s fps and the time between sampled frames is Δt = s/30 seconds. A hand-clap that lasts 0.1 s spans only 3 frames at 30 fps; sample every 8th frame (Δt ≈ 0.27 s) and you may capture zero frames of the closed hand — the action becomes invisible.帧率 / 步幅。如果你从一个 30 fps 的源里每隔 s 帧保留一帧，你的有效帧率就是 30/s fps，采样到的帧之间的时间间隔是 Δt = s/30 秒。一次持续 0.1 s 的拍手，在 30 fps 下只跨越 3 帧；每 8 帧采样一次（Δt ≈ 0.27 s），你可能一帧合拢的手都没捕捉到——这个动作变得不可见。
Clip length. How many sampled frames the model ingests at once (e.g. 8, 16, 32). This is the temporal receptive field: a 16-frame clip at stride 4 covers 16 · 4 / 30 ≈ 2.1 s of real time. Actions longer than the clip (opening a door, a tennis rally) are seen only in pieces.片段长度。模型一次吞入多少帧采样（例如 8、16、32）。这就是时序感受野：一个步幅为 4 的 16 帧片段覆盖 16 · 4 / 30 ≈ 2.1 s 的真实时间。比片段更长的动作（开一扇门、一个网球回合）只能被分段地看到。
Sampling pattern. Dense (contiguous frames — good for fine motion), uniform/sparse (spread evenly across the whole video — TSN-style, good for slow context), or random per training step (an augmentation that forces invariance to exact timing). Two-rate designs like SlowFast run two pathways at once: a slow path at high spatial resolution but a low frame rate (it captures appearance — what is in the scene), and a fast path at low spatial resolution but a high frame rate (it captures motion — how things move); the two are then fused, so each axis is sampled where it is cheap to do so.采样模式。密集（连续帧——适合精细运动）、均匀/稀疏（在整段视频上均匀铺开——TSN 风格，适合缓慢的上下文），或每个训练步随机（一种数据增强，迫使模型对确切时序保持不变性）。像 SlowFast 这样的双速率设计同时运行两条通路：一条慢路，高空间分辨率但低帧率（捕捉外观——场景里有什么），以及一条快路，低空间分辨率但高帧率（捕捉运动——东西如何移动）；两者随后融合，于是每个轴都在其代价低廉之处被采样。

The frequency view from lesson 02 applies directly: sampling in time is subject to the same Nyquist limit as sampling in space. To resolve an event that repeats at f Hz you must sample faster than 2f; below that, fast motion aliases into a slower phantom (the wagon-wheel effect — spokes appearing to spin backward). Choosing the frame rate is choosing your temporal anti-aliasing budget.

第 02 课的频率视角在这里直接适用：时间上的采样受制于与空间采样相同的奈奎斯特限。要分辨一个以 f Hz 重复的事件，你的采样必须快于 2f；低于这个速度，快速运动就会混叠成一个更慢的幻影（车轮效应——辐条看起来在倒转）。选择帧率，就是在选择你的时序抗混叠预算。

source 30 fps |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■| 30 frames / sec ↓ stride 8 (Δt ≈ 0.27 s) sampled |■.......■.......■.......■....| ~4 frames / sec ↑ a 0.1 s clap (3 source frames) can fall entirely in a gap — sampled too coarsely to ever be learned

Trap — the split leak comes back, worse陷阱——划分泄漏卷土重来，而且更糟

Adjacent frames are near-duplicates: frame t and t+1 differ by milliseconds of motion. Split a dataset at the frame level and almost every validation frame has a near-twin in training — the model memorizes, validation accuracy looks superb, production collapses. Split by source video, scene, or uploader, exactly the entity/time-level discipline lesson 07 demanded for images, now mandatory because the duplication is structural. 相邻帧是近乎重复的：第 t 帧与第 t+1 帧只相差几毫秒的运动。在帧级别划分数据集，几乎每一个验证帧在训练集里都有一个近乎孪生的兄弟——模型死记硬背，验证准确率看起来极好，生产环境却崩溃。要按源视频、场景或上传者划分，这正是第 07 课对图像所要求的那种实体/时间级别的纪律，如今更是必须的，因为这种重复是结构性的。

2 · What optical flow actually is

2 · 光流究竟是什么

"Motion signal" is vague. Optical flow makes it a number: for each pixel, the 2D displacement vector (u, v), measured in pixels per frame-step, that says where that bit of brightness moved between frame t and frame t+1 (so the "step" is one frame, Δt = 1 — the unit the Taylor expansion below quietly assumes). The whole construction rests on one assumption, the brightness-constancy assumption: a surface point keeps the same intensity as it moves a small amount over a small time. Writing the image intensity as I(x, y, t):

"运动信号"是含糊的。光流把它变成一个数：对每个像素，给出一个二维位移向量 (u, v)，以像素每帧步长为单位，说明那一小块亮度在第 t 帧与第 t+1 帧之间移动到了哪里（所以这里的"步长"是一帧，Δt = 1——正是下面泰勒展开悄悄假定的那个单位）。整个构造建立在一个假设之上，即亮度恒定假设：一个表面点在小的时间内移动一小段距离时，保持相同的强度。把图像强度写作 I(x, y, t)：

I(x, y, t) = I(x + u, y + v, t + 1) .

"A pixel's brightness doesn't change as it slides to its new location." Now expand the right side with a first-order Taylor approximation (valid when the motion is small — the assumption's fine print):

"一个像素的亮度在它滑向新位置时不会改变。"现在用一阶泰勒近似展开右边（在运动很小时成立——这就是那条假设的附加细则）：

I(x + u, y + v, t+1) ≈ I(x, y, t) + I_x u + I_y v + I_t ,

where I_x, I_y are the spatial gradients (how brightness changes left-right and up-down — the Sobel responses from lesson 02) and I_t is the temporal gradient (frame difference at that pixel). This is just the multivariate f(a+Δ) ≈ f(a) + ∇f·Δ in three variables, with steps (u, v, 1) — so the I_t term is the time-derivative times the one-frame step. Substitute into brightness-constancy (set the two sides equal): the shared I(x,y,t) cancels, leaving the optical-flow constraint equation:

其中 I_x, I_y 是空间梯度（亮度在左右和上下方向如何变化——第 02 课的 Sobel 响应），而 I_t 是时间梯度（该像素处的帧差）。这不过是三变量下的多元式 f(a+Δ) ≈ f(a) + ∇f·Δ，步长为 (u, v, 1)——所以 I_t 这一项就是时间导数乘以一帧的步长。代入亮度恒定式（令两边相等）：公共的 I(x,y,t) 相消，留下光流约束方程：

I_x u + I_y v + I_t = 0 ⟺ ∇I · (u, v) + I_t = 0 .

Worked numerically. Suppose at one pixel the horizontal gradient is I_x = 10 intensity units/pixel, the vertical gradient I_y = 0 (a vertical edge — no vertical structure), and between frames the pixel got darker: I_t = −20 units. The constraint becomes 10u + 0·v − 20 = 0 ⟹ u = 2 pixels. We recovered the horizontal motion: the edge slid 2 pixels right. But v is completely undetermined — the equation says nothing about vertical motion along a vertical edge. This is the aperture problem: one equation, two unknowns per pixel. You only ever recover the motion component perpendicular to the local edge — called the normal flow. Classical solvers break the tie with extra assumptions — Lucas–Kanade assumes neighboring pixels share one (u,v) and solves a small least-squares system; Horn–Schunck adds a global smoothness penalty. Modern systems learn it: RAFT regresses flow with iterative refinement and beats the classical methods on benchmarks.

数值演算。假设在某个像素处，水平梯度为 I_x = 10 强度单位/像素，竖直梯度 I_y = 0（一条竖直边缘——没有竖直方向的结构），而两帧之间该像素变暗了：I_t = −20 单位。约束变成 10u + 0·v − 20 = 0 ⟹ u = 2 像素。我们复原了水平运动：边缘向右滑了 2 像素。但 v 完全无法确定——方程对沿竖直边缘的竖直运动只字未提。这就是孔径问题：每像素一个方程、两个未知数。你能复原的永远只是垂直于局部边缘的那个运动分量——称为法向光流。经典求解器靠额外假设来打破僵局——Lucas–Kanade 假设相邻像素共享同一个 (u,v)，求解一个小的最小二乘系统；Horn–Schunck 则加上一个全局平滑惩罚。现代系统则把它学出来：RAFT 用迭代细化来回归光流，并在基准测试上胜过经典方法。

So flow is an explicit motion representation you can hand a model. The two-stream design (next section) does exactly that. The alternative is to let the model discover motion implicitly from raw frames — cheaper at inference (no flow to compute), but it must learn the equivalent of I_t internally and needs enough temporal data to do so.

所以光流是一种你可以直接交给模型的显式运动表示。双流设计（下一节）做的正是这件事。另一种选择是让模型从原始帧中隐式地发现运动——推理时更便宜（不必计算光流），但它必须在内部学到等价于 I_t 的东西，并且需要足够的时序数据才能做到。

2.5 · Interactive · the aperture problem

2.5 · 交互 · 孔径问题

The "one equation, two unknowns" claim is the whole reason flow is hard, and it is easiest to see. Watch a diagonal edge drift behind a small circular window: through that hole, you cannot tell whether the edge is moving sideways, up, or diagonally — every true-motion vector whose tip lands on a single dashed line produces the identical view. The constraint I_xu + I_yv + I_t = 0 is exactly that line. Only the component perpendicular to the edge (the normal flow, the blue arrow) is observable; the rest is undetermined. Add a corner — a second edge with a different orientation — and the two constraint lines intersect at one point: the ambiguity collapses to a unique vector, which is precisely what Lucas–Kanade does with a neighborhood.

"一个方程、两个未知数"这个论断正是光流之所以困难的全部原因，而它最容易被看出来。看一条对角边缘在一个小圆窗后面漂移：透过那个孔，你分不清边缘是在横向、纵向还是斜向移动——凡是尖端落在同一条虚线上的真实运动向量，产生的视图都完全相同。约束 I_xu + I_yv + I_t = 0 恰好就是那条线。只有垂直于边缘的分量（法向光流，那支蓝色箭头）是可观测的；其余部分无法确定。加上一个角点——一条方向不同的第二条边缘——两条约束线便交于一点：歧义坍缩成唯一的向量，这正是 Lucas–Kanade 用一个邻域所做的事。

The aperture problem — one constraint line, not one vector

孔径问题——一条约束线，而非一个向量

Left: a moving edge (or corner) seen through a circular aperture. Right: velocity space (u,v) — the dashed line is every true motion consistent with what you observe through the hole; the blue arrow is the normal flow (the only recoverable component). Drag the true motion slider and watch the blue arrow refuse to change. Then switch to corner: two lines, one intersection, ambiguity resolved.

左图：透过一个圆形孔径看到的一条运动边缘（或角点）。右图：速度空间 (u,v)——虚线是与你透过孔看到的一切相符的所有真实运动；蓝色箭头是法向光流（唯一可复原的分量）。拖动真实运动滑块，看着蓝色箭头拒绝改变。然后切换到角点：两条线、一个交点，歧义被解决。

scene:场景： edge angle θ:边缘角度 θ： 0° true motion dir:真实运动方向： 0° true speed:真实速度： 2.0

I_x , I_y

10, 0

I_t

−20

normal flow |v⊥|

法向光流 |v⊥|

2.0

solvable?

可解？

no — 1 eq

Show the core JS查看核心代码

// One edge gives one row of the constraint: I_x·u + I_y·v + I_t = 0.
// The spatial gradient ∇I points across the edge (normal n = ∇I/|∇I|).
// Brightness-constancy: I_t = -∇I · vTrue  (only the along-normal part of the true motion shows up).
function observe(theta, vTrue){
  const nx = Math.cos(theta), ny = Math.sin(theta);   // unit edge-normal
  const Ix = G*nx, Iy = G*ny;                          // gradient magnitude G along the normal
  const It = -(Ix*vTrue.u + Iy*vTrue.v);               // temporal gradient we'd measure
  const vn = -It / G;                                  // normal-flow magnitude (signed)
  return { Ix, Iy, It, normalFlow:{ u: vn*nx, v: vn*ny } };
}
// Single edge → the solution set is the LINE  I_x·u + I_y·v + I_t = 0  (under-determined).
// A corner adds a 2nd edge (theta2) → 2 lines intersect → unique (u,v) = the true motion.

The KPIs reproduce the worked example exactly: set θ so the edge is vertical (gradient purely horizontal, I_y=0) and a rightward true motion of 2 px gives I_x=10, I_y=0, I_t=−20 and normal flow 2 — yet "solvable?" stays no until you add the corner. That gap between a confident I_t reading and an unsolvable (u,v) is the aperture problem in one screen.

这些 KPI 精确地重现了那个演算例子：把 θ 设成让边缘竖直（梯度纯水平，I_y=0），一个向右 2 px 的真实运动给出 I_x=10, I_y=0, I_t=−20 以及法向光流 2——然而"可解？"一直是否，直到你加上角点。一个笃定的 I_t 读数与一个无法求解的 (u,v) 之间的这道鸿沟，就是一屏之内的孔径问题。

3 · Three ways to fuse time — and why 3D conv is expensive

3 · 融合时间的三种方式——以及 3D 卷积为何昂贵

An image backbone (lesson 06) maps one frame to features. To use time we must combine information across frames. There are three dominant strategies.

一个图像骨干网络（第 06 课）把一帧映射为特征。要利用时间，我们必须跨帧地组合信息。有三种主流策略。

3D convolution extends the 2D kernel of lesson 06 with a temporal depth: instead of a k×k window over one frame, a k×k×t window slides over a stack of t frames, learning spatio-temporal patterns (a flick, a bounce) directly. It is the most natural extension — and the most expensive. Here is the arithmetic, because the FLOP argument is the whole point.

3D 卷积把第 06 课的 2D 卷积核扩展出一个时间深度：不再是一个 k×k 的窗口滑过一帧，而是一个 k×k×t 的窗口滑过 t 帧的堆叠，直接学习时空模式（一次抖动、一次弹跳）。它是最自然的扩展——也是最昂贵的。下面是这笔算术，因为 FLOP 论证正是全部要点。

A 2D conv layer with C_in input channels, C_out output channels, kernel k×k, over a feature map of H×W spatial positions costs roughly

一个具有 C_in 个输入通道、C_out 个输出通道、卷积核 k×k、作用在 H×W 个空间位置的特征图上的 2D 卷积层，其开销大约是

FLOPs_2D ≈ H · W · C_in · C_out · k · k .

The 3D version adds a temporal kernel dimension k_t and processes D temporal positions (frames in the stack):

3D 版本加上一个时间卷积核维度 k_t，并且要处理 D 个时间位置（堆叠中的帧）：

FLOPs_3D ≈ D · H · W · C_in · C_out · k · k · k_t .

Compare the same clip. Take H=W=56, C_in=C_out=256, and k=3. One 2D application costs 1.85 × 10⁹ operations per frame, so applying it independently to all D=16 frames costs 16·1.85 × 10⁹ ≈ 2.96 × 10¹⁰. A 3D layer with k_t=3 over those same 16 output times costs ≈8.9 × 10¹⁰: about k_t=3× the same-clip 2D baseline, or D·k_t=48× only if compared misleadingly with a single 2D frame. Boundary padding and output strides can adjust exact counts, but the like-for-like ratio is the temporal kernel factor. (2+1)D splits k×k×k_t into spatial k×k×1 then temporal 1×1×k_t; its saving depends on the intermediate channel width, while the extra non-linearity can also improve optimization. SlowFast instead allocates different frame rates and channel budgets to slow semantic and fast motion pathways.

必须比较同一个片段。取 H=W=56、C_in=C_out=256、k=3。2D 层每帧一次的开销是 1.85 × 10⁹，因此独立处理 D=16 帧的开销为 16·1.85 × 10⁹ ≈ 2.96 × 10¹⁰。在同样 16 个输出时刻上、k_t=3 的 3D 层开销约为 8.9 × 10¹⁰：相对同片段 2D 基线约为 k_t=3×；只有在误导性地拿它与单帧 2D 比较时才是 D·k_t=48×。边界填充与输出步幅会改变精确数字，但同口径比值来自时间核因子。(2+1)D 把 k×k×k_t 拆成空间 k×k×1 再接时间 1×1×k_t；节省多少取决于中间通道宽度，中间多出的非线性也可能改善优化。SlowFast 则给慢速语义路径与快速运动路径分配不同帧率和通道预算。

Two-stream networks sidestep learning motion from scratch: one CNN branch sees RGB (appearance — what is there), a second sees precomputed optical flow (motion — how it moves), and their predictions fuse. This was the early winner on action recognition precisely because flow hands the network the I_t signal it would otherwise struggle to extract. The cost is the flow computation itself — a second, often non-trivial, pipeline and failure surface.

双流网络绕开了从零学习运动：一条 CNN 分支看 RGB（外观——有什么），第二条看预先计算好的光流（运动——如何移动），二者的预测再融合。它之所以是动作识别的早期赢家，恰恰是因为光流把网络本来难以提取的 I_t 信号直接递到了手上。代价是光流计算本身——第二条往往并不简单的流水线，以及一处失败面。

Temporal attention tokenizes frames or patches and lets the transformer attend across time. With F frames and P patches per frame, joint attention over FP tokens costs O(F²P²). Factorized attention performs spatial attention for each of the F frames plus temporal attention for each of the P patch positions, for O(FP² + PF²)—not O(P²+F²), because each subproblem is repeated. Windowing, sparse attention, token pruning, and memory mechanisms reduce it further.

时序注意力把帧或图块切成 token，让 Transformer 跨时间做注意力。若有 F 帧、每帧 P 个图块，对全部 FP 个 token 做联合注意力的开销是 O(F²P²)。分解注意力会对每个 F 帧做空间注意力，再对每个 P 图块位置做时间注意力，总开销为 O(FP² + PF²)——而不是 O(P²+F²)，因为每个子问题都要重复。窗口化、稀疏注意力、token 剪枝与记忆机制还能进一步降本。

Family家族	How it fuses time如何融合时间	Cost driver开销主因	Best when最适合于
Per-frame + pool逐帧 + 池化	Image backbone per frame, average logits每帧一个图像骨干，对 logit 求平均	cheapest; linear in frames最便宜；帧数线性	action ≈ appearance (a scene, an object)动作 ≈ 外观（一个场景、一个物体）
3D CNN / (2+1)D3D CNN / (2+1)D	conv over space and time在空间与时间上做卷积	≈ ×k_t over per-frame 2D on the same clip相对同片段逐帧 2D	fine local motion matters精细的局部运动很关键
Two-stream双流	RGB branch + explicit optical-flow branchRGB 分支 + 显式光流分支	flow precompute pipeline光流预计算流水线	motion-defined actions, smaller data由运动定义的动作、数据较少
Temporal transformer时序 Transformer	attention over frame/patch tokens在帧/图块 token 上做注意力	O(n²), memory-heavy，显存开销大	long-range temporal context, large data长程时序上下文、数据量大
Track-then-classify先跟踪后分类	detect/track (lesson 11), classify trajectory检测/跟踪（第 11 课），再对轨迹分类	tracker quality跟踪器质量	object-centric events, few actors以物体为中心的事件、行动者少

3b · The modern landscape: represent, predict, or intervene?

3b · 现代版图：表示、预测，还是干预？

Architecture names hide three different learning contracts. A video foundation encoder learns a reusable representation from large video corpora using masked prediction, contrastive objectives, or video–text alignment; it can recognize and retrieve without being a simulator. A predictive video model learns p(z_t+1:t+H | z_≤t) (or pixels) and must represent multiple plausible futures. An action-conditioned world model learns p(z_t+1:t+H | z_≤t, a_t:t+H−1): the action is essential if we want counterfactuals—same state, different action, different consequence—for planning or control. Large actionless video pretraining can supply visual and physical priors, then smaller action-labeled trajectories add controllability. Pixel generation is useful for inspection; latent prediction is often cheaper for decision-making. The state, action, and intervention contract—not photorealism alone—determines whether a system is a world model.

架构名字常掩盖三种不同的学习契约。视频基础编码器通过掩码预测、对比目标或视频–文本对齐，从海量视频中学习可复用表示；它能做识别与检索，却不因此成为模拟器。预测式视频模型学习 p(z_t+1:t+H | z_≤t)（或像素），并必须表达多个合理未来。动作条件世界模型学习 p(z_t+1:t+H | z_≤t, a_t:t+H−1)：若要为规划或控制回答反事实——同一状态、不同动作、不同后果——动作不可或缺。大规模无动作视频预训练可提供视觉与物理先验，再由规模较小、带动作标签的轨迹加入可控性。像素生成便于检查；潜在预测通常更适合低成本决策。决定系统是否是世界模型的，是状态、动作与干预契约，而不只是照片级真实感。

4 · Streaming, latency budgets, and temporal metrics

4 · 流式处理、延迟预算与时序指标

The headline task is action recognition: assign a clip a label from a fixed set ("clapping", "pouring"). Its cousins are temporal action localization (find the start/end of each action in a long untrimmed video) and spatio-temporal detection (box + action per actor per frame). Each needs a metric that respects time.

头号任务是动作识别：为一个片段从固定集合中赋予一个标签（"拍手""倒水"）。它的近亲是时序动作定位（在一段未剪辑的长视频里找出每个动作的起止）和时空检测（每帧每个行动者的边界框 + 动作）。每一种都需要一个尊重时间的指标。

Clip accuracy vs video accuracy. A model scores individual clips, but a video has many clips. Clip accuracy evaluates each sampled clip; video accuracy aggregates (average the softmax over clips, then argmax). Video accuracy is what users see and is almost always higher — averaging cancels per-clip noise. Reporting clip accuracy as if it were video accuracy understates the system; the reverse oversells it. State which one you mean.片段准确率与视频准确率。模型对单个片段打分，但一段视频有许多片段。片段准确率评估每个采样到的片段；视频准确率做聚合（对各片段的 softmax 求平均，再取 argmax）。视频准确率才是用户看到的，且几乎总是更高——求平均抵消了逐片段的噪声。把片段准确率当成视频准确率来报告会低估系统；反过来则会夸大它。请说清你指的是哪一个。
Temporal IoU and localization mAP. For localization, borrow IoU from lesson 09 but in 1D: tIoU = (overlap of predicted and true time intervals) / (their union). A prediction counts as correct if tIoU ≥ τ (e.g. 0.5), and you summarize with mean Average Precision (mAP) — exactly the precision-recall area of lesson 09, now over temporal segments, averaged across classes and often across several τ thresholds.时序 IoU 与定位 mAP。对定位，借用第 09 课的 IoU，但放到一维上：tIoU = (预测与真实时间区间的重叠) / (二者的并集)。当 tIoU ≥ τ（例如 0.5）时一个预测算作正确，你再用平均精度均值（mAP）来汇总——正是第 09 课的精确率-召回率曲线下面积，如今作用于时序片段，在各类别间（并常常在若干 τ 阈值间）取平均。
Streaming-specific quality. For online models also track per-frame latency, detection delay (how long after an action begins until you flag it), and temporal flicker (label thrashing between adjacent frames).流式处理特有的质量。对在线模型还要追踪逐帧延迟、检测延迟（动作开始后过多久你才标记出它）以及时序抖动（相邻帧之间标签来回跳变）。

The streaming latency budget. An offline model may use future frames — it can look at frame t+10 to label frame t. A real-time model cannot; it is causal, seeing only the past. Concretely, suppose a product has a 100 ms end-to-end budget per frame at 30 fps (a frame arrives every 1000/30 ≈ 33 ms). If inference takes 70 ms, you have 30 ms for decode + preprocess + post-process, and you cannot buffer 10 future frames for smoothing without adding 10·33 = 330 ms of delay — blowing the budget more than 3×. So streaming forces choices the offline benchmark never penalizes: smaller clips, causal (no-lookahead) temporal modeling, frame-skipping under load, and lightweight temporal smoothing (an exponential moving average over recent logits) that trades a little lag for stability. The trap: heavy smoothing hides exactly the quick events you may most need to catch.

流式延迟预算。离线模型可以使用未来的帧——它能看第 t+10 帧来给第 t 帧打标签。实时模型不行；它是因果的，只能看到过去。具体地说，假设一个产品在 30 fps 下有 100 ms 的端到端逐帧预算（每 1000/30 ≈ 33 ms 到来一帧）。若推理耗时 70 ms，你只剩 30 ms 用于解码 + 预处理 + 后处理，而你无法缓冲 10 个未来帧来做平滑，否则会加进 10·33 = 330 ms 的延迟——把预算超支 3 倍以上。所以流式处理迫使你做出离线基准从不惩罚的取舍：更小的片段、因果（无前瞻）的时序建模、负载下跳帧，以及轻量的时序平滑（对近期 logit 做指数移动平均），用一点点滞后换取稳定。陷阱在于：过重的平滑恰恰会掩盖你可能最需要捕捉的那些快速事件。

Handoff to 17交接到第 17 课

Everything in lessons 01–16 has analyzed pixels — classified them, localized them, tracked them through time, aligned them to text. We have been a very sophisticated reader of images. The obvious next question: can a model run the arrow the other way and synthesize pixels — turn a label, a caption, or pure noise into an image or a video? That is a fundamentally different objective, and lesson 17 builds it from p(x) up. 第 01–16 课里的一切都在分析像素——给它们分类、定位、随时间跟踪、与文本对齐。我们一直是一名非常老练的图像读者。显而易见的下一个问题是：模型能不能把箭头反过来指，去合成像素——把一个标签、一句描述或纯噪声变成一张图像或一段视频？那是一个根本不同的目标，而第 17 课将从 p(x) 出发把它建立起来。

Where this points next

这将指向何处

We added time and found that the loader is part of the model: sampling rate sets the information ceiling, flow gives an explicit motion signal, and 3D conv / two-stream / attention is a compute-and-data trade. Recognition and foundation encoders summarize video; predictive and action-conditioned models forecast it. Lesson 17 (Generative vision) now broadens the output contract from future prediction to explicit sampling from p(x) or p(x|c). That is not classification run backward: its one-to-many structure demands autoencoders, GANs, diffusion, and flow-based generation. The temporal constraint returns in video generation, where independently plausible frames are insufficient—the trajectory must also be coherent.

我们加上了时间，并发现数据加载器是模型的一部分：采样率设定信息上限，光流给出显式运动信号，而 3D 卷积 / 双流 / 注意力之间是算力与数据的权衡。识别与基础编码器负责概括视频；预测式与动作条件模型负责预报视频。第 17 课（生成式视觉）会把输出契约从未来预测扩展为从 p(x) 或 p(x|c) 中显式采样。这不是把分类反过来跑：一对多结构需要自编码器、GAN、扩散与基于流的生成。时序约束会在视频生成中再次出现——逐帧各自合理还不够，整条轨迹也必须一致。

Takeaway要点

Video is images plus time, and frame sampling sets the information ceiling. Optical flow rests on brightness constancy and yields I_xu + I_yv + I_t = 0, hence the aperture problem. For the same D-frame clip, a dense 3D convolution is about k_t× its per-frame 2D counterpart, not Dk_t×; factorized attention costs O(FP²+PF²). Keep the learning contract explicit: foundation encoders represent, predictive models forecast, and action-conditioned world models support interventions. Production then chooses offline versus causal streaming and evaluates with clip/video accuracy or temporal-IoU mAP. 视频是图像加时间，而帧采样设定信息上限。光流建立在亮度恒定上，并导出 I_xu + I_yv + I_t = 0，因此产生孔径问题。对同一个 D 帧片段，稠密 3D 卷积相对逐帧 2D 的开销约为 k_t×，而不是 Dk_t×；分解注意力的开销为 O(FP²+PF²)。必须写清学习契约：基础编码器做表示，预测模型预报未来，动作条件世界模型支持干预。生产再选择离线或因果流式，并以片段/视频准确率或时序 IoU mAP 评估。

Interview prompts

面试题

Why is video harder to split correctly than images, and what split granularity fixes it? (§1: adjacent frames are near-duplicates; split by source/scene/uploader, not frame.)为什么视频比图像更难正确划分，什么样的划分粒度能修复它？(§1：相邻帧近乎重复；按源/场景/上传者划分，而不是按帧。)
Write the optical-flow constraint equation and explain the aperture problem. (§2: I_xu+I_yv+I_t=0; one equation, two unknowns — only motion perpendicular to the edge is recoverable.)写出光流约束方程并解释孔径问题。(§2：I_xu+I_yv+I_t=0；一个方程、两个未知数——只有垂直于边缘的运动可以复原。)
Compare a 3D conv with per-frame 2D conv on the same clip. What is the cost ratio and how does (2+1)D help? (§3: roughly k_t, not Dk_t; factorize spatial then temporal, with savings set by intermediate width.)在同一片段上比较 3D 卷积与逐帧 2D 卷积。开销比是多少，(2+1)D 如何缓解？(§3：约为 k_t，而不是 Dk_t；先空间后时间分解，节省量取决于中间宽度。)
How do frame rate and clip length change which actions are learnable? (§1: stride sets temporal Nyquist; clip length sets the temporal receptive field.)帧率和片段长度如何改变哪些动作是可学习的？(§1：步幅设定时序奈奎斯特；片段长度设定时序感受野。)
How do offline and streaming video models differ, and what does a 100 ms budget at 30 fps imply? (§4: causal vs lookahead; can't buffer future frames without blowing the budget.)离线视频模型与流式视频模型有何不同，30 fps 下 100 ms 的预算意味着什么？(§4：因果与前瞻之别；不能在不超支预算的前提下缓冲未来帧。)
How would you evaluate an action-recognition model — and report results honestly? (§4: distinguish clip vs video accuracy; use temporal-IoU mAP for localization.)你会如何评估一个动作识别模型——并诚实地报告结果？(§4：区分片段准确率与视频准确率；定位用时序 IoU 的 mAP。)
What separates a video foundation encoder, a predictive video model, and an action-conditioned world model? (§3b: reusable representation vs future distribution vs intervention-conditioned consequences; action labels enable counterfactual planning.)视频基础编码器、预测式视频模型与动作条件世界模型有何区别？(§3b：可复用表示 vs 未来分布 vs 以干预为条件的后果；动作标签使反事实规划成为可能。)