Multimodal vision-language models

多模态视觉-语言模型

Lesson 14 let images supervise themselves, but that signal teaches structure and invariance, not human meaning. This lesson hands the supervisor's job to the text that already accompanies images across the web. We start with CLIP — which uses the same contrastive idea from lessons 12 and 14, now across two modalities — write its loss precisely, and then climb to the architecture that defines the field today: a vision encoder feeding a large language model.

第 14 课让图像自我监督，但那种信号教会的是结构与不变性，而不是人类意义上的含义。本课把监督者的工作交给了网络上早已伴随图像出现的文本。我们从 CLIP 开始——它使用的正是第 12、14 课中的对比学习思想，只不过现在是跨两个模态——精确地写出它的损失，然后攀升到今天定义整个领域的架构：一个视觉编码器为大语言模型供给输入。

What we are standing on

我们站在什么之上

Two threads converge here. From lesson 12 and 14: contrastive learning — pull matching pairs together, push mismatched pairs apart, via a softmax over similarities (InfoNCE). We will not redefine "contrastive," "in-batch negatives," or "temperature" — they are exactly as before, only now one element of each pair is an image and the other is text. From lesson 13: the ViT, which is the image encoder inside almost every model below. The new idea is purely the cross-modal pairing and what it unlocks.

两条线索在这里汇合。来自第 12、14 课：对比学习——通过对相似度做 softmax（InfoNCE），把匹配的样本对拉近、把不匹配的样本对推远。我们不会重新定义"对比"、"批内负样本"或"温度"——它们和之前完全一样，只不过现在每个样本对中一个元素是图像、另一个是文本。来自第 13 课：ViT，它是下面几乎每个模型内部的图像编码器。全新的思想纯粹在于跨模态配对以及它所解锁的能力。

1 · CLIP: contrastive pretraining across modalities

1 · CLIP：跨模态的对比预训练

CLIP (Contrastive Language–Image Pretraining) trains two encoders jointly: an image encoder (a ViT) producing f_i for image i, and a text encoder (a Transformer) producing g_j for caption j. Both outputs are projected to a shared dimension and L2-normalized, so a dot product is a cosine similarity (lesson 12's move). The training data is a batch of N image–text pairs scraped from the web — the caption that shipped with each image is its positive; every other caption in the batch is a negative. No manual labels.

CLIP（Contrastive Language–Image Pretraining，对比式语言-图像预训练）联合训练两个编码器：一个图像编码器（一个 ViT），为图像 i 产生 f_i；一个文本编码器（一个 Transformer），为标题 j 产生 g_j。两个输出都被投影到一个共享维度并做 L2 归一化，于是点积就是余弦相似度（第 12 课的做法）。训练数据是从网络上爬取的一个批次的 N 个图像–文本对——每张图像自带的标题是它的正样本；批次中其他每个标题都是负样本。无需人工标注。

Compute the N×N similarity matrix S, scaled by a learned temperature τ:

计算 N×N 的相似度矩阵 S，并用一个可学习的温度 τ 做缩放：

The correct matches lie on the diagonal (image i with its own caption i); everything off-diagonal is a negative. The loss is symmetric InfoNCE — a cross-entropy that, read across each row, asks "which caption matches this image?" and, read down each column, asks "which image matches this caption?" — averaged:

正确的匹配位于对角线上（图像 i 配它自己的标题 i）；一切非对角元都是负样本。损失是对称 InfoNCE——一个交叉熵：沿每一行读时，问"哪个标题匹配这张图像？"，沿每一列读时，问"哪张图像匹配这个标题？"——再取平均：

This is the InfoNCE of lesson 14, applied twice (once per direction) over the same matrix. The temperature τ controls how sharply the softmax distinguishes the diagonal from off-diagonal — small τ sharpens (heavily penalizes near-miss negatives), and CLIP learns it. With a batch of N=32768, each pair is contrasted against ~32k in-batch negatives at once — which is why CLIP wants enormous batches, exactly the negative-hungry behavior lesson 14 noted for contrastive methods.

这就是第 14 课的 InfoNCE，在同一个矩阵上施加两次（每个方向一次）。温度 τ 控制 softmax 区分对角线与非对角元的锐利程度——小 τ 使之更锐利（重罚差一点就命中的负样本），而 CLIP 会去学习它。当批量 N=32768 时，每个样本对会一次性与约 3.2 万个批内负样本做对比——这正是 CLIP 想要巨大批量的原因，恰恰是第 14 课指出的对比方法那种"嗜负样本"的行为。

One line worth stating explicitly, because the lesson leans on it everywhere: the L2-normalization is why the dot product reads as cosine. If |a| = |b| = 1, then a · b = |a||b|cos∠ = cos∠ — the dot product collapses to the cosine of the angle, bounded in [−1, 1]. That is the entire reason CLIP normalizes before scoring.

有一句话值得明确写出来，因为本课处处倚重它：L2 归一化正是点积能读作余弦的原因。若 |a| = |b| = 1，则 a · b = |a||b|cos∠ = cos∠——点积坍缩为夹角的余弦，取值被限制在 [−1, 1]。这就是 CLIP 在打分前先做归一化的全部原因。

2 · Worked by hand: a 3×3 CLIP loss

2 · 手把手算：一个 3×3 的 CLIP 损失

Numbers make the symmetric loss concrete. Take a tiny batch of N=3 image–text pairs and, for simplicity, fold the temperature in (set τ=1) so the matrix entries are the scaled similarities S_ij. Suppose after some training the model has produced this similarity matrix (rows = images, columns = captions; the diagonal is the correct match):

数字能把对称损失讲具体。取一个只有 N=3 个图像–文本对的小批次，并为简单起见把温度折进去（设 τ=1），于是矩阵元就是缩放后的相似度 S_ij。假设经过一些训练后，模型产生了这个相似度矩阵（行 = 图像，列 = 标题；对角线是正确匹配）：

Row 1, image→text. The softmax denominator is e^2.0 + e^0.5 + e^0.0 = 7.389 + 1.649 + 1.000 = 10.038. The probability mass on the correct (diagonal) caption is p₁₁ = 7.389 / 10.038 = 0.736. Its contribution to the loss is −log p₁₁ = −log(0.736) = 0.306.

第 1 行，图像→文本。softmax 的分母是 e^2.0 + e^0.5 + e^0.0 = 7.389 + 1.649 + 1.000 = 10.038。落在正确（对角线）标题上的概率质量是 p₁₁ = 7.389 / 10.038 = 0.736。它对损失的贡献是 −log p₁₁ = −log(0.736) = 0.306。

Doing the same for the other two rows: row 2 gives denominator e^0.3+e^1.8+e^0.4 = 1.350+6.050+1.492 = 8.892, so p₂₂ = 6.050/8.892 = 0.680 and −log p₂₂ = 0.385. Row 3 gives denominator 1.105+1.221+4.482 = 6.808, so p₃₃ = 4.482/6.808 = 0.658 and −log p₃₃ = 0.418. Averaging the three rows:

对另外两行做同样的计算：第 2 行的分母是 e^0.3+e^1.8+e^0.4 = 1.350+6.050+1.492 = 8.892，于是 p₂₂ = 6.050/8.892 = 0.680、−log p₂₂ = 0.385。第 3 行的分母是 1.105+1.221+4.482 = 6.808，于是 p₃₃ = 4.482/6.808 = 0.658、−log p₃₃ = 0.418。对这三行取平均：

Now the other direction. The column softmax reads down each column (caption→image): for column 1 the denominator is e^2.0+e^0.3+e^0.1 = 7.389+1.350+1.105 = 9.844, so p₁₁^col = 7.389/9.844 = 0.751, −log = 0.287. Columns 2 and 3 work out to −log = 0.388 and 0.442, giving ℒ_t→i = ⅓(0.287+0.388+0.442) = 0.372. The symmetric loss is the average of the two directions:

现在看另一个方向。列 softmax 沿每一列往下读（标题→图像）：对第 1 列，分母是 e^2.0+e^0.3+e^0.1 = 7.389+1.350+1.105 = 9.844，于是 p₁₁^col = 7.389/9.844 = 0.751、−log = 0.287。第 2、3 列算出来是 −log = 0.388 和 0.442，得到 ℒ_t→i = ⅓(0.287+0.388+0.442) = 0.372。对称损失就是两个方向的平均：

The intuition the arithmetic makes unmissable: the loss is small only when each diagonal entry dominates its row and its column. Raise a diagonal value (a positive pair scores higher) or lower the off-diagonal noise, and the diagonal's softmax probability climbs toward 1, driving −log p toward 0. The widget below lets you push exactly those knobs and watch ℒ fall.

这段算术让直觉无法回避：损失只有在每个对角元同时主导它所在的行和列时才会小。抬高一个对角值（让一个正样本对得分更高），或降低非对角的噪声，对角线的 softmax 概率就会攀向 1，把 −log p 推向 0。下面的小部件让你恰好去拨动这些旋钮，看着 ℒ 下降。

3 · Interactive · the CLIP loss matrix

3 · 交互 · CLIP 损失矩阵

A 4×4 image–text similarity matrix — the same object as section 1, now live. The four diagonal cells (green outline) are the correct image–caption pairs; the twelve off-diagonal cells are in-batch negatives. Click any cell to raise its similarity, shift-click to lower it, or hit Train step to nudge the whole matrix the way the gradient would — diagonal up, off-diagonal down. The panel computes the row softmax (i→t), the column softmax (t→i), each row's −log p_ii, and the symmetric loss ℒ = ½(ℒ_i→t+ℒ_t→i) — exactly the by-hand computation above, scaled to N=4. Watch ℒ drop as the diagonal separates from its row and column.

一个 4×4 的图像–文本相似度矩阵——和第 1 节里是同一个对象，现在是活的了。四个对角格（绿色描边）是正确的图像–标题对；十二个非对角格是批内负样本。点击任意格子可抬高它的相似度，shift-点击可降低它，或者点训练一步让整个矩阵按梯度的方向被推动——对角线上升、非对角下降。面板会计算行 softmax（i→t）、列 softmax（t→i）、每一行的 −log p_ii，以及对称损失 ℒ = ½(ℒ_i→t+ℒ_t→i)——正是上面手算的那套，缩放到 N=4。看着 ℒ 随对角线与它所在行、列分离而下降。

const N = 4;
// S is an N×N similarity matrix; diagonal = correct (image i, caption i)
function softmaxRows(S){                       // i→t: normalize across each row
  return S.map(row => {
    const mx = Math.max(...row);
    const ex = row.map(s => Math.exp(s - mx));
    const Z  = ex.reduce((a,b)=>a+b, 0);
    return ex.map(e => e/Z);
  });
}
function lossDir(P){                            // mean of −log p_ii over the diagonal
  let s = 0; for (let i=0;i<N;i++) s += -Math.log(P[i][i]);
  return s / N;
}
const Prow = softmaxRows(S);                    // image→text
const Pcol = softmaxRows(transpose(S));         // text→image (softmax down columns)
const Lit  = lossDir(Prow);
const Lti  = lossDir(Pcol);
const L    = 0.5 * (Lit + Lti);                 // symmetric InfoNCE loss

4 · The payoff: zero-shot classification, captioning, VQA

4 · 回报：零样本分类、图像描述、VQA

The reason CLIP mattered: it makes the label space itself a free text variable. To classify an image into classes "cat / dog / car" with no training on those classes, you do not need a trained classifier head. You embed each class name (wrapped in a prompt, e.g. "a photo of a {cat}") with the text encoder, embed the image with the image encoder, and pick the class whose text embedding has the highest cosine similarity to the image — argmax over the same similarity the loss optimized. That is zero-shot classification: the model has never seen those labels at train time, yet classifies into them because they are just text.

CLIP 之所以重要：它让标签空间本身成为一个自由的文本变量。要把一张图像分到"cat / dog / car"这些类，而不需要在这些类上做任何训练，你并不需要一个训练好的分类头。你用文本编码器嵌入每个类名（套进一个提示词里，例如 "a photo of a {cat}"），用图像编码器嵌入图像，然后挑出其文本嵌入与图像有最高余弦相似度的那个类——在损失优化的同一个相似度上取 argmax。这就是零样本分类：模型在训练时从未见过这些标签，却能把图像分进去，因为它们不过是文本。

CLIP is discriminative — it scores image–text matches but cannot write. The generative cousins train a decoder to produce text conditioned on the image — the conditioning mechanism is exactly §5's projector / cross-attention, built in detail below: captioning (describe the image) and VQA (Visual Question Answering — answer a natural-language question about the image). These are the bridge to the LLM-based models in the next section, and they expose the field's defining risk: a fluent language model can state things the image does not support.

CLIP 是判别式的——它给图像–文本匹配打分，但不会写东西。它的生成式表亲会训练一个解码器，以图像为条件产生文本——这个条件化机制正是 §5 的投影器 / 交叉注意力，下面会详细搭建：图像描述（描述这张图像）和 VQA（Visual Question Answering，视觉问答——回答一个关于图像的自然语言问题）。它们是通向下一节基于 LLM 的模型的桥梁，也暴露了这个领域的标志性风险：一个流畅的语言模型可能说出图像并不支持的东西。

5 · The modern paradigm: vision encoder → projector → LLM

5 · 现代范式：视觉编码器 → 投影器 → LLM

This is the architecture that defines vision-language today, and the single most important thing to know cold. The insight: we already have powerful frozen components — a vision encoder (a CLIP/DINOv2 ViT from lessons 13–14) that turns an image into patch-token features, and a large language model that reasons and generates text. The only thing missing is a translator that maps image features into the LLM's token-embedding space, so the image looks to the LLM like a handful of extra "words." That translator is the projector (or Q-Former) — and because it is the lesson's most important concept, it is worth grounding in concrete shapes.

这是今天定义视觉-语言的架构，也是必须烂熟于心的头号要点。洞见在于：我们已经拥有强大的冻结组件——一个视觉编码器（第 13–14 课里的 CLIP/DINOv2 ViT），它把图像变成图块 token 特征；以及一个能推理并生成文本的大语言模型。唯一缺的是一个翻译器，把图像特征映射进 LLM 的 token 嵌入空间，好让图像在 LLM 看来像是几个额外的"词"。这个翻译器就是投影器（或 Q-Former）——因为它是本课最重要的概念，值得用具体的形状来落地。

The shapes. Run a ViT-L on a 224×224 image at patch 16 and you get a grid of 14 × 14 = 196 patch tokens (lesson 13's exact count); higher-resolution encoders push this to 256 or more. Say each token is a vector of dimension d_v=1024, and the target LLM works in dimension d_llm=4096. The projector's whole job is to bridge 1024 → 4096 and decide how many visual tokens the LLM must read:

形状。在一张 224×224 的图像上以图块大小 16 跑一个 ViT-L，你会得到一个 14 × 14 = 196 个图块 token 的网格（第 13 课里的确切数字）；分辨率更高的编码器会把它推到 256 或更多。设每个 token 是维度 d_v=1024 的向量，而目标 LLM 工作在维度 d_llm=4096。投影器的全部工作就是打通 1024 → 4096，并决定 LLM 必须读多少个视觉 token：

• MLP projection (LLaVA). A small trained MLP applied to each patch token independently — a 1024 → 4096 map (with a hidden layer). It changes the dimension but not the count: 256 patches in → 256 visual tokens out. Surprisingly effective, but those 256 tokens sit in the LLM's context for every generated word.MLP 投影（LLaVA）。一个小的、训练好的 MLP，独立地作用于每一个图块 token——一个 1024 → 4096 的映射（带一个隐藏层）。它改变维度但不改变数量：输入 256 个图块 → 输出 256 个视觉 token。出奇地有效，但那 256 个 token 会在 LLM 生成的每一个词时都占着上下文。
• Q-Former / resampler (BLIP-2, Flamingo). Introduce a fixed set of ~32 learned query tokens that cross-attend to the 256 patch features and OUTPUT exactly 32 tokens. The patches are compressed 256 → 32 — an 8× shorter visual prefix — regardless of how many patches came in.Q-Former / 重采样器（BLIP-2、Flamingo）。引入一组固定的 约 32 个可学习的查询 token，它们交叉注意到那 256 个图块特征上，并恰好输出 32 个 token。图块被压缩 256 → 32——一个短 8 倍的视觉前缀——无论进来多少个图块都一样。
• LLM receives [visual tokens] + [text tokens] as one sequence and generates the answer autoregressively — the image tokens are just a prefix it attends to.LLM 接收 [视觉 token] + [文本 token] 作为一个序列，并自回归地生成答案——图像 token 只不过是它去注意的一个前缀。

Why "cross-attend" is just lesson 13's attention. The Q-Former is not a new mechanism — it is the exact softmax(QKᵀ/√d)V from lesson 13, with the roles assigned across modalities: the learned query tokens are the queries Q, and the patch features are both the keys K and the values V. Each of the 32 query tokens asks "which patches are relevant to me?", softmaxes over the 196–256 patch keys, and reads out a weighted sum of patch values — producing 32 summary vectors. That is why the output count equals the query count, not the patch count: attention outputs one vector per query.

为什么"交叉注意"不过是第 13 课的注意力。Q-Former 不是什么新机制——它就是第 13 课那个一模一样的 softmax(QKᵀ/√d)V，只是把角色跨模态地分配了：可学习的查询 token 就是查询 Q，而图块特征既是键 K 也是值 V。32 个查询 token 中的每一个都在问"哪些图块与我相关？"，在那 196–256 个图块键上做 softmax，再读出图块值的一个加权和——产生 32 个摘要向量。这就是为什么输出数量等于查询数量、而不是图块数量：注意力对每个查询输出一个向量。

Why the 256→32 saving matters. Recall lesson 13: the LLM's self-attention is O(n²) in sequence length, and every visual token occupies context for every generated text token. Feeding 256 visual tokens versus 32 is an 8× longer visual prefix — roughly 64× more attention work attributable to the image in the quadratic term, plus 8× the KV-cache memory. The resampler buys a fixed, small visual budget; the MLP keeps every patch (more spatial detail, better for OCR and fine grounding) at a higher context cost. That tension — detail vs. token budget — is the central design knob of modern VLMs.

为什么 256→32 这个节省很重要。回想第 13 课：LLM 的自注意力对序列长度是 O(n²) 的，而每个视觉 token 在每一个生成的文本 token 处都占着上下文。喂 256 个视觉 token 相比 32 个，是长 8 倍的视觉前缀——在二次项里归因于图像的注意力计算量大约多 64 倍，外加 8 倍的 KV 缓存内存。重采样器换来一个固定的、很小的视觉预算；MLP 则保留每一个图块（更多空间细节，更利于 OCR 和精细的视觉定位），代价是更高的上下文开销。这种张力——细节 vs. token 预算——正是现代 VLM 的核心设计旋钮。

Visual instruction tuning is how these models learn to follow instructions about images: after the projector is trained to align modalities, the whole stack is fine-tuned on (image, instruction, response) triples — often generated by prompting a text-only LLM with image captions/boxes to produce conversational Q&A. This is the "LLaVA recipe," and Qwen-VL / InternVL extend it with higher-resolution encoders, more visual tokens, and stronger LLMs. Two capabilities matter for interviews:

视觉指令微调是这些模型学会遵循关于图像的指令的方式：在投影器被训练好以对齐模态之后，整个栈会在（图像，指令，回答）三元组上做微调——这些三元组常常是用图像的标题/框去提示一个纯文本 LLM，让它生成对话式问答而来的。这就是"LLaVA 配方"，而 Qwen-VL / InternVL 用更高分辨率的编码器、更多视觉 token 和更强的 LLM 对其做了扩展。有两项能力对面试很重要：

• Grounding — tying a phrase in the output to a specific region (a box or mask), so the model can say "the red cup" and point to it. This is the antidote to hallucination: an answer with a region reference is checkable.视觉定位（grounding）——把输出中的一个短语系到一个具体的区域（一个框或掩码）上，让模型能说"那个红色的杯子"并指出它。这是幻觉的解药：一个带区域引用的答案是可核查的。
• OCR / document understanding — reading text and layout in the image (receipts, forms, charts). The bottleneck is resolution and reading order; tiny text demands high-res encoders or tiling, which collides with the token-count cost from lesson 13.OCR / 文档理解——读取图像中的文字和版式（收据、表单、图表）。瓶颈在于分辨率和阅读顺序；细小的文字需要高分辨率编码器或图像切片，而这与第 13 课的 token 数量开销相冲突。

Where this points next

这将指向何处

Everything in the track so far — classification, detection, segmentation, embeddings, ViTs, SSL, and these vision-language models — operates on a single still image. But the real world moves: a collar's color is unambiguous across frames, an action is defined by change over time, and a caption for a video must respect what happened, not just what is present. Lesson 16 adds the time axis: frame sampling as a modeling choice, optical flow (the brightness-constancy equation), 3D CNNs / two-stream / temporal attention, and the latency budgets that make streaming video its own systems problem.

到目前为止这门课里的一切——分类、检测、分割、嵌入、ViT、自监督学习，以及这些视觉-语言模型——操作的都是单张静止图像。但真实世界是运动的：一个项圈的颜色跨帧看是明确无误的，一个动作是由随时间的变化定义的，而一段视频的描述必须尊重发生过什么，而不只是当前呈现着什么。第 16 课加入时间轴：把帧采样作为一个建模选择、光流（亮度恒定方程）、3D CNN / 双流 / 时序注意力，以及让流式视频成为一个自成一体的系统问题的延迟预算。

Interview prompts

面试题

• Write CLIP's loss: what is the similarity matrix, why symmetric, what is the temperature for?写出 CLIP 的损失：相似度矩阵是什么，为什么是对称的，温度是干什么用的？
• How does CLIP enable zero-shot classification, and why does prompt wording matter?CLIP 是如何实现零样本分类的，为什么提示词的措辞很重要？
• Why does CLIP need such large batches? (Tie it to in-batch negatives from lesson 14.)为什么 CLIP 需要如此大的批量？（把它和第 14 课的批内负样本联系起来。）
• Describe the VLM→LLM stack: what does the projector / Q-Former do, and why?描述 VLM→LLM 这条栈：投影器 / Q-Former 做什么，为什么？
• MLP projector vs Q-Former: how do the visual-token counts differ (256→256 vs 256→32), and what's the context-cost trade-off? Why is the resampler's output count the number of query tokens?MLP 投影器 vs Q-Former：视觉 token 数量有何不同（256→256 vs 256→32），上下文开销的权衡是什么？为什么重采样器的输出数量就是查询 token 的数量？
• Given a 3×3 similarity matrix, compute one row's −log p_ii and the symmetric loss — and say what makes ℒ small.给定一个 3×3 的相似度矩阵，计算某一行的 −log p_ii 和对称损失——并说出是什么让 ℒ 变小。
• What is visual instruction tuning, and how is its data often created?什么是视觉指令微调，它的数据通常是怎么造出来的？
• Why can a VLM hallucinate visual details, and how does grounding help?为什么 VLM 会对视觉细节产生幻觉，视觉定位如何帮上忙？
• How would you evaluate document/OCR understanding — and what's the resolution tension?你会如何评估文档/OCR 理解——分辨率上的张力是什么？