Vision transformers and hybrid architectures

视觉 Transformer 与混合架构

Lesson 12 learned an embedding space by comparing examples — but the backbone producing those embeddings was still a CNN, and a CNN bakes in locality: each neuron sees only a small window, and long-range structure has to be assembled slowly through depth. This lesson swaps that inductive bias for a more flexible one. We build the transformer from first principles — queries, keys, values, attention — and apply it to images by cutting them into patches. The payoff is global context from layer one; the price is a quadratic cost and a large appetite for data.

第 12 课通过比较样本学到了一个嵌入空间——但产生这些嵌入的骨干网络仍然是 CNN，而 CNN 内建了局部性：每个神经元只看到一个小窗口，长程结构必须靠深度慢慢地拼接起来。本课把这种归纳偏置换成一种更灵活的。我们将从第一性原理构建 Transformer——查询、键、值、注意力——并通过把图像切成图块（patch）来把它用到图像上。回报是从第一层起就有全局上下文；代价是二次方的计算开销和对数据的巨大胃口。

What we are standing on

我们站在什么之上

Lesson 06 argued that a convolution's power is its inductive bias: it assumes useful features are local and translation-equivariant, and that assumption lets a CNN learn from modest data. But a bias is a constraint, and a constraint you cannot remove is a ceiling. If two objects on opposite corners of an image need to interact — a question word and the object it refers to, a face and the hand near it — a CNN can only relate them after enough conv-and-pool layers have grown the receptive field to cover both. The transformer asks: what if every location could look at every other location directly, in a single layer, with the strength of each look learned per input rather than fixed by a kernel?

第 06 课论证过，卷积的威力在于它的归纳偏置：它假设有用的特征是局部的、平移等变的，而这个假设让 CNN 能从不多的数据中学习。但偏置是一种约束，而一个你无法去除的约束就是一道天花板。如果图像中位于相对两角的两个物体需要相互作用——一个疑问词和它所指的物体、一张脸和它旁边的手——CNN 只有在足够多的卷积和池化层把感受野扩大到覆盖二者之后，才能把它们关联起来。Transformer 追问的是：如果每个位置都能直接看到其他每个位置，在单独一层里，而且每一"看"的强度是随输入学习出来的、而非由卷积核固定，会怎么样？

The plan计划

Five moves. (1) Cut the image into patch tokens and count them. (2) Derive self-attention from scratch — queries, keys, values, the score q·k/√d, softmax, weighted sum — and explain why the √d is there. (3) Show why attention costs O(n²) in token count, with a worked number. (4) Fix the two things raw patches lack: position (positional encodings) and a place to read out a global answer (CLS token vs mean-pool). (5) Survey the architectures that put locality back in (Swin, ConvNeXt) and the efficiency fixes (windowed attention, FlashAttention) that make high resolution affordable. 五步走。(1) 把图像切成图块 token 并数一数它们。(2) 从零推导自注意力——查询、键、值、分数 q·k/√d、softmax、加权求和——并解释为什么要有 √d。(3) 用一个具体数字说明为什么注意力在 token 数上的开销是 O(n²)。(4) 补上原始图块缺的两样东西：位置（位置编码）以及读出全局答案的地方（类别 token（CLS）与均值池化）。(5) 综述把局部性重新放回来的架构（Swin、ConvNeXt），以及让高分辨率变得可负担的效率改进（窗口注意力、FlashAttention）。

1 · From image to tokens: patchification

1 · 从图像到 token：图块化

A transformer operates on a sequence of vectors — tokens. Language gives you tokens for free (words/subwords); an image does not. The Vision Transformer (ViT) manufactures them by cutting the image into a grid of square patches and flattening each patch into a vector. Take a 224×224 RGB image and a patch size of 16×16. The grid is

Transformer 作用在一个向量序列——token——之上。语言天然给你 token（词/子词）；图像却没有。视觉 Transformer（ViT）通过把图像切成由方形图块（patch）组成的网格、并把每个图块展平成一个向量，来制造出这些 token。取一张 224×224 的 RGB 图像，图块大小为 16×16。网格是

(224 / 16) × (224 / 16) = 14 × 14 = 196 patches.

Each patch is 16×16×3 = 768 raw numbers. A single learned linear layer (a matrix E of shape 768×d) projects each flattened patch to a token embedding of dimension d (say d=768). Watch the two 768s — they are a coincidence, not the same quantity. The first 768 is the patch dimension (16·16·3, fixed by patch size and channel count); the second is the chosen embedding width d, a free hyperparameter the architect picks. With ViT-Base's d=768 they happen to be equal, so E is a 768×768 matrix — but change the patch size or pick d=1024 and they diverge (E becomes 768-rows × 1024-cols). E maps "the 768 numbers in a patch" to "a d-dimensional token"; the input and output dimensions are independent. So the image becomes a sequence of n=196 tokens, each a vector in ℝ⁷⁶⁸. That is the entire input contract: a 196 × 768 matrix.

每个图块是 16×16×3 = 768 个原始数字。一个单独的、学习出来的线性层（一个形状为 768×d 的矩阵 E）把每个展平后的图块投影成一个维度为 d 的token 嵌入（比如 d=768）。盯住这两个 768——它们只是巧合，并不是同一个量。第一个 768 是图块维度（16·16·3，由图块大小和通道数固定）；第二个是选定的嵌入宽度 d，一个由设计者自选的自由超参数。在 ViT-Base 的 d=768 下它们恰好相等，于是 E 是一个 768×768 的矩阵——但改一下图块大小或选 d=1024，它们就分道扬镳了（E 变成 768 行 × 1024 列）。E 把"一个图块里的 768 个数字"映射成"一个 d 维的 token"；输入维度和输出维度是相互独立的。于是图像变成一个 n=196 个 token 的序列，每个 token 是 ℝ⁷⁶⁸ 中的一个向量。这就是全部的输入契约：一个 196 × 768 的矩阵。

Correctness: patches are not definitionally non-overlapping正确性：图块并非按定义就不重叠

Vanilla ViT uses non-overlapping patches, and the linear projection above is mathematically identical to a convolution with kernel 16×16 and stride 16. But that is a design choice, not a law. Many strong variants use a convolutional stem (a few small conv layers before tokenizing) or overlapping patches (kernel larger than stride); both improve optimization stability and low-level feature quality. So: vanilla ViT is non-overlapping, but "ViT ⟹ non-overlapping patches" is false. 原始 ViT 使用不重叠的图块，而上面那个线性投影在数学上等同于一个卷积核为 16×16、步幅为 16 的卷积。但这是一个设计选择，而非定律。许多强力变体使用卷积 stem（在 token 化之前先接几层小卷积）或重叠图块（卷积核大于步幅）；二者都能改善优化稳定性和底层特征质量。所以：原始 ViT 是不重叠的，但"ViT ⟹ 图块不重叠"这个说法是错的。

2 · Self-attention from first principles

2 · 从第一性原理推导自注意力

This is the heart of the lesson, and we will not assume you have seen it. We have n tokens, each a vector x_i ∈ ℝ^d. We want each token to produce an updated vector that is a mixture of information from the tokens it finds relevant — and "relevant" must be learned and content-dependent, not hard-wired by position.

这是本课的核心，我们不会假设你已经见过它。我们有 n 个 token，每个是一个向量 x_i ∈ ℝ^d。我们希望每个 token 产生一个更新后的向量，它是来自它认为相关的那些 token 的信息的混合——而"相关"必须是学习出来、依赖内容的，而不是由位置写死的。

The mechanism gives every token three roles, each a learned linear projection of x_i:

这个机制给每个 token 三种角色，每一种都是 x_i 的一个学习出来的线性投影：

Query q_i = W_Q x_i — "what am I looking for?"查询（Query） q_i = W_Q x_i——"我在找什么？"
Key k_j = W_K x_j — "what do I offer, as something to be matched against?"键（Key） k_j = W_K x_j——"我提供什么，用来被匹配？"
Value v_j = W_V x_j — "what information do I actually carry, to be passed along?"值（Value） v_j = W_V x_j——"我实际携带什么信息，用来传递出去？"

The relevance of token j to token i is the dot product of the query with the key — a similarity score, exactly the inner product we used in lesson 12 to measure embedding closeness:

token j 对 token i 的相关性，是查询与键的点积——一个相似度分数，正是我们在第 12 课用来度量嵌入接近程度的那个内积：

score_ij = q_i · k_j / √d .

We turn the row of scores (score_i1, …, score_in) into a probability distribution over the tokens with the softmax (derived back in lesson 08) — these are the attention weights α_ij, non-negative and summing to 1:

我们用 softmax（在第 08 课推导过）把这一行分数 (score_i1, …, score_in) 变成 token 上的一个概率分布——这些就是注意力权重 α_ij，非负且求和为 1：

α_ij = exp(score_ij) / ∑_j' exp(score_ij') .

Finally the output for token i is the weighted sum of values — it reads a little from every token, mostly from the ones it scored highest:

最后，token i 的输出是值的加权求和——它从每个 token 都读取一点，主要从它打分最高的那些读取：

z_i = ∑_j α_ij · v_j .

Stacked over all tokens, with Q,K,V the matrices whose rows are the queries/keys/values, this is the equation you will see everywhere:

把所有 token 堆叠起来，令 Q,K,V 为以查询/键/值为行的矩阵，这就是你到处都会见到的那个公式：

Attention(Q,K,V) = softmax( Q Kᵀ / √d ) V .

Read it left to right: QKᵀ is the n×n table of all pairwise scores; √d rescales; softmax (applied row-wise) normalizes each query's row into weights; multiplying by V mixes the values. Because the weights are computed from the content of Q and K, a token can attend strongly to a far-away token whenever their content matches — no receptive-field growth required.

从左到右读它：QKᵀ 是所有成对分数构成的 n×n 表；√d 做缩放；softmax（按行施加）把每个查询那一行归一化成权重；乘以 V 就把值混合起来。因为权重是从 Q 和 K 的内容算出来的，只要内容匹配，一个 token 就能强烈地关注到一个远处的 token——无需扩大感受野。

Multi-head attention. In practice we don't run one attention over the full d — we split it across several heads running in parallel. With model dim d = 768 and h = 12 heads, each head gets its own learned W_Q, W_K, W_V that project tokens into a 64-dim subspace (768 / 12 = 64), and runs its own softmax(QKᵀ/√64)V there. The h = 12 head outputs (each 64-dim) are then concatenated back to 768 and passed through one more linear layer. The intuition: each head computes a separate attention map, so different heads can specialize — one might track nearby patches (texture/edges), another long-range structure or color — like several attention patterns computed at once instead of forcing a single map to do every job. And it is nearly free: splitting d into h heads keeps the total attention compute at ≈ n²·d — each of the h heads does n²·(d/h) work, and h · n²·(d/h) = n²·d — so multi-head buys specialization at no extra asymptotic cost over a single head of width d.

多头注意力。实践中我们不是在完整的 d 上跑一个注意力——而是把它拆到并行运行的若干个头（head）上。在模型维度 d = 768、h = 12 个头的情况下，每个头有自己学习出来的 W_Q, W_K, W_V，把 token 投影到一个 64 维子空间（768 / 12 = 64），并在那里跑自己的 softmax(QKᵀ/√64)V。h = 12 个头的输出（各 64 维）随后被拼接回 768，再经过一个额外的线性层。直觉是：每个头计算一张独立的注意力图，因此不同的头可以各有分工——一个可能追踪邻近图块（纹理/边缘），另一个追踪长程结构或颜色——就像一次算出好几种注意力模式，而不是逼一张图去干所有活。而且这几乎是免费的：把 d 拆成 h 个头，总的注意力计算量仍保持在 ≈ n²·d——h 个头中的每一个做 n²·(d/h) 的工作，而 h · n²·(d/h) = n²·d——所以相比一个宽度为 d 的单头，多头以不增加渐近开销的代价换来了分工。

Why the √d — and it is not cosmetic为什么要有 √d——而且它绝非装饰

Suppose q and k have d components, each roughly independent with unit variance. Their dot product q·k = ∑_m=1^d q_mk_m is a sum of d such terms, so its variance grows like d and its standard deviation like √d. With d=64 that gives √64 = 8, so the scores have standard deviation ≈ 8 and routinely land within ±8 (one to two standard deviations) — large enough to saturate the softmax, which is exactly why we divide by √d. The raw scores swing on the order of ±8 before softmax. Softmax of large-magnitude inputs is nearly a one-hot spike: it puts essentially all weight on the single largest score and its gradient with respect to the others goes to zero — the layer stops learning. Dividing by √d rescales the scores back to unit variance, keeping softmax in its responsive regime. The fix is exactly the standard deviation we just computed. 假设 q 和 k 有 d 个分量，每个大致独立且方差为 1。它们的点积 q·k = ∑_m=1^d q_mk_m 是 d 个这样的项之和，因此它的方差随 d 增长，标准差随 √d 增长。在 d=64 时得到 √64 = 8，所以分数的标准差 ≈ 8，通常落在 ±8 之内（一到两个标准差）——大到足以让 softmax 饱和，这正是我们要除以 √d 的原因。原始分数在进 softmax 之前会在 ±8 量级上波动。大幅值输入的 softmax 几乎就是一个 one-hot 尖峰：它把几乎全部权重都放在那个最大的分数上，而它对其余分数的梯度趋于零——这一层就停止学习了。除以 √d 把分数重新缩放回单位方差，让 softmax 保持在它响应灵敏的区间。这个修正恰好就是我们刚算出的那个标准差。

3 · Why attention is O(n²) — the cost we just bought

3 · 为什么注意力是 O(n²)——我们刚买下的开销

Look again at QKᵀ. It is an n×n matrix: every one of the n queries is dotted against every one of the n keys. That is n² score computations, n² softmax entries to store, and another n² multiply-accumulates to mix the values. Compute and memory both scale as O(n²·d) in the token count. This is the structural price of "every token looks at every token" — the very thing that bought us global context.

再看一眼 QKᵀ。它是一个 n×n 的矩阵：n 个查询中的每一个都要与 n 个键中的每一个做点积。这就是 n² 次分数计算、n² 个 softmax 项需要存储，以及另外 n² 次乘加来混合值。计算和内存都随 token 数按 O(n²·d) 增长。这是"每个 token 都看每个 token"的结构性代价——而这恰恰正是为我们换来全局上下文的那件事。

Concretely: our 224×224 image at patch 16 gave n=196, so the attention matrix is 196×196 ≈ 38k entries per head — trivial. Now keep the patch size but double the resolution to 448×448. The token count quadruples to n = 28×28 = 784, and the attention matrix is 784×784 ≈ 615k entries — 16× larger, because cost goes as n² and n itself scales with the number of pixels. Halving the patch size to 8×8 has the same effect: 4× the tokens, 16× the attention cost. High-resolution dense prediction (segmentation, detection) lives exactly where this hurts, which is why sections 5 and the efficiency note matter.

具体地说：我们那张 224×224、图块为 16 的图像给出 n=196，所以每个头的注意力矩阵是 196×196 ≈ 3.8 万项——微不足道。现在保持图块大小，但把分辨率翻倍到 448×448。token 数翻两番到 n = 28×28 = 784，注意力矩阵是 784×784 ≈ 61.5 万项——大了 16 倍，因为开销随 n² 增长，而 n 本身又随像素数增长。把图块大小减半到 8×8 也是同样的效果：token 变 4 倍，注意力开销变 16 倍。高分辨率稠密预测（分割、检测）恰恰活在这个疼点上，这正是第 5 节和那条效率提示重要的原因。

4 · Two things raw patches lack: position and a readout

4 · 原始图块缺的两样东西：位置与读出

Position. Self-attention is, by construction, permutation-equivariant: shuffle the input tokens and the outputs shuffle identically, because the equation never references where a token sits — only its content. For images that is a disaster; the patch at the top-left and the patch at the center are interchangeable to the raw mechanism. So ViT adds a positional encoding p_i to each token embedding before the first layer: x_i ← x_i + p_i. Three common choices:

位置。自注意力按其构造是置换等变的：把输入 token 打乱，输出也会同样地打乱，因为公式从不引用一个 token位于何处——只看它的内容。对图像来说这是一场灾难；左上角的图块和中心的图块，对这个原始机制而言是可以互换的。于是 ViT 在第一层之前给每个 token 嵌入加上一个位置编码 p_i：x_i ← x_i + p_i。三种常见选择：

Learned — a free parameter vector per position, trained end to end. Simple and standard in ViT; the cost is that it does not extrapolate to resolutions unseen in training without interpolation.可学习的（Learned）——每个位置一个自由参数向量，端到端训练。简单，且是 ViT 中的标准做法；代价是不做插值就无法外推到训练中未见过的分辨率。
Sinusoidal — fixed sines/cosines of varying frequency (from the original Transformer). No parameters and, because the encoding is a fixed closed-form function of the position index, a position never seen in training (a longer sequence, a higher resolution) still maps to a defined, smooth vector — whereas a learned encoding simply has no parameter for an unseen index and must be interpolated. Being defined everywhere is not the same as generalizing well, though: in practice plain sinusoidal encodings still extrapolate poorly to much longer sequences, which is exactly what motivated relative and rotary schemes (RoPE, ALiBi) that bake position into attention itself.正弦式（Sinusoidal）——不同频率的固定正弦/余弦（来自最初的 Transformer）。没有参数，而且由于该编码是位置索引的一个固定闭式函数，一个训练中从未见过的位置（更长的序列、更高的分辨率）仍然映射到一个有定义的、平滑的向量——而可学习编码对未见过的索引根本没有参数，必须做插值。不过"处处有定义"并不等于"泛化得好"：实践中朴素的正弦编码对长得多的序列外推仍然很差，这恰恰催生了把位置直接融入注意力本身的相对式和旋转式方案（RoPE、ALiBi）。
2D / relative — encode row and column separately, or encode the relative offset between two tokens directly in the attention score. This respects image geometry (a patch two cells to the left is the same relation everywhere) and is what window-attention models use.二维 / 相对式（2D / relative）——分别编码行和列，或把两个 token 之间的相对偏移直接编码进注意力分数里。这尊重了图像的几何（"往左两格的图块"这个关系在任何地方都相同），也是窗口注意力模型所采用的做法。

A readout. Classification needs one vector for the whole image, but attention produces one per patch. Two options. The CLS token (borrowed from BERT) prepends an extra learnable token to the sequence; it owns no patch but attends to all of them, and its final-layer output is fed to the classifier — a learned, attention-weighted summary. Alternatively, mean-pooling the patch tokens (averaging all 196 output vectors) is parameter-free and often matches or beats CLS for classification. CLS is the historical default; mean-pool is a perfectly respectable, sometimes better, choice — the interview-correct answer is "either works; CLS is a learned pool, mean-pool is a fixed one."

一个读出。分类需要整幅图像的一个向量，但注意力对每个图块产生一个。两个选项。类别 token（CLS token，借自 BERT）在序列前面加上一个额外的可学习 token；它不拥有任何图块，却关注所有图块，它最后一层的输出被送入分类器——一个学习出来的、注意力加权的摘要。或者，对图块 token 做均值池化（对全部 196 个输出向量求平均）是无参数的，在分类上常常能追平甚至超过 CLS。CLS 是历史上的默认做法；均值池化是一个完全体面、有时更好的选择——面试中正确的答案是"两者都行；CLS 是学习出来的池化，均值池化是固定的池化"。

5 · Putting locality back: Swin, ConvNeXt, and the data regime

5 · 把局部性放回来：Swin、ConvNeXt 与数据规模

Pure global attention discards the locality prior — and a prior you discard, you must re-learn from data. That is the central trade. On ImageNet-scale data alone (≈1.3M images), a from-scratch ViT typically loses to a comparable CNN, because it has to learn locality the CNN was given for free. Pretrain the same ViT on a much larger corpus (hundreds of millions of images, or the self-supervised signal of the next lesson) and it overtakes the CNN — it had the capacity all along; it just needed the data to fill it. The honest one-liner: ViTs are data-hungry; CNNs are data-efficient.

纯粹的全局注意力丢弃了局部性先验——而一个你丢弃的先验，你必须从数据中重新学回来。这就是核心的取舍。仅在 ImageNet 规模的数据上（约 130 万张图像），一个从零训练的 ViT 通常输给可比的 CNN，因为它必须学会 CNN 免费获得的局部性。把同一个 ViT 在大得多的语料上预训练（数以亿计的图像，或下一课的自监督信号），它就会反超 CNN——它一直都有那个容量；只是需要数据来把它填满。诚实的一句话总结：ViT 很吃数据；CNN 很省数据。

Two architectures split the difference:

两种架构在二者之间取了折中：

Swin Transformer reintroduces locality with window attention: attention is computed only within local windows (say 7×7 patches), making cost linear in token count instead of quadratic, and the windows are shifted between layers so information still crosses window boundaries. It also builds a hierarchy (merging patches to halve resolution as depth grows), recovering the feature-pyramid structure detection and segmentation rely on (lesson 09/10).Swin Transformer 用窗口注意力重新引入局部性：注意力只在局部窗口内（比如 7×7 个图块）计算，使开销在 token 数上变为线性而非二次方，而且窗口在层与层之间会移位（shift），这样信息仍能跨越窗口边界。它还构建了一个层级结构（随深度增长合并图块以把分辨率减半），恢复了检测与分割所依赖的特征金字塔结构（第 09/10 课）。
ConvNeXt runs the experiment in reverse: keep the CNN, but modernize it with transformer-era choices (larger kernels, fewer activations/norms, depthwise convs, LayerNorm). It matches ViT accuracy while staying a pure convnet — evidence that much of ViT's win was the training recipe and scale, not attention per se.ConvNeXt 把这个实验反过来做：保留 CNN，但用 Transformer 时代的选择把它现代化（更大的卷积核、更少的激活/归一化、深度可分卷积、LayerNorm）。它在保持纯卷积网络的同时追平了 ViT 的精度——这说明 ViT 的胜利很大程度上来自训练配方和规模，而非注意力本身。

Axis维度	CNN (e.g. ResNet/ConvNeXt)CNN（如 ResNet/ConvNeXt）	ViT (global)ViT（全局）	Swin (windowed)Swin（窗口）
Inductive bias归纳偏置	Locality + translation equivariance built in内建局部性 + 平移等变性	Almost none — learned from data几乎没有——从数据中学习	Locality via windows; hierarchy通过窗口获得局部性；层级结构
Data appetite数据胃口	Efficient on small/medium data在小/中等数据上高效	Hungry — shines with large pretraining很吃数据——大规模预训练时大放异彩	Between the two介于两者之间
Context range上下文范围	Grows slowly with depth/pyramids随深度/金字塔缓慢增长	Global from layer 1从第 1 层起就是全局	Local per layer; global via shifts + depth每层局部；通过移位 + 深度达成全局
Attention cost注意力开销	n/a	O(n²) in tokenstoken 数上的 O(n²)	O(n) in tokenstoken 数上的 O(n)
Best fit最适合	Limited data, low latency, edge数据有限、低延迟、边缘端	Big pretrain, global reasoning大规模预训练、全局推理	Dense prediction at high resolution高分辨率的稠密预测

Efficient attention & FlashAttention — the high-res fix高效注意力与 FlashAttention——高分辨率的解药

Section 3's O(n²) is the bottleneck at high resolution. Two distinct attacks: (1) change the algorithm — windowed/local attention (Swin), or linear/sparse approximations, reduce the number of pairs you score. (2) change the implementation without changing the math — FlashAttention computes exact softmax attention but never materializes the full n×n matrix in slow GPU memory; it tiles the computation and fuses the softmax so the dominant cost (memory I/O, not arithmetic) drops sharply. The result is identical numbers, far less memory and wall-clock — which is why it is now the default kernel for any sizeable transformer, vision or language. 第 3 节的 O(n²) 是高分辨率下的瓶颈。两种不同的攻法：(1) 改算法——窗口/局部注意力（Swin），或线性/稀疏近似，减少你要打分的成对数目。(2) 在不改数学的前提下改实现——FlashAttention 计算的是精确的 softmax 注意力，但从不在缓慢的 GPU 显存里物化完整的 n×n 矩阵；它把计算分块（tiling）并把 softmax 融合（fuse）起来，于是主要开销（显存 I/O，而非算术运算）大幅下降。结果是数值完全相同，而显存和墙钟时间都大为减少——这正是它如今成为任何规模较大的 Transformer（无论视觉还是语言）默认算子的原因。

Interactive · patch self-attention

交互 · 图块自注意力

A 5×5 grid of toy image patches. Each patch gets a fixed 3-dimensional embedding built from its (row, col) position and its brightness, and from that embedding we form a query and a key (here, identically — a self-similarity attention, which is enough to see the mechanism). Click any patch to make it the query. The widget computes score_j = q · k_j / √d for every patch, runs the softmax, and heatmaps the attention weights — bright cells are where the query "looks." Notice it attends to patches that are similar in position and brightness, exactly as the dot product dictates, and that the weights sum to 1. Toggle the √d scaling off to watch the distribution sharpen. Here d=3, so the shift is gentle; at the d=64 of a real model the unscaled scores swing on the order of ±8 and softmax collapses toward a single spike that stalls learning — the failure mode the scaling prevents.

一个 5×5 的玩具图像图块网格。每个图块得到一个固定的三维嵌入，由它的（行、列）位置和亮度构成，我们从这个嵌入里形成一个查询和一个键（这里两者相同——一种自相似注意力，足以看清机制）。点击任意图块让它成为查询。小部件为每个图块计算 score_j = q · k_j / √d，跑 softmax，并把注意力权重画成热力图——明亮的格子就是查询"看"向的地方。注意它会关注那些在位置和亮度上相似的图块，恰如点积所决定的那样，而且权重求和为 1。把 √d 缩放关掉，看分布变得更尖锐。这里 d=3，所以变化是温和的；在真实模型的 d=64 下，未缩放的分数会在 ±8 量级上波动，softmax 会坍缩成一个单一尖峰而使学习停滞——正是缩放所防止的那种失效模式。

Self-attention — click a query patch, watch the weights

自注意力——点击一个查询图块，观察权重

Left: the patch grid (brightness = pixel value); the clicked query patch is outlined in accent. Right: the same grid heatmapped by attention weight α_j = softmax(q·k_j/√d) — brighter = more attended. Each embedding is [row, col, brightness], so a patch attends to others nearby in position and intensity.

左：图块网格（亮度 = 像素值）；被点击的查询图块用强调色描边。右：同一网格按注意力权重 α_j = softmax(q·k_j/√d) 画成热力图——越亮 = 被关注越多。每个嵌入是 [行, 列, 亮度]，所以一个图块会关注那些在位置和强度上都相近的图块。

Query patch

查询图块

(2,2)

Top-attended

最受关注

—

√d

1.73

Max weight

最大权重

—

Show the core JS (≈18 lines)查看核心代码（约 18 行）

const G = 5, D = 3;                 // 5×5 grid, 3-dim embeddings
// embedding of patch (r,c): [normalized row, normalized col, brightness]
function embed(r,c){ return [r/(G-1), c/(G-1), bright[r][c]]; }

function attention(qr, qc){
  const q = embed(qr, qc);
  const scores = [];
  for (let r=0;r<G;r++) for (let c=0;c<G;c++){
    const k = embed(r,c);
    let dot = 0; for (let m=0;m<D;m++) dot += q[m]*k[m];   // q·k
    scores.push(useScale ? dot/Math.sqrt(D) : dot);        // /√d
  }
  const mx = Math.max(...scores);
  const ex = scores.map(s => Math.exp(s - mx));            // softmax
  const Z  = ex.reduce((a,b)=>a+b, 0);
  return ex.map(e => e/Z);                                 // weights sum to 1
}

The lesson on screen: attention is just a learned, content-dependent average. The query reaches across the whole grid in one step (the global-context win), the weights are a softmax over dot-product scores (the mechanism), and the √d toggle shows why the scaling exists — without it, softmax collapses to a near-one-hot and the layer loses its ability to blend.

屏幕上呈现的这一课：注意力不过是一个学习出来、依赖内容的加权平均。查询一步就够到整个网格（全局上下文的收益），权重是点积分数上的一个 softmax（机制本身），而 √d 开关展示了缩放为何存在——没有它，softmax 会坍缩成近乎 one-hot，这一层就失去了混合的能力。

Where this points next

这将指向何处

We now have an architecture that scales with data and models global structure — but section 5 exposed its weakness bluntly: it is data-hungry, and human-labeled data does not exist at internet scale. We cannot annotate a billion images. So the question that defines the next lesson is: where does the supervision come from when there are no labels? Lesson 14 answers it with self-supervised learning — letting images supervise themselves through pretext tasks, contrastive objectives (recalling lesson 12's InfoNCE), and masked reconstruction. Here is the precise hand-off: every ViT in this lesson was still trained on labeled data (ImageNet classes, the supervised pretraining that lets it beat a CNN). Lesson 14 shows how to pretrain this exact architecture with no labels at all — MAE masks out patches and reconstructs them, DINO matches two augmented views — both running on the very ViT backbone we just built. DINOv2, the current frozen-backbone default, is a self-supervised ViT.

现在我们有了一个随数据扩展、能建模全局结构的架构——但第 5 节直白地暴露了它的弱点：它很吃数据，而人工标注的数据并不存在于互联网规模上。我们没法给十亿张图像做标注。所以定义下一课的问题是：当没有标签时，监督从何而来？第 14 课用自监督学习来回答它——让图像通过前置任务（pretext task）、对比目标（回想第 12 课的 InfoNCE）和掩码重建来监督自己。这里是精确的交接：本课中的每一个 ViT 仍然是在有标签数据上训练的（ImageNet 类别，正是这种有监督预训练让它打败 CNN）。第 14 课将展示如何在完全没有标签的情况下预训练这个一模一样的架构——MAE 遮掉图块并重建它们，DINO 匹配两个增强视图——二者都跑在我们刚刚构建的这个 ViT 骨干网络上。DINOv2，当前冻结骨干网络的默认选择，就是一个自监督 ViT。

Takeaway要点

A ViT turns an image into a sequence of patch tokens (224/16 → 196 tokens) and processes them with self-attention: each token forms a query, every token a key and value, the score is q·k/√d, softmax over scores gives weights, and the output is the weighted sum of values — global context in one layer. The √d keeps softmax responsive (it cancels the √d growth of the dot product). The cost is O(n²) in tokens, which bites at high resolution; raw patches also need positional encodings (they are otherwise permutation-equivariant) and a readout (CLS token or mean-pool). ViTs beat CNNs only with large-scale pretraining; Swin (windowed, hierarchical, O(n)) and ConvNeXt (a modernized CNN) split the difference, and FlashAttention makes exact attention cheap by fixing memory I/O. ViT 把一张图像变成一个图块 token 序列（224/16 → 196 个 token），并用自注意力处理它们：每个 token 形成一个查询，每个 token 也是键和值，分数是 q·k/√d，对分数做 softmax 给出权重，输出是值的加权求和——一层之内就有全局上下文。√d 让 softmax 保持响应灵敏（它抵消了点积按 √d 的增长）。代价是 token 数上的 O(n²)，在高分辨率下会咬人；原始图块还需要位置编码（否则它们是置换等变的）和一个读出（CLS token 或均值池化）。ViT 只有在大规模预训练下才打败 CNN；Swin（窗口式、层级式、O(n)）和 ConvNeXt（一个现代化的 CNN）在二者之间取折中，而 FlashAttention 通过解决显存 I/O 让精确注意力变得廉价。

Interview prompts

面试题

Derive self-attention: what are queries, keys, values, and why is the score q·k/√d?推导自注意力：什么是查询、键、值，以及分数为什么是 q·k/√d？
Why the √d specifically — what breaks without it?为什么偏偏是 √d——没有它会坏在哪里？
Why is attention O(n²), and what does doubling input resolution do to the cost?为什么注意力是 O(n²)，把输入分辨率翻倍会对开销产生什么影响？
Why does a ViT need positional encodings? Compare learned vs sinusoidal vs 2D/relative.ViT 为什么需要位置编码？比较可学习式、正弦式与二维/相对式。
CLS token vs mean-pooling for the classification readout — what's the difference?分类读出用 CLS token 还是均值池化——两者有什么区别？
When would you choose a CNN, a global ViT, or Swin? Frame it by data regime and resolution.你什么时候会选 CNN、全局 ViT 还是 Swin？请从数据规模和分辨率的角度来阐述。
What does FlashAttention change — the math, or the implementation?FlashAttention 改变的是什么——数学，还是实现？