The real-time frame, the GPU & where graphics is going实时帧、GPU 与图形学的未来

We now hold every piece: geometry (03), the transform chain (02), rasterization and its fragments (04–06), the rendering equation and how surfaces answer it (07–09), global illumination correct and faked (10–12), motion (13), and the color pipeline that reaches the display (14). This finale assembles them into one real frame under a millisecond budget, explains the hardware that runs it, and shows the field folding back into the Computer Vision and 3D Vision tracks it began as the mirror of. 现在每一块都在手上了：几何（03）、变换链（02）、光栅化及其片元（04–06）、渲染方程与表面如何回应它（07–09）、正确的与伪造的全局光照（10–12）、运动（13），以及抵达显示器的色彩管线（14）。这最后一课把它们拼成毫秒预算下的真实一帧，讲清运行它的硬件，并展示这门领域如何折返回它一开始作为镜像的《计算机视觉》与《3D 视觉》。

1 · The GPU is a throughput machine, not a fast CPUGPU 是一台吞吐机器，不是一颗快的 CPU

Every loop in this course — "for each pixel," "for each fragment," "for each sample" — runs millions of nearly-identical, independent iterations. That workload shape is why the GPU exists. A CPU minimizes the latency of one thread with big caches and branch prediction; a GPU maximizes throughput across thousands of threads and hides latency by having so many in flight that some are always ready to run. It spends its transistor budget on arithmetic units, not on making any single thread clever.

这门课里的每一个循环——“对每个像素”“对每个片元”“对每个样本”——都要跑上百万次几乎相同、彼此独立的迭代。正是这种工作负载的形状，让 GPU 得以存在。CPU 用大缓存和分支预测去压低单个线程的延迟；GPU 则在成千上万个线程上追求吞吐，并靠“在飞”的线程足够多、总有一些随时可跑，来把延迟藏起来。它把晶体管预算花在算术单元上，而不是花在让某个单线程更聪明上。

Threads execute in lockstep groups — a warp (NVIDIA, ~32 threads) or wavefront (AMD) — under one instruction decoder: SIMT (single instruction, multiple threads). Two consequences dominate performance and cross-link the GPU Kernels track:

线程以锁步的小组执行——一个 warp（NVIDIA，约 32 个线程）或 wavefront（AMD）——共用一个指令译码器：SIMT（单指令、多线程）。有两个后果主导性能，也与《GPU Kernels》那门课相互呼应：

• Branch divergence. If threads in one warp take different paths of an if, the hardware runs both paths and masks off the inactive threads — the branches serialize. A shader where neighboring pixels wildly disagree (e.g. per-pixel loop counts) can lose most of its throughput. This is why GPU-friendly code keeps warps coherent.分支发散。若同一 warp 里的线程走了 if 的不同分支，硬件会把两条分支都跑一遍、并把非活动线程屏蔽掉——分支被串行化。一个相邻像素分歧剧烈的着色器（例如逐像素的循环次数不同）可能损失大部分吞吐。这正是 GPU 友好的代码要让 warp 保持一致的原因。
• Memory coalescing. When the 32 threads of a warp read 32 contiguous addresses, the hardware fuses them into one wide memory transaction; scattered reads become many transactions and stall. Texture caches and tiled layouts (lesson 05's mipmaps included) exist to keep access local.访存合并。当一个 warp 的 32 个线程读取 32 个连续地址时，硬件把它们融合成一次宽内存事务；散乱的读取则变成许多次事务、并造成停顿。纹理缓存与分块布局（包括第 05 课的 mipmap）之所以存在，就是为了让访问保持局部。

2 · The programmable pipeline and the API可编程管线与图形 API

Lessons 04–09 quietly assumed a machine that runs vertex work, then fixed-function rasterization, then per-fragment shading. That machine is the graphics pipeline, and the programmable stages are where your code (shaders) plugs in:

第 04–09 课默认了这样一台机器：先做顶点工作，再做固定功能的光栅化，然后做逐片元着色。那台机器就是图形管线，而可编程阶段正是你的代码（着色器）插入之处：

The other half of real-time cost is on the CPU side: the application issues draw calls and state changes through a graphics API (Vulkan, Direct3D, Metal, WebGPU). Each draw call has overhead, so ten thousand tiny draws can bottleneck the CPU while the GPU sits idle. Half of real-time performance work is reducing draw calls — batching many objects into one call, sorting by material to avoid state thrash — which motivates the culling and instancing in section 4.

实时开销的另一半在 CPU 一侧：应用程序通过图形 API（Vulkan、Direct3D、Metal、WebGPU）发出绘制调用（draw call）与状态切换。每次绘制调用都有开销，于是一万次细碎的绘制会让 CPU 成为瓶颈、而 GPU 闲着。实时性能优化有一半就是减少绘制调用——把许多物体合批进一次调用、按材质排序以避免状态抖动——这引出了第 4 节的剔除与实例化。

3 · Forward vs deferred shading前向 vs 延迟着色

Here is the central architectural fork of a real-time renderer, and it follows directly from lesson 08's shading loop: for each visible fragment, sum the contribution of each light. Where you put that loop decides everything.

这是实时渲染器的核心架构分岔，它直接从第 08 课的着色循环而来：对每个可见片元，累加每个光源的贡献。你把这个循环放在哪里，决定了一切。

• Forward. Rasterize each object and shade its fragments against every light immediately. Simple, and the only easy home for transparency and MSAA (lesson 06). But it pays for overdraw — a fragment shaded now may be overwritten by a nearer one later — and its cost scales as roughly objects × lights, which explodes with many lights.前向。光栅化每个物体，并立刻用每个光源给它的片元着色。简单，也是透明与 MSAA（第 06 课）唯一好安家的地方。但它要为过绘制付费——现在着色的片元可能稍后被更近的覆盖掉——而且其开销大致按物体数 × 光源数增长，光源一多就爆炸。
• Deferred. First pass: rasterize geometry into a G-buffer (albedo, normal, depth, material — the BRDF inputs of lesson 09), shading nothing. Second pass: for each pixel that survived the depth test, shade once against the lights in screen space. Shading is now decoupled from geometry — cost scales as pixels × lights, independent of object count — so it eats many lights cheaply. The price: fat G-buffer bandwidth, and real pain with MSAA and transparency (which need per-sample or per-layer data the single G-buffer discarded).延迟。第一趟：把几何光栅化进一个 G-buffer（基础色、法线、深度、材质——即第 09 课的 BRDF 输入），完全不着色。第二趟：对每个通过深度测试的像素，在屏幕空间里用光源只着一次色。着色至此与几何解耦——开销按像素数 × 光源数增长、与物体数无关——因此能廉价地吃下很多光源。代价是：庞大的 G-buffer 带宽，以及在 MSAA 与透明上的真正痛苦（它们需要单张 G-buffer 丢弃了的逐样本或逐层数据）。

Modern engines often use a hybrid — tiled/clustered forward+ — that culls lights to screen tiles so a forward pass only shades each fragment against the lights that actually reach it, recovering deferred's many-lights scaling while keeping forward's transparency and MSAA. The widget lets you feel the crossover: crank the light count and watch deferred overtake forward.

现代引擎常用一种混合——分块/分簇 forward+——把光源剔除到屏幕分块上，于是前向通道只让每个片元与真正照到它的光源着色，既找回了延迟的“多光源”扩展性，又保住了前向的透明与 MSAA。下面的控件让你感受这个交叉点：把光源数拉大，看延迟如何反超前向。

// Schematic RELATIVE cost — not milliseconds. Illustrates the scaling law only.
var G = 1;          // geometry cost per object
var S = 0.6;        // shade cost per fragment per light
var SCREEN = 260;   // one full-screen lighting pass (constant, object-independent)
// forward: every object's fragments shaded against every light, all paying overdraw
var forward  = obj * overdraw * (G + lights * S);
// deferred: G-buffer write (still overdraws geometry) + screen-space lighting per light
var deferred = obj * overdraw * G + SCREEN * lights;
var cheaper = (forward < deferred) ? 'forward' : 'deferred';

4 · Assembling a frame under budget在预算内拼出一帧

A 60 FPS game has about 16.7 ms per frame (schematic — 30, 90, and 120 Hz targets are all common); everything in this course must fit. A frame is a sequence of passes, each writing buffers the next consumes — the whole track, run in order:

一个 60 FPS 的游戏每帧大约有 16.7 毫秒（示意——30、90、120 Hz 的目标都很常见）；本课的一切都得塞进去。一帧是一串通道，每个通道写出的缓冲被下一个消费——正是整门课按顺序跑一遍：

To make it fit, the renderer's first job is to not do work. The toolkit:

为了塞得进去，渲染器的头等大事是不做无用功。工具箱：

5 · Where graphics is going — the loop closes with vision图形学的走向——与视觉合拢的回路

The frontier is graphics fusing with machine learning, and it arrives in three waves. Neural upscaling & frame generation (DLSS, FSR, XeSS) render fewer pixels/frames and let a network hallucinate the rest — the temporal-accumulation idea of lesson 06's TAA, learned. ML denoising (lesson 11) turns a 1–8 spp path-traced image into a clean one, finally making real-time path tracing viable on the ray-tracing hardware of lesson 10.

前沿在于图形学与机器学习的融合，它分三波到来。神经上采样与帧生成（DLSS、FSR、XeSS）只渲染更少的像素/帧，让网络“脑补”其余——正是第 06 课 TAA 的时域累积思想，被学习化了。ML 去噪（第 11 课）把 1–8 spp 的路径追踪图像变干净，终于让实时路径追踪在第 10 课的光追硬件上变得可行。

The third wave is the deep one: neural rendering. A NeRF or a 3D Gaussian splat is not a magic trick — it is a differentiable renderer. It is this entire forward pipeline (project, composite, shade) written so that the gradient of "rendered image vs photograph" can flow backwards into the scene parameters. Run that gradient descent and the images optimize the scene that produced them. And that is exactly the inverse problem lesson 01 named: vision recovers a scene from images. So differentiable rendering is the bridge — graphics (scene → image) run backwards is vision (image → scene). The Computer Vision and 3D Vision tracks pick up exactly here, from the other side of the same arrow.

第三波是更深的一波：神经渲染。一个 NeRF 或一团 3D 高斯泼溅并不是魔术——它是一个可微渲染器。它就是把这整条正向管线（投影、合成、着色）写成让“渲染图像与照片之差”的梯度能反向流回场景参数的形式。跑这个梯度下降，图像就会去优化生成它们的场景。而这恰恰是第 01 课点名的逆问题：视觉从图像还原场景。所以可微渲染就是那座桥——把图形学（场景 → 图像）倒着跑，就是视觉（图像 → 场景）。《计算机视觉》与 《3D 视觉》正是从这里、从同一支箭的另一端接手。

Where this points next接下来指向何处

This is the end of the forward road. You can now trace a scene description all the way to lit, textured, anti-aliased, animated, tone-mapped pixels, and reason about the GPU frame that ships them at 60 FPS. The one arrow left points outward: reverse the whole pipeline and you are doing computer vision. Follow the companion Computer Vision and 3D Vision tracks to walk the inverse problem — recovering geometry, materials, and light from images — the mirror image of everything here. Return to the track index for the full map.

这是正向之路的终点。你现在能把一段场景描述一路追踪成有光照、有纹理、经抗锯齿、会动、经色调映射的像素，并推理以 60 FPS 交付它们的那一帧 GPU 工作。剩下的唯一一支箭指向外面：把整条管线倒过来，你做的就是计算机视觉。跟随配套的《计算机视觉》与 《3D 视觉》去走那个逆问题——从图像还原几何、材质与光——正是此处一切的镜像。回到课程目录可查看完整地图。

Interview prompts面试题

• What is branch divergence and why does it hurt on a GPU? (§1 — a warp runs in lockstep, so divergent if paths execute serially with masking, wasting throughput.) 什么是分支发散，为什么它在 GPU 上有害？(§1 — 一个 warp 锁步执行，分歧的 if 分支被带屏蔽地串行跑完，浪费吞吐。)
• Name the programmable pipeline stages a triangle passes through. (§2 — vertex shader → (tessellation/geometry) → rasterizer → fragment shader → depth-test/blend.) 说出一个三角形经过的可编程管线阶段。(§2 — 顶点着色器 →（镶嵌/几何）→ 光栅化器 → 片元着色器 → 深度测试/混合。)
• Why does draw-call count matter even when the GPU is fast? (§2 — each draw carries CPU-side API overhead; too many draws bottleneck the CPU, so batching/instancing matters.) 为什么即便 GPU 很快，绘制调用次数仍要紧？(§2 — 每次绘制都带 CPU 侧 API 开销；调用过多会让 CPU 成为瓶颈，故要合批/实例化。)
• Forward vs deferred: when does each win, and what breaks deferred? (§3 — forward for few lights + transparency/MSAA; deferred scales to many lights (cost ∝ pixels × lights) but struggles with transparency and MSAA.) 前向 vs 延迟：各自何时取胜，延迟在什么上会崩？(§3 — 前向适合少光源 + 透明/MSAA；延迟能扩展到多光源（开销 ∝ 像素 × 光源），但在透明与 MSAA 上很吃力。)
• You are over your 16.7 ms budget. Name three ways to cut work. (§4 — frustum/occlusion culling, LOD for distant meshes, instancing/batching to cut draw calls.) 你超出了 16.7 毫秒预算。说出三种削减工作量的办法。(§4 — 视锥/遮挡剔除、对远处网格用 LOD、用实例化/合批削减绘制调用。)
• In what precise sense are NeRF and Gaussian splatting "graphics," and how do they connect to vision? (§5 — they are differentiable renderers; running the forward pipeline backwards optimizes scene parameters from images — the inverse problem, i.e. computer vision.) NeRF 与高斯泼溅在何种确切意义上属于“图形学”，又如何与视觉相连？(§5 — 它们是可微渲染器；把正向管线倒着跑，就能从图像优化场景参数——即逆问题，也就是计算机视觉。)