Shading, interpolation, and textures

Lesson 06 handed us two things and no more: coverage — which pixels a triangle fills — and a z-buffer — which triangle wins at each pixel. The pixels are flat-colored placeholders. This lesson makes them look right: interpolate the attributes each vertex carries, sample a texture image at the correct spot, and run a shading model to decide the final color. Then it confronts the one property that reshapes the entire rest of the track — the property that is the whole reason lessons 09 and 10 exist: this pipeline, for all its polish, is not differentiable. You cannot push an image loss back through it into the geometry.

The plan

Five moves. (1) Interpolate vertex attributes across a triangle with the barycentric weights from lesson 06 §4 — and fix the trap that sinks every first implementation: linear interpolation in screen space is wrong under perspective, so we derive perspective-correct interpolation. (2) Texture mapping — UVs, bilinear filtering for magnification, and minification aliasing that is the exact Nyquist problem from the CV track, fixed by mipmapping. (3) Shading — flat vs Gouraud vs Phong, and where color actually gets computed in the modern programmable pipeline (the fragment shader). (4) Antialiasing the jagged edges hard coverage produces, via MSAA. (5) The payoff: why the whole rasterizer is non-differentiable, and the two escapes — soft rasterizers, and volumetric fields — that hand us straight to lesson 09.

1 · Attribute interpolation, and the perspective-correct catch

A vertex is never just a position. It carries attributes: an RGB color, a surface normal (for lighting), a pair of UV texture coordinates (where in a texture image this vertex maps to), often the world-space position too. The rasterizer's job, after lesson 06 decided a pixel is inside a triangle, is to figure out the value of each attribute at that pixel — somewhere between the three corner values. That "somewhere between" is interpolation, and the natural tool is the barycentric weights (b₀, b₁, b₂) from lesson 06 §4: three non-negative numbers summing to 1 that express a point inside the triangle as a weighted average of its vertices.

The tempting move is to interpolate every attribute the same way you'd interpolate position on the screen:

attr_naive(P) = b₀·attr₀ + b₁·attr₁ + b₂·attr₂ .

For colors on a triangle facing the camera this looks fine. For a textured surface receding into the distance it is visibly, embarrassingly wrong — and knowing why is a rite of passage.

Where it breaks. The barycentric weights are computed in 2D screen space, after the perspective divide of lesson 06 §2 — the step where clip-space coordinates get divided by w (which is proportional to view-space depth) to land on the screen. That divide is nonlinear. Equal steps across the screen do not correspond to equal steps across the 3D surface: a pixel halfway up a receding floor is not the halfway point of the floor in the world — it is much farther than halfway, because distant geometry is compressed toward the horizon. So a quantity that varies linearly across the surface (like a texture coordinate) does not vary linearly across the screen. Interpolating it linearly on the screen smears the texture as if the surface were flat and facing you.

The fix. What is linear in screen space is the attribute divided by its depth, and also 1/w itself. So you interpolate those two linearly with the barycentric weights, then divide at the end to undo it — perspective-correct interpolation:

attr(P) = ^{Σ_i b_i·(attr_i / w_i)}⁄_{Σ_i b_i·(1 / w_i)}

where b_i are the screen-space barycentric weights and w_i is the clip-space w (proportional to view-space depth) at vertex i. The numerator interpolates attr/w; the denominator interpolates 1/w; the ratio recovers the true surface value. Note that when all three w_i are equal — an orthographic view, or a triangle at constant depth — the w's cancel and this collapses back to the naive formula. Perspective correction is exactly the correction for the vertices being at different depths.

Worked intuition. Picture a checkerboard tiled on a floor that recedes from the camera — a wide near edge at the bottom of the frame, a narrow far edge near the horizon. In the real world the checker rows shrink as they recede: near rows are tall, far rows are squished. With naive screen-linear interpolation the texture coordinate climbs at a constant rate up the screen, so every checker row comes out the same height — the board looks like a flat poster tilted in a way your eye rejects. With perspective-correct interpolation the far vertices' large w (small 1/w) drags the interpolated UV, so the rows correctly bunch toward the far edge. The widget in §5's neighborhood — actually right below in this section — lets you toggle the two and watch the checker rows go from uniform (wrong) to foreshortened (right).

The classic real bug / interview question

"My textured ground plane looks warped and the tiles are all the same size, but only when the camera tilts down." This is almost always naive (affine) interpolation where perspective-correct was needed. It is invisible on geometry that faces the camera (equal depths → the w's cancel) and worst on surfaces raking away at a shallow angle (large depth spread). Old consoles that lacked per-pixel perspective correction (the PlayStation 1 is the textbook case) show exactly this texture-swim on floors. If asked "why is perspective-correct interpolation necessary and when can you skip it?", the answer is: the perspective divide is nonlinear, so interpolate attr/w and 1/w then divide; you can skip it only when all vertices share a depth (orthographic, or a screen-space UI quad).

Perspective-correct vs naive interpolation

A receding textured floor — a trapezoid with a wide near edge (bottom) and a narrow far edge (top) — split into two triangles and painted with a UV-space checkerboard. We rasterize it pixel-by-pixel using the barycentric weights from lesson 06, interpolating the UV two ways. Toggle the mode: naive screen-linear interpolation keeps every checker row the same height — visibly wrong, the rows should shrink toward the far edge. Perspective-correct divides by depth and bunches the rows toward the horizon, matching real perspective. Drag foreshortening to change how much deeper the far edge sits than the near edge; the effect vanishes as that ratio → 1 (equal depths, the w's cancel).

foreshortening (far depth / near depth): 5.0× checker rows: 8

Interpolation mode

perspective-correct

Foreshortening w_far/w_near

5.0×

Rows in far half

—

Correct look?

yes

Show the core JS

// Trapezoid floor: 4 corners, each with a UV and a clip-space w (∝ depth).
// Near edge (bottom) is close  → small w;  far edge (top) is distant → large w.
var wNear = 1, wFar = fore;                 // fore = far depth / near depth (the slider)
var V = [
  { x: xNL, y: yBot, u: 0, v: 0, w: wNear },  // near-left
  { x: xNR, y: yBot, u: 1, v: 0, w: wNear },  // near-right
  { x: xFR, y: yTop, u: 1, v: 1, w: wFar  },  // far-right
  { x: xFL, y: yTop, u: 0, v: 1, w: wFar  }   // far-left
];
var tris = [[0,1,2],[0,2,3]];               // two triangles

// barycentric weights of pixel P in triangle (A,B,C)  — lesson 06 §4
function bary(A,B,C,px,py){
  var d = (B.y-C.y)*(A.x-C.x) + (C.x-B.x)*(A.y-C.y);
  var b0 = ((B.y-C.y)*(px-C.x) + (C.x-B.x)*(py-C.y)) / d;
  var b1 = ((C.y-A.y)*(px-C.x) + (A.x-C.x)*(py-C.y)) / d;
  return [b0, b1, 1-b0-b1];
}

for (each pixel P inside a triangle (A,B,C)) {
  var b = bary(A,B,C, P.x, P.y);
  var u, v;
  if (mode === 'naive') {                    // WRONG under perspective
    u = b[0]*A.u + b[1]*B.u + b[2]*C.u;
    v = b[0]*A.v + b[1]*B.v + b[2]*C.v;
  } else {                                   // PERSPECTIVE-CORRECT
    var iw = b[0]/A.w + b[1]/B.w + b[2]/C.w;             // Σ b_i / w_i
    u = (b[0]*A.u/A.w + b[1]*B.u/B.w + b[2]*C.u/C.w) / iw;
    v = (b[0]*A.v/A.w + b[1]*B.v/B.w + b[2]*C.v/C.w) / iw;
  }
  var checker = ((Math.floor(u*ROWS) + Math.floor(v*ROWS)) & 1);
  setPixel(P, checker ? light : dark);
}

2 · Texture mapping and aliasing — a callback to the CV track

The UV coordinates we just interpolated exist to index a texture image. Texture mapping is the map from a fragment's interpolated (u,v) ∈ [0,1]² to a location in an image, whose value (a color, or a normal, or roughness…) becomes part of the fragment's shading. A UV of (0.5, 0.5) reads the center texel; a triangle's three UVs staple a rectangular patch of the image onto the triangle. The only real subtlety is sampling — and it splits into two opposite regimes.

Magnification — the texture is too small. When the camera is close, one texel spans many screen pixels. Nearest-neighbor sampling then shows blocky texels with hard steps. The standard fix is bilinear filtering: read the four texels surrounding the sample point and blend them by the fractional position — exactly the bilinear interpolation from CV lesson 01 (../computer_vision/01_image_representation.html), now used for lookup rather than resizing. Smooth, cheap, done.

Minification — the texture is too big. When the surface is far or grazing, one screen pixel covers many texels. Naively reading a single texel per pixel means you are point-sampling a high-frequency signal below its Nyquist rate — the precise aliasing failure from CV lessons 01 and 02 (../computer_vision/02_filtering_convolution.html): downsampling without a low-pass pre-filter. The texture's fine detail folds back as false low-frequency structure. On a static frame it is moiré; in motion it shimmers and crawls as the sample points slide across texels. This is not a new problem; it is the sampling theorem again, moved onto the texture.

The fix — mipmapping. Recall the CV rule: you must low-pass before you downsample. You cannot afford to average hundreds of texels per pixel at draw time, so you do it ahead of time. A mipmap is a precomputed pyramid of the texture, each level a properly filtered (box-averaged) 2× downsample of the one above — the same Gaussian/box pyramid idea from CV lesson 02, stored:

Mip level	Resolution	Meaning
0	512 × 512	full detail (used up close)
1	256 × 256	pre-filtered 2× down
2	128 × 128	…
…	…	down to 1×1 (the average texel)

The pyramid costs only +1/3 extra memory (a geometric series 1 + 1/4 + 1/16 + … = 4/3). Per fragment, the GPU estimates the pixel's footprint in texture space from the UV derivatives — how fast (u,v) changes between neighboring pixels, i.e. ∂(u,v)/∂(x,y) — and picks the level where one pixel ≈ one texel, so the read is already pre-averaged to the right scale. Sampling between the two nearest levels and blending them gives trilinear filtering (bilinear within each level, linear across levels), which removes the visible seams as an object recedes. Where the footprint is elongated — a surface seen at a grazing angle, long in one texture direction and short in the other — a single square mip level over-blurs; anisotropic filtering takes several samples along the long axis of the footprint to keep sharpness without aliasing.

The unifying idea

Magnification, minification, mipmapping, and anisotropic filtering are all one question — how many texels does this pixel see, and how do I average them correctly? — which is the sampling/Nyquist question from CV lessons 01–02 in disguise. The rasterizer version just adds the twist that the footprint changes per fragment and must be estimated from UV derivatives, and that the pre-filtering is precomputed into a pyramid because you can't do it at draw time.

3 · Shading — where the color comes from, and the programmable pipeline

Interpolation and texturing supply ingredients; the shading model decides how they combine into a fragment's final color, and — the part worth internalizing — where in the pipeline that decision happens. Historically three models trade quality for cost by moving the lighting computation to finer granularity:

Flat shading — one lighting computation per triangle, using its face normal; the whole triangle gets a single color. Cheapest; every facet is visible, giving a low-poly look.
Gouraud shading — lighting computed per vertex, then the resulting colors are interpolated across the triangle (with the §1 machinery). Smooth over a mesh, but it interpolates the lit result, so a sharp specular highlight that falls between vertices is dimmed or lost entirely.
Phong shading — interpolate the normal per fragment, then compute lighting per fragment with that interpolated normal. The highlight is evaluated where it actually lands, so it stays crisp. More expensive (a lighting evaluation at every pixel), and the default look of modern real-time rendering.

The move from Gouraud to Phong — interpolate the input to lighting, not its output — is exactly what the programmable pipeline makes routine. Modern GPUs replace the old fixed-function color stages with two programmable ones bracketing the fixed-function rasterizer:

vertices ──▶ [ VERTEX SHADER ] per-vertex: model→world→view→clip transforms │ (lesson 06 §2), plus pass-through attributes ▼ [ RASTERIZER ] fixed-function: coverage test + z-buffer │ (lesson 06 §4) + perspective-correct │ attribute interpolation (§1 above) ▼ [ FRAGMENT / PIXEL SHADER ] per-pixel: sample textures, run the │ lighting model, apply normal maps → final color ▼ framebuffer

Phong shading is just "do the lighting in the fragment shader, using an interpolated normal." Normal mapping — faking fine surface detail by reading a per-pixel normal from a texture — is the same slot. The takeaway is structural: the vertex shader decides where geometry goes; the fragment shader decides what color a pixel is. That is the seam every real-time graphics programmer works along.

The lighting model, briefly. You do not need a full lighting course to place the pieces. A fragment's shaded color is typically a sum of a diffuse term and a specular term. Diffuse (Lambert) is the matte component — brightness proportional to how squarely the surface faces the light:

L_diffuse = k_d · max(0, n·l)

with n the unit surface normal, l the unit direction to the light, and the max(0, ·) clamping away light from behind. Specular (Blinn–Phong) adds the shiny highlight, a sharp lobe around the mirror direction controlled by a shininess exponent. Both are just plugged into the fragment shader — and both are the simplest instance of a BRDF (bidirectional reflectance distribution function), the general "how does this material scatter incoming light into outgoing directions?" object that physically based rendering makes far richer. For this lesson the point is not the formula but its location: this evaluation runs per fragment, which is why interpolating the normal (Phong) beats interpolating the color (Gouraud).

4 · Antialiasing the edges — MSAA

Texturing was not the only place aliasing crept in. The coverage test of lesson 06 §4 is a hard binary decision — a pixel's center is inside the triangle or it is not — so a slanted edge becomes a staircase of fully-on and fully-off pixels: jaggies. That is aliasing on the geometry, the same sampling failure one more time, now because a step-edge (infinite frequency) is being sampled at one point per pixel.

The brute-force fix is supersampling: render the whole image at, say, 4× resolution and average down (a low-pass then downsample — the CV rule yet again). Correct, but it multiplies the fragment-shader cost by the sample count, which is ruinous because shading is the expensive part.

MSAA (multisample antialiasing) is the clever economization. Per pixel it keeps several sub-pixel sample points (say 4), and evaluates coverage and depth at each sample — so an edge cutting through a pixel lights up, say, 2 of 4 samples, and the pixel ends up 50% blended. But it runs the fragment shader only once per pixel (typically at the pixel center), sharing that one shaded color across all covered samples. You pay for extra coverage/depth resolution, which is cheap, without paying for extra shading, which is not. Edges get smooth; interior shading cost is unchanged. (It antialiases geometry edges, not texture minification — that is what mipmaps are for; the two fixes are complementary because they target different aliasing sources.)

5 · Why this pipeline is NOT differentiable — the bridge to lesson 09

Here is the payoff of the whole rasterization pair (lessons 06–07). We have a fast, gorgeous forward renderer. Now recall the loop from lesson 01: 3D vision is inverse rendering — we want to take an image loss and push it backward into the geometry, ∂L/∂(scene), so an optimizer can improve the scene. For that the renderer must be differentiable. The standard rasterizer is not, and the reason is baked into the two operations that made it work.

The coverage test is a hard argmax; the z-test is a hard min. Whether triangle k colors pixel p is a step function of the vertex positions: nudge a vertex a hair and either nothing changes (the pixel was safely inside, and stays inside) or the pixel abruptly flips in or out at the moment the edge crosses its center. So the coverage indicator is piecewise constant in the vertex coordinates:

∂(pixel color) ⁄ ∂(vertex position) = 0 almost everywhere, undefined at the edges.

The z-buffer adds a second hard decision — the winning fragment is a \min over depths, another non-differentiable selection with a jump wherever two surfaces swap order. Both are the same pathology lesson 01 §4 flagged: a discontinuous renderer has a gradient of zero in the flat regions and no gradient at the jumps, so gradient descent on geometry through a standard rasterizer is dead on arrival. You literally cannot backpropagate an image loss into vertex positions — the chain rule multiplies your loss gradient by a Jacobian that is zero or undefined.

The wall the rest of the track is built to get around

Everything in lessons 06–07 optimizes appearance beautifully and optimizes geometry not at all — because the coverage/z decisions are step functions of the vertices. "Just render the mesh and minimize photometric error to move the vertices" fails for exactly this reason, not because the loss is wrong. Recognizing that a hard-edged renderer has no useful geometry gradient is the insight that produced the next two lessons.

There are two escapes, and they define the remainder of the track:

(a) Make the rasterizer soft. Differentiable rasterizers — SoftRas, the Neural Mesh Renderer, DIB-R — replace the hard in/out coverage test with a smooth spatial falloff: a fragment's contribution decays continuously with its distance to the triangle edge, and the depth resolution is a soft (softmax-like) blend instead of a hard min. Now every pixel has a nonzero, well-defined gradient with respect to every vertex, so you can backprop an image loss into a mesh. You keep meshes; you relax the discontinuity.
(b) Abandon hard surfaces for soft / volumetric representations. Instead of a surface that a pixel is either on or off, use a medium that composites continuously along the ray and is differentiable by construction. This is the radiance field of NeRF — colored, semi-transparent density integrated along each ray (lesson 09) — and the soft anisotropic Gaussians of 3D Gaussian Splatting (lesson 10), which is, quite literally, a differentiable rasterizer of soft primitives whose gradients flow into every Gaussian's position, shape, color, and opacity.

State it plainly: this non-differentiability is the reason lessons 09 and 10 exist. But forward rendering has one chapter left before we take either escape: we have only done local shading — a couple of lights, no shadows, no bounced light, ad-hoc materials — and real photographs carry all of it. Lesson 08 climbs that realism ladder (physically-based materials, shadow mapping, global illumination) to the ceiling of what the hard rasterizer can do — and it is there, at photoreal and still non-differentiable and still hand-authored, that the pivot to the inverse, differentiable renderers of lessons 09–10 becomes unavoidable.

Where this points next

We can now turn coverage and a z-buffer into a finished image: interpolate vertex attributes perspective-correctly, look up textures with mip-filtered sampling, shade per fragment, and antialias the edges with MSAA — but with only local shading, which is not yet photorealism. Lesson 08 finishes the forward-rendering story: physically-based materials (the microfacet BRDF), shadow mapping, and the global-illumination tricks that fake bounced light — how a flat-shaded model becomes a photoreal render, all still on the rasterizer. Crucially, lesson 08 also exposes the ceiling: even photoreal rasterization is hand-authored and non-differentiable, which is exactly what forces the inverse turn — the differentiable radiance field of lesson 09, where a scene becomes colored semi-transparent fog integrated along rays, finally closing the represent→render→compare→backprop loop that lesson 01 promised and this forward pipeline cannot.

Takeaway

Lesson 06 gave coverage + a z-buffer; this lesson made the pixels look right. Vertices carry attributes (color, normal, UV, position) that the rasterizer interpolates with lesson 06's barycentric weights — but linearly in screen space is wrong under perspective (the perspective divide is nonlinear), so you use perspective-correct interpolation: interpolate attr/w and 1/w, then divide. UVs index a texture; magnification wants bilinear filtering, minification is the Nyquist aliasing of CV lessons 01–02, fixed by mipmaps (pre-filtered pyramid, level chosen from the UV-derivative footprint) with trilinear/anisotropic refinements. Shading — flat / Gouraud / Phong — happens in the fragment shader of the programmable pipeline (vertex shader = where geometry goes; fragment shader = what color it is), with diffuse (Lambert) + specular (Blinn–Phong) as the entry point for a BRDF. Jagged edges are aliasing again, tamed by MSAA (multisample coverage/depth, shade once). The decisive fact: the coverage test is a hard argmax and the z-test a hard min, so ∂(color)/∂(vertex) is zero almost everywhere and undefined at edges — the rasterizer is not differentiable. The two escapes — soft rasterizers and volumetric/Gaussian representations — are why lessons 09–10 exist.

Interview prompts

Why is perspective-correct interpolation necessary, and when can you skip it? (§1 — the perspective divide is nonlinear, so screen-linear interpolation of surface attributes is wrong; interpolate attr/w and 1/w then divide. Skip only when all vertices share a depth — orthographic, or a screen-space quad — where the w's cancel.)
A textured ground plane looks warped only when the camera tilts down. What's the bug? (§1 — naive affine interpolation instead of perspective-correct; invisible on camera-facing geometry, worst on surfaces raking away with a large depth spread.)
What causes texture shimmer on distant/grazing surfaces, and how do mipmaps fix it? (§2 — minification aliases because one pixel covers many texels (sampling below Nyquist, CV 01–02); mipmaps store a pre-filtered pyramid and pick the level where one pixel ≈ one texel from the UV-derivative footprint; trilinear blends levels, anisotropic handles elongated footprints.)
Contrast flat, Gouraud, and Phong shading — where is lighting computed in each? (§3 — flat: once per triangle; Gouraud: per vertex then interpolate the color (loses between-vertex highlights); Phong: interpolate the normal, light per fragment (crisp highlights). Phong lives in the fragment shader.)
How does MSAA antialias edges more cheaply than supersampling? (§4 — it multisamples coverage and depth per pixel but runs the fragment shader once per pixel, so it pays for edge resolution, not shading; it fixes geometry jaggies, not texture minification.)
Why can't you backpropagate an image loss into mesh vertices through a standard rasterizer, and what are the fixes? (§5 — coverage is a hard argmax and z-test a hard min, both step functions of vertex position, so the gradient is zero a.e. and undefined at edges; fixes are soft/differentiable rasterizers (SoftRas, DIB-R) or soft volumetric representations (NeRF, Gaussian Splatting).)