Shading, interpolation, and textures

Lesson 06 handed us two things and no more: coverage — which pixels a triangle fills — and a z-buffer — which triangle wins at each pixel. The pixels are flat-colored placeholders. This lesson makes them look right: interpolate the attributes each vertex carries, sample a texture image at the correct spot, and run a shading model to decide the final color. Then it confronts the property that reshapes the rest of the track: although shading and interpolation are differentiable while visibility is fixed, the hard coverage and z-buffer decisions do not provide the boundary gradients inverse rendering needs. A conventional rasterizer therefore cannot simply be dropped into an end-to-end geometry-optimization loop.

1 · Attribute interpolation, and the perspective-correct catch

A vertex is never just a position. It carries attributes: an RGB color, a surface normal (for lighting), a pair of UV texture coordinates (where in a texture image this vertex maps to), often the world-space position too. The rasterizer's job, after lesson 06 decided a pixel is inside a triangle, is to figure out the value of each attribute at that pixel — somewhere between the three corner values. That "somewhere between" is interpolation, and the natural tool is the barycentric weights (b₀, b₁, b₂) from lesson 06 §4: three non-negative numbers summing to 1 that express a point inside the triangle as a weighted average of its vertices.

The tempting move is to interpolate every attribute the same way you'd interpolate position on the screen:

For colors on a triangle facing the camera this looks fine. For a textured surface receding into the distance it is visibly, embarrassingly wrong — and knowing why is a rite of passage.

Where it breaks. The barycentric weights are computed in 2D screen space, after the perspective divide of lesson 06 §2 — the step where clip-space coordinates get divided by w (which is proportional to view-space depth) to land on the screen. That divide is nonlinear. Equal steps across the screen do not correspond to equal steps across the 3D surface: a pixel halfway up a receding floor is not the halfway point of the floor in the world — it is much farther than halfway, because distant geometry is compressed toward the horizon. So a quantity that varies linearly across the surface (like a texture coordinate) does not vary linearly across the screen. Interpolating it linearly on the screen smears the texture as if the surface were flat and facing you.

The fix. What is linear in screen space is the attribute divided by its depth, and also 1/w itself. So you interpolate those two linearly with the barycentric weights, then divide at the end to undo it — perspective-correct interpolation:

where b_i are the screen-space barycentric weights and w_i is the clip-space w (proportional to view-space depth) at vertex i. The numerator interpolates attr/w; the denominator interpolates 1/w; the ratio recovers the true surface value. Note that when all three w_i are equal — an orthographic view, or a triangle at constant depth — the w's cancel and this collapses back to the naive formula. Perspective correction is exactly the correction for the vertices being at different depths.

Worked intuition. Picture a checkerboard tiled on a floor that recedes from the camera — a wide near edge at the bottom of the frame, a narrow far edge near the horizon. In the real world the checker rows shrink as they recede: near rows are tall, far rows are squished. With naive screen-linear interpolation the texture coordinate climbs at a constant rate up the screen, so every checker row comes out the same height — the board looks like a flat poster tilted in a way your eye rejects. With perspective-correct interpolation the far vertices' large w (small 1/w) drags the interpolated UV, so the rows correctly bunch toward the far edge. The widget in §5's neighborhood — actually right below in this section — lets you toggle the two and watch the checker rows go from uniform (wrong) to foreshortened (right).

// Trapezoid floor: 4 corners, each with a UV and a clip-space w (∝ depth).
// Near edge (bottom) is close  → small w;  far edge (top) is distant → large w.
var wNear = 1, wFar = fore;                 // fore = far depth / near depth (the slider)
var V = [
  { x: xNL, y: yBot, u: 0, v: 0, w: wNear },  // near-left
  { x: xNR, y: yBot, u: 1, v: 0, w: wNear },  // near-right
  { x: xFR, y: yTop, u: 1, v: 1, w: wFar  },  // far-right
  { x: xFL, y: yTop, u: 0, v: 1, w: wFar  }   // far-left
];
var tris = [[0,1,2],[0,2,3]];               // two triangles

// barycentric weights of pixel P in triangle (A,B,C)  — lesson 06 §4
function bary(A,B,C,px,py){
  var d = (B.y-C.y)*(A.x-C.x) + (C.x-B.x)*(A.y-C.y);
  var b0 = ((B.y-C.y)*(px-C.x) + (C.x-B.x)*(py-C.y)) / d;
  var b1 = ((C.y-A.y)*(px-C.x) + (A.x-C.x)*(py-C.y)) / d;
  return [b0, b1, 1-b0-b1];
}

for (each pixel P inside a triangle (A,B,C)) {
  var b = bary(A,B,C, P.x, P.y);
  var u, v;
  if (mode === 'naive') {                    // WRONG under perspective
    u = b[0]*A.u + b[1]*B.u + b[2]*C.u;
    v = b[0]*A.v + b[1]*B.v + b[2]*C.v;
  } else {                                   // PERSPECTIVE-CORRECT
    var iw = b[0]/A.w + b[1]/B.w + b[2]/C.w;             // Σ b_i / w_i
    u = (b[0]*A.u/A.w + b[1]*B.u/B.w + b[2]*C.u/C.w) / iw;
    v = (b[0]*A.v/A.w + b[1]*B.v/B.w + b[2]*C.v/C.w) / iw;
  }
  var checker = ((Math.floor(u*ROWS) + Math.floor(v*ROWS)) & 1);
  setPixel(P, checker ? light : dark);
}

2 · Texture mapping and aliasing — a callback to the CV track

The UV coordinates we just interpolated exist to index a texture image. Texture mapping is the map from a fragment's interpolated (u,v) ∈ [0,1]² to a location in an image, whose value (a color, or a normal, or roughness…) becomes part of the fragment's shading. A UV of (0.5, 0.5) reads the center texel; a triangle's three UVs staple a rectangular patch of the image onto the triangle. The only real subtlety is sampling — and it splits into two opposite regimes.

Magnification — the texture is too small. When the camera is close, one texel spans many screen pixels. Nearest-neighbor sampling then shows blocky texels with hard steps. The standard fix is bilinear filtering: read the four texels surrounding the sample point and blend them by the fractional position — exactly the bilinear interpolation from CV lesson 01 (../computer_vision/01_image_representation.html), now used for lookup rather than resizing. Smooth, cheap, done.

Minification — the texture is too big. When the surface is far or grazing, one screen pixel covers many texels. Naively reading a single texel per pixel means you are point-sampling a high-frequency signal below its Nyquist rate — the precise aliasing failure from CV lessons 01 and 02 (../computer_vision/02_filtering_convolution.html): downsampling without a low-pass pre-filter. The texture's fine detail folds back as false low-frequency structure. On a static frame it is moiré; in motion it shimmers and crawls as the sample points slide across texels. This is not a new problem; it is the sampling theorem again, moved onto the texture.

The fix — mipmapping. Recall the CV rule: you must low-pass before you downsample. You cannot afford to average hundreds of texels per pixel at draw time, so you do it ahead of time. A mipmap is a precomputed pyramid of the texture, each level a properly filtered (box-averaged) 2× downsample of the one above — the same Gaussian/box pyramid idea from CV lesson 02, stored:

The pyramid costs only +1/3 extra memory (a geometric series 1 + 1/4 + 1/16 + … = 4/3). Per fragment, the GPU estimates the pixel's footprint in texture space from the UV derivatives — how fast (u,v) changes between neighboring pixels, i.e. ∂(u,v)/∂(x,y) — and picks the level where one pixel ≈ one texel, so the read is already pre-averaged to the right scale. Sampling between the two nearest levels and blending them gives trilinear filtering (bilinear within each level, linear across levels), which removes the visible seams as an object recedes. Where the footprint is elongated — a surface seen at a grazing angle, long in one texture direction and short in the other — a single square mip level over-blurs; anisotropic filtering takes several samples along the long axis of the footprint to keep sharpness without aliasing.

3 · Shading — where the color comes from, and the programmable pipeline

Interpolation and texturing supply ingredients; the shading model decides how they combine into a fragment's final color, and — the part worth internalizing — where in the pipeline that decision happens. Historically three models trade quality for cost by moving the lighting computation to finer granularity:

• Flat shading — one lighting computation per triangle, using its face normal; the whole triangle gets a single color. Cheapest; every facet is visible, giving a low-poly look.
• Gouraud shading — lighting computed per vertex, then the resulting colors are interpolated across the triangle (with the §1 machinery). Smooth over a mesh, but it interpolates the lit result, so a sharp specular highlight that falls between vertices is dimmed or lost entirely.
• Phong shading — interpolate the normal per fragment, then compute lighting per fragment with that interpolated normal. The highlight is evaluated where it actually lands, so it stays crisp. More expensive (a lighting evaluation at every pixel), and the default look of modern real-time rendering.

The move from Gouraud to Phong — interpolate the input to lighting, not its output — is exactly what the programmable pipeline makes routine. Modern GPUs replace the old fixed-function color stages with two programmable ones bracketing the fixed-function rasterizer:

Phong shading is just "do the lighting in the fragment shader, using an interpolated normal." Normal mapping — faking fine surface detail by reading a per-pixel normal from a texture — is the same slot. The takeaway is structural: the vertex shader decides where geometry goes; the fragment shader decides what color a pixel is. That is the seam every real-time graphics programmer works along.

The lighting model, briefly. You do not need a full lighting course to place the pieces. A fragment's shaded color is typically a sum of a diffuse term and a specular term. Diffuse (Lambert) is the matte component — brightness proportional to how squarely the surface faces the light:

with n the unit surface normal, l the unit direction to the light, and the max(0, ·) clamping away light from behind. Specular (Blinn–Phong) adds the shiny highlight, a sharp lobe around the mirror direction controlled by a shininess exponent. Both are just plugged into the fragment shader — and both are the simplest instance of a BRDF (bidirectional reflectance distribution function), the general "how does this material scatter incoming light into outgoing directions?" object that physically based rendering makes far richer. For this lesson the point is not the formula but its location: this evaluation runs per fragment, which is why interpolating the normal (Phong) beats interpolating the color (Gouraud).

4 · Antialiasing the edges — MSAA

Texturing was not the only place aliasing crept in. The coverage test of lesson 06 §4 is a hard binary decision — a pixel's center is inside the triangle or it is not — so a slanted edge becomes a staircase of fully-on and fully-off pixels: jaggies. That is aliasing on the geometry, the same sampling failure one more time, now because a step-edge (infinite frequency) is being sampled at one point per pixel.

The brute-force fix is supersampling: render the whole image at, say, 4× resolution and average down (a low-pass then downsample — the CV rule yet again). Correct, but it multiplies the fragment-shader cost by the sample count, which is ruinous because shading is the expensive part.

MSAA (multisample antialiasing) is the clever economization. Per pixel it keeps several sub-pixel sample points (say 4), and evaluates coverage and depth at each sample — so an edge cutting through a pixel lights up, say, 2 of 4 samples, and the pixel ends up 50% blended. But it runs the fragment shader only once per pixel (typically at the pixel center), sharing that one shaded color across all covered samples. You pay for extra coverage/depth resolution, which is cheap, without paying for extra shading, which is not. Edges get smooth; interior shading cost is unchanged. (It antialiases geometry edges, not texture minification — that is what mipmaps are for; the two fixes are complementary because they target different aliasing sources.)

5 · Differentiable inside a visibility region, discontinuous at its boundary

Here is the payoff of the whole rasterization pair (lessons 06–07). We have a fast, gorgeous forward renderer. Now recall the loop from lesson 01: 3D vision is inverse rendering — we want to take an image loss and push it backward into the geometry, ∂L/∂(scene), so an optimizer can improve the scene. The precise obstacle is not that every operation in a rasterizer has zero derivative. With the set of visible triangles held fixed, barycentric interpolation, texture filtering, and shading are differentiable with respect to vertex attributes and often vertex positions. The hard part is how that visible set changes.

Coverage is a hard inside/outside test; the z-test is a hard min. Whether triangle k contributes to pixel p is a step function of the vertex positions: nudge a silhouette a hair and either the winner stays fixed or the pixel abruptly flips when the edge crosses a sample. The coverage indicator is therefore piecewise constant, even though the winning triangle's interpolated color can vary smoothly inside each region:

The z-buffer adds a second hard selection — the winning fragment is a \min over depths. Away from a visibility change, gradients may flow through the winner while occluded alternatives receive none; where two surfaces swap order, the rendered image can jump. Ordinary automatic differentiation through only the executed hard decisions therefore misses the boundary term: it cannot tell an occluded or just-outside triangle how moving its edge would reveal it and reduce the loss. Geometry optimization is not mathematically impossible, but the conventional rasterization pipeline supplies an incomplete and often unhelpful gradient exactly where silhouettes and occlusions must move.

There are two escapes, and they define the remainder of the track:

• (a) Define useful mesh-rendering gradients. Some differentiable rasterizers — SoftRas, Neural Mesh Renderer, DIB-R — relax hard coverage and/or depth with smooth spatial probabilities, so nearby edges and competing surfaces receive a gradient. Others retain a hard-looking forward image but derive or estimate gradients at visibility boundaries. In either case, not every pixel must depend on every vertex: the goal is a useful, controlled gradient for the parameters that could change the image.
• (b) Abandon hard surfaces for soft / volumetric representations. Instead of a surface that a pixel is either on or off, use a medium that composites continuously along the ray and is differentiable by construction. This is the radiance field of NeRF — colored, semi-transparent density integrated along each ray (lesson 09) — and the soft anisotropic Gaussians of 3D Gaussian Splatting (lesson 10), a differentiable rasterizer of soft primitives whose gradients can flow into the contributing Gaussians' positions, shapes, colors, and opacities.

State it plainly: the missing visibility-boundary gradient is one reason lessons 09 and 10 matter. But forward rendering has one chapter left before we take either escape: we have only done local shading — a couple of lights, no shadows, no bounced light, ad-hoc materials — and real photographs carry all of it. Lesson 08 climbs that realism ladder (physically-based materials, shadow mapping, global illumination) to the ceiling of a hand-authored forward pipeline. From there, lessons 09–10 introduce scene representations and renderers built for inverse optimization.

Where this points next

We can now turn coverage and a z-buffer into a finished image: interpolate vertex attributes perspective-correctly, look up textures with mip-filtered sampling, shade per fragment, and antialias the edges with MSAA — but with only local shading, which is not yet photorealism. Lesson 08 finishes the forward-rendering story: physically-based materials (the microfacet BRDF), shadow mapping, and the global-illumination tricks that fake bounced light — how a flat-shaded model becomes a photoreal render, all still on the rasterizer. Crucially, lesson 08 also exposes the ceiling: photoreal forward rasterization is hand-authored, and its hard visibility decisions do not directly yield the gradients needed to recover geometry. Lesson 09 turns to a differentiable radiance field, where colored semi-transparent density is integrated along rays to close the represent→render→compare→backprop loop lesson 01 promised.

Interview prompts

• Why is perspective-correct interpolation necessary, and when can you skip it? (§1 — the perspective divide is nonlinear, so screen-linear interpolation of surface attributes is wrong; interpolate attr/w and 1/w then divide. Skip only when all vertices share a depth — orthographic, or a screen-space quad — where the w's cancel.)
• A textured ground plane looks warped only when the camera tilts down. What's the bug? (§1 — naive affine interpolation instead of perspective-correct; invisible on camera-facing geometry, worst on surfaces raking away with a large depth spread.)
• What causes texture shimmer on distant/grazing surfaces, and how do mipmaps fix it? (§2 — minification aliases because one pixel covers many texels (sampling below Nyquist, CV 01–02); mipmaps store a pre-filtered pyramid and pick the level where one pixel ≈ one texel from the UV-derivative footprint; trilinear blends levels, anisotropic handles elongated footprints.)
• Contrast flat, Gouraud, and Phong shading — where is lighting computed in each? (§3 — flat: once per triangle; Gouraud: per vertex then interpolate the color (loses between-vertex highlights); Phong: interpolate the normal, light per fragment (crisp highlights). Phong lives in the fragment shader.)
• How does MSAA antialias edges more cheaply than supersampling? (§4 — it multisamples coverage and depth per pixel but runs the fragment shader once per pixel, so it pays for edge resolution, not shading; it fixes geometry jaggies, not texture minification.)
• Which parts of rasterization are differentiable, what breaks at visibility changes, and what are the fixes? (§5 — with visibility fixed, interpolation and shading can differentiate through the winning triangle; hard coverage and z-order switches are discontinuous, so ordinary autodiff misses silhouette/occlusion boundary gradients and gives none to losing surfaces. Differentiable mesh rasterizers define or approximate those gradients; NeRF/3DGS use soft compositing.)