What the label means

Lesson 6 ended on a number that depends on a definition: the same detector misses 63%, 58% or 44% of the rare case depending on which pedestrians count, and Lesson 2 had priced the label stage at nothing. Both are true. A label enters a project twice, as the target a detector is taught and as the rule that decides whom a miss rate is averaged over, and different clauses move the two: how much of a pedestrian must show moves the reported number by up to 25 points and the detector by about one, while where the mask lies and what it includes move the detector and no frame-level number. A label is a definition with a clause for each question a mask leaves open. This lesson measures each clause on both doors and writes them down as a contract. It cannot say whether a renderer we did not write keeps it.

1 · One detector, seven rules

Take the detector that the exact-stage program bbba trains (the street's scene, light and camera, the program's own label), three seeds, and the exam of Lesson 1: 3,000 real test frames, a threshold that lets a tenth of the pedestrian-free validation frames alarm. 1,517 of the frames show a pedestrian, and the exam counts 1,323 of them. A benchmark could have written another sentence about who counts. Each sentence below is a rule that scores the same scores differently, and none touches a weight; the rare case is Lesson 6's slice of 3,000 frames of pedestrians who step out from behind the van closer than 12 m, which a project cannot buy (a lab privilege).

The reported miss rate spans 7.1 points over the exam's frames and 25.0 over the rare case (the three rules of Lesson 6, any pixel, the exam's and half, span 19.2). A rule is a cut on the curve of Lesson 6's last section, the miss rate as a function of the visible fraction, and the number it reports is the average of the curve above the cut. In the rare case 49% of the pedestrians show less than half of themselves, against 13% of the exam's, so a cut there moves the average far more.

Lesson 2 priced the label stage at −0.1 points: aaab reproduced aaaa to the last digit, and the street's rule in place of the program's moved bbba to bbbb by 0.3 points. A stage that moves a reported number by 25 points cannot be worth nothing. Either the price was measured on the wrong thing or the label does two jobs, and only one of them was in the price.

2 · Two doors

A definition enters a project in two places. As a scoring rule it chooses whom a miss rate is averaged over; §1 changed that and nothing else. As a training label it chooses what the detector is taught, and that is the only door Lesson 2 swapped. The head is taught per cell of a stride-2 grid: a cell is positive when the pixel at its centre is at least half pedestrian and the frame counts as holding one under the rule, lab = c >= 0.5 && y. The rule enters through y alone, so it can change a cell only in a frame whose pedestrian is partly hidden. Count such frames in the program's scene and in the street's, for the rules of the program (any pixel), the street (15% of the silhouette) and the exam:

The program's scene never hides anyone, so its rule is never asked: that is why aaab equals aaaa to the last digit. Even in the street's scene the training rule decides about one pedestrian frame in fifty, and a frame decides only its covered cells: of the 5,579 positive cells in the 819 pedestrian frames of the seed-1 training set, the street's rule changes 23 (0.4%), a rule asking for half the silhouette 176 (3.2%). The scoring rule touches no cell; it changes the population. The exam counts 1,323 pedestrians, any pixel 1,460, at most 35% hidden 1,263, and the average runs over a curve that falls from 92% missed when under a quarter of the silhouette shows to 36% when over four fifths do (seed 1, the rare case).

Teach under three rules and score under three (the street's rule is the cell bbbb; the 50% detector is new; three seeds each, the series' protocol):

Down a column the detector changes and the rule stays: the numbers differ by at most 0.7 points over the exam's frames and 2.4 over the rare case. Along a row the detector stays and the rule changes: 4.0 and 19.2 points, 6 and 8 times as far. Lesson 2's price was the price of one door. The other is paid every time two numbers are compared.

3 · What a mask leaves open: six clauses

The label stage turns what a renderer returns, coverage, a centre-ray depth and range, an id, into a frame label and cell labels. A mask of a pedestrian answers "which pixels" and leaves six questions on which two honest implementations disagree. Each is a clause of the definition.

How far. A pixel with tan φ across and ρ up looks along (tan φ, ρ, 1), so a surface at depth z sits at z·(tan φ, ρ, 1) and its range is

r = z·√(1 + tan²φ + ρ²)

In the ground-plane law of the table f = 83.1 px, h_c = 1.4 m and V₀ = 9 are Lesson 1's, and v counts rows down. The factor is 1.155 on the horizon at the edge of the 60° field and 1.165 at the corner pixel; pedestrians stand near the axis, and among the exam's the largest is 1.06. Small as that is, it moves pedestrians across the 12 m line of Lesson 6's case R: by depth 0.822% of the street's frames are close step-outs, by range 0.683%, 17% fewer, and every weight p/q of that lesson inherits the difference.

Aligned how. The same point has coordinate u where pixel i spans [i, i + 1) and u − 0.5 where centres are integers; a mask projected with one convention onto a grid that assumes the other is displaced by half a pixel in both directions, and a pedestrian at 16 m is about 2.5 pixels wide. Counted how finely. Coverage is an area integral. With 2 × 2 rays a pedestrian narrower than the ray spacing slips between them: 57 pedestrians of the test set have zero visible area at 2 × 2 rays and 14 at 8 × 8, and "at least 6 px²" gives another answer for 33 of the 310 whose area lies between 3 and 10 px², a threshold effect that matters only near the line. Which instant. A pedestrian at 1.5 m/s, 12 m away, in a 20 ms exposure moves f·v·Δt/z = 0.21 px; the lab has one instant, but a pipeline that stamps a label at one time and exposes the picture over another has two, and the offset between them is the motion during the exposure.

4 · Each clause, both doors

Measure every clause twice. Taught: retrain the detector of bbba on the same 1,600 frames and seeds with only the label changed. Scored: keep the detector and change the clause in the rule. The dial does the second live, over the stored decisions of nine detectors, one per label.

L.counts = function (r, rule) {
    case 'pixel': return r.area >= 1;
    case 'exam': return r.area >= 6 && r.area >= 0.15 * r.full;
    case 'half': return r.area > 0 && r.area >= 0.5 * r.full;
    case 'amodal': return r.full >= 6;
  for (i = 0; i < rows.length; i++) if (L.counts(rows[i], rule)) { n++; if (scores[i] > thr) hit++; }
  return { n: n, miss: n ? 1 - hit / n : null };
L.sliverArea = function (z, w, h) { return SV.CAM.f * SV.CAM.f * w * h / (z * z); };
    sum += a - A; worst = Math.max(worst, Math.abs(a - A));
    worst = Math.max(worst, Math.abs(map[q] / (C.f * C.hc / (j + 0.5 - C.V0)) - 1)); n++;
L.rangeFactor = function (tanPhi, rho) { return Math.sqrt(1 + tanPhi * tanPhi + rho * rho); };

What to try. As the page opens, the program's detector on the rare case under the exam's rule: 2,404 pedestrians counted, 58.1% missed (seed 1; the tables' three-seed mean is 58.4%). Cut to 0 with the 6 px² box unticked: any pixel, 2,824 counted, 63.2% missed; cut at 50%: 1,518 counted, 43.7%. Switch to the exam's frames and repeat: 50.5% to 46.6%, because most pedestrians clear the cut. Back on the rare case, step through the nine detectors at the exam's rule: best to worst is 9.6 points, where the cut alone moves the program's detector by 19.5. Read the area with one ray per pixel, what an object-index pass delivers: 2,376 counted, 57.8% missed; after the default filter: 2,043 counted, 52.1%, six points from a reader with the detector unchanged. Choose range: of the 2,398 counted, all are closer than 12 m by depth and 2,022 by range. Tick half a pixel off on the exam's frames: violet marks where the displaced mask lands, and its overlap with the true mask is 57%.

The other column comes from nine retrained detectors. Six seeds, the differences paired by seed (the same 1,600 frames, only the label differs), because several differences are of the size of the seed wobble:

Three things stand out. Reporting effects are large and cost nothing: no detector was retrained to get the right-hand column. The clauses that move the detector are not the ones that move the number. The visibility rule, argued about most, decides 0.4 to 3.2% of the positive cells and moves the detector by less than its wobble; so do 8×8 rays and one ray, which change 16 to 20%. A mask half a pixel off moves 39% of the cells and costs 2.3 points in all six seeds, though no frame-level number of the exam depends on where a mask lies; the whole silhouette adds cells that show only van and costs 1.6. The sign of a clause is not known in advance. The softened mask of the default filter removes 18% of the positive cells and adds none: an edge pixel that is exactly half covered, which the lab's rule counts, reads just under one half once blurred, and thin parts are mostly edge, so 32% of the head's cells go, 24% of the feet's, and 75% of all that go lie on the outer columns. The detector taught on it is −1.8 points better on the exam, in all six seeds.

And a rule never reorders detectors. Across the 16 programs of Lesson 2 (seed 1, exam miss from 46.7 to 80.5%) and the 11 designs of Lesson 6, the order under each of the seven rules is the order under the exam's (rank correlation at least 0.998), and any pair that swaps places is within 0.6 points under one of the two rules. What reorders two detectors is the population. The amodal-taught detector misses −2.8 points fewer of the rare case than the program's and 1.6 more of the exam's pedestrians, in all six seeds; the default filter's does the opposite, −1.8 better on the exam and 1.2 worse on the rare case. Whose misses are averaged is part of the definition too.

5 · The contract

A clause that is a habit of one author's code is invisible until two pipelines meet. The evidence for the label stage is what the benchmark wrote down, an afternoon of reading, and what a renderer returns for scenes whose answers are known in closed form: a few scenes, no labels and no street. So write each clause as a setting and a test.

The listing under the dial holds the counting rules and two of these tests. The coverage test places a 1.2 × 5 pixel sliver, whose area is exactly 6.0 px², at sixteen sub-pixel offsets and reports the worst error of the counted area; the distance test reads a frame's depth pass on the ground and reports its largest deviation from the ground-plane law. The lab passes it to 0.0000, and a range map in its place fails it by 0.165. At two rays a 6 px² sliver is counted within 1.5 px², 25% of its area, so "at least 6 px²" is a coin toss for it: the rule is applied to a number only that accurate, and the contract says so.

6 · A renderer we did not write

All six clauses were settled in street.js by one author, who chose every convention. A production pipeline settles them with a renderer's defaults. In Blender 5.2 LTS, whose defaults were checked by running it, the Depth pass is the distance along the camera axis and the Mist pass the Euclidean distance; the Object Index pass is point-sampled at the pixel centre, a hard label with no coverage, where Cryptomatte stores coverage-weighted ids; and the default pixel filter, Blackman-Harris 1.5 px wide, lets a step edge that lies on a pixel boundary leak 12% into the next pixel, where the box filter gives exact area coverage. Each is a setting of a clause above, chosen by someone else.

The filter is the worked example. A Gaussian blur of standard deviation σ, read at a pixel centre half a pixel from the edge, leaks Φ(−0.5/σ) (Φ is the standard normal distribution function), so a leak of 12% means σ = 0.43 px. The lab's own pixel is a box of standard deviation 0.29 px; variances add, so the default filter's extra blur is 0.31 px. Read the exam's pedestrians' area through each of these, same detector (seed 1), the exam's rule:

None of these is wrong. Each is a different definition chosen by a default, and each changes a number a team will report, 54 pedestrians of the exam's frames and six points of the rare case for the filter alone, or a detector it will train, in a direction nobody chose.

Common mistakes / failure modes

Checkpoint exercise

Where this points next

The clauses of a label are written down and measured. The visibility rule moves the number a team reports by up to 25 points and the detector by about one: the same detector scores 44.7% or 51.8% on the exam depending on the rule. What moves a trained detector is a clause nobody argues about, a mask half a pixel off, the hidden part of a silhouette, a softened edge, by 2.3 and 1.6 points worse and −1.8 better. All of it was settled in a renderer we wrote ourselves, where we chose every convention, and a production renderer's defaults decide, silently, how a sliver is counted: an exam's pedestrians change status in 5.2% of cases when their area is read from an index pass and in 4.1% through the default filter. A production pipeline uses a production renderer whose conventions we did not choose. How do we make a renderer we did not write obey the same contract?

Interview prompts

• Two papers report miss rates for the same detector five points apart. What do you check first? (§1 — the counting rule: visible fraction, minimum area, whether hidden pedestrians count; seven rules moved one detector's scores 7.1 points.)
• A synthetic label stage changed and no model moved. Is the stage irrelevant? (§2 — it acts through the training door only where partly hidden pedestrians exist; the scoring door was not tested, and it moves what is reported.)
• Would you teach an amodal mask to a detector scored on visible pedestrians? (§4 — it contradicts the pixels: false alarms move to the van and the exam loses 1.6 points, though the rare case gains.)
• What is the difference between depth and range, and when does it matter? (§3 — along the axis against along the ray, r = z√(1 + tan²φ + ρ²); it matters at any distance gate, case table or weight: 17% of close step-outs.)
• How do you test a mask-building convention without a reference renderer? (§5 — scenes with closed-form answers: a sliver of known area at sub-pixel offsets, a ground plane for the depth law, a target at a known place.)
• Why does a coverage threshold at 6 px² need a tolerance? (§5 — at 2 × 2 rays the counted area of a 6 px² sliver errs by up to 1.5 px²; only near the line do decisions flip.)

Companion reads: Computer Graphics · 06 Sampling and antialiasing (coverage, rays per pixel and pixel filters), Computer Vision · 09 Object detection (overlap and the matching rule) and Computer Vision · 04 Cameras and projection geometry (pixel centres and the pinhole).