Same scene, other pixels

Lesson 3 gave the program the street's camera and took the miss rate from 80.3% to 65.8%, but it still lights every frame with one sun at one brightness, while the street shows the same scenes at dusk, in glare and under cloud, and the label does not care. Which of the light's ten parameters must the detector see vary, and how far? Here one does, the brightness, because the camera turns light into noise; its range can be read from the sky of unlabelled frames. Widening the whole look past the street's range costs the program its own labels.

1 · One scene, many lights

The program of Lesson 3 has the street's camera and the light of SIM0: a sun 55° high behind the camera, one brightness, a white van, bright shirts, a blue sky. The gap shows in the camera's exposure control: over 1,000 street logs its gain runs from 1.9 to its cap of 8 and sits at the cap in 34% of them; over the program's frames (the 1,000 that Lesson 3 sent through this camera) it runs from 1.8 to 2.3, so the detector has never met a frame taken at the cap.

None of the ten enters the label, a function of the scene and the pinhole: which pixels the pedestrian covers, how much of the silhouette shows, whether the pedestrian counts. Draw 8 scenes under 8 looks each: the masks of one scene agree to the last bit (largest difference 0), and so do labels and areas, while the pictures differ. All ten are nuisances, factors the label does not depend on, the setting of augmentation theory, which averages over transformations that keep the data distribution, and with it the label relation, approximately invariant (Chen, Dobriban and Lee, 2020).

A learner cannot be told which factors are nuisances. It fits whatever separates its training pictures, and if the light never changes, how this pedestrian looks in this light separates them as well as what a pedestrian is. Only a scene shown in other light says that the factor moves and the label does not, and every factor shown costs frames: which of the ten does the detector need?

2 · Which lights the detector can tell apart

Take four looks from the street's range, each a single point: noon, the program's own (sun 55° high behind the camera, ambient 0.45, brightness 1); low (15° high, 90° to the side, ambient 0.30, brightness 0.7); against (30° high, 150° round, nearly facing the camera, ambient 0.30, brightness 0.8); dim (noon's geometry at brightness 0.15). Train the fixed detector on 800 frames of one look and grade it on the program's frames of each, at the threshold that lets 10% of that look's empty frames alarm. Cells are miss rates in per cent; no street, no real label.

Read it by columns. Among the three bright looks no off-diagonal cell exceeds 0.8%, though the sun moved from behind the camera to the side and into the lens: to this detector they are one look. The dim look is another world: detectors trained on the bright looks miss 14.7 to 25.4% of its pedestrians, the dim-trained one at most 4.8% anywhere else, so transfer runs from hard light to easy and not back. Training on 200 frames of each look mends the dim column to 5.9% but costs 2.0 points against the specialist: the first sight of the price of invariance.

Why brightness and not the sun? The detector reads bars and steps in both polarities, so a pedestrian against a road is an edge whichever side is lit; a sun in another direction only moves light and shade about. Brightness changes what the filters cannot read around, the noise. A pixel of radiance L collects ev·L electrons and its relative noise is √(ev·L + r²) / (ev·L), with ev = 600 electrons per unit radiance and read noise r = 3 electrons (Lesson 3): 7.6% at L = 0.30 and 22.2% at 0.045, the same road at brightness 0.15, 2.9 times as much, and auto-exposure lifts the picture back to the same average level with the noise in it. Light reaches the picture by two doors, pattern, which this detector ignores, and noise, which it does not.

3 · Where a nuisance stops being one

Why not show every nuisance as widely as it goes? Because at some brightness the pedestrian is no longer in the picture. The pedestrian's signal is the picture with the pedestrian minus the same picture without, as the camera records it, blur included: s_i at each pixel and channel. The camera adds independent noise of variance σ_i² = (ev·L_i + r²) / ev² there, L_i the radiance behind the pixel. The signal's length in units of noise is the contrast-to-noise ratio:

CNR = √( Σ_i s_i² / σ_i² )

It is the detectability d′ of an observer who knows the signal exactly, and under this noise model no detector that sees only these pixels does better: a matched filter run on 4 scenes of the lab's own noisy frames, at brightnesses from 0.02 to 0.5, measures a d′ within about 5% of it. A frame has K = 576 cells that can alarm and one empty frame in ten may, so each cell may alarm with probability 1 − 0.9^1/K: a threshold of z = 3.56 noise sd, confirmed by simulating the largest of 576 normals. The ideal observer finds a pedestrian of CNR c with probability Φ(c − z): half at 3.56, nearly all at 8.

The detector is not ideal. Grade the detector of the street's own look on the program's frames, drawn in the street's look (open pedestrians 8 to 16 m away), and bin its pedestrians by CNR. This is a different population and a different detector from Lesson 3's line of about 14, which was the street-trained detector on the street's own test frames, where some pedestrians step out from behind the van; the two lines need not agree:

A logistic fit of hits against log CNR crosses one half at c₀ = 8.5 (90% of 200 bootstrap fits lie between 6.5 and 10.4; only 23 pedestrians are under 12, so the line is a soft gauge). Widen every numeric range of the street's look 1.5 times about its centre, brightness down to 0.01: at the street's own width 0.8% of the program's pedestrians fall below the line, at 1.5 times 6.2%, positives with little evidence behind them that free labels cannot reveal. The empty dim frames cost as much: 17% of the frames are dim (brightness under 0.2), and at the threshold pooled over all frames an empty one alarms 25% of the time against 7% for a bright one, so the threshold sits too high for the bright frames and their miss rate rises from 7.7% to 13.6%. So a width has a limit the program can measure with no street: the share of pedestrians below the line, and its own miss rate, the exam run on frames it draws.

4 · The street's range, from pixels whose content is known

The street's range of brightness is in no label, but it is in pixels whose content is known without one. The top row is sky (the far wall ends below it): the brightness times the sky's colour. The bottom rows are road: the brightness times the ground albedo times the road's light, ambient + (1 − ambient) · sin(elevation). Read each band through the calibrated camera, DN^γ / (κ · gain) with the logged gain, which turns the picture back into scene radiance, and take the band's median, so that a tree, a van or a pole over part of it moves nothing.

The sky tells two things. Its colour, the channel ratios with the level divided out, names which of three skies the frame shows (blue, grey, warm; the program is the calibration standard of its own probes and supplies their colours): right in 99.9% of the 96% of frames it classifies. Its level, divided by the level the program's probe reports for that sky at brightness 1, is the frame's brightness: against the hidden truth (a lab privilege) it is 0.95 of it, middle 80% from 0.90 to 0.99, a little low because the blur leaks the darker wall into the top row. If the brightness is uniform on [lo, hi], its p-quantile is lo + p(hi − lo), so the 5% and 95% quantiles give the whole range and a few outliers move neither:

The street's own range, which no project could read, is 0.12 to 1.00. The sd shrinks like 1/√n; the offset of the upper end does not, for it is the probe's low reading. A hundred logs are enough.

The road tells less. Dividing its level by the brightness leaves G = ground albedo × road light, a product of three things it cannot separate: a road of albedo 0.30 under a 55° sun and ambient light 0.45 and one of albedo 0.45 under a 30° sun and ambient light 0.20 have the same G, 0.27 and 0.27. The probe reads them alike (ratio 1.00) and the wall behind barely tells them apart (1.17). The probes constrain these parameters jointly and identify only the brightness and the sky; the sun's bearing and the clothes have no probe. The principle is to match the distribution of what you can observe, only through a cause you can attribute: fitting a simulator's parameter distribution to real observations is the idea of Chebotar et al. (2019, who matched real-robot trajectories), here with pixel statistics as the observation.

L.uniformRange = function (v, a, b) {
  var qa = quantile(v, a), qb = quantile(v, b), span = (qb - qa) / (b - a);
  return [qa - a * span, qb + (1 - b) * span];
};
L.read = function (frame, tab) {
  var p = L.probes(frame), k = L.classify(p, tab), l;
  if (k < 0) return null;
  l = p.level / tab[k].level;
  return { sky: k, lum: l, G: p.ground / l };
};

  var db = L.blur(d, cam.blur), rb = L.blur(rest, cam.blur);
  for (i = 0; i < d.length; i++) { v = (cam.ev * Math.max(rb[i], 0) + cam.read * cam.read) / (cam.ev * cam.ev); s2 += db[i] * db[i] / v; }
  return Math.sqrt(s2);
L.zAlarm = function (K, fa) { return L.PhiInv(Math.pow(1 - (fa === undefined ? 0.1 : fa), 1 / K)); };

What to try. Open as set: width 1, 100 logs, the 3rd scene (a pedestrian 11 m away, CNR 42). That width's detector has an exam miss rate of 61.5% and an own miss rate of 0.4%; 0 of the 8 frames fall below the line, 0 missed. Slide the width to 0: the eight frames share one brightness, 0.53, yet the exam is 62.0% against 61.5%: the level mattered, not the width. Go to 1.5 and 3: the range reaches 0.01 and 1.8, the exam is 60.7% and 60.7% and the own miss rate at or below 2.5%, while the dashed curves, the whole look widened, climb to an own miss rate of 16.7% at 1.5. Switch the top row to the logs: width 0 alarms on 5 of the first eight frames, width 1.5 on 2, and over 1,000 logs the alarm rate falls from 74.2% to 30.6% while the exam does not move. The farthest scene (15 m) has CNR 20 against 42 at 11 m: distance costs CNR as dim light does (§6).

5 · How wide? The sweeps

Two sweeps scale the width s of a range about its centre: one for the brightness alone (the street's 0.12 to 1, kept above 0.01), the other for every numeric range of the street's look together (sun elevation and bearing, ambient fraction, brightness, ground albedo, stripes) with the street's clothes, vans and skies, a lab privilege. "Line" is the share of pedestrians below c₀. Cells are per cent, means of three seeds at 1,600 frames; the exam wobbles by about 1.5 points between seeds.

Brightness alone: level before width. The exam is flat in the width (the six values lie within 1.3 points) and the own miss rate stays at or below 2.5%: at 1,600 frames the width of the brightness range buys nothing. What helps is the level: the program's one brightness, 1, sat at the top of the street's range, and moving it to the centre, with no variation, takes the exam from 65.8% to 62.0% (3.8 points), the whole range then adding 0.5 more. The matrix says why: trained at 1 the detector misses 25.4% at 0.15, and the street's frames lie anywhere in the range, nearly all dimmer than 1; a brightness at the centre is at most 4.7 times from any of them, one at the top 8.3 times from the dimmest.

The whole look: a shallow U and a cliff. Widened together the look gives 60.9% at width 0.5, 61.6% at the street's own width and 63.4 to 63.6% beyond it, while the program's own miss rate, which needs no street, stays at or below 6.7% to width 1 and jumps to 16.7% at 1.5, as the share below the line goes from 0.8% to 6.2% and the dim frames lift the threshold (§3). That is a stopping rule that needs no real label: widen until the program's own error jumps. The alarm rate on 1,000 unlabelled frames, a meter, does not give it: it falls from 74.2% to 30.6% as the brightness alone widens to 1.5 and from 25.5% to 17.8% as the whole look widens from 1 to 2, while the exam stops improving. It says how alien the frames are, not whether more variety helps the labels.

The price of invariance. Whole look, width 3 against width 1: exam 69.2 against 66.2% at 200 frames, 64.6 against 64.1% at 400, 62.6 against 62.7% at 800 and 62.9 against 61.6% at 1,600 (one seed below 1,600). The exam gap is a few points at 200 frames and within the seed wobble beyond; the wide program's own miss rate is higher at every size, 18.2 against 12.3% at 200 and 12.8 against 6.7% at 1,600.

6 · The price, and what the repaired pixels leave

The program built from the evidence draws the brightness from [0.12, 0.94], read from the sky of 956 usable frames of 1,000 unlabelled logs; nothing else changes. Trained on 1,600 frames and graded with the Lesson 1 exam (three seeds), it misses 61.2% of the street's pedestrians, against 65.8% for the program before the repair (aaba) and 61.6% for the exact-stage look (abba): 0.4 points from the exact one (a lab comparison: a project cannot build abba), inside the 1.5-point wobble between seeds.

What the repaired pixels leave is the scene. With the light right, repairing it is worth more: the scene stage was worth 5.5 points while the pictures were wrong (baaa against aaaa) and is worth 14.8 now (bbba against abba). And the scene is where the program is silent. It draws every pedestrian 8 to 16 m from the camera, in the open, none stepping out from behind the van: of 600 program pedestrians that count, the nearest is at 8 m, the farthest at 16 m, 0% step out. The street's are weaker: the median contrast-to-noise ratio of its exam pedestrians is 22 against 38 for the program's, 1.7 times as strong, and the 5% quantile 7 against 17, below the line 8.5 for the street and above it for the program. A median hides which cases are missing and how many, and a project has no labels of the street's pedestrians to find out.

Common mistakes / failure modes

Checkpoint exercise

Where this points next

The light is repaired as far as this exam can feel it: a program shown the street's range of brightness, read from the sky of unlabelled frames, misses 61.2% against 61.6% for the exact-stage look and 65.8% before. With the pixels right, a scene repair worth 5.5 points while the pictures were wrong is worth 14.8 now, and the scene is where the program is silent about whole kinds of case: every pedestrian it draws stands in the open, 8 to 16 m from the camera, 1.7 times as strong as the street's median pedestrian. Which real cases does the scene never draw, and how do we count them?

Interview prompts

• What is a nuisance factor, and how does a learner come to depend on one? (§1: a factor the label does not depend on; a learner fits whatever separates its training pictures, so a factor that never changes is as good a feature as the object.)
• How do you decide which nuisances to randomise? (§2: train under one setting, grade under another on free labels; randomise those with a large off-diagonal miss rate.)
• Why can randomising a nuisance more widely make synthetic data worse? (§3, §5: some frames hold a pedestrian the noise swallows, and the empty frames among them lift the alarm threshold for the rest.)
• How would you set the range of a lighting parameter without labelled real data? (§4: read sky and road through the calibrated camera and take a robust range.)
• What can a sky-and-road probe not tell you? (§4: the road level is a product of three causes; the sun's bearing and the clothes have no probe.)
• Why is the program's own error a stopping rule for widening, and the alarm rate on unlabelled frames not? (§5: own error jumps when frames leave the pixels; the alarm rate keeps falling as the program looks more like the street.)

Companion reads: Computer Graphics, from first principles (the terms of a look) and Computer Vision · 07 Training CV models (augmentation).