Sensor measurement models

A scene engine produces a world trajectory. A sensor does not read that world directly: it turns radiance, propagation time, wavelength, and motion into finite, delayed, noisy numbers. This lesson builds that observation process one transformation at a time.

0 · The contract: what is a sensor simulator for?

A useful simulator is not the one with the most effects. It is the simplest observation model that preserves the evidence, uncertainty, and failure conditions relevant to the downstream decision. If a detector relies on four-pixel pedestrians, the point-spread function and resize kernel may matter greatly. If the task is lane geometry near the image edge, distortion calibration may dominate. If a planner consumes LiDAR sweeps, firing time and ego-motion may matter more than photorealistic intensity.

From the model’s perspective, the input is a random variable, not a picture. A compact description is

where W_k is the acquisition interval (or set of per-ray times), c_k is sensor control state such as exposure and gain, and θ contains calibrated intrinsics, optics, noise, scan pattern, and processing. This notation makes three facts explicit:

1. An observation can depend on many world states. Exposure integrates motion; a scanner fires rays sequentially.
2. The process is stochastic. Two captures of the same static world need not have identical electrons, events, detections, or compressed bytes.
3. Sensor control is stateful. Auto-exposure, autofocus, temporal denoising, and clock correction can depend on earlier observations.

A running rig

We will carry one case through the lesson. At dusk, an autonomous shuttle approaches a parked van just as a pedestrian emerges from behind it. At a defined checkpoint, the policy must decide whether to brake or coast; later lessons can branch both futures from exactly that checkpoint. The front rig has a 1920×1200 rolling-shutter RGB camera with 6 μm pixels, a 12 mm lens, 4 ms exposure, and 12 ms top-to-bottom readout; a 64-channel spinning LiDAR at 10 Hz; and a 77 GHz FMCW radar with 4 GHz sweep bandwidth. Every sensor has a calibrated rigid transform from sensor coordinates to the shuttle body, T_bs, plus a clock model. The engine supplies continuous—or safely interpolated—body, van, and pedestrian poses T(t).

This scene exposes why “render the same scene in three modalities” is not enough. The camera receives few photons on the pedestrian’s shadowed clothing and may skew their boundary as the shuttle yaws. LiDAR beams strike the occlusion edge at different moments. Radar receives a weak moving-human return near a strong static reflector. Whether the pedestrian is observable before the brake/coast checkpoint depends on each sensor’s measurement process, not only on whether the world state contains a pedestrian.

1 · Linearized thinking: follow the evidence forward

This order is more than presentation. Operations generally do not commute. Blur followed by clipping is not clipping followed by blur. Poisson noise added to linear photoelectrons does not have the same distribution as Gaussian noise added to gamma-encoded pixels. A rolling-shutter image cannot, in general, be recovered by applying a two-dimensional warp to a finished global-shutter image because occlusion and illumination can change during readout.

The practical rule is: place an effect where its cause occurs. Approximate aggressively when an experiment shows the approximation is harmless, but record the approximation and its validity range.

The fundamental mismatch: state is not observation

The engine has privileged state. At any time it can answer: “the pedestrian’s hip is at this exact 3D coordinate; this triangle belongs to instance 17; its surface has this material; its velocity is 1.8 m/s.” The deployed model receives none of those variables directly. It receives a finite array of camera values, a sparse set of LiDAR returns, or radar samples after those facts have been filtered by visibility, physics, time, and electronics. Synthetic vision therefore has a precise fundamental problem: turn privileged state into the same partial evidence available at deployment without leaking privileged state through a shortcut.

To see why a sensor is an operator over space–time, begin with a physical field F(x,ω,λ,t;s) produced by world trajectory s(t). It may represent optical radiance, a returned laser field, or an RF echo. Measurement element i accepts only part of that field. Its aperture, spectral response, beam, pixel footprint, and acquisition schedule define a kernel k_i and support W_i:

The first line transports, selects, and integrates physical evidence. The second transduces it with stochastic device variables ξ_i and quantizes it with Q_i. A camera row has a rectangular time support; a LiDAR return uses a narrow transmitted pulse at one firing angle; radar coherently combines phase across a chirp sequence; an event pixel reports a threshold crossing rather than an integral value. These are different kernels, so one modality is not merely another modality with a different file extension.

Every integral and sampling step is many-to-one. Spatial integration removes detail smaller than the footprint. Spectral integration makes distinct spectra share one channel value. Exposure discards the exact ordering of changes inside the interval. Occlusion removes all direct evidence about a hidden surface. Clipping maps every signal above a threshold to one number. Quantization maps a continuum to a bin. Once discarded, information cannot be recovered by a prettier renderer or larger neural network. A learned model can infer a likely hidden cause from priors and history, but that is inference, not measurement.

Thought experiment: two worlds, one observation

At the decision checkpoint, construct World A with the pedestrian entirely behind the opaque van and World B with no pedestrian. Give both worlds the same visible van surfaces, illumination, weather, rig motion, and sensor noise seed. Suppose no LiDAR beam reaches around the van and the pedestrian creates no separable radar return. The complete current camera–LiDAR–radar observation is then identical in A and B even though the engine’s semantic state differs.

No perception architecture can determine from that observation which world it occupies. If a dataset labels the single current observation “brake” in A and “coast” in B, it has assigned contradictory targets to identical evidence. The solution is not more photorealism. The task needs earlier history, a belief over hidden occupancy, an information-gathering action, or a risk-sensitive policy that brakes when the hidden-pedestrian probability and cost justify it. This is the first-principles meaning of partial observability.

Now reveal two centimeters of the pedestrian. The worlds cease to be exactly observationally equivalent, but the distinction may still fall below camera resolution, between LiDAR beams, or under radar clutter. “Different state” does not imply “decision-relevant observable difference.” A sensor simulator should preserve this boundary rather than handing the model perfect engine truth.

An information-loss ledger

Why simulator-only shortcuts are so dangerous

Learning finds any stable predictive cue, not the causal cue the dataset author intended. Suppose every pedestrian asset was authored by one vendor and carries a faint material-frequency pattern, or semantic instances receive subtly different antialiasing, while backgrounds come from another pipeline. In synthetic training, that pattern may predict “pedestrian” more reliably than shape under occlusion. Empirical risk minimization is correct to use it. In the real camera, the correlation disappears, so the learned rule fails.

This failure is not repaired by matching image-level averages. Test for shortcuts by intervention: keep geometry and labels fixed while changing asset source, material IDs, render path, seeds, compression, and background; then keep nuisance style fixed while changing pedestrian presence. A valid measurement pipeline should make the prediction follow decision evidence, not engine bookkeeping. Never expose instance colors, depth precision, object order, or deterministic noise patterns to the input path unless the deployed sensor exposes an equivalent signal.

2 · From world light to radiance at the entrance pupil

Camera simulation starts with radiometry. The terms are easy to confuse, so define both meaning and units:

In lossless free space, radiance is conserved along a ray. That makes it the natural bridge between a renderer and a lens. At a pixel, the lens integrates radiance over the pupil’s solid angle and over the pixel footprint. In a simplified thin-lens system the resulting irradiance scales approximately as 1/N², where N=f/D is the f-number, but off-axis vignetting, transmission, focus, and pupil geometry modify this relation.

Let E_λ,i(t) be the spectral irradiance that the ideal lens geometry would deliver at pixel i before wavelength-dependent losses. For pixel area A_px, optical transmission τ(λ), quantum efficiency QE_i(λ), and exposure interval [t_o,t_c], the expected number of signal electrons is

The factor λ/(hc) converts optical energy at wavelength λ into photon count; quantum efficiency converts incident photons into expected electrons. Equivalently, one may use a calibrated responsivity in electrons per joule. Do not multiply by both a responsivity that already includes photon conversion and λ/(hc).

Worked example: photons, electrons, noise, and SNR

Consider one 6 μm × 6 μm green photosite in the running rig. Suppose the wavelength-integrated irradiance near 550 nm is approximately 0.01 W m⁻² during a 5 ms calibration exposure, and approximate all photons by 550 nm. The pixel area is 36×10⁻¹² m², so incident energy is

A 550 nm photon has energy hc/λ≈3.61×10⁻¹⁹ J. The pixel therefore receives about 4,985 photons. At 60% quantum efficiency, μ_sig≈2,991 electrons. If dark current contributes 0.5 electron and read noise has standard deviation 3 electrons, the approximate variance is 2,991+0.5+3²=3,000.5 electrons squared. The signal-to-noise ratio is

Doubling exposure nearly doubles signal. In the shot-noise-dominated regime, noise grows only as the square root, so SNR improves by about √2. But exposure can also increase blur or saturation; it is not free signal.

3 · Optics and projection: where does the evidence land?

Choose conventions before equations. Let camera coordinates use +Z forward, +X image-right, and +Y image-down. Let pixel coordinates refer to pixel centers, with the top-left center at (0,0). Other conventions are valid, but silent mixing is not.

A world point becomes a camera point through the world-to-camera extrinsic transform:

Normalized pinhole coordinates are x=X_c/Z_c and y=Y_c/Z_c. With focal lengths in pixels f_x,f_y, principal point (c_x,c_y), and optional skew s,

Because f_x=f/p_x, the running rig’s 12 mm lens and 6 μm pixel pitch give f_x≈2,000 px. With c_x=960,c_y=600, a point at (2,0.5,20) m projects to (1160,650) px before distortion.

Radial and tangential distortion

A common Brown–Conrady model first distorts normalized coordinates. Let r²=x²+y² and a=1+k₁r²+k₂r⁴+k₃r⁶. Then

Apply K to (x_d,y_d). With only k₁=−0.12, a point at normalized x=0.5,y=0 moves from u=1960 to u=1930: a 30-pixel inward shift. Distortion is therefore not a cosmetic post-effect for edge geometry.

Real optics also distribute a point over a point-spread function (PSF). Focus distance, aperture, diffraction, aberrations, and motion determine that footprint. A spatially varying measured PSF or modulation transfer function can be more task-useful than an elaborate uncalibrated lens model. Flare and ghosting are nonlocal: a bright source can alter distant pixels, so a small local blur kernel cannot express them.

4 · Exposure and shutter: a frame is a space–time measurement

An ideal instantaneous image samples one state. A real camera integrates irradiance over time. For a global shutter, all pixels share an exposure interval:

For a rolling shutter with row time Δt_row, row v opens at

Its radiance must be evaluated throughout [t_open(v),t_open(v)+T_exp]. A practical renderer may use several temporal samples per row band, but it should converge-test both number of bands and exposure samples around fast motion, thin objects, flashes, and occlusion boundaries.

Worked example: rolling-shutter skew and motion blur

The rig reads 1,200 rows in 12 ms, so Δt_row=10 μs. The midpoint of row 0 is 2 ms after frame start; row 1,199 is centered about 13.99 ms after start. During a 90°/s yaw, their viewing directions differ by about 1.079°. Near the optical axis, that is roughly f_xtan(1.079°)≈37.7 pixels. One pose for the whole frame cannot explain both rows.

Separately, a target moving sideways at 10 m/s at 20 m range moves about

during a 4 ms exposure. Readout duration governs row-to-row skew; exposure duration governs within-row blur. They are different parameters and create different failures.

In the shuttle case, the pedestrian’s leading edge may become visible halfway through readout. Upper rows can record the van before emergence while lower rows record part of the pedestrian. A finished-frame warp cannot recreate the newly revealed surface or its correct blur; the renderer must evaluate visibility during acquisition.

Auto-exposure adds a feedback loop: a controller estimates brightness from prior frames and chooses exposure, analog gain, aperture, or illumination. If the real system darkens after leaving a tunnel over several frames, independently choosing a plausible exposure for each synthetic frame will miss the temporal transient. Model the controller when the downstream system sees or reacts to it.

fr       = row / rows                      // 0 at top, 1 at bottom
captured = x_ref + speed · fr · readout    // rolling: each lower row is later
skew     = speed · readout                 // total top-to-bottom shift (px)
global model ⇒ captured = x_ref (straight) ⇒ sim↔real mismatch = skew

5 · From photons to RAW numbers

After optical integration, the sensor converts photons into charge and charge into digital numbers. A compact pixel model is

Here ε_PRNU is photo-response non-uniformity, μ_dark includes temperature- and exposure-dependent dark current, n_read includes read and row/column noise in electrons, γ is conversion gain in digital numbers per electron, b_DN is black offset, and B is ADC bit depth. Full-well capacity can clip charge before ADC clipping. Real cameras may switch conversion gain, so a single noise curve over all ISO settings is often wrong.

Shot noise is Poisson: its variance equals its expected count. At high counts it resembles a Gaussian with signal-dependent variance; in shadows, the discrete distribution and read noise matter. Fixed-pattern terms are persistent across frames and therefore learnable as texture. Resampling fresh independent “sensor noise” every frame erases that temporal and spatial structure.

The ISP is part of the observation model

A Bayer array samples only one color filter at each photosite. Demosaicing invents missing channels from neighbors, creating color zippering or false detail near edges. Denoising and sharpening modify small-object evidence. Temporal ISP stages can produce ghost trails. JPEG or video codecs create block and motion artifacts. If production inference sees processed YUV or RGB, reproducing its actual ISP and resize path is generally more relevant than exporting pristine renderer RGB. If inference sees RAW, preserve the mosaic and metadata rather than passing through a display transform.

Color encoding is nonlinear. For normalized linear sRGB channel C, the forward display encoding is 12.92C when C≤0.0031308, otherwise 1.055C^1/2.4−0.055. Noise, blur, and alpha compositing should not be moved across this transform without understanding the change in statistics.

6 · LiDAR: time-stamped rays, not a depth image

A time-of-flight LiDAR emits a pulse and estimates round-trip delay. For propagation speed c,

A useful return record is (r,α,β,I,t_emit,t_return,channel,returnIndex). Angles and time follow a device firing table, not an image grid. Transforming a return to world coordinates uses the sensor pose at its acquisition time:

Whether t_j means emission, reflection, or reception must be declared; for ordinary automotive ranges the intra-ray travel-time motion is small but not mathematically zero. Across a sweep, it is often scan time—not light travel time—that causes large distortion.

Worked example: timing precision and motion distortion

A target at 30 m returns after 2r/c≈200.14 ns. A 200 ps timing error corresponds to range error c·200 ps/2≈3.0 cm. Meanwhile a 10 Hz spinning sensor takes 100 ms per revolution. A rig moving at 15 m/s travels 1.5 m during one sweep. If every point is transformed with the end-of-sweep pose, static walls bend and moving objects receive an additional, different distortion. Deskewing requires the rig trajectory; it does not correct an independently moving target unless that target motion is also modeled.

Important LiDAR effects include finite beam divergence and spot shape, per-channel angle/range offsets, minimum range, range-dependent precision, partial beam hits at depth discontinuities, dropout, saturation, multiple returns, atmospheric attenuation, and multipath. Return intensity depends on wavelength, incidence, range, receiver gain, and material response; visible RGB albedo is not a calibrated substitute. A depth-buffer-to-point-cloud conversion can be an intentional low-fidelity baseline, but it misses native angular sampling, per-ray timing, mixed pixels, and return logic.

At the van boundary, one finite beam may cover both the van and emerging pedestrian. Depending on waveform and return logic, the shuttle may receive the stronger surface, first return, two returns, or no accepted pedestrian return. That conditional behavior is more relevant to the braking decision than a uniformly dense, perfectly aligned point cloud.

7 · Radar: delay, Doppler, angle, and ambiguity

Automotive FMCW radar transmits frequency chirps and samples complex in-phase/quadrature (IQ) echoes. A simplified target contributes a delayed, attenuated, phase-shifted copy. Round-trip delay gives range r=cτ/2. For a chirp slope S, the range beat frequency is approximately f_b,r=2Sr/c; target motion also affects beat frequency and must be separated using multiple chirps. Monostatic Doppler is

under a declared sign convention. Radar directly observes radial velocity, not the full 3D velocity vector. Angle is inferred from phase differences across antennas and has finite resolution and sidelobes.

Worked example: range resolution and Doppler

At 77 GHz, wavelength is about 3.893 mm. A target closing at 20 m/s has Doppler magnitude 2×20/0.003893≈10.27 kHz. With 4 GHz swept bandwidth, ideal range resolution is c/(2B)≈3.75 cm. This does not mean every returned range is accurate to 3.75 cm: SNR, calibration, windowing, multipath, bin interpolation, and target extent affect error. Maximum unambiguous range and velocity depend on sampling rate and chirp schedule.

Choose the output level to match the production interface: raw IQ, range–Doppler/angle tensors, clustered detections, or tracks. A geometry ray caster with one hit per visible triangle cannot naturally reproduce sidelobes, speckle, interference, multipath ghosts, elevation ambiguity, clutter, or missed detections. Yet full electromagnetic simulation may be wasteful for a camera-only task. Match fidelity to the consumer and validate conditional distributions such as detection probability versus range, aspect, and occlusion.

For the shuttle, the parked van can create a strong zero-Doppler response and multipath while the pedestrian contributes a weaker nonzero radial component. A model trained on one clean radar point per visible object may learn an unrealistically easy separation. Conversely, if the deployed planner consumes only vendor tracks, matching the tracker’s miss and false-alarm behavior may be more useful than simulating raw fields.

8 · Event cameras: threshold crossings in log intensity

An ideal event pixel stores a reference log intensity and emits an event (u,v,t,p) when the change reaches a polarity-dependent contrast threshold:

After an event, the ideal reference advances by pC, so a large continuous change can emit several events. Suppose intensity rises from 100 to 125 arbitrary linear units. The log change is log(1.25)=0.223. With C⁺=0.10, the ideal pixel emits two positive events and retains about 0.023 log units of residual. A fall from 100 to 80 gives log(0.8)=−0.223, or two negative events at a symmetric threshold.

Real event cameras add per-pixel threshold variation, refractory time, timestamp quantization, latency, leak, background activity, low-light shot noise, and bandwidth saturation. Differencing two RGB frames cannot recover the times—or even the correct count—of crossings between them. Render radiance at enough temporal resolution, adaptively find crossings, or use an analytic continuous signal where possible. The offset ε is a numerical model choice, not a replacement for a calibrated low-light response.

9 · One rig means geometry, clocks, and correlated observations

Multimodal data is not a bag of independent files. Let T_bs be each sensor’s extrinsic relative to the body, and let its local timestamp τ_s map to reference time as

where a_s models clock-rate error, b_s offset, and j_s timestamp jitter or latency uncertainty. For synthetic data, one can set clocks perfectly—but doing so may train a fusion model that depends on impossible alignment. Calibrate real offset and drift, then simulate their supported distribution if sensitivity warrants it.

In the running shuttle rig, an RGB frame nominally stamped at 12:00:00.100 may integrate rows over 12 ms while a LiDAR sweep spans 100 ms and radar chirps form a frame over another interval. “Nearest timestamp” is not a physical synchronization contract. Store start/end, row or firing schedule, time reference, and interpolation method. Weather and object trajectories must be shared across sensors so dropout and visibility can be correlated for the right causes. Define the brake/coast checkpoint on the reference clock; otherwise two dataset samples described as “the decision frame” can contain different amounts of pedestrian evidence.

10 · Analytic calibration first; learned residual second

Some structure is known and identifiable: projective geometry, exposure intervals, photon counting, range-delay conversion, antenna layout, and coordinate transforms. Encode that structure analytically. Then use data to estimate parameters. A learned residual is valuable for effects that are expensive or incomplete—complex flare, conditional noise, radar detection residuals, or ISP texture—but it should not silently rewrite geometry or timing.

This division is an authority boundary, not a claim that hand-written physics is always more accurate. The analytic core owns invariants for which the device supplies an auditable coordinate system: a pixel ray follows calibrated projection; a LiDAR firing has a time and direction; round-trip delay maps to range; exposure cannot include a state outside its interval. These constraints extrapolate beyond the exact calibration images because they express how the instrument is constructed. They also keep measurement geometry consistent with labels.

The residual owns the discrepancy after that core: conditional noise correlations, imperfect material return, flare structure, or a vendor processor that is difficult to reproduce. One useful factorization is

where C_θ is the calibrated causal chain and r_φ is a constrained learned correction. Addition is illustrative: a dropout model may instead sample acceptance, and an ISP residual may transform RAW. The important point is that the residual is conditioned on the physical signal and controls, cannot access semantic labels or hidden instance IDs, and is prevented from changing protected geometry or timestamps.

Why not learn the entire mapping from state to realistic sensor output? First, it is underdetermined: a perceptual loss may be reduced by shifting, erasing, or hallucinating the pedestrian. Second, it is hard to identify which learned behavior is physics and which is training-set correlation. Third, it can extrapolate arbitrarily when range, weather, motion, or hardware changes. Finally, if it moves evidence without moving labels, it corrupts supervision. The analytic core makes errors localizable; the residual spends learning capacity only where the tractable model is wrong.

Train residuals on paired or tightly matched calibration data when possible, condition them on causal variables, and constrain their outputs. Validate geometry, temporal consistency, physical range, and failure slices—not only perceptual similarity. A neural translator that makes synthetic frames “look real” can hallucinate or remove the very object to be labeled. Keep the clean analytic output beside the residual output so the contribution remains auditable.

Fidelity is a property of a decision, not an image

Let the downstream system choose action a=π(o) and incur cost J(a,s). A sensor mechanism matters when changing its parameter within calibrated deployment support changes the action, uncertainty, or cost distribution. In shorthand, its decision sensitivity is the change

measured under controlled scenario and training seeds. This is not a derivative requirement; discontinuous effects such as clipping or detection thresholds are often exactly what matters. It is a reasoning tool: if plausible distortion changes lane localization and braking, model it; if a costly diffraction simulation changes neither evidence nor policy, defer it. Fidelity is sufficient when remaining measurement error no longer changes conclusions within the target operating domain and uncertainty budget.

Worked sensitivity example: 100 ms can reverse the braking outcome

Suppose the shuttle travels at 10 m/s, the controller adds 150 ms of sensing-and-actuation latency after a brake decision, and comfortable emergency deceleration is 6 m/s². Ignoring jerk, the distance required after evidence triggers the decision is

In an ideal sharp, bright synthetic camera, suppose the emerging pedestrian crosses the detector’s evidence threshold when the shuttle is 10.5 m from the conflict point. The nominal stopping margin is only 10.5−9.83=0.67 m. In the real dusk camera, low photon count, motion blur, and temporal denoising delay a stable detection by 100 ms. The shuttle travels 1.0 m during that delay, leaving 9.5 m: now it is about 0.33 m short under the same simplified braking model. A visually modest sensor discrepancy has reversed a safety outcome.

The right experiment is causal. Render the same scene trajectory and labels with calibrated dusk noise/ISP on and off; retrain or evaluate under a declared protocol; measure detection-time distribution, confidence, brake time, and minimum separation. Then vary exposure or denoising latency across measured support. If the outcome changes, the mechanism deserves fidelity and coverage. If adding chromatic aberration changes pixels but not detection time, calibration, or minimum separation across relevant cases, it is presently lower priority.

11 · An implementable sensor pipeline

A declarative configuration makes units and assumptions reviewable. This abbreviated example is not a universal schema, but it illustrates what should be explicit:

rig:
  reference_clock: tai_seconds
  body_trajectory: cubic_se3             # interpolation is versioned
sensors:
  rgb_front:
    type: rolling_shutter_camera
    extrinsic_T_body_sensor: [...]        # named axis convention
    resolution_px: [1920, 1200]
    pixel_pitch_um: [6.0, 6.0]
    focal_length_mm: 12.0
    principal_point_px: [960.0, 600.0]
    distortion: {model: brown, k: [-0.12, 0.01, 0.0], p: [0.0, 0.0]}
    exposure_ms: 4.0
    row_readout_us: 10.0
    spectral_response_asset: qe_rgb_v3.csv
    full_well_e: 18000
    read_noise_e: 3.0
    adc_bits: 12
    isp_asset: production_isp_v7.json
  lidar_top:
    type: pulsed_scanning_lidar
    extrinsic_T_body_sensor: [...]
    firing_table_asset: lidar64_revC.csv
    rotation_hz: 10.0
    range_m: [0.8, 120.0]
    timestamp_reference: emission
  radar_front:
    type: fmcw_radar
    carrier_ghz: 77.0
    bandwidth_ghz: 4.0
    waveform_asset: chirps_v2.json

Parameter files are dataset inputs and must be versioned like meshes or code. Record units in names or schema, reject missing conventions, and distinguish a scalar nominal value from a sampled distribution.

Pipeline pseudocode

for scenario in manifest:
    state = load_continuous_trajectory(scenario)
    rig   = load_and_validate_calibration(scenario.rig_version)

    for acquisition in make_sensor_schedule(rig, scenario.time_range):
        sample_rng = keyed_rng(scenario.id, acquisition.sensor, acquisition.id)
        times = acquisition.exposure_or_ray_times()
        poses = interpolate_poses(state, times)

        physical_signal = transport_signal(
            state=state, poses=poses, sensor=acquisition.sensor,
            times=times, spectral_or_waveform_mode=True)
        raw = transduce_sample_clip_quantize(
            physical_signal, acquisition.controls, sample_rng)
        observation = apply_device_processing(raw, acquisition.metadata)

        labels = derive_labels(
            state=state, support_times=times,
            convention=acquisition.label_contract)
        assert_geometry_timing_and_range_invariants(observation, labels)
        write(observation, raw, labels, provenance=full_config_and_seeds)

A keyed random generator makes a sample reproducible without forcing every effect to share a random stream. Preserving RAW, processed output, clean latent signal, controls, time support, and seed allows a failed example to be replayed and decomposed.

12 · Calibrate, ablate, and diagnose

Calibration experiments

1. Camera geometry: capture a calibrated planar target across the field and focus range; fit intrinsics and distortion; inspect held-out reprojection residual versus radius, temperature, and focus.
2. Camera response and noise: use dark frames across exposure, temperature, and gain; uniformly illuminated flat fields across intensity; estimate black level, hot pixels, fixed pattern, saturation, photon-transfer curve, and gain-dependent read noise.
3. Optics and ISP: use slanted edges or point sources for spatial PSF/MTF, bright sources for flare, color targets under measured illuminants, and RAW/processed pairs to separate sensor from ISP.
4. Timing: image a time-coded LED or moving edge to estimate exposure, row readout, latency, and timestamp convention. Rotate the full rig to estimate cross-sensor offsets from aligned motion.
5. LiDAR: measure planes and edges at multiple ranges/incidence angles; use reflectance targets, fog/rain when required, and moving platforms to verify firing order and deskew.
6. Radar: use corner reflectors at known range/angle, controlled radial motion for Doppler, empty-scene recordings for clutter, and multi-object setups for resolution and ghosting.
7. Events: use controlled flicker and moving edges across brightness to fit positive/negative threshold distributions, latency, refractory behavior, and background activity.

Fit on one set of runs and validate on different runs, scenes, devices, and environmental conditions. Otherwise a flexible model can memorize the calibration fixture.

Ablation ladder

Hold scenario seeds, labels, training budget, and evaluation fixed. Add one mechanism at a time:

1. Ideal projection or ideal geometric rays.
2. Correct resolution, field of view, scan pattern, and time support.
3. Measured dynamic range, clipping, quantization, and baseline noise.
4. Optical/beam footprint and motion effects.
5. Production processing, dropout, clutter, or device residual.
6. Clock and calibration uncertainty within measured support.

Compare not just average accuracy but calibration, small-object performance, range/brightness/weather slices, temporal stability, and closed-loop outcomes. Remove an effect if it adds cost without repeatable benefit. Randomize a parameter only over credible support; an implausibly broad range may teach invariance, or may destroy useful evidence.

Failure diagnosis table

13 · The bridge to labels: observation time is label time

A label is only “ground truth” relative to a declared measurement. For an instantaneous pinhole frame, a 3D box can be projected at one timestamp. For the running rolling-shutter camera, row v sees the box over that row’s exposure. For LiDAR, a point’s semantic identity and object pose belong to that ray time. Radar Doppler supervision needs relative radial velocity during the coherent measurement interval. Event labels must respect each event timestamp.

There are several legitimate camera box contracts, but they answer different questions:

• Reference-time box: project object state at a named frame time, such as center of readout. Simple, but not the exact support of rolling rows.
• Per-row geometry: intersect projected geometry with each row at its time. Closest to the measurement and useful for geometric learning.
• Exposure-support box: bound all visible projections during exposure/readout. Matches all observed energy but may be larger than an instantaneous object.
• Visibility-weighted label: define the label from time-integrated visible evidence. Task-relevant, but requires an explicit threshold and implementation.

None is universally correct. Store the contract, reference time, exposure interval, scan table, pose interpolation version, and whether ego/object motion compensation was applied. For the 12 ms camera and 100 ms LiDAR sweep, a single shared timestamp field is insufficient.

For the shuttle case, freeze the pre-decision state and all sensor histories at the checkpoint before branching brake and coast. Observations that end after the branch action begins are action-conditioned future evidence, not pre-decision input. Keeping that boundary explicit prevents label leakage and lets later lessons compare counterfactual futures from the same information set.

14 · Exercises and self-test

1. Projection. In the running camera, where does (X,Y,Z)=(−1,1,10) m project before distortion? Assume f_x=f_y=2000, c=(960,600).
   Answer: x=−0.1,y=0.1, hence (u,v)=(760,800) under the stated +Y-down convention.
2. Exposure. A target at 25 m moves laterally at 5 m/s. Estimate blur for a 10 ms exposure and 2000 px focal length.
   Answer: 2000×(5/25)×0.01=4 px, assuming constant depth and velocity near the optical axis.
3. Shot noise. Ignoring read/dark noise, how does SNR change if expected signal falls from 10,000 to 100 electrons?
   Answer: Poisson SNR is μ/√μ=√μ, so it falls from 100 to 10.
4. LiDAR timing. What range bias corresponds to 1 ns round-trip timing bias?
   Answer: c·1 ns/2≈0.15 m.
5. Radar. Why can two vehicles with the same speed produce different Doppler?
   Answer: Doppler depends on the component of relative velocity along the radar line of sight; lateral motion can have near-zero radial velocity.
6. Events. With threshold 0.2, can a smooth doubling of intensity generate three positive events?
   Answer: log 2≈0.693, so the ideal model produces three events and leaves about 0.093 residual, assuming no refractory or bandwidth loss.
7. Design. Your lane model fails at the image edge but is stable across brightness. Which mechanism would you test first, and what held-out measurement would falsify it?
   Answer: Test intrinsics/distortion/crop consistency before a complex noise model; held-out reprojection residual versus image radius can falsify the hypothesis.
8. Fidelity. A camera-only classifier receives center crops from bright global-shutter images. Must the dataset include radar multipath and rolling-shutter simulation?
   Answer: No. Neither affects the consumed observation under the stated deployment. Add them only if the interface or operating support changes.

Interview prompts

• Why is additive Gaussian noise on sRGB usually insufficient? Photon noise is signal-dependent and occurs before clipping, mosaicing, nonlinear ISP, resize, and compression; real noise also has fixed and correlated components.
• What is the difference between rolling-shutter readout and exposure time? Readout determines row-to-row time offset; exposure determines each row’s integration and motion blur.
• Why not make LiDAR from a rendered depth image? A real scanner has a device-specific angular pattern, per-ray times, beam footprint, multiple-return logic, motion distortion, and dropout.
• What does radar Doppler directly measure? Relative radial velocity through phase/frequency change, not the full velocity vector.
• When is a learned sensor model appropriate? After preserving analytic geometry and timing, as a constrained residual for calibrated effects that the tractable physical model misses.
• How should fidelity be chosen? By calibration evidence and controlled downstream ablations on deployment-relevant slices—not by visual impressiveness.

Further reading

• EMVA 1288 — an industrial standard for characterizing image-sensor sensitivity, noise, and spatial non-uniformity.
• OpenCV camera calibration and 3D reconstruction — practical projection and distortion conventions.
• v2e: From Video Frames to Realistic DVS Events — an example of calibrated event-camera simulation and its nonidealities.
• RADIATE — multimodal radar data illustrating the importance of difficult weather and sensor-specific outputs.