Labels, synchronization, provenance, and QA

A simulator can reveal every internal variable and still export the wrong dataset. Ground truth becomes useful only when its meaning, coordinates, time support, visibility, lineage, and tolerances form one testable contract.

Learning path and running example

We will proceed in the order in which information exists: measurement event → semantic contract → transform graph → geometric labels → temporal alignment → authoritative bundle → replay → QA → leakage-safe split. Each step removes one class of ambiguity before the next step is built.

One example will run through the lesson. At dusk, an autonomous shuttle must decide whether to brake when a pedestrian emerges from behind a parked delivery van. A checkpoint just before emergence later branches into otherwise matched brake and coast rollouts. The shuttle has a rigid sensor rig:

• a 1,280 × 720 rolling-shutter camera at 30 Hz, with 4 ms exposure and 12 ms top-to-bottom readout;
• a spinning LiDAR at 10 Hz, whose individual beams span a 100 ms sweep;
• a radar at 20 Hz, whose chirps measure range and radial velocity over a frame interval;
• ego pose estimates at 200 Hz and controls at 50 Hz;
• a simulated street containing the parked van, the initially occluded pedestrian, and static buildings.

The simulator uses a monotonic nanosecond clock. Its world and shuttle-body frames are right-handed with x forward, y left, and z up. The camera uses the common vision convention x right, y down, and z forward. We will make every conversion explicit rather than relying on the engine's implicit conventions.

1 · First principle: truth belongs to a measurement event

A physical scene has a time-varying state s(t): poses, joint angles, velocities, materials, lights, weather, contacts, and latent controller state. A sensor does not observe all of s(t). It integrates or samples a subset through a measurement process. Represent an acquisition event as

The acquisition support A may be one instant, an exposure interval, one interval per image row, or one timestamp per LiDAR beam. A label is then a declared query over the same event:

Here κ is the label contract: ontology, units, frame, visibility, invalid values, and reference time. Including o_e matters because some labels depend on what the sensor resolved. A modal mask depends on the camera visibility test; an amodal 3D box can come from state alone.

This formulation separates three notions often collapsed into “ground truth”:

1. Engine state truth: the variables stored by the simulator at particular times.
2. Measurement truth: the quantity the sensor model produced over its acquisition support.
3. Task truth: the target defined by a human ontology and evaluation protocol.

They are related but not identical. The engine may know that an articulated prop has internal object ID 814, while the task ontology calls it “background.” It may store a camera pose at the beginning of a rolling exposure, while the annotation contract requires row-time geometry. It may return a precise nonlinear depth-buffer value, while the task expects metric camera-z. Every number can be computed without numerical error and still answer the wrong question.

1.1 What supervision actually asks a learner to recover

Supervised learning does not have access to the label we intended. It only sees exported pairs (o,ỹ) and minimizes empirical risk:

Suppose the intended task target is y=L(s,o,κ), but the exporter silently uses another contract κ′ or another time, producing ỹ=L(s′,o,κ′). Nothing in the loss tells the model that y was intended. With enough data and capacity, the learner becomes a better predictor of ỹ. Optimization faithfully amplifies the exporter's definition.

This is why systematic mismatch can be more dangerous than modest annotation noise. If scalar labels are ỹ=y+ε with conditionally zero-mean noise, squared-loss regression can still approach E[y|o]; noise mainly raises variance and sample demand. If instead ỹ=y+b(o), then the optimum shifts to E[y|o]+b(o). A depth error that grows with image radius, a box offset that grows with velocity, or a class mapping tied to asset family will not average away. More perfectly rendered data makes the biased rule more statistically certain.

“Exact labels” therefore has two independent axes:

• numerical precision: how accurately the implementation evaluates its chosen function;
• semantic validity: whether that function is the target required by the task and measurement.

The second comes first. Computing the wrong function to 32-bit precision is not better supervision than a slightly noisy estimate of the right quantity. It is often worse operationally because clean overlays and stable checksums create false confidence.

1.2 Thought experiment: two exact exporters

Freeze the shuttle scene, engine build, and random seeds. Exporter A declares camera-z, row-time modal masks, stable logical IDs, and camera-from-world poses. Exporter B writes the raw graphics depth buffer as depth, evaluates boxes at simulation tick start, uses transient render handles as instance IDs, and stores world-from-camera poses under the same field name. Both are deterministic. Both can reproduce every byte. Both can truthfully say “the values came directly from the engine.”

Now train on B without knowing its hidden choices:

• The depth head learns the projection-buffer transfer curve, including the training near/far planes, rather than metric geometry. Changing the projection configuration at deployment changes the target function even when the physical scene is identical.
• The detector sees dusk pixels integrated after the pedestrian begins moving but boxes from an earlier state. It learns a motion-conditioned offset: fast emerging actors are labelled closer to their old position than slow actors.
• The tracker sees an identity change when draw order or level-of-detail changes an engine handle. It is penalized for maintaining the real pedestrian identity and learns that appearance continuity is sometimes untrustworthy.
• The pose consumer interprets T_wc as T_cw. A reconstruction can still look structured in simple scenes while being mirrored, translated, or rotated into the wrong frame.
• A random-frame test split contains neighbouring views of the same pre-emergence checkpoint. The model appears to generalize because the evaluation repeats content it has memorized.

None of these failures is fixed by additional samples, higher resolution, photorealism, or deterministic replay. Those investments reproduce the same mistaken supervision at greater scale. The fundamental problem is identification: an array does not identify which semantic query produced it. The contract, event, frame graph, and lineage supply the missing information.

1.3 Worked error propagation: how time becomes pixel bias

For a camera-frame point, horizontal projection is u=f_xX/Z+c_x. Let relative point velocity in the camera frame be (V_X,V_Z). Differentiating with the quotient rule gives

The two terms have different causes: lateral motion changes X; closing motion changes perspective scale through Z. In the shuttle scene, take a pedestrian point at X=1.5 m, Z=18 m, lateral velocity V_X=2 m/s, closing velocity V_Z=−12 m/s, and f_x=800 px. Then

Using a state 8 ms early shifts the expected horizontal label by about 1.07 px; 16 ms produces about 2.13 px. The sign and magnitude are correlated with lateral direction, closing speed, depth, and image position. The resulting label error is not white noise. A trained detector can learn it as a stable motion prior, then brake too late or too early when real annotations use exposure-time geometry. This calculation is why timestamp semantics and interpolation tolerances must be derived from downstream spatial error—not chosen because “one frame is close enough.”

2 · Turn label semantics into an API contract

A tensor name is not a definition. “Depth,” “pose,” “flow,” and even “RGB” each have multiple legitimate meanings. Treat every exported field as a public API: a second team should be able to implement a compatible reader without inspecting the exporter source.

2.1 The minimum declaration

For every field, declare:

• quantity and units: metres, radians, pixels, seconds, reflectance, category ID, or dimensionless probability;
• shape and storage: dimensions, channel order, data type, endianness, compression, and numeric range;
• frame and axes: source/target frames, handedness, positive directions, origin, and whether coordinates are at pixel centres or corners;
• time support: instant, interval, per-row, per-ray, reference time, and interpolation policy;
• semantics: ontology version, modal/amodal status, geometric/shading definition, and inclusion rules;
• validity: valid mask, void category, NaN/zero/infinity policy, clipping, saturation, and confidence;
• derivation: authoritative source, algorithm version, calibration ID, and checksum.

2.2 Ontologies are executable policy

An ontology is more than a list of class names. It determines which engine entities are labelable, how parts compose into instances, how states change categories, and how ambiguous pixels become void. In the running scene, the van's chassis, doors, and wheels may be separate engine nodes but one task instance. A person seen through reflective glass needs an explicit visibility policy.

Version mappings from engine tags to task categories. Never infer category from mutable display names such as Van_Blue_Final2. Give each logical object a stable 64-bit ID for its trajectory lifetime, and give each class a stable ontology ID. Do not recycle instance IDs after deletion within the same sequence; temporal association would become ambiguous.

2.3 Why the contract must be executable

A prose definition can still be interpreted differently by exporter, loader, evaluator, and researcher. Make the contract executable in three forms: a schema rejects incomplete metadata; a reference derivation turns authoritative state into the field; numerical fixtures assert expected outputs for known scenes. “Depth is in metres” then becomes a test such as “the point (1,−1,8) in camera coordinates exports z=8, range=√66, and reprojects to pixel centre (740,260).”

This is minimal because no array inspection can recover omitted intent. A floating-point image cannot reveal whether its values are z or range when looking only at the principal ray, and a 4 × 4 matrix cannot reveal which direction it maps when the pose is identity. The executable contract deliberately includes off-axis, rotated, occluded, moving, invalid, and boundary cases where competing interpretations disagree.

The contract removes definition ambiguity; it does not prove the chosen definition is useful. A perfectly implemented amodal-box contract is still wrong for an evaluation expecting visible boxes. Contract review must therefore begin from the downstream decision and real-data annotation policy, then make that choice reproducible.

3 · Coordinate frames form a typed transform graph

Most multimodal dataset failures are not difficult geometry; they are unstated geometry. Fix one notation first. In this lesson, T_ab maps homogeneous coordinates from frame b into frame a:

Composition and inversion now follow from types rather than memory:

Read the inner frame letters during composition: T_abT_bc maps c → b → a. If the adjacent letters do not match, the product is ill-typed. This tiny discipline prevents many silent inverse errors.

3.1 The graph for a multimodal rig

Store nodes for world, map, robot body, rig, camera, LiDAR, every dynamic object, and articulated parts. An edge says whether a transform is static calibration or time-varying state. For example:

The arrows above describe coordinate mapping, not physical containment. Store calibration covariance or source when relevant, and identify the exact calibration revision. Convert engine coordinates into the canonical dataset convention at one tested boundary. Allowing each modality writer to perform its own axis swaps eventually creates camera/LiDAR disagreement.

3.2 Worked projection and round trip

At one reference time, put the camera centre at C_w=(10,2,1) m in the world, looking along world +x. A surface point on the van is X_w=(18,1,2) m. The vector from camera to point in world coordinates is (8,−1,1). Converting world x-forward/y-left/z-up into camera x-right/y-down/z-forward gives

Let f_x=f_y=800 px and principal point (c_x,c_y)=(640,360). With no distortion, perspective projection is

To test the contract, back-project pixel centre (740,260) using camera-z 8 m:

This numerical fixture belongs in the automated tests. It catches axis, origin, matrix-order, and inverse mistakes. Also test non-central pixels, non-identity rotations, negative coordinates, distortion, and points behind the camera. A convention written only in prose is likely to drift.

4 · Derive each geometric label from the contract

4.1 Camera-z, ray range, inverse depth, and disparity

For a valid camera-frame surface point X_c=(x,y,z), common depth-like targets are

Camera-z is distance along the optical axis; range is Euclidean distance along the ray. For our point, d_z=8 m while r=√(1²+1²+8²)=√66≈8.124 m. Calling range “depth” creates only a 1.5% error here, but the discrepancy grows toward the image edge. In normalized image coordinates x_n=(u−c_x)/f_x and y_n=(v−c_y)/f_y:

The relationship assumes undistorted normalized image coordinates. A graphics depth buffer is generally a nonlinear encoding created after projection and clipping. Its decoding depends on projection matrix, near/far planes, reversed-z choice, and API depth range. The safest exporter obtains camera-space hit position or ray range from the renderer, derives named products, and retains a valid mask. Do not assign sky the number zero unless zero is excluded from the valid domain and readers are required to consult validity. NaN is expressive in floating-point products, but many formats and accelerators mishandle it; a separate validity plane is often safer.

4.2 Surface positions and normals

Exporting a world- or camera-space surface-position pass makes many labels independently derivable. A normal must additionally say whether it is a geometric normal, interpolated vertex normal, or shading normal after normal mapping. These answer different questions: geometric normals supervise shape; shading normals reproduce lighting detail that may not exist in geometry.

Under a rigid transform, normals rotate but do not translate:

Under a linear transform A containing non-uniform scale, use the inverse transpose and renormalize: n′=normalize(A^−⊤n). Using An fails to preserve perpendicularity to transformed tangents. Declare outward, viewer-facing, or two-sided orientation.

4.3 Semantic, instance, part, and panoptic masks

A semantic mask assigns a class; an instance mask assigns a persistent object; a part mask identifies components; a panoptic label combines category and instance while reserving “stuff” regions. Store these as integer IDs, not visualization colours. Anti-aliasing can blend colours into nonexistent IDs. If soft edge coverage is useful, export it as a separate floating-point coverage field alongside a hard ownership rule.

Modal means the surface that wins the sensor's visibility process. Amodal means the complete object under a defined counterfactual visibility operation. For the partly occluded pedestrian:

• the modal mask contains only visible head, shoulder, and legs;
• the amodal mask is rendered with external occluders removed but keeps the pedestrian's own geometry and self-occlusion policy explicit;
• visible fraction can be defined as modal pixel area divided by amodal projected pixel area;
• image truncation is separate: part of the complete projection may lie outside the sensor bounds.

Amodal truth is not always uniquely physical. Should the back surface of a solid object count? Should a transparent windshield reveal the driver? The answer comes from the task contract, not the renderer. Prefer retaining modal mask, amodal projection, external-occlusion fraction, self-occlusion state, and truncation as distinct fields rather than compressing them into one “visibility” boolean.

4.4 Boxes, keypoints, and poses

A modal 2D box is the tight axis-aligned rectangle around valid modal pixels. An amodal projected box encloses the full projected object geometry. Keep an unclipped box in continuous image coordinates and a clipped box for image consumers. Suppose the pedestrian's full projection is [690,210,790,580], but the parked van hides the lower-left portion so visible pixels occupy [720,210,790,390]. Both are correct labels with different semantics; evaluating one against the other creates a systematic localization penalty.

A 3D box should specify centre frame, dimensions and their order, orientation representation, bottom- versus centre-anchor, and whether it is an object-template box, a tight geometry box, or an axis-aligned world box. An oriented object box transformed into the camera is not generally recovered by taking per-axis minima in the camera frame. Keypoints need location plus visibility, truncation, and validity flags; an amodal projected keypoint can remain geometrically valid even when its pixel is occluded.

For poses, publish a transform with named source and target frames rather than a bare translation and quaternion. Declare quaternion order (w,x,y,z) or (x,y,z,w), active/passive meaning, and normalization. Since q and −q encode the same rotation, temporal consumers may enforce a sign-continuity rule such as choosing the sign with non-negative dot product to the previous quaternion.

4.5 Optical flow and scene flow

Forward optical flow at source pixel p₀ is the projected displacement of the same material surface point from event 0 to event 1. Let X_o be a point fixed in object coordinates. Then

This derivation includes both object motion and camera motion. For static world geometry, X_w,1=X_w,0. In our example, the source point projects to (740,260). If the camera advances 0.5 m toward a static point with unchanged orientation, its new camera coordinate is (1,−1,7.5), which projects to approximately (746.67,253.33). Forward flow is therefore (6.67,−6.67) pixels.

That geometric destination exists even if another object occludes the point in frame 1. Export separate flags:

• source valid: the source pixel belongs to a resolved surface;
• destination in bounds: the projected point lies inside the second image;
• destination visible: it wins the second event's depth/visibility test within tolerance;
• correspondence defined: the material point persists rather than being spawned, destroyed, or topologically remeshed.

Photometric warping usually requires destination visibility; motion learning may still benefit from occluded geometric flow. Backward flow is not generally the negative of forward flow at the same array index because it is defined on a different pixel grid. Three-dimensional scene flow similarly requires a declared frame: X_w,1−X_w,0 is world-frame displacement, while expressing both endpoints in their contemporaneous camera frames mixes object and ego motion.

5 · Synchronize events, not filenames

“Frame 0042” is an ordering label, not a timestamp. Give every subsystem a monotonic master time or a calibrated mapping into one. Preserve at least five time concepts:

• state time: when a pose, velocity, or joint state is evaluated;
• command time: when a controller requests an action;
• application time: when the actuator or dynamics begins applying it after latency;
• acquisition time: the exposure, row, beam, chirp, or event interval;
• delivery time: when the completed sample reaches the model or logger.

These distinctions are necessary for causal learning. A policy trained with delivery-time images and command-time actions may accidentally see future consequences unless latency is represented correctly.

5.1 Worked rolling-shutter and LiDAR timing

Let the camera begin readout at 10.000 s, take 12 ms from first to last row, and expose each row for 4 ms. The centre time of row v in a 720-row image is approximately

The first row's centre is about 10.00201 s, the middle row's centre about 10.00801 s, and the last row's centre about 10.01399 s. A box covering rows 250–520 does not have one exact pose; its pixels refer to a band of times. The exporter can render row-time labels, or define a reference-time label and record the approximation. It must not silently use the frame-start pose.

The LiDAR sweep runs from 9.950 to 10.050 s. Each beam keeps its own timestamp and is transformed by T_wlidar(t_j). Naming the sweep “10.000 s” does not make all points simultaneous. Relative motion of 15 m/s creates 3 cm state error in 2 ms; if transverse, that is roughly 1.2 px at 20 m depth and 800 px focal length. Pairing a point near 10.050 s with state at sweep start creates up to 1.5 m of error. The radar frame likewise spans chirps: range and radial-velocity truth must use each chirp's time support, not an arbitrary frame stamp.

5.2 Interpolate states, not meanings

Retain modalities at native acquisition times. Evaluate continuous state using the simulator's integration checkpoints or an interpolation model. Linear interpolation is reasonable for translation over small intervals; rotations require spherical interpolation or a Lie-group spline. Do not linearly interpolate the 16 entries of a transform matrix: the result need not be a rigid transform. For high dynamics, interpolation must respect velocities and integration scheme.

Nearest-neighbour pairing is acceptable only under a declared maximum skew justified by task sensitivity. A useful association record says, for example, “camera reference time 10.00801 s; nearest LiDAR beam 10.00807 s; skew 60 μs; pose interpolated between checkpoints 10.005 and 10.010 s.” If no event satisfies tolerance, emit no pair rather than inventing simultaneity.

d   = speed · offset_ms                             // pixel displacement of the box
IoU = max(0, w − |d|) / (2w − max(0, w − |d|))      // same-size boxes shifted by d
// every engine number is "correct"; only the timestamp is wrong

6 · Build one authoritative sample bundle

The authoritative bundle is not “a folder where filenames line up.” It is a typed manifest connecting scenario lineage, measurement events, state snapshots or splines, calibrations, label contracts, and content-addressed payloads. Raw state and sensor products that are expensive to reproduce should be retained; convenient labels should be deterministic, versioned derivations whenever possible.

6.1 Concrete manifest

This abbreviated JSON-like record is intentionally explicit. A production schema should validate types, units, required fields, and referential integrity.

{
  "schema": "synthetic-vision/sample@4.1.0",
  "sample_id": "scene17.branch03.camera_front.10008010000",
  "scenario": {
    "id": "scene17", "branch_id": "branch03",
    "lineage_group": "city_block_08/traffic_seed_921",
    "split": "train"
  },
  "build": {
    "engine": "engine-6.2.1+9f1c", "exporter": "labels@31ac",
    "asset_manifest_sha256": "…", "shader_set_sha256": "…",
    "ontology": "street-v3.2", "platform": "gpu-model/driver"
  },
  "event": {
    "sensor": "camera_front", "clock": "sim_monotonic_ns",
    "frame_start_ns": 10000000000,
    "row_readout_ns": 12000000, "exposure_ns": 4000000,
    "reference_time_ns": 10008010000,
    "timing_model": "rolling_top_to_bottom@2"
  },
  "calibration": {
    "id": "rig-cal-007", "image_size_px": [1280, 720],
    "K": [[800,0,640],[0,800,360],[0,0,1]],
    "distortion": {"model":"brown5", "coefficients":[…]},
    "T_camera_rig": [[…],[…],[…],[0,0,0,1]]
  },
  "state": {
    "transform_spline_uri": "sha256/…/transforms.bin",
    "objects_uri": "sha256/…/objects.parquet",
    "actions_uri": "sha256/…/actions.parquet"
  },
  "modalities": {
    "rgb": {"uri":"sha256/…/rgb.exr", "shape":[720,1280,3],
      "dtype":"float16", "encoding":"linear_scene_rgb", "checksum":"…"},
    "camera_z": {"uri":"sha256/…/z.exr", "units":"m",
      "frame":"camera_front", "invalid":"validity_mask", "checksum":"…"},
    "flow_0_to_1": {"uri":"sha256/…/flow.exr", "units":"pixel",
      "grid":"event0_pixel_centres", "visibility_uri":"sha256/…/valid.png"},
    "instance_modal": {"uri":"sha256/…/instance.png", "dtype":"uint32",
      "ontology":"street-v3.2", "background_id":0, "void_id":4294967295}
  },
  "qa": {"profile":"automotive-rig-v5", "report_uri":"sha256/…/qa.json"}
}

A manifest should make invalid states unrepresentable where possible. A flow tensor without direction, grid, units, endpoint event, and validity plane should fail schema validation. A pose without source frame, target frame, and timestamp should not be admitted as a pose.

6.2 Exporter pseudocode

for scenario in assigned_scenarios(split_manifest):
    state_log = simulate_and_checkpoint(scenario, seed_tree)

    for event in sensor_scheduler(state_log):
        calibration = calibration_registry.resolve(event.sensor, event.time_support)
        observation = sensor_model.measure(state_log, event, calibration)

        # Query the same continuous state and visibility process as the event.
        surfaces = renderer.authoritative_surfaces(observation.render_record)
        labels = derive_labels(
            state_log=state_log,
            event=event,
            calibration=calibration,
            surfaces=surfaces,
            contract=label_contract_version)

        bundle = assemble_manifest(event, observation, labels, provenance)
        schema_validate(bundle)
        qa_report = run_cross_modal_invariants(bundle)

        if qa_report.has_hard_failure:
            quarantine(bundle, qa_report)       # preserve evidence and rejection reason
        else:
            publish_content_addressed(bundle, qa_report)

Notice the order: split assignment precedes rendering; sensor scheduling defines the event; labels query that event; schema validation precedes semantic QA; failures are quarantined rather than silently discarded. Silent retries bias the accepted distribution toward scenes the renderer handles easily.

7 · Provenance and replay are part of the dataset

A single random seed is not provenance. Changing the number of vegetation draws should not alter pedestrian behaviour. Use a seed tree with independent streams for scene layout, assets, behaviours, physics disturbances, sensor noise, learned refiners, and counterfactual branches. Store the derivation path, not just leaf values.

Replay also needs engine/exporter revisions, asset and shader manifests, scenario configuration, ontology, calibration, physics settings, checkpoints, dependencies, hardware/driver facts, and acceptance decisions. Content hashes detect mutation.

7.1 Define reproducibility by level

“Reproducible” must state an equivalence relation:

• bitwise exact: integer IDs, ontology tables, split assignments, manifests, and checksums should normally match exactly;
• numerically equivalent: physics states, floating-point transforms, radiance, and ray intersections may use declared absolute and relative tolerances across hardware;
• statistically equivalent: stochastic learned refinement may be accepted only if exact replay is impossible and distributional tests are explicitly defined;
• semantically equivalent: derived outputs may change encoding while representing the same quantity, but require a versioned migration and invariant checks.

Use a mixed floating-point condition such as |a−b| ≤ ε_abs+ε_rel|b|. Absolute tolerance controls values near zero; relative tolerance scales with magnitude. Tolerances should come from downstream sensitivity and expected numerical variation, not from whatever makes a failing test pass. A one-millimetre pose difference may be irrelevant for a distant building but material for close-range robotic grasping.

8 · QA proves cross-modal invariants

Schema checks prove that fields are present. Invariants test whether independently exported fields agree with the same physical and semantic contract. Run them per sample, per trajectory, and at dataset scale.

8.1 Per-sample and temporal checks

These numbers are examples for the running rig, not universal constants. Test authoritative values before serialization and decoded values with encoding-aware tolerances. Exclude silhouette discontinuities from smooth reprojection, but test them separately for ownership and visibility. Report full error distributions and slices by distance, image radius, motion, material, and class—not only one mean.

8.2 Dataset-level checks

Track class/asset frequencies, visibility, depth, motion, saturation, invalid rates, rejections, sensor dropouts, weather, branch counts, and near-duplicate fingerprints. Compare builds with a pinned baseline. A fall in pedestrian pixels may come from scenario logic, camera pitch, ontology, or sample rejection; one global count cannot localize it.

Visual overlays remain essential: axes drawn on objects, LiDAR projected into the correct camera row, modal and amodal contours, colourized depth, normal orientation, forward-flow warps, exposure timing bands, and box/keypoint visibility. Numerical checks can be consistently wrong if they share the same faulty convention. Human inspection supplies an independent semantic check.

8.3 Failure triage

1. Freeze the evidence. Quarantine the manifest, payload hashes, seed tree, QA residuals, and minimal replay command.
2. Classify the boundary. Is the failure schema, semantics, coordinates, timing, visibility, numeric encoding, sensor physics, or scenario validity?
3. Reduce the case. Replay one sensor, one object, one pixel/ray, and two timestamps. Disable learned refinement and noise without changing state.
4. Test authoritative intermediates. Compare state point, camera-space point, projected point, visibility result, and serialized value in order.
5. Find the first divergence. Fix that boundary, add the sample as a regression fixture, and rerun dataset statistics.

Residual shape is diagnostic. A constant pixel offset suggests centre-versus-corner convention. Error growing radially suggests distortion or range/z confusion. Error proportional to motion suggests timing. Mirroring suggests axis handedness. Errors only at silhouettes suggest visibility or anti-aliasing. Correct geometry with wrong class IDs points to ontology mapping, not rendering.

9 · Split by causal lineage before export

Randomly splitting frames is almost always leakage. Adjacent views share geometry, textures, lighting, trajectories, and object identities. The test set then measures interpolation or memorization rather than generalization.

Construct a lineage graph before rendering. Group together:

• the same base scene and procedural ancestor;
• near-duplicate assets from one scan, CAD family, texture source, or generated parent;
• trajectory segments from one rollout or checkpoint;
• counterfactual branches that share history before an intervention;
• frames and modalities from the same acquisition event;
• learned-refinement outputs derived from the same clean render.

Assign the connected lineage group to exactly one split, then schedule exports. Audit exact content hashes plus geometric, texture, and perceptual near-duplicate fingerprints. IDs should not encode split in a way a model can exploit. Keep a locked test manifest whose scenarios and acceptance rules are not tuned after observing results.

10 · The running example, assembled end to end

1. Define event. Camera frame starts at 10.000 s; row timing and exposure define a space-time acquisition surface. LiDAR beams and radar chirps remain at native times.
2. Resolve state. A transform spline evaluates robot, rig, van, pedestrian, and articulated-part poses throughout both acquisitions.
3. Measure. The sensor models produce rolling-shutter radiance and timed LiDAR returns with their own noise and visibility.
4. Label. Camera-space hits produce named camera-z, ray range, geometric normals, IDs, and material correspondence. Object state produces amodal geometry. Both carry validity.
5. Transform. The typed graph converts world/object data into each sensor convention. The known point round-trips from world (18,1,2) to pixel (740,260) and back.
6. Synchronize. Camera rows and LiDAR beams use their own acquisition times. Any fused pair records interpolation support and actual skew.
7. Bundle. One schema-valid manifest connects payloads to event, calibration, ontology, state, build, seeds, and split lineage.
8. Prove. Projection, flow, identity, time, and cross-sensor residuals pass profile thresholds; visual overlays receive sampled review.
9. Publish or quarantine. Hard failures preserve their evidence and rejection cause. Published bundles are content-addressed and replayable.

The order is deliberate. Precise transforms cannot fix unresolved semantics; a perfect transform at the wrong time cannot fix synchronization; missing provenance prevents failures from becoming regression tests.

11 · Design review: common precisely-wrong datasets

12 · Necessity ledger: what each mechanism buys—and cannot buy

The design can look like metadata overhead until each mechanism is paired with the ambiguity it removes. None is sufficient alone. Together they make the dataset's claim falsifiable: we can say which event was measured, which target was requested, how it was computed, whether modalities agree, and what kind of generalization the split tests.

The residual column is as important as the mechanism column. Engineering discipline should not turn into a new claim of omniscience. After each ambiguity is removed, the remaining uncertainty becomes the next explicit validation problem. This creates a linear assurance argument rather than the vague assertion “the engine gives us ground truth.”

13 · Exercises and self-tests

Exercise 1 · Depth conversion

A pixel has normalized coordinates x_n=0.6, y_n=0.8, and ray range 10 m. What is camera-z? What error results from treating range as z?

Answer. z=10/√(1+0.6²+0.8²)=10/√2≈7.071 m. Treating range as z overestimates by 2.929 m, or about 41.4% relative to true z.

Exercise 2 · Transform typing

You have T_wr (rig → world), T_cr (rig → camera), and an object point X_o with T_wo (object → world). Write the object-to-camera mapping.

Answer. Since T_cw=T_crT_rw=T_crT_wr⁻¹, the point is X_c=T_crT_wr⁻¹T_woX_o. Adjacent frame letters match right to left.

Exercise 3 · Flow validity

A source surface persists and projects to (300,200) in event 0 and (315,194) in event 1, but a passing van covers that destination in event 1. What should be exported?

Answer. Geometric forward flow is (15,−6) px; source-valid and correspondence-defined are true; destination-in-bounds is true; destination-visible is false. A photometric-warp validity mask should reject it without erasing the geometric motion.

Exercise 4 · Timing sensitivity

An actor moves laterally at 12 m/s at 24 m depth in a camera with 900 px focal length. A label is evaluated 5 ms early. Approximate the pixel displacement.

Answer. Motion is 12·0.005=0.06 m. Small-angle image displacement is approximately fΔx/z=900·0.06/24=2.25 px—large enough to corrupt a tight boundary or flow target.

Exercise 5 · Modal versus amodal

A pedestrian's complete projected area is 1,000 pixels. External occlusion leaves 350 visible pixels; 100 pixels of the complete projection are outside the image. Propose separate fields rather than one visibility value.

Answer. Export modal area 350, amodal projected area under the declared canvas convention, external visible fraction, and truncation fraction separately. If amodal area counts the full unbounded projection, visible fraction is 0.35 and truncation fraction is 0.10. Also retain modal/amodal masks and keypoint-specific visibility because one scalar cannot recover their spatial pattern.

Exercise 6 · Split audit

A procedural city has five weather variants and three camera trajectories per base layout. A neural refiner produces four appearances per clean frame. What is the safest default split unit?

Answer. The base-layout lineage group, including its weather variants, trajectories, clean frames, and refined descendants, should remain in one split. A benchmark may intentionally relax this to measure weather or view interpolation, but must name that task rather than call it unseen-scene generalization.

Self-test checklist

• Can every pose be read aloud as “coordinates from ___ into ___ at time ___”?
• Can you state whether each depth product is z, range, inverse depth, disparity, or encoded buffer?
• Can flow remain geometrically defined while photometrically invalid?
• Can a second machine replay a failed event with its engine, assets, calibrations, seed tree, and acceptance rule?
• Can any two samples in different splits share a procedural ancestor, rollout prefix, asset family, or refined parent?
• Does every QA threshold have a physical or downstream sensitivity rationale?

14 · Bridge to the implementation lab

Lesson 05 will turn these contracts into a working Blender image-model pipeline. The order matters: a render script cannot decide after the fact whether a pedestrian box means visible pixels or the hidden full body, which camera time a mask belongs to, or whether neighboring variants may cross a split boundary. Those are semantic decisions, not Blender settings.

The implementation lab therefore treats the task manifest, measurement event, label query, lineage, and QA invariants as inputs. Blender evaluates a declared scene and exports evidence; it does not silently invent the experiment. After that end-to-end path is proven on one sample and calibration scenes, Lesson 06 asks whether the resulting synthetic supervision improves real performance. Otherwise a measured “domain gap” may actually be a label-definition gap, time-skew gap, calibration gap, or split leak.

Once correctness is established, the authoritative bundle becomes a diagnostic tool. Clean and corrupted measurements can share exact state; sensor effects can be ablated; real calibration can replace nominal calibration; error can be sliced by motion, occlusion, depth, or material. This is how synthetic data becomes an instrument for causal debugging rather than a pile of rendered images.

Interview prompts

• Why can engine ground truth be precisely wrong? The queried engine value can be exact while its time, coordinates, visibility, ontology, or physical quantity differs from the task label.
• How does camera-z differ from range? Camera-z is optical-axis distance; range is Euclidean ray length and grows relative to z away from the principal ray.
• Why is backward flow not simply negative forward flow? They are defined on different source grids, with different visibility and correspondence domains.
• What makes a rolling-shutter frame a space-time object? Rows begin exposure at different times, so pixels correspond to different camera and object states.
• What belongs in a replay claim? Manifest, versions, assets, calibration, state/checkpoints, seed tree, acceptance decisions, and a declared exact or tolerance-based equivalence.
• What is the best first response to a QA failure? Freeze the evidence, reduce to one event and one geometric path, then find the first boundary where authoritative and serialized values diverge.
• Why split by lineage? Frames, branches, and generated variants can share causal content; keeping their connected ancestry together prevents memorization from masquerading as generalization.