The data-generating process and task contract

Before touching a renderer, define what the deployed model must decide, what can make that decision wrong, and which controllable process will produce evidence about those failures.

0 · The destination and the running example

By the end of this lesson, you should be able to start with a deployed decision and derive a generator specification. In particular, you will be able to:

• write a task contract that makes the model input, target, loss, latency, label semantics, and deployment boundary explicit;
• factor a simulated trajectory into initial state, behavior, dynamics, observation, and labeling mechanisms;
• distinguish support, coverage, and density, which are often incorrectly collapsed into “diversity”;
• find shortcuts with paired interventions rather than visual inspection;
• separate training, development, and locked acceptance data; and
• turn all of the above into a small, auditable generator program.

We will carry one example through every section. An autonomous shuttle approaches a parked van at dusk. A pedestrian may emerge from behind the van. The shuttle has a front rolling-shutter camera, a scanning LiDAR, and radar. Its perception-and-planning stack must decide whether to brake now or continue. Later lessons will make the scene simulation-ready, model the three sensors, synchronize its labels, transfer to reality, and branch “brake” and “coast” futures from the same checkpoint.

0.5 · The fundamental problem: finite evidence, open-world consequences

A deployed vision model must act on situations it has not seen exactly before. Yet training supplies only a finite dataset D={(x_i,y_i)}_i=1^N, sampled from some collection process Q. Deployment evaluates the model on another, only partly known process P_real. The fundamental problem is therefore epistemic: how can finite observations justify a claim about costly decisions over a much larger distribution?

They cannot do so by themselves. Infinitely many predictors can agree on every element of D and disagree everywhere else. One may detect pedestrians by body geometry; another may exploit the fact that all positive training scenes used one dusk preset. Both achieve zero training error. The observed files do not identify which rule was learned. Generalization requires additional assumptions about which variations preserve truth, which causes the generator covers, and how deployment differs from collection.

A numerical thought experiment makes the asymmetry visible. Suppose a dangerous emergence occurs once per 100,000 real encounters and a collision costs 100,000 units. Its contribution to expected cost is approximately (1/100,000)×100,000=1 unit per encounter. A benign error that happens in 1% of encounters but costs one unit contributes only 0.01×1=0.01. The rare case matters one hundred times more to risk. Nevertheless, a random validation set of 100,000 independent encounters has probability

In roughly 36.8% of such validation sets, a model that fails every dangerous emergence is empirically indistinguishable from one that handles it. Average accuracy does not merely downweight the problem; the sample may contain no evidence about it.

Synthetic data helps because a generator can deliberately create the missing encounter, reveal privileged state, repeat controlled variations, and branch alternative actions. But it introduces a second epistemic gap: the generator is a hypothesis about worlds and sensors, not reality itself. Its exact labels are exact only inside that hypothesis. The task is therefore not “replace scarce reality with unlimited simulation.” It is to use simulation for controlled coverage and causal experiments, then use independent real evidence to test whether those assumptions transfer.

1 · Begin with a decision, not a dataset

1.1 The model sees observations, not the world

Let s_t denote the complete state of the simulated world at time t. It might include every actor pose and velocity, the pedestrian's intent, road friction, sun position, the shuttle's brake temperature, and the internal timing of each sensor. A deployed model cannot access that privileged state. It receives an input x_t, assembled from observations such as recent camera images, LiDAR returns, radar detections, timestamps, and possibly ego motion.

Let f_ω be the learned model with parameters ω. It maps the available input to a prediction:

The target y_t could be a pedestrian box, depth map, occupancy grid, collision probability, future trajectory, or discrete action. The symbol ŷ means the model's estimate; y means the declared reference answer. For our shuttle, suppose the immediate output is a calibrated probability ŷ_t∈[0,1] that continuing will cause a collision within 2.5 seconds. A downstream rule brakes when ŷ_t≥0.02.

1.2 Deployment risk is the objective

A benchmark average is not automatically the product objective. Write the real objective as an expected deployment loss:

Read the equation from the inside outward:

• P_real is the deployment distribution over model inputs x, correct targets y, and consequence weights c.
• ℓ(ŷ,y) is a per-example error measure. It could be cross-entropy, depth error, trajectory displacement, or a decision loss.
• c(x,y)≥0 increases the influence of consequential cases. Missing a pedestrian at stopping distance can matter more than shifting a distant box by two pixels.
• The expectation E means an average over what deployment actually presents, not over whatever examples happen to be easy to generate.

This expression is a design compass, not a claim that all consequences can be reduced to one perfect number. Safety constraints, worst-case slices, latency limits, and calibration may remain separate acceptance conditions. The important move is to make them explicit before data generation.

1.3 A worked decision-cost example

Suppose the shuttle's simplified action costs are:

If p is the true collision probability, the expected cost of braking is 2. The expected cost of continuing is 100p. In this toy model, brake when 2<100p, or p>0.02. A 0.5 probability threshold would therefore be indefensible. Real systems add comfort, rear-collision risk, uncertainty, control feasibility, and safety rules, but the calculation illustrates why the data contract must name the consuming decision. A probability can be statistically accurate yet operationally useless if it arrives too late or is uncalibrated near the decision threshold.

2 · Write the task contract

A task contract is the first durable artifact. It is a versioned agreement among the product, data, simulation, labeling, training, and evaluation systems. It prevents each team from silently solving a slightly different task.

Why must the contract come before data?

There is no task-neutral definition of “enough detail” or “correct label.” Consider the same shuttle scene used for three consumers. A pedestrian detector needs identity, visibility, and image-space boxes. A free-space mapper needs metric occupancy and coordinate frames. A brake decision needs future motion, friction, action latency, uncertainty, and collision cost. The RGB frame can be identical in all three cases, while the sufficient hidden state and useful generator distribution are different.

Starting with rendered files reverses the dependency. The team first commits to scenes, sensors, sampling frequencies, and exported fields, then tries to invent a task those files support. Missing privileged state is often unrecoverable: if the run did not retain per-row exposure time, road friction, pedestrian intent, or the pre-action checkpoint, no later annotation pass can reconstruct the desired counterfactual exactly. And data cannot decide whether a late false negative is worse than an early false positive; that comes from the consumer and consequence model.

The contract therefore acts as a claim boundary. It says what must be predicted, from which evidence, for which decision and operating regime. That boundary tells the simulator what state to preserve, tells the labeler what truth means, tells training which correlations should be invariant, and tells evaluation where success may be claimed. Without it, “more data” can make the wrong task more precisely optimized.

The contract should also identify nuisance variables: factors that can change the sensor input without changing the desired answer. For pedestrian occupancy, the van's paint color may be a nuisance. For stopping distance on a wet road, road condition is not a nuisance because it changes the outcome. “Nuisance” is therefore task-relative, not an intrinsic property of a scene variable.

2.1 Labels are definitions, not merely exports

A simulator can expose exact internal state, but “exact” does not mean “correct for the task.” Is pedestrian depth the camera-axis coordinate Z, Euclidean range, first-surface range, or object-center range? Is a rolling-shutter box defined at the first row's exposure, the middle row, or per row? Does “visible” require one pixel, ten percent of the body, or unoccluded torso? Two pipelines can export numerically perfect answers to different questions.

For the shuttle, a useful collision label could be: “Under the recorded future actor behavior and the candidate action continue, do the ego and pedestrian collision volumes intersect within 2.5 seconds?” This label includes a counterfactual action and a horizon. It is different from “Is the pedestrian currently in front of the vehicle?” Lesson 4 will make these timing, coordinates, visibility, and provenance rules operational.

2.2 Convert desired failures into generator requirements

For every costly deployment failure, ask four questions in order:

1. What latent causes can produce it? For a late brake: occlusion, high pedestrian speed, poor contrast, scan timing, wet-road dynamics, or planner latency.
2. Which causes change truth? Pedestrian velocity and road friction can change collision outcome. Van color should not.
3. Which causes change measurement? Glare, motion blur, LiDAR incidence angle, radar multipath, and clock offset alter observations.
4. Which variable and test will represent each cause? If “dusk” is only a filename tag with no measured illumination or sensor response, it is not a controllable experiment.

3 · Derive the trajectory model one mechanism at a time

3.1 Define the variables

A still image is insufficient for behavior, motion, timing, and action consequence. Define a trajectory τ over discrete simulation steps t=0,…,T. The step may be much finer than a sensor frame.

3.2 Start with the chain rule

Probability's chain rule says that a joint distribution can be written as a product of conditional distributions. If the simulator state contains all information needed for the next step, and the current observation depends on the current state plus sensor randomness, a useful factorization is:

The product symbol ∏ means “multiply the same set of factors for every time step.” The indicator 𝟙[condition] equals one when the condition is true and zero otherwise. It encodes a deterministic label exporter L. If labels contain annotator ambiguity or measurement error, replace the indicator with a label distribution P(y_t|s_t,o_t).

The displayed ordering is a compact convention; implementations often observe, then act, then transition. One can write the observation factor before the policy factor and include a terminal observation at T. What matters is the declared dependency: the policy receives only h_t, the transition receives the current state and action, and the observation model measures state. No module should silently read future or privileged information.

3.3 Interpret the five mechanisms

1. Initial-state distribution, p_θ(s₀). This chooses which worlds begin an episode: base scene, assets, layout, time of day, weather, initial poses, velocities, goals, sensor mounting errors, and other episode-level variables. In the shuttle case it chooses the road segment, parked van geometry, pedestrian body and clothing, spawn position, dusk illumination, and initial shuttle speed.

2. Behavior policy, π_θ(a_t|h_t). A policy is a distribution over actions given available history. It determines which portions of state-action space are visited. The ego policy might be a recorded driver, scripted controller, planner under test, exploration policy, or mixture. Pedestrian behavior can be represented by its own policy or folded into the transition model, provided the boundary is explicit. For our example, behavior determines when the pedestrian steps out and whether the shuttle brakes, coasts, or recovers.

3. Transition or dynamics model, T_θ(s_t+1|s_t,a_t). This describes how state changes after an action: rigid-body motion, tire-road interaction, contacts, articulation, actuator delay, actor response, and stochastic disturbances. A simulator can render an impeccable wet road while retaining dry-road braking dynamics; that is a dynamics gap, not an appearance gap.

4. Observation model, O_θ(o_t|s_t). This maps world state into measurements through geometry, illumination, optics, exposure, rolling shutter, LiDAR emission and scan pattern, radar response, noise, quantization, compression, timing, and the image signal processor. It is a probability distribution because photon arrival, electronics, environment returns, and parts of the rendering approximation may be stochastic. It produces sensor data—not generic “realistic pixels.”

5. Label function, L(s_t,o_t). This maps authoritative state and observation metadata into the task's declared target: coordinates, visibility, segmentation, optical flow, collision outcome, or uncertainty fields. It owns ontology and convention. The label must share the observation's time basis and calibration lineage.

3.4 Why keep the factors separate?

Each mechanism creates a different reality gap and requires a different remedy:

This separation is diagnostic, not metaphysical. A learned video generator may entangle dynamics and observation. It can still be useful, but then interventions, truth ownership, and failure attribution become harder. The engineering question is whether each assumption can be controlled and audited at the level required by the task.

3.5 Why this decomposition—and why is it close to minimal?

The chain rule permits many mathematical factorizations. These five factors are useful for engineering because they answer five distinct questions that must all be answered before a labeled trajectory exists:

1. Which world did we start in? That is p(s₀).
2. Which choices made this trajectory occur? That is the behavior policy π.
3. What consequences followed those choices? That is the transition T.
4. How did the evolving world become device measurements? That is the observation model O.
5. Which answer did the task ask us to derive from that evidence? That is the label contract L.

Each question admits a different intervention and an independent test. We can change an occluder while fixing behavior; change brake action while cloning state; compare stopping curves while ignoring rendered color; replay one trajectory through two camera models; or re-export one state with two visibility conventions. This is why the factors are a debugging boundary: a failed test points toward an assumption we can change without rebuilding everything else.

Now try merging them. If initial state and behavior are one opaque “scenario” sampler, a missing near-collision could mean the geometry never spawned or the policy always avoided it. If policy and dynamics are merged, we cannot tell whether late braking is absent from the data or produces the wrong deceleration. If dynamics and observation are merged into video, a wet street that looks correct can hide dry stopping physics. If observation and labels share an implicit timestamp, a 40 ms error can be blamed on either sensing or truth export. The monolithic generator may still emit beautiful clips, but it cannot tell us which causal claim failed.

Consider two simulators that render the same photoreal wet-road frame. Simulator A merely darkens asphalt and adds reflections; it retains dry friction. Simulator B changes both the optical measurement and tire-road dynamics using calibrated ranges. Before an action, their camera distributions may be nearly identical. After the shuttle brakes, their stopping distances diverge. An image-quality metric cannot distinguish the model that answers the decision correctly because the relevant difference lives in T, not O. Separating the factors makes the necessary experiment obvious: hold initial state and brake command fixed, measure the state transition, then separately test sensor appearance.

The decomposition is “minimal” in a practical rather than universal sense. A particular engine may split actor policies, physics, lighting, optics, and electronics further; another may implement several with one learned network. But collapsing any of the five questions removes a major intervention class or confuses truth ownership. Expand a factor when diagnosis requires finer control; do not collapse the conceptual contract merely because code shares a module.

4 · Support, coverage, and density are different

4.1 Support: can the event happen at all?

The support of a distribution is the set of outcomes to which it assigns nonzero probability. Let A be a decision-relevant slice such as “a partially occluded pedestrian emerges from behind a van at dusk while the shuttle is within stopping distance.” A necessary condition for learning from synthetic data is:

If the generator forbids that combination, collecting a billion synthetic frames cannot reveal it. Importance weighting cannot repair zero support because no example exists to weight. Support comes before photorealism and before matching frequencies.

4.2 Coverage: did the finite dataset actually sample it?

A distribution may assign an event nonzero probability, yet a finite dataset may miss it. Coverage describes what the realized dataset spans at useful resolution. If an event is sampled independently with probability p in each episode, the probability that N episodes contain no such event is (1−p)^N.

Suppose the shuttle event has synthetic probability p=0.001, one in a thousand. With N=1,000 episodes, the probability of seeing zero is approximately 0.999¹⁰⁰⁰≈0.368, or 36.8%. To have at least a 95% chance of seeing it once, solve (1−p)^N≤0.05; this gives about 2,995 episodes. One example is still inadequate for learning variation, so rare consequential slices should usually be sampled deliberately rather than left to chance.

4.3 Density: how often should it occur?

Density describes relative frequency within support. The real event might occur in only 0.1% of encounters, but we may synthesize it in 10% of training episodes to obtain enough examples. This can be sensible for learning, provided the sample probability is recorded and the model is calibrated and evaluated against the real distribution.

For a simple numerical example, generate 10,000 encounters:

• 1,000 rare emergence cases from P_syn(A)=0.10;
• 9,000 ordinary cases from P_syn(¬A)=0.90.

If deployment estimates say P_real(A)=0.001, an unbiased average-risk estimate under ideal sampling would weight an A example by 0.001/0.10=0.01, while a non-A example receives approximately 0.999/0.90=1.11. This does not prove the simulator is correct inside either slice; it only corrects their relative mixture under strong assumptions. For training, weighting and sampling also change optimization variance and calibration, so validate the choice empirically.

4.4 Think in a support ladder

1. Semantic support: do the relevant entities and events exist—pedestrian, van, crossing, braking?
2. Geometric support: do range, scale, occlusion, pose, and spatial relationship span the decision boundary?
3. Temporal and action support: do emergence timing, velocities, brake/coast choices, delays, and recoveries occur?
4. Measurement support: do exposure, glare, scan phase, noise, dropout, and faults cover plausible devices?
5. Joint support: can important combinations occur together without violating physics or semantics?

Marginal checklists are not enough. A dataset may contain children, vans, dusk, wet roads, and braking separately while never combining them. Conversely, taking a Cartesian product of all variables creates implausible combinations and consumes budget. Use conditional scenario structure to place probability on meaningful joints; Lesson 2 develops that machinery.

4.5 Why scale and photorealism cannot repair missing support

Sample count reduces uncertainty about outcomes the process can already produce. It cannot change the process's support. If P_syn(A)=0, then for every finite sample count N, the chance of observing A remains zero; the infinite-sampling limit is also zero:

Higher resolution adds measurements of the same supported states. More adjacent frames add highly correlated views of the same trajectory. Better materials can reduce part of the observation gap O. None creates a pedestrian trajectory forbidden by π, wet-road braking absent from T, or a collision label omitted by L.

Compare two budgets. Dataset A contains ten million cinematic frames from 100 safe approaches in which the pedestrian always waits. Dataset B contains 20,000 moderately rendered frames from 2,000 independent encounters spanning emergence time, occlusion, brake/coast branches, and sensor phase. Dataset A may be better for learning generic appearance. For the declared brake decision, however, it supplies zero direct evidence about the consequence of continuing when the pedestrian crosses. Dataset B places evidence around the decision boundary. The correct choice follows from the task contract, not from the larger byte count.

// more frames ⇒ variance ↓ : the SAME curve in both modes
synAcc(N)      = 0.995 − 0.45/√(N/100)
// structural error: real recall on the rare danger slice
danger(N, on)  = on ? 0.90 − 0.82/(N/100)^0.42   // support ⇒ climbs with N
                    : 0.05                          // NO support ⇒ flat, ∀ N
// the danger slice is 0.1% of reality, so the average barely moves
realAvg(N, on) = 0.999·0.965 + 0.001·danger(N, on)

5 · Correlation, causation, and synthetic shortcuts

5.1 Why empirical risk permits shortcuts

Training rewards features that predict labels on the training distribution. It does not ask whether those features are causal, stable, or sensible. If every synthetic pedestrian wears a bright material and no background object does, brightness predicts the mask more cheaply than body shape. A larger network and sharper renderer may exploit the artifact even more reliably.

A shortcut is a predictive rule that performs well under training correlations but fails when those correlations change. Common synthetic shortcuts include:

• semantic class encoded by material, instance color, asset family, or file naming artifacts;
• positive scenes rendered with one weather preset and negative scenes with another;
• action labels correlated with initial danger because expert policies brake only in hazards;
• object masks aligned with unrealistic anti-aliasing boundaries;
• depth or flow inferred from a fixed camera path rather than image evidence;
• padding, compression, exposure, or resolution revealing which generator branch produced a sample.

5.2 Use counterfactual pairs

Visual inspection finds obvious artifacts but cannot establish what a model uses. Build paired tests that hold task-relevant truth fixed while changing a suspected nuisance. In the shuttle scene, replay the same geometry and pedestrian trajectory while changing van color, clothing texture, background, renderer seed, or exposure within the valid envelope. If collision truth is unchanged, a robust decision should remain sufficiently stable.

Here z contains the task-relevant state and n is the nuisance. A large Δ_nuisance is evidence of sensitivity, not automatically proof of an invalid model: appearance can affect genuine observability. Interpret it relative to the task. Changing van paint at fixed sensor signal is impossible; changing paint in the world legitimately changes reflected light. The question is whether the decision changes more than the information change warrants.

5.3 Actions require stronger interventions

Suppose we compare ordinary recorded “brake” episodes to “continue” episodes. Expert drivers brake in dangerous contexts, so action is confounded with initial hazard. The data may suggest that braking causes collisions simply because braking is observed where collision risk was already high.

A simulator can do better. Save the full state and random-number state immediately before the decision, clone it, and apply do(a_t=brake) in one branch and do(a_t=coast) in another. The notation do(·) means forcibly set the action rather than merely observe it. Matched branches isolate action consequence under shared context. Lesson 2 will implement these branchable scenarios, and Lesson 6 will use them for world-model evaluation.

6 · Training, validation, and the locked real test

A generator is a model with tunable choices. If the same examples guide those choices and report final performance, evaluation leaks. Separate the roles before producing data:

A random frame split is usually invalid for temporal synthetic data. Adjacent frames share nearly everything. Two clips may share the same base scene, CAD family, trajectory checkpoint, or random-seed lineage. Split at the highest shared cause that the evaluation intends to generalize beyond. Keep matched counterfactual siblings together unless the test explicitly studies branch generalization.

6.1 A concrete split for the shuttle

Training receives towns A and B, asset families 1–8, and scenario seed lineages 1–80. Development validation receives different routes and times within town C plus asset families 9–10. The locked test uses separately collected real runs in town D, including a prespecified dusk-emergence slice. If all splits reuse the identical parked van mesh and street background, high performance may measure memorization of appearance rather than generalization to occlusion geometry.

Do not let the locked set become the generator's steering wheel. Generator changes can be selected on a bounded development set, ideally with uncertainty estimates and a written hypothesis. Final acceptance remains real because a synthetic benchmark necessarily grades the assumptions that created it.

7 · Distribution design as constrained experimental design

Let θ collect generator parameters and q_φ(θ) be the distribution used to sample them, controlled by design parameters φ. Under render-and-simulate budget B, the conceptual goal is:

This is a nested objective: a generator produces data, training produces a model, and deployment evidence judges the model. It is rarely differentiable end to end, and real risk is only partially observable. Treat it as disciplined experimental design rather than pretending to solve the equation exactly.

7.1 Classify parameters before sampling

Independent uniform sampling is not neutral. It asserts a distribution and often breaks dependencies. Heavy rain should influence illumination, surface reflectance, LiDAR returns, braking distance, and behavior together. Sample a high-level condition, then condition downstream variables on it. Preserve the distinction between plausible distribution coverage and deliberate stress testing; both are useful, but they answer different questions.

7.2 Allocate budget by information, not frame count

Ten thousand adjacent frames of one approach do not equal ten thousand independent encounters. Count base scenes, asset families, trajectories, decisions, interventions, sensor regimes, and failure cells. Spend expensive high-fidelity rendering where measurement fidelity affects transfer; use cheaper structural passes to reject invalid scenarios and explore broad support.

A practical loop is:

1. diagnose a real validation failure slice;
2. state a mechanism-level hypothesis, such as “rolling-shutter timing creates late localization”;
3. change only the relevant generator factor and create controlled variants;
4. retrain with fixed compute and seeds where feasible;
5. measure the prespecified real slice plus regression slices;
6. retain the change only if transfer improves with acceptable uncertainty and no material regression.

8 · From contract to generator: an implementable recipe

8.1 A minimal scenario schema

The representation below is engine-agnostic. It is intentionally explicit about units, dependencies, provenance, sampling probability, and the split unit:

task:
  decision: "brake if collision-under-continue risk exceeds threshold"
  horizon_s: 2.5
  latency_budget_ms: 80
  outputs: [collision_probability, pedestrian_occupancy_ego]

scenario_family: pedestrian_from_behind_parked_van
split_unit: {base_scene_id, asset_family_ids, seed_lineage}

initial_state:
  ego_speed_mps:        {distribution: truncated_normal, mean: 7.0, sd: 1.2}
  van_pose:             {conditional_on: lane_geometry, constraints: [legally_parked]}
  pedestrian_profile:  {conditional_on: region_and_scenario}
  illumination:         {condition: dusk, calibrated_range: true}

behavior:
  pedestrian_intent:   {choices: [wait, cross], probabilities: [0.35, 0.65]}
  emergence_time_s:    {conditional_on: time_to_arrival}
  ego_action:           {branches: [brake, coast], branch_from_checkpoint: true}

dynamics:
  road_friction:        {conditional_on: surface_wetness}
  actuator_delay_ms:    {distribution: measured_empirical}

observation:
  camera: {shutter: rolling, rate_hz: 30, calibration_version: cam_2026_04}
  lidar:  {scan_model: packet_level, rate_hz: 10, calibration_version: lid_2026_03}
  radar:  {rate_hz: 20, calibration_version: rad_2026_02}

labels:
  ontology_version: shuttle_v3
  coordinate_frame: ego_at_decision_time
  visibility_rule: visible_projected_fraction
  counterfactual_action: coast

provenance:
  generator_version: required
  asset_hashes: required
  master_seed: required
  proposal_probability: required

A real schema should also record engine version, code commit, unit conventions, rejected-scenario reason, every derived seed, and per-sensor capture time. The point is not YAML syntax; it is replayability and auditability.

8.2 Linearized generator-design algorithm

INPUT: task contract C, real development slices V, compute budget B

1. Enumerate costly failure slices A_1 ... A_K from C.
2. For each A_k:
     list latent causes;
     map every cause to p(s0), policy, transition, observation, or label;
     mark it known, uncertain, or adversarial;
     define a measurable coverage cell and an acceptance metric.
3. Build the smallest conditional scenario program with nonzero support
   for every decision-relevant cell. Reject invalid states before rendering.
4. Assign proposal probabilities and a split lineage before sampling.
5. For each accepted scenario until budget B is spent:
     save configuration, seeds, versions, and proposal probability;
     simulate authoritative state;
     checkpoint before the decision and branch required actions;
     render timestamped sensor observations;
     derive labels from the same state and timing contract;
     run geometry, physics, sensor, label, and leakage QA;
     export sample plus provenance.
6. Train with a declared real/synthetic mixture.
7. Evaluate synthetic invariance tests and frozen real development slices.
8. Change one mechanism in response to a written failure hypothesis.
9. Freeze generator and model choices; run the locked real acceptance suite.

The word smallest matters. Do not build every weather effect, city, or sensor failure before testing whether the basic pipeline transfers. Begin with a generator that spans the hypothesized causes. Add fidelity when an ablation shows that the missing mechanism changes real performance or closes a verified measurement gap.

8.3 A worked mini-design

Suppose real validation reveals late braking specifically when the pedestrian first becomes visible between camera rows during a LiDAR scan. We linearize the response:

1. Decision: collision probability must cross the brake threshold early enough to satisfy the 80 ms budget.
2. Causes: occluder geometry and emergence timing affect visibility; camera row timing and LiDAR scan phase affect measurement; pedestrian and ego speeds affect truth; actuation delay affects consequence.
3. Factor map: geometry and initial speeds enter p(s₀); emergence controller enters π; motion and braking enter T; rolling shutter and LiDAR schedule enter O; collision horizon and time frame enter L.
4. Support: ensure emergence occurs across all phases of both sensor schedules and around the stopping boundary, not only at convenient frame times.
5. Interventions: replay identical state with global versus calibrated rolling shutter as a diagnostic; vary van texture separately; branch brake and coast at the same checkpoint.
6. Acceptance: measure time-to-correct-threshold, calibration near 0.02, false negatives in the frozen real slice, and closed-loop stopping margin. Do not select on synthetic average accuracy.

If calibrated rolling shutter improves the targeted real slice while texture randomization does not, invest in timing fidelity rather than additional materials. If neither improves transfer, revisit the hypothesis instead of hiding the result inside a larger synthetic mixture.

9 · Diagnose failures by asking what was wrong

10 · What not to optimize first

11 · Self-test and exercises

11.1 Quick checks

1. What is the first artifact to write? A versioned task contract connecting sensors and model outputs to a deployed decision, loss, label semantics, failure slices, and real acceptance evidence.
2. Why separate π from T? The policy determines which actions and states are visited; dynamics determines the consequences of those actions. Perfect physics cannot recover an action never sampled.
3. Why is support more fundamental than density matching? Reweighting can adjust frequencies only where samples exist. A critical event with zero synthetic probability contributes no evidence.
4. Is road wetness a nuisance? It depends on the task. It may be nuisance for semantic identity, a measurement cause for camera/LiDAR, and a task-relevant dynamics cause for stopping distance.
5. Why can a locked synthetic test not be the sole acceptance set? It remains generated by the same family of assumptions. It tests consistency with those assumptions, not adequacy for reality.

11.2 Worked exercises

Exercise A—factor mapping. A warehouse robot misses a transparent door, then collides because its controller has a 120 ms delay. Place the glass geometry, transmitted/reflected light, controller delay, and collision label into the five-factor model.

Solution. Glass geometry begins in p(s₀); optical transmission/reflection belongs to O(o|s); controller/actuator delay belongs to T(s′|s,a) if represented as physical/controller state evolution; collision semantics and horizon belong to L. The navigation policy that selects an action is π.

Exercise B—finite coverage. A target slice has proposal probability 0.02 per independent episode. Approximately how many episodes give at least a 95% chance of seeing it once?

Solution. Solve (1−0.02)^N≤0.05. Thus N≥log(0.05)/log(0.98)≈148.3, so at least 149 independent episodes. This guarantees only one occurrence with high probability, not enough variation for robust learning.

Exercise C—shortcut test. All positive shuttle scenes were generated at dusk and all negative scenes at noon. Propose a minimal diagnostic set.

Solution. Create a balanced 2×2 intervention: positive and negative states rendered at both noon and dusk, with state and sensor validity controlled. Compare predictions across illumination at fixed label and across label at fixed illumination. Then verify the suspected dependence on real slices; synthetic balance alone does not establish real robustness.

Exercise D—split leakage. Frames 1–100 of a trajectory are training data and frames 101–120 are validation data. Why is the estimate optimistic, and what is the repair?

Solution. Adjacent frames share scene, assets, state history, lighting, behavior, and often pixels. Split entire trajectory lineages—and, when the claim requires it, base scenes and asset families—before rendering.

Exercise E—design challenge. Draft a task contract and support ladder for a robot that estimates bin-picking grasp success. Name one cause in every factor and one matched nuisance intervention. There is no single correct answer; the grading criterion is whether every variable connects to the deployed grasp decision and a measurable acceptance test.

12 · The linear chain to remember

Every construct in this lesson exists because a simpler pipeline breaks in a specific way:

Bridge to lesson 02

We now know what distribution we need, but not yet how to make a prebuilt 3D scene produce it. A visual scene may lack metric units, stable identities, collision geometry, behavior interfaces, semantic constraints, and replayable state. Lesson 02 turns that asset into a simulation-ready world template, expresses the shuttle encounter as a conditional scenario program, rejects invalid samples before rendering, and creates matched brake/coast branches from a shared checkpoint.