Simulation-ready scenes and structured scenarios

A prebuilt 3D scene is an appearance. A data engine needs a world: persistent identity, metric state, physical rules, controllable actors, valid scenario programs, and checkpoints from which alternative futures can be compared.

0 · Begin with the decision, not the renderer

Suppose an autonomous shuttle is approaching a residential crosswalk at dusk. A delivery van is parked near the curb. A pedestrian walks from behind it. The shuttle must decide whether and when to brake. A front rolling-shutter camera, scanning LiDAR, and radar will eventually observe the event.

It is tempting to open the engine, place a pedestrian, move a camera, and export frames. That produces pixels quickly, but it does not yet produce trustworthy evidence. Was the pedestrian actually supported by the pavement or floating two centimeters above it? Did the collision shape match the visible van? Was the pedestrian reachable from the chosen sidewalk? Did “dusk” change only the sky color, or also headlights, exposure, road reflectance, and pedestrian behavior? Were brake and coast clips generated from the same world state? Can the exact sample be replayed after the scene changes?

The fundamental gap: an image-producing description is not an experiment

A prebuilt visual scene usually implements a function like I=R(g,ℓ,c): render geometry g under lighting ℓ from camera c, and obtain an image I. This is enough for illustration. It is not enough for an experiment about decisions, because an experiment needs an initial state, a legal intervention, a transition rule, and a measurement process:

The renderer supplies only part of O. It does not tell us which rendered node remains the same object over time, whether “behind the van” is a meaningful relation, which actions are legal, or what braking does to future state. An engine may contain a physics solver, but imported art does not automatically declare the masses, collision volumes, joints, or controllers that make that solver describe our intended world. The missing work is not cosmetic metadata. It is the work of defining s, a, and T.

Why does a learning system care? Because two latent worlds can produce the same first image and different supervision. In world A, the visible pedestrian mesh, semantic instance, and collider share one physical root moving at 1.2 m/s. In world B, an animator keyframes the mesh at the same apparent speed while its collider and reported object pose remain at the curb. At the first frame the RGB can be identical. One second later the image says the pedestrian moved 1.2 m, while state-derived velocity and collision labels say it did not. A tracker is punished for following the visible pedestrian; a world model is asked to assign two futures to apparently identical state and action; a policy can learn that passing through the pedestrian has no consequence. More rendering samples make this contradiction more statistically confident.

This gives the first-principles criterion for synthetic truth: a property is ground truth only if the generator has one authoritative representation of it and every observation and label is derived from that representation. A convenient engine variable is not automatically truth. We must decide what owns identity, pose, contact, behavior, and time, then force all downstream products to agree.

These questions reveal the order in which the system must be built:

This is linearized thinking: each stage establishes an invariant required by the next. If metric transforms are unreliable, colliders cannot be aligned. If colliders are unreliable, trajectories are not physical. If trajectories are not physical, a sensor-perfect rendering still teaches the wrong world. We therefore resist “render first, repair metadata later.”

1 · An art scene is not yet a simulation state

An art scene is optimized to look convincing. Hidden geometry can be missing; scale can be adjusted by eye; lighting may be baked into textures; a moving door can be a single static mesh. A simulation state must answer a stronger question: what exists, where is it, what can it do, and what laws determine its next state?

Let the authoritative state at time t be

Here G_t is the scene graph with identities and transforms; q_t and v_t are poses and velocities; m contains material and mass properties; c contains collision geometry; j contains joints and limits; b_t is behavior/controller state; and e_t contains environmental state such as illumination and wetness. Pixels are not in this definition. They will be observations sampled from it.

Scene graph: identity plus composition

A scene graph is a hierarchy of named entities. Each node has a stable identity, a parent, a local transform, and typed components such as render mesh, collider, rigid body, light, or semantic class. A child transform is interpreted relative to its parent. In the running case, the van body may be a parent of four wheel nodes; the shuttle body may be the parent of a calibrated sensor rig; the camera, LiDAR, and radar mounts are children of that rig.

The world transform of node i is the ordered product of local transforms along its ancestry:

Order matters. If the shuttle is at world position (20,3,0) m and its camera mount is (1.8,0,1.45) m in the shuttle frame, simply adding the vectors is correct only when the shuttle has no rotation or scale. With a 90° yaw, the mount offset must first rotate with the vehicle. The safe implementation composes 4×4 transforms; it does not scatter ad hoc coordinate additions through exporters.

Choose and document one world convention: for example, meters, right-handed axes, +z up, and yaw about +z. Importers must explicitly convert source conventions. Freeze or eliminate non-uniform scale before physics; otherwise a visible body, its collider, its inertia, and its children can silently disagree. A unit test should place a known point on each sensor axis and verify its world location after import.

Ontology: what the nodes mean

An ontology is a versioned vocabulary of entity types and relations. It distinguishes at least semantic class (“vehicle”), instance (“van_47”), asset family (“delivery_van_B”), and part (“front_left_door”). It may also record relations such as parked_on(van, curb_lane), occludes(van, pedestrian, shuttle_camera), and affordances such as openable(door).

Stable UUIDs must originate in the scene contract, not in renderer load order. Renderer object index 12 may become 13 when a lamp is added; UUID asset:van_B/instance:0047 should not. At export time, compact per-frame segmentation IDs can be derived from those UUIDs and accompanied by a lookup table. This lets a mask stay small without making identity unstable.

Ontology design affects what can be learned. If all people are labeled “pedestrian,” a model cannot be evaluated on children unless age group is retained as an attribute. Conversely, an ontology should not claim facts the source assets cannot support. The rule is: preserve the most detailed authoritative truth, then map it into task-specific label taxonomies later.

A practical asset contract

Derive the contract backward from what a model must learn

Each layer above exists to remove a specific ambiguity from supervision. The derivation is easiest to see from the model’s side:

Stable identity is therefore not “extra annotation”; it makes temporal statements logically possible. An ontology is not a nicer class list; it gives variables and relations names that a scenario program can condition on. Metric frames make independent subsystems agree on the same geometry. Physics and controllers make actions denote transitions. Provenance and checkpoints make claims replayable. Each is the smallest repair for a concrete non-identifiability in the data.

Worked thought experiment: a small metric error reverses the decision label

The shuttle travels at 7 m/s. Suppose braking produces a roughly constant 4 m/s² deceleration after a 0.8 s controller and actuation delay. The distance traveled before stopping is approximately

If the pedestrian’s conflict point is truly 11.5 m ahead, immediate braking is barely necessary and even then the margin is only −0.225 m. Now imagine the road was authored in centimeters, the shuttle in meters, and an importer “fixes” their visual alignment with a hidden 1.03 scale. A scenario script measuring in the wrong frame reports the conflict point as 11.845 m. That 34.5 cm error flips the calculated margin from −22.5 cm to +12 cm and can flip a safe/unsafe or brake/no-brake target, even though a human cannot detect a 3% scene-scale error in one rendered image.

The error then propagates. The collider reaches the crossing at the wrong time; time-to-collision coverage bins are wrong; matched branches are checkpointed at the wrong decision boundary; camera depth and LiDAR range disagree if only one transform path contains the scale. This is why coordinate contracts and worked numerical sanity checks come before high-volume rendering. “Looks aligned” is a test of appearance; the task requires a test of consequences.

2 · Make physical state agree with visible state

A render mesh answers “what surface should be shaded?” A collider answers “where may physical contact occur?” The two need not have the same tessellation, but they must represent the same occupied volume under the same transform. A complex van can use a few convex collision pieces for stability. It must not use a single box extending through the visually open space under the chassis if that difference changes pedestrian or LiDAR interactions relevant to the task.

Mass, center of mass, and inertia

Mass alone does not determine rotational response. The inertia tensor describes how mass is distributed around the center of mass. For a uniform box of mass m and side lengths w,h,d, principal moments are

Consider a 1,800 kg van approximated by a 2 m × 2.2 m × 5 m box. Around the vertical axis, I_z ≈ 1,800(2²+5²)/12 = 4,350 kg·m². If an import bug interprets centimeters as meters, lengths grow by 100 and inertia by 10,000 for the same declared mass. The van may look acceptable after a camera adjustment while its steering and collision response become absurd. This is why unit normalization precedes physics.

Static road, curb, and building nodes have fixed transforms. The parked van may be kinematic if its pose is controlled but it does not respond to forces. The pedestrian and shuttle are dynamic or controller-driven bodies whose velocities are part of state. Record the body type explicitly; do not infer it from whether an animation happens to exist.

Joints and controllers

A joint constrains relative motion between bodies. It declares an axis or subspace, limits, damping, and possibly a motor target. A wheel needs a rotational joint; a door needs a hinge with realistic limits; a pedestrian skeleton has linked joints with anatomical ranges. A controller converts a goal or action into forces, torques, velocities, or joint targets. It is part of the transition model, not merely presentation code.

For the shuttle, a longitudinal controller may map desired acceleration to throttle and brake under a vehicle dynamics model. For the pedestrian, a navigation controller follows a path while an animation system matches gait to physical root velocity. If the animation translates the visible mesh while the collider remains at the skeleton origin, labels, contact, and appearance diverge. The authoritative root pose must drive both.

Affordances connect semantic objects to possible actions

An ontology says what an entity is; an affordance says what interactions the current state permits. The road is drivable, the sidewalk is walkable, a crosswalk connects two walkable regions, the van blocks lines of sight, and a closed door may be openable around a particular hinge. Affordances are state-dependent: a route blocked by construction is not currently traversable even if its semantic class remains “sidewalk.”

Why not let each behavior script discover this directly from triangles? Raw geometry underdetermines intention. A flat shop roof and a sidewalk can have similar polygons, yet only one belongs in the pedestrian’s route distribution. Conversely, hard-coding one path into an animation makes the path possible by decree but gives no way to sample a nearby legal route or reject a newly blocked one. Affordances are the minimal bridge between ontology and control: they reduce “choose any motion in 3D” to “choose an action available to this actor in this state.”

For an action-conditioned model, this bridge is essential. If a pedestrian goal is sampled through walkable and connected affordances, changes in trajectory follow explainable changes in goal, obstacles, or behavior. If clips are merely selected by filename, the model sees motion but not the state variables that make motion possible. It must memorize visual correlations instead of learning a transition rule.

3 · Separate the reusable world template from the sampled scenario

Once assets are authoritative, organize them into two layers. A world template defines persistent possibilities: road graph, sidewalks, crosswalks, curb geometry, legal parking regions, buildings, sensor mounts, illumination controls, actor types, and available behavior modules. It is a reusable stage with typed slots and legal relations.

A scenario is a particular initial condition and temporal program within that template: this van is parked in this region; this pedestrian intends to cross along this route; the shuttle begins at this speed; dusk has this sun angle; the pedestrian emerges at this time; the clip ends after the conflict resolves.

The distinction prevents two opposite failures. If everything is baked into one scene, coverage requires hand-authoring thousands of files and causal factors are hard to identify. If everything is treated as an unconstrained random variable, impossible worlds dominate. A template encodes what remains structurally true; a scenario program samples what should vary.

This separation is not software architecture for its own sake. It is the minimal decomposition required to say what is shared and what was sampled. Let B denote a base world template and θ a scenario manifest. Then a trajectory is generated as

If there is no B, every sample duplicates roads, coordinate conventions, legal regions, and sensor mounts. Those copies drift, so a difference attributed to “pedestrian speed” may also contain a subtly moved curb. There is no stable unit for base-scene splitting or regression testing. If there is no explicit θ, variation lives in editor files, scripts, and hidden random calls. The sample cannot state why this van, route, timing, or action occurred, so it cannot be replayed or stratified. Template plus scenario is the smallest pair that supports both reuse and experimental variation.

The distinction also prevents accidental causal edits. Fixing a broken curb collider belongs to version B and should invalidate or regenerate all dependent trajectories. Changing emergence time belongs to θ and should leave the base world hash unchanged. Without that boundary, maintenance changes and experimental interventions become indistinguishable.

world_template:
  id: residential_crosswalk_v3
  coordinates: {unit: meter, handedness: right, up: +z}
  regions:
    curb_lane: {type: drivable, speed_limit_mps: 8.3}
    van_parking: {type: parking, polygon: [...]}
    pedestrian_spawn: {type: sidewalk, polygon: [...]}
  relations:
    - adjacent(pedestrian_spawn, van_parking)
    - connects(crossing_route, pedestrian_spawn, far_sidewalk)
  sensor_rig: shuttle_front_rig_v5

scenario_program:
  intent: pedestrian_crosses_from_occlusion
  initial: {shuttle_speed_mps: 7.0, weather: dry_dusk}
  events:
    - when: time_to_collision in [1.2, 2.5]
      do: pedestrian.enter(crossing_route)
  terminate: conflict_resolved or horizon_s == 6

This schema is a contract, not necessarily the engine’s native file format. An adapter can instantiate it in Blender, Unreal, Unity, Isaac Sim, or another engine. Engine-independent intent and identities make the dataset definition reviewable even when renderer or physics implementation changes.

4 · A scenario generator is a conditional program

Independent randomization says “draw every knob from a range.” Structured generation says “draw causes in an order that respects their dependencies.” The latter is necessary because real worlds occupy a thin, structured subset of all numerical combinations. A van is parked in a parking region, not uniformly in 3D. A pedestrian’s route begins on a walkable support surface. Wetness co-varies with tire friction, reflections, spray, and stopping distance.

Let z be high-level intent, such as pedestrian_emerges_behind_van. A useful factorization is

ℓ selects a compatible layout branch; x gives asset identities and poses; b gives goals and behavior parameters; and e gives environment state. This is a conditional grammar: earlier choices enable and constrain later productions.

def sample_occluded_crossing(rng, template, target_cell):
    intent = "pedestrian_emerges_behind_van"
    parking_region = rng.choice(template.regions.compatible_with(intent))
    van = sample_asset(family="delivery_van", split=target_cell.split, rng=rng)
    van_pose = sample_supported_pose(van, parking_region, rng)

    # Occlusion is defined relative to a prospective shuttle trajectory.
    shuttle_route = template.road_graph.route_to(parking_region)
    pedestrian_route = sample_walkable_crossing(behind=van, rng=rng)
    emergence_time = solve_conflict_timing(
        shuttle_route, pedestrian_route, target_ttc=target_cell.ttc_range)

    environment = sample_dusk_state(rng, correlated=True)
    return Scenario(intent, van, van_pose, shuttle_route,
                    pedestrian_route, emergence_time, environment)

The code samples relations before exact coordinates. “Behind the van from the shuttle’s viewpoint” is the semantic requirement; a geometric solver then finds poses satisfying it. This is more robust than hard-coding x ranges that are valid in only one street orientation.

Why conditional generation is the minimum, not an optional refinement

Suppose independent sampling has a 0.15 chance of placing the van in a legal parking area, a 0.20 chance of choosing a pedestrian spawn connected to the crossing, a 0.10 chance of making the van actually occlude the pedestrian, a 0.08 chance of producing the target conflict timing, and a 0.70 chance of keeping the event in the future sensor field of view. Even pretending these events are independent, full validity is only

Compute is not the deepest problem. The few accepted samples come from accidental intersections of the ranges and may cluster in one geometric corner. The rejection process silently reshapes the distribution, so “we sampled van pose uniformly” is false of the final dataset. Worse, if validation is relaxed to improve yield, invalid co-occurrences become shortcuts.

A conditional grammar samples a legal parking region first, then a supported van pose; selects a spawn connected to a crossing; defines occlusion relative to the shuttle route; and solves timing for a requested conflict cell. This turns rare accidental coincidence into an explicit semantic construction. Constraints are still necessary because numerical geometry and dynamics can fail, but conditional sampling places proposals near the valid manifold. Thus the minimal useful generator is not a bag of knobs. It is a dependency graph plus constraint checks.

Structural assumptions must be explicit

A conditional program implies a causal story. In the running case, time of day causes illumination and headlight activation; road wetness causes both reflectance changes and lower friction; pedestrian intent and free path cause motion; shuttle action causes future speed. Camera exposure does not cause the sun angle, even though they are correlated.

These are structural causal assumptions: declarations about which variables may directly determine others. They need not be philosophically perfect, but they must be stated because intervention data depends on them. If “brake” also triggers a random weather draw, the branch no longer isolates braking. If hazard severity directly selects action in every rollout, observational action labels are confounded.

Structured Domain Randomization (Prakash et al., ICRA 2019) uses probabilistic scene grammars to capture this kind of conditional structure. The broader principle is independent of a particular tool: randomize uncertain causes, preserve real dependencies, and avoid using randomness as a substitute for a world model.

5 · Turn semantics into constraints, then reject invalid worlds cheaply

A sample from the grammar is a proposal, not yet an accepted scenario. A constraint is a test or equation that must hold. Some can be enforced constructively; others require rejection after checking. The acceptance rule is

For the van, a supported-pose constraint places its tire contact points on the road and keeps its footprint inside the parking polygon. A non-penetration test checks the van against curb, poles, and other vehicles. For the pedestrian, the route must lie on the walkable navigation mesh and avoid static obstacles. For the event, the van must occlude the pedestrian from the shuttle sensor origin for a specified pre-emergence interval.

One useful occlusion test casts rays from the prospective sensor origin to sampled points on the pedestrian’s body. Before emergence, require the visible fraction v to be below a threshold, for example v≤0.1; after emergence, require v≥0.5 for enough frames to supervise recognition. The exact thresholds come from the task contract, not aesthetics.

Timing can be solved from distances and speeds before running expensive physics. In one sampled variant, if the shuttle is 17.5 m from the conflict point at 7 m/s, constant-speed time to conflict is 2.5 s. If the pedestrian is 2.4 m from that point and walks at 1.2 m/s, its time is 2.0 s. The program can delay the pedestrian by 0.5 s to target simultaneous arrival, then let the controller and physics engine determine the exact trajectory. This simple calculation creates a hard case deliberately instead of hoping random spawn times produce one.

Order checks by cost and repairability

1. Schema and semantic checks: required IDs exist, regions are compatible, licenses and split assignments are legal. These take microseconds and should fail before engine loading.
2. Static geometry checks: bounds, support, overlap, route reachability, and approximate occlusion. Repair by resampling local pose variables.
3. Short physics settle: detect exploding joints, tunneling, non-finite velocities, or spontaneous energy gain. Repair asset physics or reject the proposal.
4. Coarse temporal rollout: verify that the event occurs within the horizon and actors remain feasible. Repair timing or behavior parameters.
5. Capture feasibility: verify sensor schedules cover the necessary pre-event and post-event context. Only then pay for full-fidelity rendering.

Naive rejection can become wasteful when valid space is narrow. Log rejection reasons and rates. If 95% of pedestrian routes fail reachability, do not buy more compute; change the proposal distribution to sample from connected navigation components. Constraint solving should progressively move probability mass toward valid worlds without erasing the desired edge cases.

// independent: x, y ~ U(0,1) drawn separately
// conditional: x ~ U(0,1); gain = clamp((1 − x) + noise)   // gain follows the cause
valid = |y − (1 − x)| ≤ 0.22                                // physically plausible band

6 · Roll out behavior and physics as the transition model

After initialization, the engine advances state according to actions, controllers, contacts, and dynamics:

a_t contains commanded actions such as braking; ξ_t contains exogenous disturbances such as a sampled reaction delay or wind perturbation. The fixed time step, substeps, collision settings, controller versions, and solver version are part of the dataset provenance. Changing them changes T.

The pedestrian controller should receive a route and behavioral parameters—walking speed, reaction delay, stop tendency—not a sequence of world-space poses blindly copied into the engine. The shuttle controller should receive brake/coast commands through the same interface used by downstream policy evaluation when practical. This makes actions operationally meaningful.

Perfect determinism is useful but not always available. Parallel physics, GPU simulation, or floating-point order can introduce small divergence. Define a replay tolerance: stable IDs and event order must match exactly; poses and velocities must match within declared numerical bounds. Hash configuration and initial state, record engine build, and test replay regularly rather than assuming a seed is sufficient.

During rollout, log the authoritative state at a higher or equal rate than any future sensor. Never reconstruct ground-truth motion from rendered frames. Record continuous poses or enough state for interpolation, because the rolling-shutter camera and scanning LiDAR in Lesson 03 sample different points in time within a nominal frame.

7 · Seeds reproduce draws; checkpoints reproduce worlds

A seed initializes a pseudo-random-number stream. Replaying the same seed reproduces draws only if code paths, stream consumption, asset ordering, and software versions are unchanged. Adding one random draw early in a monolithic generator can shift every later choice.

Use named or hierarchical streams—such as layout, actors, behavior, environment, and sensor—derived from a root sample ID. Then changing texture randomization does not silently change the pedestrian’s emergence time. Store the sampled parameters as well as the seeds; seeds are compact recipes, while realized manifests are auditable facts.

A checkpoint is a serialized snapshot sufficient to resume evolution: entity transforms and velocities, articulation and controller state, simulation clock, event queue, random-stream states, environment state, and engine/configuration hashes. A save file containing only object poses is not a causal checkpoint if controller integrators or random states are missing.

Matched brake/coast interventions

At 1.0 s before the pedestrian emerges, save checkpoint s_t*. Clone it into two branches. In branch A issue brake; in branch B issue coast. Keep the pedestrian’s intent, reaction delay, wind, illumination, and all prior history matched:

Because the branches share state and exogenous randomness, their difference is attributable primarily to the action. This is the simulator analogue of a controlled experiment. Comparing a brake clip from a dangerous scene with a coast clip from an easy scene would instead mix action effect with context: expert drivers brake precisely when danger is greater.

From the learner’s perspective, this is the difference between correlation and a usable action derivative. A matched pair supplies two futures near the same point in state space, approximating how outcome changes when action changes. Unmatched clips may contain more total diversity, but their state difference can be much larger than their action difference. A sequence model can minimize loss by reading hazard cues from the initial image and mostly ignoring the action token. Checkpoint control makes that shortcut insufficient: the same visible history is followed by meaningfully different, physically generated consequences.

Generate two complementary sets:

• Common-random-number branches share disturbances across actions. They reduce nuisance variance and are ideal for learning local action effects.
• Independent-disturbance repeats redraw uncertain pedestrian reactions or friction for each action. They estimate the distribution of outcomes and tail risk.

Do not confuse a matched intervention with a claim that the simulator is causally complete. Unmodeled real-world causes still create a sim-to-real gap. The value is internal identification: within the stated model, the changed variable and held-fixed variables are known.

8 · Measure coverage over causes, not only exported frames

A million adjacent frames can describe almost one event. Coverage must be tracked over scenario variables that alter the decision: van asset family and pose, pedestrian morphology and clothing, emergence side, shuttle speed, time-to-collision, occlusion fraction, road friction, illumination, sensor rig, and action branch.

Define coverage cells before generation. One cell might request: pedestrian, left-side emergence, 1.2–1.6 s time-to-collision, heavy occlusion, dry dusk, brake branch. The generator targets underfilled cells, while rejection logs reveal cells that are infeasible under the current template. Keep both proposal counts and accepted counts; otherwise a “balanced” accepted dataset can conceal severe generator bias and wasted compute.

Coverage is not the same as uniformity. Deployment prevalence, safety importance, model uncertainty, and cost may imply different weights. Common easy cases teach the base distribution; rare boundary cases teach decision surfaces. Report the mixture rather than pretending one sampling rule serves every purpose.

Split at the highest shared cause

Assign train, validation, and test before rendering, using the highest-level lineage that could leak reusable appearance or geometry:

Frame-level random splits leak nearly identical views. Instance-level splits can still leak the same CAD model with another color. Brake and coast siblings share a checkpoint and therefore normally belong in the same split. Keep an immutable split assignment in each asset and scenario manifest, and make the sampler refuse incompatible assets rather than moving samples afterward.

9 · Implementation blueprint from import to accepted trajectory

def build_sample(sample_id, target_cell, catalog, engine):
    streams = NamedStreams.from_root(sample_id)

    # 1. Select only assets already certified by the asset contract.
    assets = catalog.select(target_cell, split=target_cell.split,
                            rng=streams["assets"])
    assert all(a.units == "meter" and a.ontology_version for a in assets)

    # 2. Instantiate persistent structure, then sample conditional intent.
    world = instantiate_template("residential_crosswalk_v3", assets, engine)
    proposal = sample_occluded_crossing(streams["scenario"], world, target_cell)

    # 3. Reject or locally repair from cheap checks to expensive checks.
    require(schema_valid(proposal))
    proposal = solve_support_and_timing(proposal)
    require(static_geometry_valid(proposal))
    engine.load(proposal)
    require(settle_and_check_physics(engine, seconds=0.5))
    require(coarse_event_rollout_valid(engine, target_cell))

    # 4. Replay from initial state and save a complete pre-decision checkpoint.
    engine.restore_initial()
    advance_until(engine, event="pre_emergence_decision")
    checkpoint = engine.checkpoint(include_rng=True, include_controllers=True)

    # 5. Clone matched action branches; render only after world QA passes.
    branches = {}
    for action in ["brake", "coast"]:
        engine.restore(checkpoint)
        engine.apply_action(action)
        branches[action] = rollout_and_log_authoritative_state(engine)

    manifest = write_manifest(sample_id, streams, assets, proposal,
                              checkpoint.hash, branches, engine.version)
    return manifest

Notice that this function returns a manifest of world trajectories, not images. A downstream measurement job consumes the manifest and sensor calibration to create observations. This boundary enables re-rendering the same causal event with a corrected camera model without resampling behavior, and it makes disagreements traceable to either the world or measurement stage.

10 · What existing generation systems contribute

These systems demonstrate a shared architecture: structured 3D owns world truth; render passes are derived views. They can save enormous implementation effort. They are not automatic answers to “what data should exist?” A prebuilt commercial engine similarly supplies rendering, physics, assets, and tooling, while your scenario contract supplies meaning, coverage, and evidence.

11 · Failure modes and what each one teaches

12 · Pre-render QA checklist

Use this as a release gate for the world-trajectory artifact. Each check exists because a later stage assumes it.

• Coordinates: units, handedness, up axis, transform order, and sensor mounts pass known-point tests; no hidden non-uniform scale remains.
• Identity: every relevant entity and part has a stable UUID, ontology version, asset family, and split assignment; compact render IDs have an explicit mapping.
• Geometry and physics: render geometry and colliders align; support, clearance, joint limits, mass, inertia, friction, and body type are plausible in SI units.
• Behavior: routes are reachable; controller, root, collider, and visible actor remain synchronized; events occur inside the requested horizon.
• Scenario semantics: the van actually occludes the pedestrian from the relevant sensor origin; time-to-collision and visibility fall inside the requested coverage cell.
• Numerics: settle and rollout contain no non-finite state, tunneling, unexplained energy gain, or out-of-bounds pose; physics configuration is hashed.
• Reproducibility: realized parameters, named seeds, asset versions, code version, engine build, and a complete checkpoint are stored; replay meets tolerance.
• Interventions: brake and coast begin from the identical checkpoint; shared and independent disturbances are identified, not implied.
• Coverage and leakage: proposal, rejection, and acceptance counts are recorded per causal cell; no asset, base scene, seed, or checkpoint lineage crosses forbidden splits.
• Handoff: authoritative poses and velocities are logged densely enough for rolling-shutter and scanning-sensor interpolation in Lesson 03.

13 · Self-test: derive, do not memorize

1. Transform exercise. A camera mount is 2 m forward and 1.5 m up in a shuttle frame. Explain why adding this vector to world position fails after the shuttle turns. The local offset must be rotated by the parent orientation before translation; homogeneous transform composition performs both in the correct order.
2. Ontology exercise. Why store semantic class, instance, asset family, and part separately? They answer different questions: category learning, tracking, leakage control, and articulated/part supervision. One renderer index cannot preserve all four.
3. Constraint exercise. Design three checks for “pedestrian emerges from behind van.” For example: the pedestrian route is supported and reachable; van–pedestrian line-of-sight occlusion exceeds a pre-event threshold; the pedestrian becomes sufficiently visible within a target time-to-collision window.
4. Causal exercise. Why is brake-versus-coast comparison across random scenes confounded? Action selection co-varies with danger and context. A matched checkpoint holds prior state and chosen disturbances fixed while intervening on action.
5. Reproducibility exercise. Why is a root seed insufficient? Draw order, code, assets, engine version, solver nondeterminism, controller memory, and event queues can change. Store named streams, realized parameters, versions, and full checkpoint state.
6. Split exercise. A blue and red van share the same CAD mesh. May one enter train and one test? Not if evaluation claims novel geometry or asset generalization; color variants share an asset-family cause and can leak shape. Split according to the intended claim at the highest shared lineage.

14 · Forced bridge to Lesson 03: the world is not the measurement

At this point the shuttle, van, pedestrian, road, and illumination have authoritative trajectories. We know where every surface and body is at every simulation time. We still do not have camera images, LiDAR returns, or radar detections.

A pinhole-style frame sampled at one instant would ignore the front camera’s rolling shutter. A depth map is not a scanning LiDAR point cloud: beams have an angular pattern, fire at different times, may miss dark or grazing surfaces, and return noisy ranges. Ground-truth velocity is not radar: radar measures range, radial velocity, angular response, multipath, and detection uncertainty. Even perfect world state does not license perfect observations.

The handoff artifact must therefore expose continuous-time transforms, geometry/material state, environment state, and calibrated sensor mounts without collapsing them into measurements. Lesson 03 will define the observation model

where τ can vary across rows of an image or beams of a scan. The scene engine answers “what happened?” The measurement model answers “what could this sensor have observed?” Keeping those questions separate lets us diagnose the right system when synthetic data disagrees with reality.