Sim-to-real and hybrid training

Transfer is not a beauty contest between rendered and real pixels. It is the disciplined process of matching the parts of the data-generating process that affect a decision, covering what remains uncertain, and proving the resulting model on measurements the generator never saw.

1 · Start from the objective, not from photorealism

Suppose an autonomous shuttle approaches a parked van at dusk. A pedestrian begins to emerge from behind it. The shuttle observes a front rolling-shutter camera, scanning LiDAR, and radar, and must decide whether to brake. Simulation can vary the pedestrian, van, road, weather, sensor timing, and braking action almost without limit. Yet none of that variation is valuable merely because it exists. It is valuable only if it reduces deployment risk.

Let f be the learned perception-and-decision model, o its sensor observations, y the target contract, and c(y,f(o)) the cost of a prediction. The real objective is

The cost must reflect the application. Missing a pedestrian with 1.2 seconds to collision may cost far more than braking unnecessarily for an empty roadside. Average accuracy hides this asymmetry. A useful evaluation therefore includes safety-weighted risk, calibration, relevant slices such as dusk and partial occlusion, and ultimately closed-loop outcomes such as collision rate, jerk, and stopping margin.

Synthetic data changes the training procedure in the hope of lowering R_real. Pixel similarity is at most an intermediate diagnostic. A perfectly realistic image of an easy empty road may add little information; a visibly imperfect but geometrically faithful counterfactual of a rare emergence may teach exactly the decision boundary the model lacks.

1.1 · The fundamental transfer problem

The model is fitted with evidence from one causal and measurement process but judged under another. Synthetic empirical risk estimates

while the quantity we care about is R_real(f). Their difference is

Read this identity carefully. A distribution mismatch matters only through the loss induced by the particular model and decision. Large pixel changes can be harmless when they alter irrelevant background detail. A tiny, visually inconspicuous timestamp bias can be catastrophic when it shifts estimated time-to-collision across the braking threshold. Therefore a generic image-distance score cannot, by itself, identify or bound the risk difference.

One can construct mathematical bounds using a discrepancy measured over the relevant class of models and losses. But a perceptual metric such as feature or image similarity is not automatically that discrepancy. It may ignore rolling-shutter timing, LiDAR dropout, action selection, label convention, or rare-event support. Conversely, deliberately exaggerated illumination variation may increase visual distance while producing a detector whose real risk is lower. “Looks real” is neither necessary nor sufficient for “transfers.”

There are only three logically complete ways to close the difference. First, make the synthetic process match every decision-relevant part of the real process. Second, make the learned decision invariant to the remaining differences. Third, estimate performance on independent real evidence and reject the change when the first two attempts were insufficient. Sim-to-real engineering is an auditable combination of all three.

1.2 · Think from the model's perspective: the cheapest predictor wins

A learner does not know which synthetic mechanism the designer intended it to use. Optimization rewards any feature that predicts the target in the training distribution. If a renderer artifact is simpler, cleaner, or more consistent than pedestrian evidence, using the artifact is a rational empirical-risk solution.

Consider a deliberately simple thought experiment. A synthetic generator produces 10,000 dusk scenes. Because its “critical emergence” branch uses a different post-processing preset, a faint purple bloom appears in 90% of positive scenes and 10% of negative scenes. The actual partially occluded silhouette predicts the target correctly only 75% of the time because dusk noise and occlusion make it hard. A low-capacity classifier can reach about 90% synthetic accuracy from bloom alone, so gradient descent has little incentive to learn the harder 75%-accurate physical cue.

In the real camera, purple bloom is independent of emergence. The bloom classifier falls to roughly 50%—chance—even if the individual rendered frames looked convincing to reviewers. A generative “beautifier” might lower a perceptual distance while preserving the branch-specific color cast, leaving the shortcut untouched. In contrast, a paired intervention that independently varies post-processing while holding the emergence state fixed destroys the accidental correlation. Synthetic accuracy may fall from 90% to 75%, yet real accuracy can rise toward 75% because the remaining usable cue is present in both processes.

This also explains negative transfer. Suppose a small real-only model reaches 72% on a locked emergence slice. Mixing its 1,000 real examples with 10,000 naive synthetic examples dominated by the bloom shortcut can pull it down to 55%. After removing the branch artifact, balancing matched counterfactuals, and calibrating the sensor model, the same volume might raise it to 78%. These numbers are illustrative, not a promised learning curve. Their lesson is structural: more labeled data can hurt when it increases confidence in a feature whose predictive relationship changes at deployment.

From the model's perspective, the data pipeline is the specification. Documentation saying “van color is a nuisance” has no force unless the samples break its correlation with the target and counterfactual tests verify prediction stability.

2 · Replace “the domain gap” with five testable gaps

The phrase domain gap compresses several independent failures into one vague diagnosis. Return instead to the full generator. A trajectory is sampled from an initial scene, an action policy, transition dynamics, a sensor measurement process, and a label operator:

Real and synthetic data can differ at every factor. The conceptual difference

is not a subtraction that produces one number. It is a debugging checklist:

This decomposition prevents category errors. Increasing texture realism cannot create a missing pedestrian-emergence event. More scene layouts cannot repair incorrect rolling-shutter timing. A larger network cannot reconcile two contradictory definitions of an occluded pedestrian box.

The factors also interact. Camera exposure depends on illumination; observed motion depends on both dynamics and scan timing; policy changes which content appears. “Decompose” therefore means form separate hypotheses and interventions, not assume complete independence. Make the smallest controlled change that distinguishes competing explanations.

2.1 · Why these five factors are the minimum useful fault isolation

A supervised trajectory cannot arise without answering five questions: what worlds exist; which actions visit which parts of them; how worlds change after actions; how state becomes measured evidence; and how evidence/state becomes the declared target? These correspond to content, policy, transition, observation, and label factors. Remove one question and a distinct failure becomes unlocatable. Merge action with dynamics and one can no longer distinguish “the controller never tried braking” from “braking has the wrong consequence.” Merge observation with labels and one can no longer distinguish a blurred edge from a box exported at the wrong time.

The decomposition is minimal for engineering fault isolation, not a claim that nature has exactly five independent boxes. A neural video model may entangle transition and observation; actor behavior may be represented inside the transition; content and policy can share a program. The practical requirement is that the system expose enough interventions and provenance to test each question separately. If a component cannot be independently probed, its errors remain confounded and must be treated as a larger uncertainty set.

3 · Write a transfer contract for the running case

Before tuning the simulator, specify what must transfer. For the shuttle, imagine the model estimates pedestrian occupancy and time-to-collision, then emits a brake probability. A minimal contract might be:

• Deployment envelope: urban roads, 0–50 km/h, day through dusk, rain up to a stated rate, specified camera/LiDAR/radar hardware and firmware.
• Decision: brake when the posterior probability of entering the stopping corridor within 2 seconds exceeds a cost-derived threshold.
• Critical factors: pedestrian path, occlusion, ego speed, actuator delay, road friction, exposure, sensor synchronization, and radar/LiDAR observability.
• Nuisances: paint color or background texture only after demonstrating that changing them while holding state fixed should not change the target.
• Acceptance: improve rare-emergence miss rate and expected safety cost without exceeding a false-brake and comfort budget on locked real routes.

This contract determines simulator fidelity. Exact pedestrian identity may be irrelevant to braking, while silhouette, clothing reflectance, gait, and intention cues can matter. A beautiful interior dashboard may be wasted effort. A 12 ms timestamp error can be decisive. Fidelity is always fidelity with respect to a measurement and decision.

4 · Calibrate identifiable structure

Calibration estimates simulator parameters from controlled real measurements. Let θ include focal length, lens distortion, exposure response, camera readout time, LiDAR firing offsets, radar noise, brake latency, and tire friction. Given controlled input u_i and real observation o_i^real, estimate

The summary g should isolate the subsystem: calibration-board corners for intrinsics; bright and dark flat fields for photon/read noise; a flashing target for temporal offset; known trajectories for scan timing; controlled braking on instrumented surfaces for actuator delay and friction. The regularizer Ω excludes physically implausible solutions and encodes prior measurements.

4.1 · A worked camera-noise estimate

A common raw-sensor approximation says variance grows with mean signal:

where a represents signal-dependent shot/gain noise and b represents signal-independent read noise in the chosen units. Suppose repeated flat-field frames give variance 8 at mean 20 and variance 28 at mean 120. Then

The simulator can sample zero-mean noise with variance 0.20μ+4 before the appropriate clipping and image-signal-processing operations. But this two-point fit is only a teaching approximation. A production calibration uses many intensities, separates spatial fixed-pattern noise from temporal noise, checks color channels and temperature, models saturation, and validates on held-out exposures. Applying the same Gaussian noise after tone mapping would describe a different measurement process.

4.2 · Identifiability: what did the experiment actually determine?

A parameter is identifiable when distinct parameter values lead to distinguishable observations under the calibration inputs. Consider the approximate stopping distance

where τ is actuation delay and μ_road is effective friction. One stop at one speed can be explained by a longer delay and higher friction or a shorter delay and lower friction. Fitting both to that single distance is underdetermined. Stops at multiple speeds help because the delay term scales with v and the braking term with v²; direct pressure and wheel-speed measurements help more.

Likewise, camera response, illumination intensity, and surface albedo can trade off in one image. A moving edge observed by a rolling-shutter camera confounds object velocity, ego motion, clock offset, and readout time unless the experiment constrains some of them. An optimizer returning six decimal places does not make an unidentifiable estimate true.

Use sensitivity analysis or a posterior, not only a point estimate. If the loss valley is flat in one direction, report an interval or correlated joint distribution. Calibrate parameters that the evidence identifies; randomize the remaining plausible set. This is the first bridge from system identification to robust learning.

4.3 · Why calibration requires intervention

Passive driving logs contain correlations created by weather, route, driver behavior, and the existing controller. They do not necessarily excite the system enough to separate causes. A calibration experiment deliberately changes an input while measuring the response: vary chart distance while holding the lens fixed; sweep illumination while holding surface and exposure known; command braking at several speeds while measuring pressure and wheel motion. Intervention is valuable because it breaks competing explanations.

Locally, let predicted calibration summaries be m(θ) and define the sensitivity matrix J=∂m/∂θ. If some nonzero parameter change δθ satisfies Jδθ≈0, the experiment barely changes when parameters move in that direction. That direction is not locally identifiable. More repetitions of the identical experiment reduce random noise but do not repair the missing sensitivity; one needs a different target, speed, illumination, or direct measurement that changes the rank of J.

A principled output is therefore a posterior or feasible set,

not forced certainty. Narrow, well-identified directions can be fixed near their estimates. Broad or correlated directions should be propagated into generation and downstream uncertainty. This makes calibration honest: it converts evidence into constrained uncertainty rather than converting optimizer output into presumed truth.

5 · Randomize uncertainty without breaking the world

After calibration, some variation is genuine deployment variation—weather, clothing, traffic density—and some is epistemic uncertainty—poorly known radar multipath or friction. Domain randomization samples these variables so the model cannot depend on brittle simulator-specific cues. However, independent uniform sliders usually create implausible worlds and spend compute far from deployment.

Let φ be scenario and sensor parameters and q(φ) the synthetic proposal distribution. Structured randomization represents conditional dependence:

At dusk, illumination spectrum, headlight state, camera gain, and pedestrian visibility should change together. Wetness affects reflectance and friction. Pedestrian height changes occlusion geometry and motion statistics. Rolling-shutter distortion depends jointly on readout time and relative motion. Sample from a causal or constraint-aware grammar, then reject violated scene constraints before rendering.

For each variable, choose one of four treatments:

1. Fix it when the deployment hardware or contract truly fixes it.
2. Calibrate it when controlled measurements identify it.
3. Randomize it conditionally when it varies or remains uncertain.
4. Intervene on it deliberately when it is a decision-critical factor whose causal effect must be learned.

The representation goal is sometimes summarized as learning z=f(o) with low nuisance information I(z;n|s*) and high target information I(z;y). Treat this as intuition, not an instruction to erase every domain-correlated feature. Texture is task-relevant for wet-road recognition; lighting reveals geometry; sensor artifacts can encode range. Declare a variable nuisance relative to the contract, then test with paired interventions: hold geometry and target fixed, vary the nuisance, and measure prediction stability.

Seen this way, randomization should approximate a posterior predictive experiment, not a decorative variety generator. Draw uncertain hardware and physics parameters from p(θ|D_cal); draw genuine operating variation from an estimated deployment distribution; and deliberately broaden only named robustness dimensions. Uniformly sampling every slider over its software limits asserts an unjustified prior, breaks dependencies, and trains on worlds the calibration evidence may have ruled out. Sampling only θ̂ makes the opposite mistake: it hides uncertainty and encourages the model to specialize to one convenient simulator.

Distinguish three reasons for a wide range in provenance: real aleatoric variation, epistemic uncertainty about a fixed quantity, and intentional stress testing beyond expected deployment. They answer different questions and should not be pooled when estimating real frequency or calibrating probabilities.

realPos   = 0.62
simCenter = calibrated ? realPos : 0.30      // calibration centers sim on reality
gap       = |realPos − simCenter|
deficit   = max(0, gap − width)              // reality outside support ⇒ > 0
over      = max(0, width − (gap + 0.08))     // width beyond what's needed
acc       = clamp(0.92 − 1.6·deficit − 0.42·over, 0.42, 0.92)

6 · Select hard scenarios actively, but keep the objective honest

Uniformly producing more frames eventually repeats easy evidence. Active scenario selection treats generation as experimental design. After training model f_k, score candidate parameters using loss, uncertainty, disagreement, novelty, constraint proximity, or estimated safety cost, then allocate the next generation budget.

For the shuttle, a search may discover that misses cluster when the pedestrian becomes visible 18–24 pixels wide, camera gain is high, LiDAR first sees only an arm, radar is masked by the van, ego speed is 35–40 km/h, and friction is low. That is more actionable than “night is hard.” Turn the cluster into a named, parameterized scenario family and generate matched variants.

Three safeguards matter. First, constrain search to physically and operationally plausible states; an adversary that teleports pedestrians or violates exposure limits teaches the wrong robustness. Second, preserve diversity; chasing only the current model’s largest loss can oscillate or oversample annotation bugs. Third, never optimize against a locked real test set. Use training failures, a renewable development pool, simulator-internal counterfactuals, or production incidents that are subsequently assigned to a new evaluation version.

Active selection does not eliminate coverage sampling. A practical allocation might reserve 50% for deployment-matched baseline coverage, 25% for rare predeclared hazards, 15% for model-mined hard cases, and 10% for exploratory novelty. Those numbers are not universal; the important property is a visible budget whose changes can be ablated.

7 · Treat generative refinement as a constrained residual

A structured renderer often has better geometry and labels than appearance. A learned image or video model often has better appearance statistics but weaker guarantees about identity, topology, and timing. Combine them by assigning ownership explicitly:

The engine owns state, object identity, geometry, action, dynamics, timestamps, and authoritative labels. The learned model is permitted to supply a residual: material microstructure, distant background detail, weather appearance, illumination complexity, or calibrated camera artifacts. NVIDIA’s documented Isaac Sim Replicator → Cosmos workflow (accessed 2026-07-18) illustrates exporting structured render controls, and the Cosmos world-foundation-model line has since made this the productized default. It splits the labor into exactly the three roles this lesson keeps separate: Transfer (3D-sim structure → photorealistic video — the constrained-residual appearance role described here), Predict (action-conditioned future world states, the subject of Lesson 07), and Reason (a physics-aware plausibility critic). Cosmos-Transfer1 (NVIDIA, 2025) introduced generation conditioned on combinations of segmentation, depth, edge, and motion; Cosmos 3 and Transfer 2.5 (NVIDIA, 2026) scaled it. The taxonomy matters here because "condition the generator on the engine's structure" is precisely how appearance stays a residual while geometry and labels stay engine-owned.

This pattern is useful, but it is not a truth machine. Conditioning reduces drift; it does not prove absence of drift. The model can move the pedestrian boundary, hallucinate a reflection shaped like a person, erase a limb, change the van’s identity, or introduce temporal flicker. If so, the engine label no longer exactly describes the refined pixels.

Use the least powerful transformation that solves the measured gap. If relighting and calibrated sensor post-processing suffice, do not invoke unconstrained video generation. Crop- or layer-aware refinement can isolate backgrounds from critical actors. For exact segmentation or optical-flow tasks, renderer-native images may be preferable. For representation pretraining, a small amount of bounded label noise may be acceptable—but it must be measured, not assumed.

8 · Verify label preservation as a release gate

Every refined sample should pass an automated verifier before entering training. The verifier should use signals sufficiently independent from the generator and compare task structure, not only caption similarity or subjective realism.

Choose thresholds against the real labeling contract. If human segmentations vary by three pixels at the boundary, a subpixel requirement may waste data; if a safety corridor is only five pixels wide, a four-pixel drift is unacceptable. Estimate false accept and false reject rates on a manually audited sample. A verifier trained on the same generated outputs can share blind spots, so combine deterministic controls, independent pretrained models, multiview/temporal tests, and human audits.

In the running case, the engine mask says the first visible pedestrian occupies 312 pixels. After refinement, an independent correspondence check finds that 41 boundary pixels moved more than the allowed 3 pixels, and the estimated first-visibility time is two frames earlier. Even if the frame looks excellent, reject it for training time-to-collision. Preserve the original, refined output, controls, generator/version, seed, verifier scores, and rejection reason. This provenance turns drift from invisible corruption into a measurable production yield.

9 · Neural digital twins: high fidelity, bounded support

A neural digital twin reconstructs a real captured environment and renders novel views or actor variations. UniSim (Yang et al., 2023) constructs neural sensor simulations from logged autonomous-driving data for closed-loop testing. NeuRAD (Tonderski et al., 2023; CVPR 2024) models dynamic driving scenes and synthesizes camera and LiDAR observations. By 2025–2026 the dominant form is 3D/4D Gaussian-splatting reconstruction at production scale: NVIDIA Omniverse NuRec productizes 3DGS scene reconstruction, feed-forward 4D-Gaussian methods reconstruct a dynamic drive in seconds rather than hours, and OmniDreams (2026) renders reconstructed scenes as a real-time (~105 FPS per view) closed-loop simulator for evaluating a driving policy; DriveDreamer4D (2024) shows world models acting as "data machines" for 4D driving scenes.

For the shuttle, a twin of the actual dusk route can preserve building appearance, road markings, vegetation, sensor pose, and recorded traffic. One may replay the route, alter ego trajectory locally, or insert an actor. This closes an important appearance gap. It does not imply unlimited counterfactual validity. Regions never observed remain uncertain; large viewpoint changes expose unseen surfaces; actor removal may reveal unknown backgrounds; inserted objects may lack correct shadows, interactions, radar response, or behavioral context.

The current strong practice is therefore hybrid, not a winner-take-all method: structured simulation owns controllable truth; procedural programs own coverage; digital twins anchor places and sensors; conditional generation supplies verified appearance residuals; real data calibrates the system and decides acceptance. Calling this “SOTA” should mean this systems pattern has the best practical division of labor—not that one product, paper, or benchmark has solved sim-to-real generally.

9.1 · Why the hybrid is structurally unavoidable

Each evidence source is strong precisely where another is weak. A structured simulator permits interventions and exact internal labels but is an incomplete model of open-world appearance, behavior, and measurement. A learned generator compresses real visual regularities but has no inherent obligation to preserve causal state or exact labels. A neural twin captures one place faithfully but cannot guarantee unseen surfaces or large edits. Passive real logs measure deployment but provide sparse rare events, costly labels, and actions confounded by the behavior policy.

A single source would suffice only under an extreme assumption. Either the simulator exactly reproduces every decision-relevant real mechanism, which is essentially the original world-modeling problem already solved, or real data exhaustively covers every costly intervention with trustworthy labels, which defeats the reason to synthesize. In practical finite systems, neither condition holds. Hybridization is not a fashionable ensemble; it follows from complementary missing information.

The same argument explains why refinement must remain a constrained residual. Image generation is an ill-posed inverse problem: many visually plausible scenes satisfy the same depth, mask, and prompt controls. The generator's learned prior selects among them and can silently overwrite a weak control. If it owns both the pixels and the claim that labels survived, the system is self-certifying. Structural controls, bounded edit regions, independent verification, and engine-owned truth limit that circularity.

Real data must retain independent authority for the same reason. Simulator calibration, procedural choices, refinement, and synthetic QA share assumptions and can share blind spots. Passing their internal tests proves consistency with the engineered world, not correspondence with deployment. Only held-out measurements produced by O_real under real content, behavior, and dynamics can reveal the residual joint error. The locked real test is therefore not merely one more benchmark; it is the independent reference that prevents a synthetic ecosystem from grading itself.

10 · Combine real and synthetic evidence deliberately

A hybrid learner can be written as

The schedule λ(t), sample weights, and consistency term express which source is trusted for what. Synthetic supervision may be exact for geometry but approximate for appearance. Real labels may be sparse and noisy but come from the target measurement distribution. Treating every sample equally discards that information.

10.1 · Training schedules

10.2 · A worked mixture

Assume each optimization batch has 64 examples. Start pretraining with 56 structured synthetic examples and 8 unlabeled real examples used for temporal consistency. During adaptation, use 24 synthetic rare-event examples, 32 labeled real examples, and 8 unlabeled real examples. In a final short calibration phase, use 8 synthetic replay examples and 56 labeled real examples with a low learning rate, while retaining periodic synthetic validation to detect forgetting. These are auditable starting ratios, not laws.

Suppose deployment analysis estimates that dusk occluded emergences are 2% of relevant encounters, but the targeted synthetic set makes them 20%. For an unbiased estimate of average deployment risk, their nominal importance factor is 0.02/0.20=0.10; ordinary synthetic scenes occurring 98% in deployment but 80% in generation get 0.98/0.80=1.225. This correction prevents a deliberately enriched dataset from pretending to be the deployment frequency distribution.

But training need not minimize average frequency-weighted risk alone. Safety learning may intentionally upweight rare high-cost events. If a missed emergence has 50 times the cost of an ordinary error, a cost-aware rare-event factor could be 0.10·50=5 before normalization or clipping. State clearly whether a weight corrects sampling bias, encodes application cost, or emphasizes a curriculum. Combining all three into an unexplained “sample weight” makes results impossible to interpret.

Importance weighting also assumes synthetic and real examples share relevant support and that density ratios can be estimated. Very large ratios yield high-variance gradients. Clip ratios, stratify known scenario variables, or use doubly robust/representation-level methods only with appropriate validation. No weight can repair a missing real mode or a wrong label contract.

11 · Keep validation, test, and generation feedback separate

A synthetic-data pipeline is adaptive: real failures influence what is generated next. Without disciplined data boundaries, the generator can overfit the evaluation set even when the model never directly trains on its images.

1. Training pool: real labeled/unlabeled data and synthetic data may be sampled repeatedly.
2. Development validation: selects ratios, thresholds, checkpoints, and generator revisions. Because it is repeatedly inspected, it is not final evidence.
3. Locked test: fixed before the experiment, access-controlled, and evaluated sparingly. Its scenarios must not be copied into generation during the study.
4. Post-deployment monitoring: finds new failures. Once used to update training, those events join the training/development history and require a new future evaluation cohort.

For the shuttle, the locked test may be route- and time-separated, include unseen van models and pedestrians, and retain raw synchronized sensor streams. Model selection uses a different development route. If test results reveal “red van at civil twilight” and the team creates thousands of that exact setup, the old test has become development feedback. Freeze its reported result, version the benchmark, and judge the update on a new untouched cohort.

Acceptance should include: overall expected cost; pedestrian-emergence recall at fixed false-brake rate; reliability/calibration; slices by visibility, distance, speed, sensor dropout, weather, and demographics where appropriate; robustness to matched interventions; and closed-loop stopping margin and comfort. Report confidence intervals over independent scenes or routes rather than pretending millions of correlated frames are millions of independent trials.

12 · An end-to-end implementation recipe

# Inputs: versioned simulator, real calibration set, training pool,
#         renewable validation set, access-controlled locked test

contract = define_task_cost_slices_and_label_rules()
theta_id, posterior_uncertain = calibrate_subsystems(
    experiments=designed_camera_lidar_radar_brake_tests,
    identifiability_checks=True)

scenario_program = build_conditional_distribution(
    calibrated=theta_id,
    uncertain=posterior_uncertain,
    deployment_priors=estimated_real_frequencies,
    causal_interventions=critical_emergence_factors)

real_baseline = train(real_train, contract)
simple_syn_baseline = train(render(scenario_program, mode="engine"), contract)

for round in range(K):
    base = sample_coverage_and_predeclared_hazards(scenario_program)
    hard = search_valid_scenarios(model, budget=active_budget)
    scenes = deduplicate_and_balance(base + hard)

    engine_obs, labels, provenance = simulate_and_label(scenes)
    refined_obs = refine_only_if_needed(engine_obs, structural_controls)
    accepted = verify_and_filter(refined_obs, labels, provenance)

    weights = separate_frequency_cost_and_curriculum_weights(accepted)
    model = train_with_versioned_mixture(
        real_train, accepted, weights, schedule, seed)

    metrics = evaluate(model, renewable_validation)
    diagnose_by_gap_and_slice(metrics)
    change_one_generator_or_training_family()

final_metrics = evaluate_once(model, locked_test)
run_closed_loop_safety_and_comfort_trials(model)
publish_versions_costs_rejections_ablations_and_intervals()

The order matters. If calibration, generator revision, refinement, verification, weighting, and training all change at once, a gain cannot be attributed and a regression cannot be debugged. Version every layer and make each sample traceable to it.

13 · Ablate the system, not only the network

A useful ablation matrix isolates each claim. Keep architecture and optimization fixed where possible, repeat seeds, and evaluate the same real slices.

Also compare equal compute, equal total frames, and equal dollar budgets. A method that wins using ten times the rendering and training compute may still be worthwhile for safety, but it answers a different question from sample efficiency.

14 · Failure triage: follow the evidence upstream

Triage in this order: label contract and data integrity; support/coverage; sensor calibration; action and dynamics; training mixture; model capacity. Otherwise a larger model can mask a pipeline fault long enough for it to reappear under deployment shift.

15 · Cost and sample efficiency

Synthetic frames are not free. Costs include asset creation, scene engineering, calibration experiments, renderer time, generative refinement, verifier rejects, storage, training compute, human audits, and maintenance when hardware or software changes. Real data costs include collection, privacy review, synchronization, labeling, rare-event search, and safety exposure. Optimize total decision value, not frame count.

Suppose one million engine frames cost $8,000 to generate and train, refinement adds $12,000, and verification rejects 25%, yielding 750,000 usable refined frames. The effective refinement cost is $16 per thousand accepted frames before engineering. If 10,000 labeled real frames cost $40,000, synthetic data appears cheaper—but only an experiment can establish equivalence. If the refined set reduces the real-label requirement from 10,000 to 6,000 at the same locked-test risk, it saves about $16,000 in labels; against $12,000 incremental refinement, the narrow monetary gain is $4,000, excluding reusable infrastructure and safety benefits.

Plot learning curves against several x-axes: accepted synthetic samples, real labeled samples, GPU-hours, wall-clock time, and total cost. Report the verifier yield and unique scenario coverage. Useful claims are: “same locked real risk with 40% fewer real labels,” “30% lower emergence miss rate at fixed false-brake rate and equal compute,” or “twofold more distinct critical states per dollar.” “Trained on a billion frames” says little without diversity and risk reduction.

Stop generating a family when its marginal validation gain per unit cost is below alternatives, or when uncertainty indicates the bottleneck lies elsewhere. Sometimes the best next purchase is a calibration rig, ten carefully chosen real sequences, or a label-contract audit—not another million renders.

16 · Bridge to action-conditioned trajectories

So far, most metrics could be computed frame by frame. But braking changes the future. From the same simulator checkpoint—pedestrian partly occluded, shuttle at a known speed—branch at least two interventions: brake and coast. Preserve the same sampled latent scene and sensor uncertainty where causally appropriate, then allow the action to change dynamics and later observations.

The branch is meaningful only if calibrated brake latency and friction produce credible motion, rolling-shutter/LiDAR timing follows that motion, labels remain synchronized, and any neural refinement preserves the action consequence. A world model trained on these trajectories must not merely make attractive video; it must distinguish how the chosen action changes collision probability, stopping margin, and sensor future. Lesson 07 makes that action-conditioned, closed-loop evaluation explicit.

17 · Exercises and self-checks

1. Gap attribution. The shuttle detects the pedestrian in clear daylight but fails in wet dusk; its detector is correct on recorded sensor frames, yet closed-loop stopping distance is optimistic. Name at least two separate gaps and one experiment for each. Check: sensor calibration may explain dusk perception; dynamics/actuator calibration may explain stopping distance. Test them separately.
2. Identifiability. Why can one final stopping distance not identify both delay and friction? Design a better experiment. Check: several parameter pairs match one scalar. Use multiple speeds/surfaces plus time-resolved pressure, wheel speed, and pose.
3. Noise calibration. With variances 11 and 31 at mean signals 30 and 130, fit a and b in aμ+b. Answer: a=0.20, b=5.
4. Structured sampling. Draw a dependency graph for time, rain, road wetness, friction, illumination, exposure, and visibility. Which arrows are causal or engineering approximations? Check: one defensible graph has time → illumination; rain → road wetness; road wetness → friction; illumination and rain → visibility; illumination → exposure through the camera controller; exposure → recorded visibility. “Time → exposure” is normally an indirect engineering dependency, and exact arrows depend on the declared controller and weather model.
5. Refinement gate. A generated frame preserves the pedestrian mask intersection-over-union but changes first-visible time by three frames. Can it train single-frame detection? Can it train time-to-collision? Check: possibly for the former after auditing; not with the original temporal label for the latter.
6. Mixture weighting. A scenario is 5% of deployment and 25% of the synthetic set. What is the frequency-correction weight? If its miss cost is 20 times larger, what cost-aware unnormalized factor results? Answer: 0.2 and 4.0. Explain why clipping may still be needed.
7. Leakage. The team reads every error on the locked test, generates matching scenes, and re-evaluates. What is wrong? Check: the test became development data through generator adaptation; version it and obtain a new untouched cohort.
8. Ablation. Design the minimum experiment to distinguish “photorealism helped” from “more rare scenarios helped.” Check: control scenario identities and volume, compare engine versus refined observations, and include verifier outcomes.
9. Economics. Calculate cost per accepted frame when 200,000 refinements cost $3,000 and 30% are rejected. Then state why the number is insufficient for a purchase decision. Answer: $0.0214 per accepted frame; it lacks marginal real-risk or label-efficiency value.
10. Counterfactual validity. List which random variables should be shared and which should evolve differently across brake/coast branches from one checkpoint. Check: share all realized pre-checkpoint state, pedestrian intent, road condition, calibrated parameters, and exogenous noise draws chosen before the action. Let ego motion and observations caused by it diverge. If the pedestrian reacts to the shuttle, that response is a mediator and should diverge according to the same reaction mechanism, not be frozen or redrawn arbitrarily.

18 · Compact mental model

1. Define real decision risk, label semantics, deployment envelope, and acceptance slices.
2. Decompose failures into content, action, dynamics, sensor, and label hypotheses.
3. Identify parameters with designed experiments; retain posteriors for ambiguity.
4. Cover remaining variation with conditional randomization and valid counterfactuals.
5. Search for informative hard cases without optimizing to the locked test.
6. Refine appearance only as a bounded residual, with explicit ownership.
7. Verify geometry, identity, timing, and label preservation before release.
8. Mix real and synthetic evidence with interpretable schedules and weights.
9. Ablate each system component under equal budgets.
10. Accept only improvements on untouched real evidence and closed-loop outcomes.

Interview prompts

• Why is there no single domain gap? Because content, policy, dynamics, observation, and label processes can differ independently and imply different interventions.
• When should a parameter be randomized instead of calibrated? When it genuinely varies in deployment or available experiments do not identify it; randomization should respect its posterior and dependencies.
• What is the contract for generative refinement? The generator may change bounded appearance under structural controls; the engine owns geometry, identity, action, and labels, which an independent release gate verifies.
• How do frequency and cost weights differ? A density ratio corrects deliberate sampling shift; a cost weight changes the target objective. They should be logged separately.
• What is the strongest synthetic-data claim? Lower untouched real or closed-loop risk, lower tail risk, or equal risk with fewer real labels at a stated total cost—not synthetic-only benchmark performance.