Synthetic Vision Data, from first principles
A linear path from a prebuilt 3D scene and engine to useful evidence for vision and world models. Specify the deployment distribution, generate structured trajectories, simulate measurements, preserve exact truth, implement the complete image pipeline in Blender, cross the reality gap, and validate decisions—not screenshots.
The fundamental problem: a model needs evidence, not merely generated content
Why is building synthetic vision data difficult when a 3D engine can already render an unlimited number of images? Because the scarce object is not an image. It is evidence that identifies a useful rule for deployment. A renderer answers a forward question: given a particular scene, camera, light configuration, and rendering algorithm, which pixels should appear? Training asks a different inverse question: from finite measurements, which regularities should a model retain so that it behaves correctly in worlds it has not seen?
Those questions are not equivalent. Infinitely many different simulated datasets can be rendered from the same engine. Some teach the intended relation; others make a shortcut easier. Suppose every dusk pedestrian in the synthetic set wears a bright red coat, while every empty road is rendered in blue-gray tones. A detector can minimize its training loss by treating red pixels as the danger signal. The scene may be physically valid, every mask may be exact, and every image may look convincing. Yet the learned rule is wrong because the generator made clothing color more predictive than pedestrian geometry and motion.
Write the deployment loop in its smallest useful form. A real world has hidden state s. Sensors turn that state into an observation history h. A learned system maps the history to a prediction or action, and the environment assigns a cost:
s ∼ preal(s), h ∼ Oreal(h|s), a=fφ(h), risk(φ)=E[C(s,a)]A synthetic pipeline replaces the first two distributions with configurable approximations pθ and Oθ, then supplies labels through a query Lθ. More samples reduce sampling noise inside that chosen process. They do not prove that the process matches the real states, measurements, action effects, or costs that matter. If the generator omits partially occluded pedestrians, ten billion clean synthetic frames make the model increasingly certain about a world in which that failure mode does not exist.
This is the central distinction between variance and structural error. More samples can reduce variance: the learner estimates the synthetic relationship more precisely. They cannot automatically remove structural error: the synthetic relationship itself may be wrong. Photorealism can reduce one part of observation mismatch, but it cannot create missing action support, correct an invalid collision model, align a rolling-shutter label in time, or tell us whether a planner benefits from the result.
Three impossibilities force the architecture
- No support, no learning. If a deployment-relevant state–action event never occurs in the generated process, its consequence cannot be estimated from that data. Architecture scale cannot recover evidence that is absent.
- One observation, several worlds. The same image can come from a stationary pedestrian and an approaching pedestrian, or from a dark surface under bright light and a bright surface under dim light. When those hidden causes imply different futures, a single deterministic label or current frame is insufficient. The system needs history, multiple sensors, interventions, or explicit uncertainty.
- Observation is not intervention. Logged drivers brake mostly in dangerous situations. A passive dataset may therefore show more collisions after braking than after coasting. It cannot identify the effect of braking without matched context, exploration, structural assumptions, or controlled intervention.
These are not defects of a particular engine. They are information limits. The seven lessons remove them in a defensible order. Scenario programs create state and action support. Sensor models declare which hidden distinctions survive measurement. Label contracts say exactly which event is being supervised. The Blender lab then turns those contracts into an executable image-data pipeline, so implementation details remain consequences of the reasoning rather than substitutes for it. Calibration and real anchors constrain process mismatch. Checkpointed alternative futures provide interventional evidence. Closed-loop tests ask whether the remaining approximation is good enough for the decision.
Why not solve everything with one end-to-end generator?
An end-to-end learned generator can be extremely useful, but it cannot serve as its own authority. If its video says the shuttle stopped safely, we still need an independent definition of position, collision, action timing, and uncertainty to know whether that statement is true. If the same model generates the sample, invents the label, and grades the result, a shared error can pass every internal check.
The remedy is not to reject learned generation. It is to assign authority according to what each component can establish:
- A structured engine is strong at explicit geometry, identity, timestamps, controllable state, and repeatable interventions.
- Procedural programs are strong at deliberately covering combinations and rare event families.
- Calibrated sensor models are strong at preserving known optics, scan timing, and units.
- Learned residuals and neural reconstructions are strong where hand-written appearance or dynamics become too expensive.
- Real held-out evidence is the only component that can establish that the entire approximation improved the deployment distribution.
This separation is a debugging theorem as much as a software architecture. When the real miss rate rises, we need to ask whether the relevant world was absent, the dynamics were wrong, the sensor erased the cue, the label described a different event, the learner used a shortcut, or the planner exploited an optimistic error. A monolithic quality score entangles those hypotheses. Explicit boundaries make each one falsifiable.
One running problem, seven linear stages
The lessons reuse one case so that each abstraction has a concrete job. An autonomous shuttle approaches a parked van at dusk. A pedestrian may emerge from behind it. The shuttle observes the scene through a front rolling-shutter camera, scanning LiDAR, and radar, then must decide whether and when to brake. The expensive failure is not “an image looks unrealistic.” It is a late or unjustified braking decision caused by missing state, misleading measurements, bad supervision, or an incorrect prediction of what braking changes.
| Layer | Question the case forces | Artifact handed to the next lesson |
|---|---|---|
| 01 · Distribution | Which real failures matter, and which state–action situations must have support? | A task contract and factorized data-generating process. |
| 02 · World | How do the van, pedestrian, road, intent, physics, and brake/coast alternatives become controllable state? | Valid checkpointed scenario trajectories. |
| 03 · Measurement | What does each sensor actually integrate or scan while the actors and shuttle move? | Timed camera, LiDAR, and radar observations. |
| 04 · Supervision | Which state and coordinate frame define depth, flow, boxes, visibility, pose, and action labels? | A replayable, synchronized, leakage-safe dataset. |
| 05 · Build lab | How do those contracts become a repeatable Blender pipeline for an image detector or segmenter? | Headless RGB, instance-mask, COCO, QA, lineage, training, and feedback artifacts. |
| 06 · Transfer | Which differences from reality should be calibrated, randomized, reconstructed, or learned? | A hybrid synthetic–real training and validation loop. |
| 07 · Decision | Does the learned model predict brake versus coast consequences well enough to reduce real closed-loop risk? | An evaluation ladder and deployment acceptance decision. |
The object being engineered
An engine is a controllable data-generating process. A trajectory follows an initial-state distribution, a behavior policy, dynamics, and a sensor model; labels are deterministic queries of authoritative engine state:
s0 ∼ pθ(s0), at ∼ πθ(at|ht), st+1 ∼ Tθ(st+1|st,at), ot ∼ Oθ(ot|st), yt=L(st,ot)The parameters θ include scene layout, assets, actor intent, physics, lighting, optics, timing, noise, and post-processing. The goal is not to maximize variety in every parameter. It is to choose a conditional joint distribution whose support contains the deployment cases that change the model’s decisions.
Why the sequence is linear
- Contract before content. Until sensors, outputs, decisions, failure costs, and real evaluation slices are fixed, “more realistic data” has no target.
- World before measurement. A sensor can only measure the simulated state. Scene structure and behavior determine whether the needed situations ever occur.
- Measurement before labels. Depth, flow, exposure, and timestamps must refer to a declared measurement event; otherwise exact engine truth becomes precisely wrong supervision.
- Truth before implementation scale. Reproducibility, invariants, and leakage-safe splits are prerequisites for a trustworthy exporter.
- One verified sample before a render farm. Prove that scene state, RGB, labels, metadata, and replay agree end to end; scaling an unverified exporter only multiplies hidden error.
- Implementation before transfer claims. Until the actual training bytes and labels pass end-to-end checks, a sim-to-real comparison cannot distinguish renderer mismatch from pipeline corruption.
- Reality before acceptance. Synthetic metrics select generators; frozen real-world and closed-loop metrics decide whether the system improved.
The lessons
A compact operating rule
Start with one expensive deployment failure. Reconstruct or author the smallest family of scenes that can produce it. Randomize the uncertain causes, preserve the known structure, replay matched action branches, export synchronized labels, train with a small trusted real anchor, and accept the new generator only if it reduces held-out real risk. Then repeat. This turns synthetic data from a one-time content dump into a measured feedback controller.
Primary-source map
The series links sources at the claim they support. Key systems include the Blender Python API and headless command-line interface, Infinigen (CVPR 2023), Kubric (CVPR 2022), BlenderProc (IJRR 2023; first released 2019), Structured Domain Randomization (ICRA 2019), Isaac Sim + Cosmos workflows (NVIDIA documentation, accessed 2026-07-18), Cosmos-Transfer1 (2025), UniSim (2023), NeuRAD (2023 / CVPR 2024), V-JEPA 2 (2025), Genie 3 (2025), and WorldSimBench (2024).