Real-World Data vs Synthetic and Simulation Data

The decision behind the question

"Real versus synthetic" hides three separate choices: what seeds the data, what is generated from the seed, and what you will be judged on. Most programs (see the requirements specification template) end up buying a seed set of real data, generating around it, and keeping a real, held-out set for evaluation.

This page is about that split. For the human-video branch of the stack, see what human demonstration video can support; for sizing, see how many demonstrations published work reports.

Four kinds of "not real" data

What published work reports

• MimicGen generated more than 50,000 demonstrations from about 200 human demonstrations across 18 tasks, varying scene configurations, object instances and robot arms (Mandlekar et al., 2023). The seed is human; the output is robot-executable in simulation or on the setup used.
• DreamGen trains policies on synthetic neural trajectories from image-to-video world models adapted to the robot, recovering pseudo-actions with a latent action model or an inverse-dynamics model. It reports a humanoid learning 22 new behaviors from teleoperation data of a single pick-and-place task (Jang et al., 2025).
• GR00T N1 places synthetic simulation and neural-video trajectories in the middle of a data pyramid, between web and human egocentric video at the base and real-robot teleoperation at the top, and reports neural trajectories expanding teleoperation data from 88 hours to 827 hours (NVIDIA, 2025).
• Isaac Sim is a simulation and synthetic-data framework whose Replicator tool outputs annotated data such as bounding boxes, segmentation masks and depth, and which imports CAD and URDF into USD (NVIDIA, n.d.).

Notice the pattern: every generated set above sits on top of real demonstrations. None of these sources says generation removes the need for them.

When to buy real data

• Target embodiment and sensors. Your hand, camera placement and control stack are specific; a sim model of them is only as good as its calibration.
• Contact-rich and deformable work. Contact, friction, compliance and cloth are the places where simulated and real behavior diverge most; check this against your own tasks rather than assuming.
• Evaluation. A held-out real set, split by site, operator or object as your claim requires, is the evidence that anything works. See training vs evaluation datasets.
• Workplace realism. Clutter, lighting, wear and unscripted variation in operating businesses are hard to author. Real workplace video can help with perception, with the limits explained in the embodiment gap.

When simulation or generation is enough, or better

• Early algorithm work where the question is about method, not hardware.
• Perception tasks where annotated renders of CAD-known objects supplement real photos.
• Varying scenes, objects and arms around demonstrations you trust.
• Stress cases that are unsafe or too rare to collect, as a supplement and not a substitute.

Illustrative specification split

# Illustrative data-strategy split (synthetic values)
real_seed:
  source: teleoperation on target embodiment
  purpose: policy seed + calibration of sim
real_eval:
  held_out_by: site, operator
  never_used_for: generation or training
generated:
  method: demonstration generation from real_seed
  must_report: seed count, generator version, rejection rule

Limits and extra preparation

Generated data inherits the seed's biases and mistakes. Ask for the seed count, the rejection rule and the share of generated episodes that pass a success check. Pseudo-actions from inverse-dynamics models are estimates, not recorded commands. Generated and real data should be tagged separately so you can ablate them.

Questions to ask a supplier or tool vendor

1. What real data seeds the generated set, and can we see its provenance?
2. Which fields are measured and which are inferred or pseudo-labeled?
3. What fraction of generated episodes was filtered out, and by what rule?
4. How is the sim-to-real gap measured, on which tasks and with what held-out real data?
5. Are assets and scenes licensed for training and redistribution?
6. Can you tag every episode as real, replayed, generated or neural?

For rights questions on synthetic outputs, see licensed vs synthetic vs scraped data and model weights, derivatives and redistribution.

Acceptance and documentation for mixed real and generated sets

Ask that every delivered episode carry a provenance tag (real teleoperation, human video, replayed, generated, neural) and that generated episodes reference their seed episodes. Without the tag, you cannot ablate generated data later or explain a regression. Ask also for the generator version, the random seeds and any filtering rule, so a rerun produces the same set.

For perception data, check that rendered assets match real objects in scale, texture and material. Annotated renders from CAD are cheap, but a detector trained only on them may meet a different image distribution on a real camera; a real validation set from your own cameras is the check.

A staged approach

1. Collect a small real seed on the target embodiment, with calibration recorded.
2. Use simulation or generation to vary scenes, objects and start states around the seed.
3. Train, then evaluate on a held-out real set; compare against a seed-only baseline.
4. Buy more real data where the failure analysis points, rather than generating more of what already works.
5. Repeat, keeping the real evaluation set fixed.

This keeps the generated data accountable to real results. See real-world policy evaluation protocols for how to run step 3 on hardware.

Discuss your requirements

Describe the real seed and evaluation data you need in a request to SourceX. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match. See also the procurement hub, sourcing routes and the humanoid data guide.

Discuss your humanoid training-data requirements with SourceX.

Sources

• MimicGen — generation of 50K+ demonstrations from about 200.
• DreamGen — neural trajectories and pseudo-actions.
• Isaac Sim — simulation and synthetic-data tooling.
• GR00T N1 — data pyramid and neural-trajectory expansion.