Short answer
A coverage plan lists the factors your model must generalize across, such as sites, demonstrators, object instances, layouts, lighting, task variants and failure conditions, then sets how many distinct values of each the collection will include and how they combine. It turns 'diverse data' into countable quotas, reserves held-out factors for evaluation, and lets you check each delivery against the plan.
Why diversity has to be planned
Left alone, a collection drifts toward convenience: the site that hosts most easily, the operator who volunteers most often, the objects already on the bench, the morning shift. The result looks large but varies along few axes, and a policy or encoder trained on it learns that narrow world.
Published datasets treat variety as an explicit target and report it. DROID collected 564 scenes and 86 tasks with 50 data collectors across three continents over 12 months (Khazatsky et al., 2024). BridgeData V2 spans 24 environments (Walke et al., 2023). Open X-Embodiment pooled data from 22 robot embodiments covering 527 skills, contributed by 21 institutions (Open X-Embodiment Collaboration, 2023). On the human side, Ego4D recorded 931 camera wearers in 74 locations across 9 countries (Grauman et al., 2021), Nymeria 264 participants in 50 locations (Ma et al., 2024), and EgoDex 194 tabletop tasks (Hoque et al., 2025).
These are what particular studies collected, not targets. How much variety you need along each axis depends on your architecture, tasks, embodiment and intended use.
Coverage factors
| Factor | Example levels (illustrative) | Why it matters | How to verify on delivery |
|---|---|---|---|
| Site | site_017, site_024, site_031 | Layout, lighting, fixtures, local practice | site_id on every clip; counts per site |
| Demonstrator | operator_A12, operator_B07; experience band | Technique, speed, hand size, handedness | Pseudonymous operator_id and band |
| Object instance | Part variants; new and worn units | Appearance and geometry variation | object_id per clip; reference photo of each instance |
| Layout and start state | Tote positions, shelf heights, clutter level | Policies overfit to start configurations | Start-state code or photo per episode |
| Lighting and shift | Day, night, task light on or off | Exposure and color shifts | Light-condition flag; timestamp |
| Task variant | Quantity, size or sequence variants | Coverage of sub-skills | Variant code |
| Hands and PPE | Bare, nitrile, cut-resistant gloves | Hand appearance and tracking | Glove-type field |
| Pace | Normal, rushed, deliberately slowed | Blur, technique, visibility | Pace flag |
| Outcome | Success, recovery, failure | Recovery behavior; success detection | Outcome label |
| Capture unit | Camera unit IDs | Sensor-specific artifacts | unit_id per clip |
Not every factor applies to every program. Strike the ones your model does not need to generalize over, and say so in the plan, so a supplier does not spend effort on them.
Combining factors without confounding
A full factorial design multiplies quickly and is rarely possible on real sites. Instead, set a quota for each factor on its own and add pairwise constraints for the pairs that matter most: each demonstrator works at two or more sites, each object instance appears under more than one lighting condition, and each task variant is performed by several people.
The pairwise rules exist to prevent confounds. If one operator works only at one site, the model cannot separate that person's technique from that site's layout. If every failure was recorded in a staged cell, "failure" and "staged" become the same signal. Rotate people across sites and stations, and record the setting on every clip.
Decide also whether training should follow natural frequencies or balanced quotas. Natural frequencies keep realism, but rare conditions then barely appear; a common compromise is natural frequency with a floor for each rare level. Conditions that occur too seldom to schedule need their own capture strategy, set out in capturing rare events and edge cases.
Demonstrator quality is a coverage factor too
Operators differ in skill, and that difference reaches the model. In a study of offline learning from human demonstrations, the multi-human datasets held 300 demonstrations from 6 operators grouped as worse, okay and better, and results depended on demonstration quality (Mandlekar et al., 2021). Record a proficiency band for every demonstrator and decide in advance whether you will balance, filter or condition on it; the options are compared in expert vs novice demonstrations.
Reserve held-out factors before collection starts
Choose now which factor values will never be used for training: a whole site, a set of operators, a group of object instances or a later time window. Put them in the plan so quotas cover both the training and the held-out data. Splitting after the fact rarely works, because the factor you want to hold out is often mixed through every delivery. Demonstration-specific leakage risks are covered in splitting demonstration data by site and operator, and the general separation rules in training datasets vs evaluation datasets.
Real collection vs synthetic expansion
Some factors can be widened after collection. MimicGen generated more than 50,000 demonstrations from about 200 human demonstrations, varying scene configurations, object instances and robot arms in simulation (Mandlekar et al., 2023). Programmatic variation of object pose or layout is cheaper than recording it. It does not reproduce real sites, real people, real lighting or real wear, so plan real coverage for those factors.
Illustrative coverage plan
# Illustrative only — synthetic IDs and example quotas, not a recommended size
coverage_plan: tote_kitting_v1
unit: one complete task episode
factors:
site: {levels: [site_017, site_024, site_031, site_042], min_share_each: 0.15}
demonstrator: {levels: 12 pseudonymous IDs, max_share_any_one: 0.12}
object_instance: {levels: 40 SKUs incl. 8 worn or damaged, min_episodes_each: 10}
start_state: {levels: [sparse, mixed, cluttered], min_share_each: 0.2}
lighting: {levels: [day, night], min_share_each: 0.25}
outcome: {natural, with floor: {recovery: 0.05, failure: 0.03}}
pairwise_rules:
- every demonstrator records at >= 2 sites
- every object instance appears under both lighting levels
held_out:
site: site_042 # evaluation only
demonstrators: 2 IDs # never in training
time_window: final two weeks of collection
reporting: counts per level and pairwise matrix with every delivery
Tracking each delivery against the plan
Ask for a coverage report with every delivery: counts per factor level, the pairwise matrix for constrained pairs, a list of unfilled cells and the reason for each gap. Compare it with the plan before accepting the delivery. When a host cannot provide a level, such as a night shift, agree a substitute in writing rather than letting the quota quietly lapse.
Limitations
- A coverage plan makes variety measurable; it cannot guarantee that a model will generalize.
- Many factor levels are unknown until a pilot shows what the sites can host.
- Coverage across human demonstrators says nothing about variation in robot embodiments; that needs robot data.
- Deciding how many tasks to cover at all, rather than how much variety within each, is a separate question addressed in depth vs breadth collection strategy.
- Keeping capture consistent across many sites is an operational problem covered in running demonstration collection across multiple sites.
Questions to ask collection providers
- How many distinct sites, stations and demonstrators can realistically be scheduled?
- Can demonstrators rotate across sites and stations, or are they tied to one location?
- Which object instances are available, and can worn or damaged units be included?
- Which lighting and shift conditions can be recorded?
- Will factor levels be recorded per clip by people or captured from systems?
- Can you deliver a per-level coverage report with each delivery?
- Which planned levels are at risk, and what substitutes would you propose?
- How will held-out sites or demonstrators be kept out of training deliveries?
Discuss your requirements
If you can name the factors your model must generalize over, send SourceX the coverage you need along with the tasks and permitted uses. SourceX reviews the requirement, investigates whether US businesses can supply that variety, and follows up about potential sources; for new recordings, the site describes a physical-world path where each opportunity has its own requirements. A request does not guarantee a match. Related topics are listed on the workplace task demonstration hub and the humanoid training data guide.
Describe the workplace demonstration data your robotics project needs
Sources
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — reported scene, task and collector diversity.
- BridgeData V2: A Dataset for Robot Learning at Scale — environment diversity in robot data.
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models — pooled embodiments and skills across institutions.
- Ego4D: Around the World in 3,000 Hours of Egocentric Video — wearer and location diversity in human video.
- Nymeria: A Massive Collection of Egocentric Multi-modal Human Motion in the Wild — participant and location diversity in human motion data.
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video — task diversity in egocentric dexterity data.
- What Matters in Learning from Offline Human Demonstrations for Robot Manipulation — operator quality and its effect on learning.
- MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations — synthetic expansion of scene and object variation from a small human set.