Splitting Demonstration Data by Site and Operator to Prevent Leakage

The buyer's problem

A video model that scores well on a random split of workplace clips may be recognizing the kitchen, not the task. A manipulation policy evaluated on the same operator and site it trained on says little about a new site. Demonstration data is especially leak-prone because it comes as many short clips cut from a few long sessions, recorded by a few people. Splits are also where supplier choices matter: a dataset delivered with no site, operator or session IDs cannot be split correctly by the buyer.

The general principle is in training vs evaluation datasets. This page covers leakage routes specific to demonstration data.

Leakage routes in demonstration data

Decide which of these your claim excludes. A claim of "generalizes to new sites" requires site-level holdout; "works for new workers" requires operator-level holdout. You can hold out on several axes at once, at a cost in data.

What large video datasets show

Ego4D recorded 3,670 hours of daily-life video from 931 camera wearers at 74 locations in nine countries, with consenting participants, de-identification procedures and five benchmark tasks (Grauman et al., 2021). Because each wearer appears in many clips, a split that ignores wearer identity would share people and places across training and test. Ego-Exo4D recorded 740 participants in 13 cities across 123 scene contexts (1,286 hours in arXiv v4), with simultaneous egocentric and exocentric views (Grauman et al., 2023); the same clip may exist in several views, so a view-level split would also leak. Whether and how a given benchmark's official splits handle these questions is a matter for its documentation; check before reuse.

Operator effects are measurable. Mandlekar et al. studied offline learning from human demonstrations and report that results depend on demonstration quality; their multi-human datasets of 300 demonstrations from six operators were grouped as Worse, Okay and Better, and a separate proficient-human set held 200 demonstrations from one operator (Mandlekar et al., 2021). Mixing operators of different skill in train and test changes what a score means, so record operator metadata and consider holding out by operator and skill group.

Illustrative split manifest

# Illustrative - synthetic IDs
dataset: wp_demo_v1
split_policy:
  unit: [site_id, operator_id]
  rule: "no site or operator appears in more than one split"
  time_holdout: "last 2 weeks of each site go to test only if drift matters"
  dedup: "content hash plus near-duplicate review before splitting"
splits:
  train:  {sites: [site_001, site_002, site_003, site_004], operators: [op_A01 .. op_A18]}
  val:    {sites: [site_005], operators: [op_B01, op_B02]}
  test:   {sites: [site_006, site_007], operators: [op_C01 .. op_C05]}
checks:
  - site_overlap: 0
  - operator_overlap: 0
  - object_instance_overlap: "report"
  - session_overlap: 0
sealed_test: true
test_access: "evaluation team only; no label inspection during tuning"
documented_in: datasheet_or_data_card

Practical rules

1. Split first, then clip. Assign sessions to splits before cutting clips, so clips from one session stay together.
2. Hold the test set out of curation. Do not use test footage for choosing filters, prompts, taxonomy changes or hyperparameters; see task understanding evaluation sets.
3. Report on several axes. Show results for seen sites and new sites, seen and new operators, so a reader sees the gap.
4. Balance, but keep it honest. Small test sets make site-level splits noisy; report counts and uncertainty rather than hiding it.
5. Seal and version. Store split files with hashes. Changes to taxonomy or labels bump a version.
6. Plan for the real world check. Video splits do not replace robot evaluation; see real-world policy evaluation protocols.

What the supplier must deliver for splits to work

Splits need metadata at delivery: site, operator, session, camera, object IDs, recording date and a stable clip ID. Without these the buyer cannot reconstruct units. Put the fields in the demonstration clip metadata schema and document how groups were defined in a datasheet or data card. Data Cards are structured summaries of the essential facts about a dataset needed by stakeholders across its lifecycle, based on the lessons from deploying over 20 of them (Pushkarna et al., 2022); see also datasheets for robotics datasets.

If pseudonymized IDs are used for people, ask whether the mapping is stable across batches so the same operator is not treated as two. Privacy and consent constraints may limit what operator metadata a supplier can share; this is not legal advice, so raise it early with counsel and see worker consent and identifiable people.

Limitations

A clean split makes an estimate honest; it does not make the data representative. If every site in the dataset is a single company with one process, a held-out site is still similar to the others. Few operators mean few independent samples. Splits cannot detect label errors, and held-out video performance does not by itself predict robot success, since human video has no robot actions. Time-based splits capture drift but may remove rare events from training. Finally, once a test set is used repeatedly it stops being held out; plan to refresh it.

Questions to ask suppliers

1. Does every clip carry stable site, operator, session and camera IDs?
2. How many distinct sites and operators does the data include?
3. Are operator IDs consistent across delivery batches?
4. Were clips cut with overlap, and can you provide session boundaries?
5. Are there duplicates or re-encoded copies in the delivery?
6. Were labels produced with pre-labeling models, and which data trained them?
7. Can you hold back an unseen site or operator group for us as a sealed test set?
8. Were any recordings scripted or repeated from a shared scenario?
9. Is operator skill or experience level recorded?
10. What documentation (datasheet or data card) describes how groups were defined?

Discuss your requirements

If your evaluation depends on held-out sites or operators, describe the groups and claims you need to test to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates license and delivery; whether held-out groups can be arranged depends on the suppliers involved. A request does not guarantee a match. Start at the humanoid training data guide or the workplace task demonstration hub. For evaluation-focused sourcing, see evaluation sets.

Tell SourceX what your evaluation set needs to cover

Sources

• Ego4D: Around the World in 3,000 Hours of Egocentric Video — Ego4D scale of camera wearers, locations and de-identification.
• Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI — Data Cards as structured dataset documentation.
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation — Mandlekar et al. multi-operator datasets graded by operator quality.
• Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives — Ego-Exo4D records participants across cities and scene contexts.