Short answer
Imitation learning trains a policy on paired observations and actions from demonstrations, so it needs robot-specific, synchronized, quality-controlled episodes. Visual pretraining trains an encoder on images or video without actions, so it values diversity and scale of visuals. Buyers often conflate the two; specify which stage the data serves, because acquisition, labels and rights differ for each.
Two jobs that share the word "demonstration"
A "demonstration" can mean a recorded robot episode with joint states and commands, or a video of a person doing a task. The first can supervise a policy. The second can, at most, help with the stages that do not need actions. A requirement that says only "demonstration data" leaves suppliers guessing which one you mean. For the broader view of human video, see what human demonstration video can do; this page contrasts the two training stages.
What imitation learning consumes
Behavior cloning style methods learn a mapping from observations to actions. Diffusion Policy frames a visuomotor policy as conditional denoising of actions and reports a 46.9% average improvement over the prior state of the art across 12 tasks from 4 benchmarks (Chi et al., 2023). Action Chunking with Transformers learned six fine-grained real-world tasks to 80-90% success from only 10 minutes worth of demonstrations (Zhao et al., 2023).
The consequences for data are specific:
- Actions are the label. Each episode needs the commands or states the policy will predict, in a declared action space and at a declared control rate. See action space and control frequency.
- Embodiment match. Observations come from the cameras the deployed robot has; actions from the robot it will run on, or something retargeted to it.
- Synchronization. Frames and actions must share a clock.
- Quality of the demonstrator. Offline learning results depend on demonstration quality. A study of six algorithms across five simulated and three real tasks used multi-human datasets of 300 demonstrations from six operators graded Worse, Okay and Better, and a 200-demonstration set from one proficient operator (Mandlekar et al., 2021). See expert vs novice demonstrations.
- Task coverage near deployment. Start states, object variety and recovery behavior in the data bound what the policy handles.
These come from teleoperation, handheld capture devices or instrumented rigs; see human demonstrations vs teleoperation.
What visual pretraining consumes
R3M pretrained a representation on Ego4D human video using time-contrastive, video-language alignment and sparsity objectives, then used it as a frozen perception module; a real Franka Panda learned tasks in a cluttered apartment from 20 demonstrations (Nair et al., 2022). Masked visual pretraining trained a 307M-parameter ViT on 4.5 million images from the internet and egocentric videos, frozen under a learned control module (Radosavovic et al., 2022).
Notice that both still needed robot demonstrations afterwards. The requirements differ:
- Diversity over pairing. Many scenes, objects and hands matter more than exact synchronization with robot state.
- No actions needed. Video-language text or temporal structure may be the supervision.
- Viewpoint breadth. Egocentric footage can help, though viewpoint mismatch with the robot remains a concern.
- Rights at scale. Many sources, each with its own terms; see public dataset license checks.
Illustrative requirement comparison
| Requirement | Imitation learning data | Visual pretraining data |
|---|---|---|
| Core label | Action per timestep | Often none, or captions |
| Source | Target robot, teleop or instrumented rig | Human or web video, images |
| Embodiment | Must match or be retargeted | Not required |
| Synchronization | Frame-to-action, per episode | Loose |
| Calibration | Intrinsics, extrinsics, hand-eye | Rarely needed |
| Demonstrator quality | Critical | Less critical |
| Scale driver | Tasks and conditions | Visual diversity |
| Typical rights issue | One robot-data supplier | Many contributors, people in frame |
Specifying both in one request
# Illustrative (synthetic values)
stage_a_visual_pretraining:
modality: egocentric video
labels: optional task text
diversity_axes: [site, hand_type, lighting]
stage_b_imitation:
modality: robot episodes
action_space: ee_delta_pose
control_hz: 15
sync: all streams on one clock
eval_holdout: site_017
Sizing for stage B is covered in how many demonstrations published work reports. A full trajectory spec is in robot trajectory dataset specification.
What neither can do alone
Pretraining does not teach control. Imitation data from one robot does not generalize to another by default. Business footage without actions can serve stage A, not stage B, as discussed under when video is not enough.
Questions to ask suppliers
- Which stage is this data labeled and recorded for?
- If it is video, what supports linking it to actions, if anything?
- If it is robot data, what are the action space, rate and synchronization?
- How were demonstrators selected and graded?
- What variation is in the data: sites, objects, operators, start states?
- What rights apply to each stage's use, and do they differ?
Matching data to a humanoid stack
Many humanoid programs combine stages: a visual encoder or video-language model pretrained on large video, a policy head trained on teleoperated robot episodes, and evaluation on hardware. Each stage has its own supplier type. Pretraining video can come from existing footage with clear licenses; policy episodes come from robot teleoperation or instrumented capture; evaluation sets need held-out sites or operators. Writing the stages separately lets you buy each from the supplier best placed to provide it, and helps avoid paying robot-data prices for video you will only use for representation learning.
It also helps with acceptance. Video can be accepted on resolution, coverage and rights; robot episodes on synchronization, action validity and quality grade. See robot trajectory dataset specification and demonstration video acceptance criteria.
Keep in mind that the same clip can serve different stages under different rights: a license that covers representation learning may not cover policy training or distribution of weights, so state the intended stage in the license request.
Common mistakes
- Asking for "demonstrations" and receiving unlabeled video when episodes were needed.
- Expecting a pretrained encoder to remove the need for robot demonstrations.
- Accepting robot episodes without a stated action space.
- Mixing a held-out evaluation set into pretraining collections.
Discuss your requirements
Say which stage the data serves when you describe your requirements to SourceX. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match. More in the demonstrations hub, the requirements specification template and the humanoid data guide.
Discuss your humanoid training-data requirements with SourceX.
Sources
- Diffusion Policy — demonstration-trained visuomotor policy.
- Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware — ACT and demonstration quantity.
- R3M — visual representation from human video.
- Real-World Robot Learning with Masked Visual Pre-training — frozen encoder pretraining.
- What Matters in Learning from Offline Human Demonstrations — demonstration quality.