Short answer
World-model and video-prediction teams need large volumes of task video with stable viewpoints, continuous frame timing and clean scene context, not necessarily action labels. Workplace video can feed pretraining and generation, but robot actions are inferred, not recorded, so a humanoid still needs real robot data to ground what the model predicts.
Why world-model teams buy video differently
A team training a policy on human demonstrations cares about hands, labels and embodiment. A team training a video world model cares about something else: whether the footage lets a network learn how scenes evolve when someone acts on them. The model predicts the next frames given the current ones and, often, a text instruction. Curation criteria follow from that objective.
The consequences for sourcing are practical. Cuts and montage break the prediction target. Variable frame rates distort learned dynamics. A camera that moves for no task-related reason adds motion the model cannot explain. Narration that does not match the image teaches the wrong association. None of these matter much for a team that only wants a few hundred hand-tracked clips, which is why this page exists separately from what human demonstration video can and cannot do for robot learning.
What published work shows video world models doing
Three patterns recur in published work, and each implies a different purchase.
- Video prediction as pretraining. GR-1 pretrained a GPT-style model on video prediction using Ego4D, about 800,000 three-second clips (8 million frames), then fine-tuned on robot data. The authors report CALVIN success rising from 88.9% to 94.9% and zero-shot unseen-scene success from 53.3% to 85.4% (Wu et al., 2023).
- Policies as video generation. UniPi treats decision-making as text-conditioned video generation, then uses an inverse dynamics model to recover actions from the generated frames; internet video and pretrained language embeddings aided transfer to real robots (Du et al., 2023).
- Synthetic trajectories from a world model. DreamGen adapts image-to-video world models to a robot, generates "neural trajectories", recovers pseudo-actions with a latent action model or inverse-dynamics model, and trains policies on them. A humanoid learned 22 new behaviors from teleoperation data of a single pick-and-place task (Jang et al., 2025). GR00T N1 places such trajectories in the middle of its data pyramid and reports expanding 88 hours of teleoperation data to 827 hours of neural trajectories (NVIDIA, 2025).
Notice what sits under every example: robot data. The world model is adapted to a robot or the policy is fine-tuned on one. Workplace video helps the model learn plausible scene dynamics; it does not replace the robot recordings that tie imagery to actions.
What workplace video can and cannot supply
| Need of a world-model team | Workplace video can support | Workplace video cannot support |
|---|---|---|
| Diverse real-world scene dynamics | Many tasks, objects, lighting and sites in one archive | Coverage of your robot's own workspace |
| Instruction-conditioned generation | Captions, narration or procedure text linked to clips | Reliable alignment unless it has been checked |
| Pseudo-action recovery | Frame pairs for latent-action or inverse-dynamics models | Ground-truth actions; these are inferred and need robot grounding |
| Contact and deformation | Visible object response to a hand | Forces, slip or tactile state |
| Evaluation of generations | Held-out real clips to compare against | Evidence that a generated video is physically executable |
The last row matters in procurement. A convincing generated video is a visual claim; whether a humanoid can execute it is a control question answered only with robot rollouts. Keep that distinction in the evaluation plan, and hold evaluation clips out by site and operator, as described in demonstration dataset splits.
Curation criteria that matter for video prediction
The same raw archive can be near-useless or very useful depending on selection. The properties below are examples buyers set, not universal thresholds.
- Shot continuity. Long uncut takes per task instance, with scene cuts flagged. Training-video archives are often edited, so ask for the edit decision list or for cut timestamps.
- Frame timing. Constant frame rate, or recorded timestamps for every frame. Variable-frame-rate screen captures and re-encoded exports are common failure points.
- Camera stability. Fixed or consistently mounted cameras. Handheld or auto-tracking footage is usable, but the camera motion should be logged or at least flagged.
- Task-relevant action. The share of frames in which an agent changes the scene. Hours of idle waiting inflate volume without adding dynamics.
- Caption fidelity. Narration transcribed automatically can be misaligned: HowTo100M pairs web videos with automatically transcribed narrations (Miech et al., 2019); later work trained a separate network because those narrations are noisy and only weakly aligned (Han et al., 2022). Ask who wrote captions and whether alignment was verified.
- Overlay and watermark content. Burned-in captions, logos, picture-in-picture presenters and screen content can leak into generated frames.
For turning raw hours into usable segments, see curating raw workplace video. Teams that use video mainly to infer actions should also read sourcing unlabeled task video for latent-action pretraining, and the physical reasons a generated human video does not transfer directly are in the human-to-humanoid embodiment gap.
Illustrative selection report for a world-model purchase
# Illustrative only - synthetic values, not an available dataset
archive: vendor_course_library_A07
videos_inventoried: 2400
after_filter:
constant_fps: yes
scene_cuts_flagged: per-video timestamps
uncut_segments_over_20s: retained
presenter_to_camera_segments: excluded
burned_in_text: flagged, not removed
captions:
source: human-written step captions
alignment_checked: sample reviewed by buyer
metadata: [site_id, camera_mount, fps, resolution, task_label, language]
holdout: whole vendors/sites reserved for evaluation
rights: model training and internal evaluation; generated outputs policy to be agreed
Treat the "generated outputs policy" line seriously. If a world model trained on licensed footage can reproduce recognizable people, products or branding, the license should say what is allowed. This is a contract question to settle with the supplier; see licensing terms for robotics training data. This is not legal advice.
Where this video comes from
Operating businesses and learning-and-development teams hold the largest existing volumes: course libraries, onboarding clips, process recordings and safety demonstrations. The corporate training video archive guide covers how to evaluate them. Egocentric corpora built for research, such as Ego4D, show what a consent and de-identification process looks like at scale: consenting camera wearers and documented de-identification procedures (Grauman et al., 2021). Workplace archives rarely come with that paperwork.
Existing footage was made for another purpose. Viewpoints, coverage, labels and consent were not designed for model training. If a world model needs footage in your robot's workspace, or from a camera matching your robot's head, consider a commissioned recording instead.
Permissions and provenance
- Who is on camera: employees, actors, customers? Was notice given, and for what purpose?
- Are any videos vendor-produced, licensed from a third-party course provider, or containing third-party music, brands or screen content?
- Does the license cover training generative models and, separately, the models' outputs?
- Can the supplier prove it controls rights to every video, or only to the library as a whole?
Redaction or face blurring changes pixels and may hurt a world model, since the network learns from every frame; it does not by itself settle whether the footage may be used.
Questions to ask suppliers
- How many hours are uncut single-take recordings versus edited compilations?
- Is frame rate constant, and what are the resolutions and codecs across the archive?
- How were captions produced, and what share has been checked against the visuals?
- What share of frames shows a person acting on objects rather than presenting to camera?
- Are camera positions fixed per task, and is mount information recorded?
- Which videos contain on-screen text, logos or third-party content, and how are they flagged?
- How many sites, vendors and task families are covered, and can I hold whole sites out?
- Who can grant training and generative-model rights, and for which videos?
Discuss your requirements
If your team is building a video world model and needs task footage with particular continuity, caption and rights properties, describe the requirement to SourceX. SourceX reviews the requirement, looks into whether US businesses hold suitable material, checks the data and the supplier's licensing permissions, and follows up about potential sources. A request does not guarantee a match. Start from the humanoid training data guide or the workplace task demonstration hub, and see the catalog page on video recordings.
Describe the workplace demonstration data your robotics project needs
Sources
- Learning Universal Policies via Text-Guided Video Generation — policies as text-conditioned video generation with inverse dynamics.
- DreamGen: Unlocking Generalization in Robot Learning through Video World Models — neural trajectories and pseudo-actions from video world models.
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (GR-1) — video-prediction pretraining on Ego4D clips.
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots — data pyramid and neural trajectories.
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips — automatically transcribed narration and its alignment limits.
- Temporal Alignment Networks for Long-term Video — weak alignment of HowTo100M narration.
- Ego4D: Around the World in 3,000 Hours of Egocentric Video — consent and de-identification at corpus scale.