Short answer
Latent-action pretraining learns from video without robot action labels, so business task video can feed it. Buyers should look for many clear, hands-and-objects-visible clips across varied tasks, with captions or step boundaries and clean rights. The video supplies no robot actions, forces or consistent viewpoints, so models still need fine-tuning on robot data for control.
What the method needs from video
Latent-action methods infer the change between frames as a learned code, so the model can be pretrained on video where nobody recorded robot commands. Latent Action Pretraining (LAPA) trains a discrete latent-action model, using a VQ-VAE-style quantizer, on pairs of frames, pretrains a vision-language-action model to predict those latent actions, and then fine-tunes on a small robot dataset. The authors report that pretraining on human manipulation video alone outperformed pretraining on Bridge V2 (Ye et al., 2024). This is one study's result under its settings, not a prediction for another model.
Related pipelines use the same idea in different places. GR00T N1 describes a data pyramid with web data and human egocentric video at the base, synthetic data in the middle and real-robot teleoperation on top, and uses a latent-action codebook and inverse-dynamics pseudo-actions to label action-less video (NVIDIA, 2025). DreamGen trains policies on synthetic "neural trajectories" from video world models, recovering pseudo-actions with a latent action or inverse-dynamics model (Jang et al., 2025). GR-1 pre-trained on video prediction using Ego4D clips, about 800,000 three-second clips, then fine-tuned on robot data; the authors report CALVIN success rising from 88.9% to 94.9% and zero-shot unseen-scene success from 53.3% to 85.4% (Wu et al., 2023).
Read these as evidence that video pretraining helped in those experiments. In each, robot data was still required for the final policy. The page on demonstrations for imitation vs pretraining explains the broader distinction, and what human demonstration video can and cannot do covers the full picture.
What the data can support - and what it cannot
| Capability | Action-free task video | Condition |
|---|---|---|
| Visual representation of tasks, hands, objects | Yes | Variety of scenes and objects |
| Learning latent transitions between frames | Yes | Clear motion, adequate frame rate |
| Step and task recognition | Yes | Captions or step boundaries help |
| Video-language pretraining | Yes | Instruction or caption pairing |
| Robot joint or end-effector actions | No | Come from robot data or inferred pseudo-actions |
| Force, contact and tactile signals | No | Not in video |
| Consistent viewpoint matching the robot | Rarely | Mixed cameras are typical |
| Complete task coverage | No | Skews to routine, successful work |
The method sidesteps the missing labels but not the embodiment gap. Latent actions inferred from human hands must still be mapped to robot controls through fine-tuning data. Video world models are a related route covered in workplace video for world models. Do not describe a model pretrained on business video as having learned to control a humanoid.
What to look for in candidate video
Hands and objects in view. Latent-action learning depends on visible change. Clips where the actor's hands, tools and objects are rarely in frame add little.
Breadth over polish. Pretraining benefits from variety in tasks, objects, scenes and people. Many modest recordings across sites often beat a few high-production ones. Production-value tutorials may have staged lighting and cut-aways that differ from real work.
Reasonable motion clarity. Frame rate, motion blur and resolution set how well frame-to-frame change can be encoded. Compressed screen recordings and heavily edited clips are weaker.
Continuous segments. Edits, jump cuts, picture-in-picture slides and overlays break the frame pairs the method uses. Prefer unedited segments, or segment at cuts.
Language or step structure. Captions, narration, or matching written steps allow pairing with instructions. See instruction-paired demonstration datasets.
Metadata. Task, site, camera type and duration let you balance the corpus and hold out evaluation data by site or task.
Supplier types
- Corporate learning and development teams hold training courses and how-to libraries. See corporate training video archives and the catalog entry for training materials and LMS content.
- Operators with process or quality recordings hold footage of real work, usually from fixed cameras; see fixed-camera footage limits.
- Franchise networks and facility contractors may hold standardized training across many locations.
- Vocational and certification programs hold recorded skills assessments.
For a comparison of footage already in archives with new recording, see existing footage vs commissioned recording.
Preparation steps
- Inventory and select: catalog titles, durations, tasks and formats; sample before licensing everything. See curating raw workplace video.
- Segment and caption: cut into task or step clips and add short descriptions.
- Redact: faces, screens, logos and identifying text; check that redaction does not obscure hands and objects. See redaction in demonstration video.
- Rights check per video: actors, employees, vendor-produced courses and third-party content each have different owners.
- Split: hold out entire sites, tasks or series for evaluation, not random clips.
Illustrative selection criteria
# Illustrative only - synthetic thresholds the buyer would set
corpus: task_video_pretraining_v0
include_if:
hands_visible_fraction: >= buyer_defined
continuous_segment_length: >= buyer_defined seconds
resolution_min: 480p
frame_rate_constant: true
exclude_if:
heavy_overlay_or_slides: true
stock_footage_or_third_party_music: unresolved rights
actors_without_release: true
metadata_required: task_label, site_id, camera_type, duration, language
balance_by: task_family, site_id, object_category
holdout: 3 sites and 5 task families never used in training
Questions to ask suppliers
- How many videos, how much total duration, and across how many tasks and sites?
- Were they produced for training, recorded from real operations, or bought in from a vendor?
- Who appears on camera, and under what release or notice?
- Is any content third-party: stock footage, music, licensed software screens?
- Are the videos edited, with cuts, overlays and slides, or continuous recordings?
- Do captions, transcripts or written procedures exist, and can they be licensed?
- Which formats and frame rates are available, and are originals retained?
- Can the supplier provide a random sample, not a showcase selection?
- Can the license cover derived artifacts such as latent-action labels and embeddings?
Discuss your requirements
If you plan a video pretraining stage, describe the tasks, scale and permitted uses to SourceX. SourceX looks for US businesses that hold suitable video, checks the data and the supplier's licensing permissions, and follows up about potential sources; a request does not guarantee a match. If no existing archive fits, SourceX can scope a commissioned collection. Start from the humanoid training data guide or the workplace task demonstration hub, and see the catalog entry for video recordings.
Describe the workplace demonstration data your robotics project needs.
Sources
- Latent Action Pretraining from Videos — VLA pretraining without robot action labels.
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots — data pyramid and pseudo-actions for action-less video.
- GR-1: Large-Scale Video Generative Pre-training for Visual Robot Manipulation — video-prediction pretraining on Ego4D then robot fine-tuning.
- DreamGen: Generalization in Robot Learning through Video World Models — synthetic neural trajectories with recovered pseudo-actions.