Short answer
Language annotations attach text to demonstration video at three levels: one instruction for the whole episode, step-level instructions or captions for each segment, and optional narration recorded during the task. Language-conditioned policies and vision-language-action models train on these pairs, so the specification must fix the level, wording style, vocabulary, time alignment and number of paraphrases, and test them against the instructions your robot will actually receive.
Why language labels matter
Vision-language-action models are conditioned on text, so the words attached to each training example shape what the model learns to respond to. RT-2, which introduced the class, expresses robot actions as text tokens and co-trains on robot trajectories and web-scale vision-language data (Brohan et al., 2023). Large robot datasets therefore ship text with their episodes: DROID includes a language instruction with each one (Khazatsky et al., 2024), and BridgeData V2 is compatible with goal-image as well as natural-language conditioned methods (Walke et al., 2023). Benchmarks then test whether policies cope with phrasing they have not seen; CALVIN evaluates on novel instructions, environments and objects (Mees et al., 2021).
Two levels of language are common. ALFRED pairs expert demonstrations with 25k directives written both as high-level goals and as low-level step instructions (Shridhar et al., 2019). Human video with captions supports representation learning too: Voltron learns from human videos and their associated captions (Karamcheti et al., 2023).
Text on human video connects footage to instructions and steps. It does not turn that footage into control data; the robot still needs its own actions.
Kinds of text to specify
| Type | Illustrative example | Granularity | Main use | Usual source |
|---|---|---|---|---|
| Task instruction | "Restock the top shelf with the cereal boxes from the cart." | Episode | Policy conditioning, retrieval | Annotators, work orders |
| Step instruction | "Take the next box from the cart." | Segment | Hierarchical policies, planners | Annotators, SOP steps |
| Caption | "The worker slides two boxes forward to the shelf edge." | Segment or clip | Video-language pretraining, evaluation | Annotators, reviewed model drafts |
| Narration | "Checking the date before I face it." | Free, time-stamped | Intent and reasoning cues | Recorded during the task |
| Outcome statement | "Shelf faced; one damaged box removed." | Episode | Success detection, reward | Annotators, system logs |
Instructions are imperative and say what should happen. Captions are descriptive and say what did happen. Mixing the two in one field confuses both training and evaluation.
Narration and post-hoc text
Narration recorded by the person doing the work is cheap per minute and captures intent that no annotator can see. EPIC-KITCHENS-100 used participants' pause-and-talk narration as its first annotation stage, then transcribed, parsed and temporally segmented it (Damen et al., 2020). Ego-Exo4D recorded expert commentary on skilled activity (Grauman et al., 2023). Recording-time narration as a capture protocol is covered in narrated demonstrations.
Narration found in existing video is weaker. HowTo100M used automatically transcribed narrations of 1.22M instructional web videos as captions, with no additional manual annotation (Miech et al., 2019); follow-up work found that a large share of those narrations do not line up with what is on screen. The trade-offs of web video against licensed business footage are set out in public instructional video vs licensed business footage.
Wording rules
- Mood and tense. Imperative for instructions; present tense for captions; first person allowed only in narration.
- Visible referents. Name objects by what a robot can perceive (color, shape, position, label text) rather than internal part numbers; keep part numbers in metadata.
- Explicit spatial relations. "Top shelf, left of the price tag", not "over there".
- One action per step instruction. Split chained instructions at the step boundary.
- No orphan pronouns. Each segment's text must make sense on its own.
- Only what is visible. Mark non-visual knowledge, such as "the part is the old revision", with a flag instead of mixing it into captions.
- Controlled verbs. Use the verbs from your action vocabulary; see action taxonomies for workplace video.
- Paraphrases. Several independent paraphrases per instruction, written by different annotators, so models see varied phrasing.
Illustrative annotation record
{
"_comment": "Illustrative only — synthetic IDs and text",
"clip_id": "clip_site031_0144",
"task_instruction": {
"text": "Restock the top shelf with the cereal boxes from the cart.",
"paraphrases": [
"Move the cereal boxes from the cart onto the top shelf.",
"Fill the top shelf using the boxes on the cart."
]
},
"steps": [
{"segment_id": "seg_002201", "instruction": "Take the next box from the cart.",
"caption": "The worker lifts a red box from the cart's lower tray.",
"non_visual_note": null},
{"segment_id": "seg_002202", "instruction": "Place the box at the left end of the top shelf.",
"caption": "The box is set down and pushed flush with the shelf edge.",
"non_visual_note": "box is the new package design"}
],
"narration": {"present": false},
"outcome": "Top shelf restocked; no items damaged.",
"language": "en-US",
"authoring": {"annotator_ids": ["ann_03", "ann_11"], "model_drafted": false},
"guideline_version": "lang_guide_v0.7"
}
Alignment to time and to other labels
Tie every step instruction and caption to a segment ID from the step segmentation and task boundary annotation, not to free-floating timestamps. That keeps text and boundaries consistent when either is revised. Hierarchical text is common; Nymeria, for example, provides hierarchical language descriptions of human motion (Ma et al., 2024). Flag any step whose executed order differs from the instruction sequence.
If a business's procedures supply the step wording, treat that text as a vocabulary source, not as a description of what happened; procedural text for instruction grounding covers how procedure documents feed language models.
Model-drafted text
Drafting captions with a vision-language model and having people correct them can cut effort. If you allow it, require the model name and version, the prompt, the share of items reviewed, and a measured rate of statements not supported by the video. Check that the model's terms allow its output to be used for training.
Acceptance tests
| Check | Method | Example pass criterion (buyer sets) |
|---|---|---|
| Grounding | Reviewer watches the clip and checks each statement | Share of fully supported items above threshold |
| Completeness | Count required text per segment | Every required segment has text |
| Vocabulary | Automated check against verb and object lists | Off-list terms below threshold |
| Paraphrase diversity | Distinct word sequences across paraphrases | Above threshold |
| Style | Sampled review against the wording rules | Pass rate above threshold |
| Personal or confidential data | Search for names, sites, customers | None found |
Limitations
- Text does not add robot actions, joint states or forces.
- Annotator phrasing differs from how your end users will speak; collect a sample of real instructions from the intended deployment and compare.
- Narration in existing footage is often late, early or off-topic.
- Spoken narration can contain names and customer details that need redaction.
- Instruction-paired assets from suppliers vary widely in quality; sourcing them is covered in instruction-paired demonstration datasets.
Questions to ask suppliers and annotation vendors
- Which text levels will you write, and in which mood and tense?
- How many independent paraphrases per instruction, from how many different annotators?
- How are object names chosen: visible attributes, catalog names or both?
- Is any text drafted by a model, and how is it reviewed?
- How is each text item tied to segment IDs and timestamps?
- How do you measure grounding errors, and what rate did your last comparable project reach?
- Does existing footage carry narration or voice-over, and who owns those recordings?
- Which languages and locales can annotators write in?
Discuss your requirements
If your models need instruction-labeled demonstrations, describe the text levels, languages and permitted uses to SourceX. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case. SourceX reviews the requirement and follows up about potential sources, and a request does not guarantee a match. For related capture and labeling topics, see the workplace task demonstration hub and the humanoid training data guide; the catalog page on expert annotations and labels covers labeled assets in general.
Explore sourcing and licensing options with SourceX
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control — the VLA model class and co-training with language.
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — per-episode language instructions in robot data.
- BridgeData V2: A Dataset for Robot Learning at Scale — support for language-conditioned methods.
- CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks — evaluation on novel instructions.
- ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks — goal-level and step-level directives paired with demonstrations.
- Language-Driven Representation Learning for Robotics — learning from human videos and captions.
- Rescaling Egocentric Vision — narration-first annotation pipeline.
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives — expert commentary on skilled activity.
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips — automatic narration as captions and its alignment limits.
- Nymeria: A Massive Collection of Egocentric Multi-modal Human Motion in the Wild — hierarchical language descriptions of motion.