Short answer
Narrated demonstrations record spoken commentary while the task happens, either think-aloud narration by the performer or commentary by an expert observer. Captured live, it carries intent, step names and reasons that post-hoc captions miss, but it can slow the task, misalign with the visuals and add audio privacy risk. Specify protocol, timing, transcription and alignment checks up front.
Language captured at recording time
Language for demonstrations usually arrives after the fact: an annotator watches a clip and writes a caption or instruction. Narration is different. The person doing or watching the task speaks while it happens, so the language reflects what they were attending to, why they chose a step, and what they were checking. That is information a caption writer would have to guess.
Post-hoc language annotation is covered in language annotations for demonstrations, and paired instruction data in instruction-paired demonstration datasets. This page covers language collected during recording.
Four narration modes
| Mode | Who speaks, when | Gives you | Main cost |
|---|---|---|---|
| Concurrent think-aloud | Performer, during the task | Intent, checks, decisions | Slows and changes the task; speech is fragmentary |
| Pause-and-talk | Performer, after short segments, with the task paused | Clean, step-level sentences | Interrupts flow; segment lengths set by the protocol |
| Retrospective narration | Performer, watching their own recording | Reasons, fuller sentences | Reconstructed memory; misses real-time attention |
| Expert commentary | Observer, during or after | Quality judgments, tips, error spotting | Separate person's time; viewpoint is the observer's |
Published work shows several of these. EPIC-KITCHENS-100 used "pause-and-talk" narration as its first annotation stage, followed by transcription, parsing and temporal segmentation, across 100 hours, 20 million frames and 90,000 actions (Damen et al., 2020). Ego-Exo4D adds expert commentary to its skilled-activity recordings, along with multichannel audio (Grauman et al., 2023).
Live narration versus automatically harvested captions
Scraped narration is cheap and large, and it is noisy. HowTo100M captions are automatically transcribed narrations of 1.22 million instructional web videos, with no extra manual annotation (Miech et al., 2019). Later work found the pairing weak: a temporal alignment network was trained because the narrations are noisy and only weakly aligned, and its authors curated HTM-Align, a manually annotated 10-hour, 80-video subset, to evaluate it (Han et al., 2022). People often describe what they will do or did, not what is on screen now.
A designed narration protocol can reduce that mismatch by controlling when and how people speak, and by logging timing. It does not eliminate it. Plan alignment checks regardless.
What narrated demonstrations can and cannot support
Can support
- Step names and segment boundaries, speeding up step segmentation.
- Instruction-to-video pairs in the vocabulary of practitioners.
- Reasons and conditions: "I check the seal before closing", which no video shows.
- Mistake and quality cues from expert commentary.
- Language grounding for vision-language-action pretraining, where the language is checked.
Cannot support
- Robot actions, forces or joint states. Speech does not change the physical modality.
- Reliable timing, unless narration is timestamped against the video.
- Natural pace. Think-aloud and pause-and-talk alter pace; see working pace in demonstrations.
- Unbiased intent. People rationalize and omit; speech describes what they say they do.
- Transfer to a robot that cannot use the concepts spoken in the narration.
Protocol parameters
- Mode and prompts. Choose per task. Provide prompts such as "say what you are about to do" or "name the object you pick up".
- Language and vocabulary. Allow trade terms, but ask for a glossary mapping synonyms to canonical names.
- Style. Short imperative or descriptive sentences are easier to use than long reflections.
- Timing. Ask for narration to precede or accompany actions, and mark which; log word-level timestamps from transcription.
- Audio capture. Close-talking microphone, noise handling, separate channel for narration, and ambient audio kept or discarded by rule.
- Training the narrators. Practice sessions, a few exemplar clips, feedback on clarity.
- Transcription. Automatic first pass with human correction on a sample; record word error on the sample.
- Segmentation. Align spoken steps with video segments and record the offset.
Illustrative narration package
# Illustrative only - synthetic values the buyer must adapt
recording:
mode: pause_and_talk # or concurrent | retrospective | expert_commentary
segment_prompt: "after each step, say what you just did and what is next"
microphone: close-talking, dedicated channel
video: constant 30 fps; audio sample rate 48 kHz
transcripts:
asr: automatic, then human-corrected on sampled clips
word_timestamps: yes
language: en-US; glossary: canonical object and step names
alignment:
step_boundaries: derived from speech, reviewed against video
max_offset_between_speech_and_action: buyer-set
alignment_review_sample: buyer-set share of clips
metadata: [take_id, speaker_id (pseudonymous), speaker_role, mode, site_017]
privacy:
ambient_audio: removed or flagged; names and personal data in speech: scrubbed or excluded
delivery: video, audio, transcript json, glossary, alignment report
Acceptance tests
- Sample clips and check that spoken steps match the visible action within the agreed offset.
- Verify word error rate on a hand-corrected sample.
- Check that narration style is consistent across speakers; retrain narrators if not.
- Measure the task-time penalty by comparing narrated and silent takes of the same task, if you can afford a small paired set.
- Confirm audio does not contain names, customer details or private conversation.
The task-time check matters because narration changes behavior. Pair silent and narrated takes in a pilot, scoped as in pilot collection scoping.
Permissions and provenance
Speech adds a new risk. Voices identify people, narration may mention names or customers, and ambient audio may capture conversations by people who did not consent. Some jurisdictions have rules for recording audio that differ from video. These are diligence questions, not conclusions, and this is not legal advice. Ask what the consent form says about audio and voice, who can hear raw recordings, and whether transcripts are scrubbed. Each narrated clip also needs the fields in the demonstration clip metadata schema. Audio in workplace footage in general is covered in workplace audio in demonstrations. Redaction does not by itself resolve permission questions.
Questions to ask collection teams
- Which narration mode do you propose, and why for this task?
- How do you train narrators, and can I hear sample recordings?
- How is narration timestamped and aligned with video, and how do you measure the offset?
- What was the measured task-time penalty?
- Who corrects transcripts, and what is the word error rate on the sample?
- How do you handle ambient speech and non-participants?
- Does the consent cover voice, and who can access raw audio?
- Is expert commentary done by someone qualified in the task?
Discuss your requirements
If you need demonstrations with language recorded alongside the task, describe the narration mode, task and permitted uses to SourceX. If no existing asset fits, SourceX can scope a commissioned collection of new recordings; SourceX's role is sourcing, rights review, licensing and delivery coordination, with annotation and specialist capture partners arranged case by case. A request does not guarantee a match. See the humanoid training data guide and the workplace task demonstration hub, or the catalog page on expert annotations and labels.
Describe the workplace demonstration data your robotics project needs
Sources
- Rescaling Egocentric Vision — pause-and-talk narration as the first annotation stage.
- HowTo100M — automatically transcribed narration and its alignment limits.
- Temporal Alignment Networks for Long-term Video — narration alignment and the HTM-Align subset.
- Ego-Exo4D — expert commentary and multichannel audio.