Instruction-Paired Demonstration Datasets

Why pairing is worth paying for

Unpaired footage tells a model what work looks like. Paired footage tells it which words describe which segment, which is what language-conditioned policies, planners and video-language models train on.

Published representation work leans on that link. R3M combined time-contrastive learning with a video-language alignment objective when pretraining on Ego4D human video (Nair et al., 2022), and Voltron learned language-driven representations from human videos and their captions (Karamcheti et al., 2023). For instruction following, ALFRED paired expert demonstrations with 25,000 natural-language directives that give both a high-level goal and low-level step instructions (Shridhar et al., 2019). A business SOP with matching step video has the same two-level shape: a task goal and an ordered list of steps.

Five kinds of pairing

Each has published precedents. EPIC-KITCHENS-100 collected "pause-and-talk" narration and then transcribed, parsed and temporally segmented it (Damen et al., 2020). Ego-Exo4D recorded expert commentary alongside skilled activities (Grauman et al., 2023). COIN annotated a hierarchy of tasks and step-level descriptions with temporal boundaries (Tang et al., 2019).

Loose voiceover is the cheapest and the noisiest. HowTo100M paired clips with automatically transcribed narration and no manual annotation (Miech et al., 2019); later work found much of that narration did not match what was visible.

Where existing paired assets live

The most promising suppliers are businesses where a documentation or quality team maintains procedures and a training team films the same tasks:

• Manufacturers with controlled work instructions and line-training video.
• Food-service and franchise operators with recipe and prep cards plus prep videos.
• Facility services and housekeeping contractors with cleaning SOPs and onboarding video.
• Laboratories with method SOPs and recorded training.
• Field service organizations with service procedures and recorded jobs.
• Users of connected-worker platforms, where each step may already store text, photos and short clips.

In most of these, the two halves live in different systems owned by different teams. Pairing is usually something you add, not something you receive. For the training-video half, use the corporate training video archive checklist.

Illustrative package contents

# Illustrative only — one paired record and the package around it; names and values are synthetic
record:
  pair_id: pair_000431
  task_id: task_case_sealing
  procedure: {doc_id: SOP-PK-017, revision: C, step_index: 4,
              step_text: "Fold the minor flaps, then the major flaps, and seal the seam with one strip of tape."}
  video: {clip_id: clip_site02_0193, start_s: 41.2, end_s: 53.8, camera: head_mounted, fps: 30}
  alignment: {source: annotator, guideline: v1.2, reviewer: annot_B07, confidence: high}
  deviation: "taped before folding the second major flap; corrected at 50.1 s"
  narration: null
  outcome: completed
package:
  procedures: PDF originals + structured step lists, with revision dates
  video: source-resolution clips with constant frame rate
  alignment_file: one row per step instance, including steps with no visible segment
  deviation_notes: skipped, reordered, repeated and added steps
  docs: annotation guideline, agreement report, rights summary per document and per clip
excluded: robot actions, joint states, forces (not present in either source)

The field-level alignment specification, including how to mark partial and missing matches, is owned by SOP-to-video step alignment. Captions written by annotators follow language annotations for demonstrations.

What the pairs can and cannot support

Can support: video-language pretraining; step recognition and localization; task-progress estimation; grounding a planner's step vocabulary in real footage; evaluation sets that test whether a model can follow, order or verify instructions; detection of skipped or reordered steps when deviations are labeled.

Cannot support: low-level control. Neither the document nor the video contains joint states, actions or forces, and the text describes the intended procedure rather than the motion. What SOPs can and cannot teach a humanoid sets out that boundary.

Quality and acceptance checks

• Revision match. The procedure revision in force when each clip was recorded is identified. Mismatches are the main source of false pairs; see procedural drift.
• Step coverage. A buyer-set share of procedure steps has a matched segment, and unmatched steps are listed, not dropped.
• Agreement. A second annotator aligns a random subset; boundary disagreement stays within a tolerance you set.
• Deviation labels. Skipped, reordered and repeated steps are marked, so the data does not teach that every execution matches the text.
• Visibility. For each step, the state change it describes is visible in the segment.
• Held-out split. Evaluation pairs are separated by site, procedure or performer, depending on the claim you want to test.

Limitations and preparation

Most paired data is built, not found. Expect to structure PDF procedures into steps, match revisions to recording dates, align and review, and write language for steps that the document leaves implicit. Narration recorded during the task, as in narrated demonstrations, is often more faithful to what happened than a document written for a different purpose.

Questions to ask suppliers

1. Do the procedures and the videos cover the same tasks at the same sites?
2. Which procedure revision was in force when each video was recorded?
3. Are procedures structured into numbered steps, or free text and images?
4. Is narration recorded live, scripted afterwards, or absent?
5. Do videos show every step, or only the ones a trainer chose to film?
6. Are deviations from the procedure visible and, if so, are they ever corrected on camera?
7. Who owns the procedures, and were any copied from equipment makers or consultants?
8. Who appears in the videos, and do their releases extend to licensing for AI training?

Permissions and provenance

Paired data has two rights chains. Procedures may contain proprietary parameters or text adapted from equipment manuals, and video involves the people recorded. Clear both before pairing, since alignment work on content you cannot license is wasted.

For ownership of procedure text, see who owns a procedure; for people on camera, see worker consent and identifiable people. This is not legal advice.

Discuss your requirements

If you need instruction-paired data, tell SourceX which tasks, pairing type and alignment granularity you need. SourceX looks for US businesses that hold both procedures and video, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case. A request does not guarantee a match; SourceX follows up about potential sources.

See also the workplace task demonstration hub, the humanoid training data guide, and the catalog pages on licensing SOPs and playbooks and video recordings.

Describe the workplace demonstration data your robotics project needs

Sources

• R3M: A Universal Visual Representation for Robot Manipulation — video-language alignment in robot representation pretraining.
• Language-Driven Representation Learning for Robotics — representations from human video and captions.
• ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks — demonstrations paired with goal- and step-level instructions.
• Rescaling Egocentric Vision — live narration as a pairing source.
• Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives — expert commentary on skilled activity.
• COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis — task hierarchy and step boundaries.
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips — loose narration pairing at scale.