SOP-to-Video Step Alignment: Annotation Specification

What alignment adds to segmentation

Step segmentation places boundaries in the video and labels each segment from a step vocabulary; its rules are on step segmentation annotation. Alignment is the next layer. It answers which documented step each segment carries out, which documented steps never appear, and where the worker did something the document does not mention.

That layer is what turns paired assets into usable data. It supports procedure-following evaluation, deviation detection, step-level success labels, video-language training with document text, and measurement of how far practice has drifted from the written procedure. Where to find businesses that hold both halves is covered on instruction-paired demonstration datasets; this page specifies the labels.

Published precedents

Instructional-video research has used text-defined step lists for years, and its design choices carry over:

• Order as supervision. CrossTask took an ordered list of steps for each task from wikiHow and annotated temporal boundaries only for its primary tasks (Zhukov et al., 2019). A procedure's step order is useful even when boundaries are sparse.
• Hierarchy. COIN organizes tasks hierarchically with step-level descriptions and temporal boundaries (Tang et al., 2019). Ego4D Goal-Step added hierarchical goal and step annotations to unscripted egocentric video using a data-driven taxonomy, with its NeurIPS abstract reporting 48K step segments over 430 hours (Song et al., 2023).
• Not everything aligns. Temporal Alignment Networks first decide whether a narration sentence is alignable to the video at all, then where; the authors built HTM-Align, an 80-video, 10-hour subset with manual alignments, to measure it (Han et al., 2022). Treat "no visual counterpart" as a first-class outcome.
• Deviations are the signal. IndustReal frames procedure step recognition around omissions and execution errors (Schoonbeek et al., 2023), and CaptainCook4D records normal and intentionally erroneous executions (Peddi et al., 2023). Workplace video contains unplanned deviations; the specification must capture them rather than force a clean match.

Inputs to fix before annotation starts

• Structured procedure. Each step has a stable ID, text and expected position, following the procedure step schema. Conditional steps are flagged; how to represent decision points is covered on branching procedures and decision points.
• Revision in force. A table linking every recording date and site to the procedure revision in effect. Aligning video to the wrong revision manufactures false deviations; procedural drift explains why this happens often.
• Video master. Constant frame rate, with any existing step segments and their guideline version.
• Order model. Either the document's linear order or, better, a task graph that marks which steps may legitimately swap; see task graphs from procedures.

Alignment unit and match types

Use one row per documented step per task instance, so every step is accounted for in every instance, plus one row per observed action that matches no step. Annotators choose from a closed set of match types.

The most important rule is the line between skipped and not_visible. A step may be called skipped only when the stretch where it would occur is fully in view. Otherwise the data will teach a model that workers omit steps they simply performed off-camera.

Evidence rules per step

For each procedure step, write down before annotation what visible evidence counts as completion: a state change (gasket seated in its groove), a contact (tool on fastener), or an object leaving or entering a place. Steps that set a parameter, such as a torque value or a dwell time, are marked performed with the parameter unverifiable unless an instrument reading is linked. For conditional steps, record which branch was taken and what triggered it.

Illustrative alignment record

{
  "_comment": "Illustrative only — synthetic IDs and values",
  "alignment_id": "aln_0007731",
  "procedure": {"doc_id": "WI-ASM-212", "revision": "E", "step_id": "WI-ASM-212.s07", "expected_position": 7},
  "instance": {"task_instance_id": "ti_site031_0458", "clip_id": "clip_site031_0458", "recorded": "2026-03-14"},
  "revision_in_force": true,
  "match_type": "reordered",
  "observed_position": 5,
  "segments": [{"segment_id": "seg_019221", "start_s": 102.4, "end_s": 109.9}],
  "evidence": "cover on housing before gasket tab is checked",
  "parameter_check": "not_applicable",
  "deviation_note": "operator fitted cover, then lifted it to check the tab",
  "proposal_source": "align_model_v0.3",
  "verified_by": "ann_05",
  "reviewer": "rev_02",
  "guideline_version": "align_guide_v1.0"
}

Model-proposed alignments

Pre-aligning with a text-video model can cut effort. If the vendor uses one, require the model name and version on every row, human review of every row that is not "matched", and a reviewed random sample of "matched" rows, with the measured error rate reported. Proposals must never fill the not_visible and non_visual types automatically.

Agreement and acceptance tests

Thresholds below are illustrative; set your own.

Report results per match type. Agreement on "matched" rows is easy; agreement on "skipped", "merged" and "added" is what tells you the guideline works.

Limitations

• Alignment is only as good as the document match; procedures that no longer describe practice yield deviation-heavy tables that say more about the document than the worker.
• Steps without a visual signature cannot be verified from video alone.
• Edited training films hide skipped steps behind cuts; flag rows that fall on an edit.
• Aligned human video still contains no robot actions, joint states or forces.

Questions to ask annotation vendors

1. How will you determine which procedure revision was in force for each recording?
2. Will you use our match-type list unchanged, and how would you propose changes?
3. How do annotators decide between skipped and not visible?
4. How are merged and added actions recorded?
5. Will you pre-align with a model, and which rows will humans review?
6. What share of instances is double-aligned, and how is agreement reported per match type?
7. How do you handle conditional steps and steps with no visual signature?
8. How are guideline changes versioned and applied to rows already delivered?

Discuss your requirements

If you hold or plan to license procedures and matching video, describe the documents, footage and alignment depth you need to SourceX. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case. SourceX reviews each requirement and follows up about potential sources; a request does not guarantee a match. The procedures hub and the humanoid training data guide cover related assets, and the catalog pages on expert annotations and labels and video recordings describe the underlying asset types.

Explore sourcing and licensing options with SourceX

Sources

• Cross-task weakly supervised learning from instructional videos — ordered step lists from text and selective boundary annotation.
• COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis — hierarchical tasks with step boundaries.
• Ego4D Goal-Step: Toward Hierarchical Understanding of Procedural Activities — hierarchical goal and step annotation on unscripted egocentric video.
• Temporal Alignment Networks for Long-term Video (TAN) — deciding whether text is alignable before aligning it.
• IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting — step recognition with omissions and execution errors.
• CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities — normal and erroneous executions with step annotations.