Short answer
Step segmentation marks where each task instance and each step begins and ends in a video, labels every segment from a fixed step list, and flags errors, repeats, pauses and occluded stretches. A usable specification defines the step vocabulary, a start and end rule for each step, how overlaps and interruptions are handled, a minimum segment length and an agreement test run on a gold set before scaling.
What temporal labels feed
Raw workplace footage is hours of continuous activity. Step segments turn it into units a model can use: step recognition, progress estimation, evaluation of procedure following, task-graph learning, clip extraction for pretraining, and segment-level pairing with language (covered in language annotations for demonstrations).
Two neighboring topics have their own pages. Which hand touches which object, and when, is spatial labeling, specified in hand-object interaction annotation. How to design the verb-noun vocabulary itself is covered in action taxonomies for workplace video. This page is about placing boundaries and applying labels consistently.
Three levels of temporal labels
Most specifications use three nested levels: the task instance (one complete job, such as kitting one tote), the step (a meaningful stage, such as "pick insert from bin") and, optionally, the fine-grained action (a single verb-object motion, such as "grasp insert").
Public datasets show how these levels are used and what they cost:
- COIN covers 11,827 videos across 180 tasks in 12 domains, with a hierarchical structure and step-level descriptions with temporal boundaries (Tang et al., 2019).
- CrossTask takes an ordered step list for each task from wikiHow and annotates temporal step boundaries for its primary tasks only (Zhukov et al., 2019), a reminder that boundary labels can be applied to a chosen subset rather than to every video.
- YouCook2 annotates procedure segments with temporal boundaries and descriptions (Zhou et al., 2017).
- Assembly101 provides more than 100,000 coarse and 1 million fine-grained action segments (Sener et al., 2022), showing how quickly volume grows at the finer level.
- EPIC-KITCHENS-100 starts from narration recorded by participants, then transcribes, parses and temporally segments it (Damen et al., 2020).
- IndustReal frames procedure step recognition with omissions and execution errors in an industrial-like setting (Schoonbeek et al., 2023), and CaptainCook4D provides 5.3K step annotations and 10K fine-grained action annotations over normal and erroneous executions (Peddi et al., 2023).
Specify only the levels your models consume. Fine-grained actions multiply annotation effort and disagreement.
Start and end rules
Annotators disagree most about where a step starts and stops, not about what it is called. Write a rule for each step type, anchored to a visible event.
| Step type (illustrative) | Start rule | End rule | Common ambiguity |
|---|---|---|---|
| Pick | Hand begins its final approach to the object | Object leaves its support surface | Regrasps and hesitations |
| Place | Object begins its final descent toward the target | Hand releases and starts to withdraw | Adjustments after release |
| Fasten | Tool or fastener contacts the part | Tool withdraws, or the next fastener begins | Several fasteners in one motion |
| Inspect | Object or head turns into the inspection pose | Object is set down, or the next step starts | Inspection blended with handling |
| Wipe | Cloth first contacts the surface | Cloth lifts for the last time | Brief lifts mid-stroke |
Add general rules: whether gaps between steps are labeled as idle or absorbed into neighbors; a minimum segment length set by the buyer; how two hands doing different steps at once are recorded (parallel tracks per hand, or one primary step with a secondary flag).
Special cases the guideline must cover
- Pauses and interruptions. Label the pause separately and resume the same step instance afterwards.
- Repeated attempts. Give each attempt an index; mark failed attempts with a status rather than deleting them.
- Errors. Distinguish omissions (recorded at task level), wrong order and execution errors, plus any correction that follows. Success labels and task outcome annotation covers the outcome layer.
- Not visible. When the hands or object leave the frame, label the stretch as not visible instead of guessing; this matters most in egocentric footage.
- Edited footage. Training films contain cuts, retakes and voice-over. Flag any boundary that falls on an edit.
Illustrative annotation record
{
"_comment": "Illustrative only — synthetic IDs and values",
"segment_id": "seg_000412",
"clip_id": "clip_site017_0093",
"level": "step",
"label": "pick_insert_from_bin",
"vocabulary_version": "steps_v1.2",
"parent_id": "task_000057",
"attempt": 1,
"status": "completed",
"error_type": null,
"hands": "right",
"start_s": 41.733, "end_s": 44.100,
"start_frame": 1252, "end_frame": 1323,
"visibility": "full",
"annotator_id": "ann_07",
"review": {"double_annotated": true, "adjudicated": false},
"guideline_version": "seg_guide_v0.9"
}
Time ranges should be expressed in seconds and frames against a constant-frame-rate master copy, so labels survive re-encoding.
Agreement and quality control
- Gold set. Experts annotate a reference subset; new annotators qualify against it before production.
- Double annotation. A buyer-set share of clips is labeled twice, independently.
- Boundary agreement. Measure the offset between paired boundaries against a tolerance you set in frames, and segment overlap as temporal intersection-over-union.
- Label agreement. Report chance-corrected agreement on step labels, by step type.
- Adjudication and versioning. Disagreements go to a reviewer; guideline changes are versioned and back-applied to earlier batches.
Metric definitions and audit procedures are covered in measuring annotation quality in task video, and contract terms for vendors in annotation vendor requirements and acceptance criteria.
Tooling and source step lists
General-purpose tools such as CVAT support boxes, polygons, polylines and points, with interpolation across video frames (CVAT.ai). Many tools are built around spatial shapes, so confirm how a tool represents and exports time ranges before you choose it.
A business's written procedures are a natural source for the step vocabulary. Aligning SOP steps to video is a specification of its own, set out in SOP-to-video step alignment; in short, the SOP supplies names and expected order, and the annotator records what actually happened, including deviations.
Limitations
- Segmented human video is still human video: no robot actions, joint states or forces are added.
- Boundaries are partly a convention. Models trained on one rule set will disagree with datasets that used another.
- Segments derived from narration inherit the narrator's timing, which often leads or lags the action.
- Long continuous recordings cost more per useful minute to segment than curated clips.
- Low resolution, gloves and occlusion reduce how finely steps can be split.
Questions to ask annotation vendors
- Will you work from our step list, and how do you propose changes to it?
- What start and end rule will annotators apply to each step type?
- How are pauses, repeats, errors and out-of-frame stretches labeled?
- What share of clips is double-annotated, and how is boundary agreement measured?
- How are annotators qualified on the gold set, and how often are they re-checked?
- Which tool will you use, and in what format are time ranges exported?
- How do you handle source files with variable frame rates or edits?
- How are guideline changes versioned and applied to batches already delivered?
Discuss your requirements
If you hold or plan to license workplace footage and need it segmented to a defined step list, describe the footage, the label levels and the permitted uses to SourceX. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case. SourceX reviews the requirement and follows up about potential sources, and a request does not guarantee a match. Other annotation and capture topics are listed on the workplace task demonstration hub and the humanoid training data guide; the catalog page on expert annotations and labels describes labeled assets more generally.
Explore sourcing and licensing options with SourceX
Sources
- COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis — hierarchical step annotations with temporal boundaries.
- Cross-task weakly supervised learning from instructional videos — ordered step lists and selective boundary annotation.
- Towards Automatic Learning of Procedures from Web Instructional Videos — procedure segments with boundaries and descriptions.
- Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities — coarse and fine-grained segment volumes.
- Rescaling Egocentric Vision — narration-first annotation pipeline with temporal segmentation.
- IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting — step recognition with omissions and execution errors.
- CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities — step and fine-grained annotations over normal and erroneous executions.
- CVAT (Computer Vision Annotation Tool) — open-source video annotation tool.