Short answer
Granularity is the level at which you cut a task: goal, procedure step, or atomic action. Steps suit planning and language grounding; atomic actions suit manipulation and hand-object analysis. Hierarchical datasets annotate several levels at once. Choose the lowest level your downstream use needs, keep the levels linked, and write definitions before annotators start.
Why granularity is a purchasing decision
Labels are expensive and hard to redo. A supplier asked for "step annotations" can return clips of three seconds or three minutes, and both are legitimate readings. If the level does not fit your use, you have bought video or documents with labels that cannot be reused for your model without re-annotation.
Granularity also decides what gets compared. A planner that predicts the next procedure step needs steps. A model that studies hand-object contact needs atomic actions. Evaluations that mix levels produce numbers that cannot be compared with each other.
The levels
| Level | Example (synthetic) | Typical duration | Typical use |
|---|---|---|---|
| Goal / task | "Restock a shelf" | Minutes | Task recognition, language conditioning |
| Procedure step | "Remove old stock" | Tens of seconds to minutes | Planning, progress tracking, SOP alignment |
| Atomic action | "Pick up carton" | About a second to a few seconds | Manipulation analysis, action recognition |
| Primitive / motion | "Close gripper" | Sub-second | Control; not available from labels alone |
Durations are indicative only; each project must define them. The lowest level in the table is robot-specific. Human video can show a person grasping, but it does not provide joint commands, and that gap is covered in the human-to-humanoid embodiment gap.
What published datasets annotated
Ego4D Goal-Step adds goal and procedural-step annotations to Ego4D with a data-driven taxonomy. The NeurIPS abstract reports 48,000 procedural step segments covering 430 hours, and goal annotations for 2,807 hours of video; tasks include goal inference and step prediction (Song et al., 2023). Goals were annotated much more widely than steps, which is consistent with fine segments taking more annotation effort than a label per video.
EPIC-KITCHENS-100 works at the action level. It has 100 hours of head-mounted footage, 20 million frames and 90,000 actions across 45 kitchens. Annotators first narrated what they did in a "pause-and-talk" pass, and the narration was then transcribed, parsed and temporally segmented (Damen et al., 2020). That workflow, narration first and timing second, is one way to get atomic labels more cheaply.
FineBio annotates a hierarchy for mock biological experiments: protocols, steps, atomic operations, object locations and manipulation states. In its data the step order is strict while the order of atomic operations is flexible, and the set covers 14.5 hours of multi-view video from 32 participants (Yagi et al., 2024). The strict-versus-flexible distinction is the practical point: steps constrain the plan, and atomic actions vary by person.
Granularity and the data source
The right level also depends on where the labels come from. Labels derived from written procedures arrive at step level, because that is how documents are cut, and they carry the author's vocabulary. Labels derived from video by annotators can reach atomic level, but only when hands and objects are visible, and they carry the annotators' judgment. Labels from narration, as in the pause-and-talk workflow, inherit the narrator's habits: some people describe every motion and some describe only goals.
Mixing sources is common, and it is where inconsistency creeps in. If procedure steps and video segments are to be aligned, agree on a boundary rule once and apply it to both; see task graphs from procedures for how steps become a graph whose nodes need stable definitions.
Choosing a level
- Predicting or planning steps from video or text. Procedure steps, with a link to the goal. See step segmentation annotation.
- Aligning a written procedure to recordings. Steps, because that is the level a document uses. See SOP-to-video step alignment.
- Hand-object interaction or action recognition. Atomic actions with object labels; see the action taxonomy for workplace video.
- Evaluating long tasks. Both: steps for coverage, goals for success.
- Pretraining on large unlabeled video. Coarse goals may be enough, with fine labels on a subset.
When unsure, buy two levels on a small subset, check which one your model benefits from, then extend the winner. Hierarchical task analysis is a standard way to define sub-goals before you label; see hierarchical task analyses.
Illustrative granularity specification
# Illustrative only - names and thresholds are placeholders
annotation_spec: granularity_v0
levels:
goal:
definition: "one named outcome from the task taxonomy"
cardinality: 1 per clip
step:
definition: "a segment that changes task state, named from the procedure"
source_vocabulary: procedure_step_ids
order: constrained by procedure; deviations flagged
boundary_rule: "starts when hand first contacts target; ends at release"
atomic_action:
definition: "verb + object, no further decomposition"
vocabulary: closed list of 40 verbs, extensible with review
order: free
links:
atomic_action.parent: step_id
step.parent: goal_id
quality:
boundary_tolerance: "set per project, e.g. within 0.5 s of consensus"
double_annotation_share: "set by buyer, e.g. 10% of clips"
adjudication: required on disagreement
The decisive lines are definition, boundary_rule and the links between levels. A step whose boundaries are not defined will be cut differently by each annotator.
Limitations
Labels at any level describe what happened. They do not add robot actions, forces or contact information, and they cannot make human video into control data. Finer labels also do not mean better data: atomic annotation of clips with occluded or gloved hands will contain guesses, and it typically costs more per hour than step labels.
Granularity does not travel well across sites. A step at one business may be two at another, and a vocabulary built for one task may not cover others. Hierarchical labels can also hide disagreement: annotators who agree on steps may differ widely on the atomic actions inside them. Report agreement per level.
Finally, changing granularity later is not free. Coarser labels can be derived from finer ones if the hierarchy is linked; the reverse requires new annotation.
Questions to ask suppliers
- Which levels of annotation do you provide, and how are they defined in writing?
- Is the vocabulary closed or open, and who maintains it?
- How do you set segment boundaries, and what is the observed agreement between annotators?
- Are levels linked, so each atomic action points to its step and goal?
- Do labels come from narration, written procedures, or annotators watching alone?
- Can you provide a small sample at two granularities for us to compare?
- How are mistakes, retries and out-of-order actions labeled?
- What happens to labels when the video has occluded or gloved hands?
- What is the process and cost, in terms you can describe, for adding a finer level later?
Describe your annotation requirements
If you know the level of labels your model needs and want to find out whether suitable material exists, describe the tasks, modalities and annotation levels to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery; it can also arrange annotation, scoped case by case. A request does not guarantee a match; SourceX follows up about potential sources. Start from the humanoid training data guide or the procedures hub, and see the procedure step schema and the site's page on expert annotations and labels.
Describe the workplace demonstration data your robotics project needs
Sources
- Ego4D Goal-Step: Toward Hierarchical Understanding of Procedural Activities — goal and step annotations on egocentric video.
- Rescaling Egocentric Vision — action-level annotation with narration-first workflow.
- FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation — protocol, step and atomic-operation hierarchy.