Action Taxonomies for Workplace Video: Verb-Noun Labels That Scale

The buyer's problem

Annotation vendors can label a very large number of clips and still deliver unusable data if the vocabulary drifts. Two annotators call the same motion "place", "put" and "set down"; one labels at the task level and another at the sub-motion level. Buyers who need to train or evaluate task and step understanding, planners, or language-conditioned models need a taxonomy that stays stable across vendors, sites and months. Segment timing is a separate subject covered in step segmentation annotation; this page is about the vocabulary.

Reference vocabularies in published datasets

• EPIC-KITCHENS-100 has 100 hours, 20 million frames and 90,000 actions across 700 variable-length videos in 45 kitchens, recorded with head-mounted cameras. Its annotation began with "pause-and-talk" narration, followed by transcription, parsing and temporal segmentation, and it reports 54 percent more actions per minute than the earlier version (Damen et al., 2020). The pipeline shows a practical order: narrate, transcribe, parse into a controlled vocabulary, then segment.
• Something-Something defines 174 labels as caption templates, in which "something" stands in for the object noun; the original paper describes more than 100,000 crowdsourced videos, and the version 2 release is reported at 220,847 videos from more than 1,300 crowd actors (Goyal et al., 2017). Templates fix the verb phrase while the object varies, which separates motion type from object identity.
• HowTo100M uses automatically transcribed narration as captions (Miech et al., 2019), with no extra manual annotation. Later work by other authors reported that this narration is often misaligned with the visuals: the MIL-NCE work estimated roughly 50 percent of clip-narration pairs were not aligned, and Han et al. (CVPR 2022) found roughly 30 percent visually aligned. Free-form speech is a weak substitute for a taxonomy.
• O*NET publishes Task Statements and Detailed Work Activities, with a file mapping tasks to activities, and retains task statements only if their relevance rating is 25 percent or higher (O*NET Resource Center). It is a public occupational vocabulary that a taxonomy can map to, with a license check for the text.

Design principles

1. Fix the level. Choose one primary level (for example, "step", between whole task and low-level motion) and state it. Allow a second layer only if you will annotate it fully.
2. Closed verbs, open nouns. Keep verbs to a controlled list with definitions; let nouns come from a managed object list so "wrench", "spanner" and "torque tool" are normalized.
3. Define start and end. Each verb needs an observable start and end event, not a gloss.
4. Separate what from why. Label the visible action ("tighten bolt"), not the intent ("fix pump").
5. Allow "other" with discipline. Cap the share of "other" and review it for new verbs.
6. Version everything. Taxonomy versions are part of the dataset's metadata, and changes get migrations.
7. Map outward. Link verbs and tasks to external references such as O*NET activities, procedure step IDs, or your own task ontology.

Illustrative taxonomy extract

Values are synthetic examples for a buyer to adapt.

Illustrative label schema

# Illustrative - synthetic
taxonomy_version: wv-actions-0.3
label:
  verb: place            # closed list
  noun: wrench_open_end  # from object list v0.2
  secondary_noun: bench_surface
  hand: right            # left|right|both
  tool: null
  outcome: success       # success|fail|partial|unknown
  onet_dwa_ref: "mapped where one exists"
  procedure_step_ref: "SOP-014/step-3"
  annotator_id: ann_041
  confidence: high
agreement_targets:
  verb_exact_match_pct_min: "set after pilot"
  noun_exact_match_pct_min: "set after pilot"

Quality and acceptance

Ask the annotation team for a pilot on a sample, run double annotation on a share of clips, and report agreement per verb rather than one global figure; rare verbs may show the most disagreement. Review confusion pairs (for example "pick" versus "take") and fix definitions rather than blaming annotators. Check that nouns are normalized, that "other" stays below your cap and that every label carries a taxonomy version. For vendor-level acceptance, see annotation vendor acceptance criteria and annotation quality for task video.

Sourcing and licensing notes

A taxonomy comes from you, from public standards or from the annotation provider. Public vocabularies carry their own terms: check whether the text of standards or databases may be reused, and whether you need to attribute. Labels created by a vendor need an assignment clause. Where the work needs domain experts, see expert annotations and labels. Hand-contact labels, which sit below the verb level, are covered in hand-object interaction annotation. Related label types include language annotations for demonstrations and the demonstration clip metadata schema.

What a taxonomy cannot do

A taxonomy is a label vocabulary, not training data. It supplies no motion, force or perception signal. Verb-noun labels do not tell a robot how to move, and they cannot describe nuance such as grip force or speed. Labels created from video inherit the limits of the video: occluded hands, gloves and off-screen events make some labels uncertain, so allow an "unknown" value. Labels from narration may disagree with the visuals, as the HowTo100M alignment estimates show. A taxonomy tuned to one site may not fit another, and a taxonomy that is too fine may not reach acceptable agreement.

Questions to ask suppliers

1. Who authored the taxonomy, and can we see the definitions and start and end rules?
2. At what level of granularity are labels applied, and is there a second layer?
3. How are nouns normalized across sites and annotators?
4. What was inter-annotator agreement on a pilot, reported per verb?
5. How do you handle occluded or ambiguous actions: unknown label, or guess?
6. Is the taxonomy versioned, and how are changes migrated into existing labels?
7. Can labels map to our procedure step IDs or an external vocabulary such as O*NET?
8. What share of labels is "other", and how is it reviewed?
9. Who owns the labels and any taxonomy text we pay to develop?
10. Can annotators work from video only, or do they get SOPs and expert guidance?

Discuss your requirements

If you need labeled workplace video with a stable vocabulary, describe the label scheme and tasks to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates license and delivery; it can also arrange annotation, scoped case by case. A request does not guarantee a match. Start at the humanoid training data guide or the workplace task demonstration hub.

Discuss your humanoid training-data requirements with SourceX

Sources

• Rescaling Egocentric Vision — EPIC-KITCHENS-100 narration-first annotation and verb-noun style action labels.
• The "something something" video database for learning and evaluating visual common sense — Something-Something caption-template labels with 'something' in place of the object.
• ONET Database (ONET 29.1 per search snapshot): Task Statements and Detailed Work Activities — O*NET task statements and Detailed Work Activities as a public task vocabulary.
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips — HowTo100M narration captions are ASR and often misaligned.