Evaluation Sets for Task Understanding From Workplace Video

The buyer's problem

You have a video-language model or a task-progress estimator, and public benchmarks say little about your tasks. Kitchens and toy assembly are not packing stations, and accuracy on one does not tell you how a model will behave at a new site. You need a test set from the environments you care about, labeled to your definitions, and trustworthy enough to settle arguments.

This page is about sourcing evaluation video for models that understand what is happening in a task. Tests of robot policies (success rates on physical hardware) are a different matter, covered in real-world policy evaluation protocols.

What published benchmarks test

Existing benchmarks give a vocabulary for what to ask:

• Goal and step understanding. Ego4D Goal-Step adds hierarchical goal and procedural-step annotations to Ego4D with a data-driven taxonomy; the NeurIPS abstract reports 48,000 procedural step segments (430 hours) and goal annotations for 2,807 hours, with tasks including goal inference and step prediction (Song et al., 2023).
• Step localization in instructional video. COIN has 11,827 videos across 180 tasks and 12 domains with step-level descriptions and temporal boundaries (Tang et al., 2019).
• Skill and proficiency. Ego-Exo4D includes fine-grained activity understanding and proficiency estimation among its benchmarks, with synchronized egocentric and exocentric views (Grauman et al., 2023).
• Mistake detection. Assembly101 records unscripted assembly with mistakes and defines a mistake-detection task with more than 100,000 coarse and one million fine-grained action segments (Sener et al., 2022).

These show the types of labels and splits to request. They do not tell you what accuracy is acceptable for your use.

What an evaluation set can and cannot show

It can show whether a model recognizes steps, orders them, anticipates the next step, detects omitted or wrong steps, and generalizes across site, camera and operator. It can expose bias, such as a model that only works with one camera angle.

It cannot show that a robot can perform the task. Understanding a task from video is a precondition for some pipelines, not control. A model that scores well on step recognition may still lack any representation of force, contact or reachability. Evaluation sets also cannot be reused indefinitely: once a model's developers tune against them they stop being held-out, and a leaked set no longer evaluates. See training vs evaluation datasets.

Evaluation targets and the data each needs

Sourcing routes

Existing footage from operating businesses. Training libraries, process recordings and quality-audit video can be re-labeled. It has natural variety but was not designed as an evaluation set, so viewpoints are unplanned and consent is not designed for training; see existing footage vs commissioned recording.

Commissioned recording. Designed coverage: chosen sites, tasks, variants and error cases, with consent built in. It costs time and money and still needs site access.

Annotation of either. Experts relabel with your rubric. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case.

Whichever route you use, source evaluation footage from different sites or periods than training footage. A set drawn from the same recordings as training data tests memorization of the site.

Illustrative package

# Illustrative only - values are examples a buyer would set, not an available dataset
eval_set: task_understanding_eval_v0
purpose: step recognition, next-step anticipation, mistake detection (non-robot)
tasks: 10 workplace tasks; each with 3 allowed variants
footage:
  sites: 6 held-out sites (site_021 .. site_026) never used in training
  operators: at least 8 per task, none in training
  cameras: 1 egocentric + 1 fixed per session where possible
  per_task_clips: reported, not fixed by this spec
labels:
  step_taxonomy: 12-25 steps per task, definitions with example frames
  boundaries: dual-annotated, tolerance reported in frames
  mistakes: type, onset, whether recovered; natural vs instructed flagged
  goals: one task label per clip, plus optional free-text
  verification: expert adjudication of disagreements; adjudicator ID recorded
governance:
  access: held-out; test labels not released to model developers
  refresh: new subset each evaluation cycle; retired subsets moved to training only after sign-off
rights: evaluation use; no training; no redistribution

Acceptance criteria

1. Separation. No site, operator or recording session appears in both evaluation and training; the supplier provides a manifest to verify it.
2. Label reliability. Agreement on double-annotated samples is reported by label type, with a gold subset adjudicated by experts. See annotation quality for task video.
3. Coverage report. Counts by task, variant, site, operator, camera and error type, including rare classes.
4. Rubric versioning. Taxonomy and step definitions are frozen and versioned for the evaluation cycle.
5. Difficulty spread. Includes normal, ambiguous and hard clips, not only the cleanest.
6. Leak check. Confirm clips are not public footage or from sources a model may have seen. Public instructional video may already appear in web-scale pretraining sets.

Keeping evaluation sets useful over time

An evaluation set degrades with use. Teams look at errors, adjust prompts and training data, and gradually fit the set. Plan for refresh from the beginning: reserve additional held-out sites or periods, keep a locked core for regression tracking and a rotating subset for new claims. Record which model versions were evaluated and when. Keep annotation of the evaluation set separate from annotation of training data, with different annotators where feasible, so shared habits do not inflate scores. If you intend to publish results, check that the license permits public reporting of metrics and any examples. Distinguish clearly between a set that measures task understanding and one that measures robot behavior; reporting the former as evidence of the latter misleads readers.

Questions to ask suppliers

1. Can you guarantee no overlap of site, operator and session with other datasets you supply? If not, which overlaps exist?
2. Who wrote the step taxonomy, and has it been tested on a pilot?
3. Were mistakes natural or instructed, and are they labeled as such?
4. Is footage new, or has it been used for other training or published?
5. How were labels verified, and by whom?
6. How many clips exist per task variant and per site?
7. What access controls keep evaluation labels from model developers?
8. What permissions cover use of the footage for evaluation only, and for re-evaluation over time?

Discuss your requirements

If you need held-out workplace video for evaluation, tell SourceX what your evaluation set needs to cover. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match. For supporting choices see demonstration dataset splits, step segmentation annotation and workplace video for task planning, or start from the humanoid training data guide and the workplace task demonstration hub. Mistake labels are covered further in human procedural mistake videos.

Tell SourceX what your evaluation set needs to cover

Sources

• Ego4D Goal-Step: Toward Hierarchical Understanding of Procedural Activities — goal and step benchmarks.
• COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis — step-level instructional video benchmark.
• Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives — proficiency and fine-grained activity benchmarks.
• Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities — mistake detection in unscripted procedural video.