Short answer
Audit task-video annotations with four instruments: a guideline with worked edge cases, a gold set with pre-decided answers, double annotation on a sample to measure agreement, and a stratified audit of delivered labels. Temporal labels need boundary tolerance and spatial labels need overlap measures. Set pass thresholds before delivery, and request evidence of annotator training and rework rates, not only a headline accuracy.
The buyer's problem
A vendor reports "98% accuracy". Accuracy against what, measured on which sample, by whom? For task video that figure is nearly meaningless without the definition. Video labels are harder to check than image labels: a step boundary can be right at the second but wrong at the frame, a track can be perfect for ten seconds and swap identity at the eleventh, and a caption can be fluent and wrong.
This page is a checklist for judging labels you are about to accept or already hold. It covers audit design and metrics; for the labels themselves see step segmentation, hand-object interaction annotation and object tracks and masks. Delivery-level checks for the video itself are in demonstration video acceptance criteria.
How published datasets organize annotation
Public datasets show that annotation is a pipeline, not a single pass. EPIC-KITCHENS-100 began with "pause-and-talk" narration, where the recorder pauses the video and describes the action, then transcribed, parsed and temporally segmented the narrations, covering 100 hours, 20 million frames and 90,000 actions (Damen et al., 2020). COIN organizes 11,827 videos across 180 tasks and 12 domains with hierarchical structure and step-level descriptions with temporal boundaries (Tang et al., 2019).
Each stage has its own error: narrators misdescribe, parsers normalize wrongly, and segmenters place boundaries differently. A vendor should be able to say which stage produced each field and how each is checked. Documenting this is what dataset documentation formats such as Data Cards aim at, and they are a reasonable template for a vendor's quality report (Pushkarna et al., 2022).
What annotation QA can and cannot establish
It can establish that labels are consistent with a written guideline, reproducible across annotators and stable over a sample. It cannot establish that the guideline was right for your model, that the labels reflect robot-relevant distinctions or that the underlying footage is representative. Agreement among annotators can also be high and wrong, when everyone follows an ambiguous rule the same way. Use the gold set, written by your own experts, to detect that.
Checklist
| Check | Pass criterion (you set the number) | Evidence to request |
|---|---|---|
| Guideline quality | Written, versioned, with edge-case images/clips for every class | The guideline and change log |
| Annotator training | Annotators passed a qualification on gold clips before production | Qualification results, retraining records |
| Gold set | Pre-decided answers from your experts, hidden in production batches | Per-annotator accuracy on gold |
| Inter-annotator agreement | Agreement by label type above your threshold on double-labeled sample | Raw double labels, not only a summary |
| Temporal boundary error | Median and 90th percentile boundary offset within a tolerance | Offsets in frames or ms per label class |
| Spatial overlap | Intersection-over-union by class above threshold | Per-class distribution |
| Track continuity | ID switches per object per minute below threshold | Audit list with clip IDs |
| Audit sampling | Random and risk-targeted sample, stratified by site and class | Sampling plan with seed |
| Rework rate | Share of labels corrected after review, tracked per batch | Rework logs |
| Schema validity | Files parse, enumerations valid, timestamps inside clip duration | Validator output |
Metrics by label type
Temporal segments (steps, actions). Report boundary error as a distribution, and use a tolerance in frames matched to the frame rate and action speed. Count merged and split segments. Compare label sequences, since an extra step is a different error from a shifted one.
Spatial (boxes, masks, keypoints). Use intersection-over-union for masks and boxes, a normalized distance for keypoints, and report by class and by size because small objects behave differently.
Tracks. Count identity switches and fragmentation; check IDs across occlusion.
Text (captions, narration). Use human review on a sample for accuracy, specificity and verb choice; automatic text metrics hide factual errors. If the text came from speech recognition, treat it as machine output. Captions in HowTo100M are automatically transcribed narrations with no manual annotation (Miech et al., 2019), and follow-up work reports that many clip-narration pairs are not visually aligned; see language annotations for demonstrations.
Categorical outcomes. Report agreement with a chance-corrected statistic such as Cohen's kappa, per class, because a majority class inflates raw agreement.
Red flags
- Quality reported only as a single percentage with no sample size, definition or class breakdown.
- No gold set, or gold clips visible to annotators as such.
- Agreement computed between a model's pre-label and the human who corrected it; this measures automation bias, not independence.
- Boundary tolerance wide enough to hide errors, such as one second on a two-second action.
- Rework applied silently, with no change log.
- Class definitions that changed mid-project without relabeling earlier clips.
- Annotators who saw the other annotator's labels before labeling.
- No handling of "can't tell", so every clip has a forced label.
Illustrative audit plan
# Illustrative only - sample sizes and thresholds are examples a buyer would set
audit_plan: video_labels_v0
gold_set: 150 clips, labeled by buyer experts, 5% injected per batch
double_annotation: 10% of each batch, independent annotators
random_audit: 3% of delivered clips, reviewed by buyer or third party
targeted_audit: all clips with low annotator confidence or flagged "undecidable"
metrics:
step_boundary_offset_ms: {median, p90} per step class
mask_iou: per class
id_switches_per_min: per clip
caption_factual_error_rate: from 100-caption review
accept_if:
batch_pass: all metric thresholds met on audit sample
otherwise: vendor reworks and resubmits; no partial acceptance
reporting: per batch, per annotator, per site
Sampling plans and sample sizes
Audit samples should be large enough to estimate the rates you care about and chosen so that rare classes are not missed. Stratify by site, camera, label class and annotator. Add a targeted sample of the hardest cases: low-confidence labels, clips with occlusion, and boundaries near other boundaries. Report error rates with an indication of uncertainty rather than a single point. If a rare class has too few audit items to measure, say so, and review every instance of it. Repeat the audit on each batch, because quality drifts when annotators change or deadlines tighten, and a good first batch does not predict the fifth. Agree in the contract what happens on failure: rework at the vendor's cost, re-audit and no partial payment for failed batches are common terms to discuss with your own counsel.
Questions to ask suppliers
- Who wrote the guideline, and was it tested on a pilot batch?
- How are annotators qualified and monitored?
- Which part of each label is machine-generated and which is human-confirmed?
- Can I see raw double-annotation data?
- What was the disagreement on the hardest classes?
- How often has the guideline changed, and were earlier labels updated?
- What happens to clips the annotators mark unclear?
- How are annotators handling footage with people in it, and what access controls apply? See worker consent and identifiable people.
For vendor-level terms, use annotation vendor acceptance criteria.
Discuss your requirements
If you are specifying or auditing labels for workplace video, describe the label types, quality thresholds and intended use to SourceX. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case. SourceX reviews the requirement and follows up about potential sources, and a request does not guarantee a match. Explore the humanoid training data guide or the workplace task demonstration hub for the surrounding decisions.
Explore sourcing and licensing options with SourceX
Sources
- Rescaling Egocentric Vision — staged narration-to-segmentation annotation pipeline.
- COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis — hierarchical step annotation with temporal boundaries.
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips — ASR captions with no manual annotation; alignment caveat.
- Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI — structured dataset documentation.