Short answer
Existing footage can usually gain step boundaries, captions, outcome flags, object masks and 2D hand labels through annotation, but annotation cannot add what the camera never captured: occluded hands, missing viewpoints, depth, robot actions or contact forces. Test feasibility by labeling a small sample first, and confirm the footage owner's license covers derivative labels before you commission any work.
The buyer's problem
A business has years of recordings of people doing work, such as training sessions, process cameras or quality reviews. A buyer wants to know whether a labeling project can turn that archive into usable demonstration data, and what it will cost in rejected clips along the way. Annotation vendors will quote on hours of video; they rarely quote on how much of the video can carry the labels you need.
The core point is that labels are a function of the pixels. A step boundary can be drawn on any clip where the action is visible. A 3D hand pose cannot be recovered reliably where the hand is behind a part. For the broader choice between archives and new recording, see existing workplace footage vs commissioned demonstration recording.
What annotation can and cannot add
| Label type | Usually feasible on archive footage | Where it breaks down |
|---|---|---|
| Task and step segmentation | Yes, when actions are visible and the procedure is known | Interleaved steps, cuts that remove transitions |
| Captions and instructions | Yes; narration or procedure text speeds it up | Narration that does not match the visuals |
| Outcome and error flags | Yes, if the result is visible | Outcomes judged off-camera or later |
| Object masks and tracks | Yes for large, distinct objects | Small, translucent or fully occluded parts |
| 2D hand boxes and keypoints | Often | Gloves, motion blur, hands leaving frame |
| 3D hand or body pose | Estimated, with uncertain accuracy | Single fixed view, depth ambiguity |
| Contact and force | No | Not visible in RGB video |
| Robot actions or states | No | Do not exist in human footage |
Published datasets show what annotation effort looks like when the footage is designed for it. EPIC-KITCHENS-100 used head-mounted cameras and a "pause-and-talk" narration pass as its first annotation stage, followed by transcription, parsing and temporal segmentation (Damen et al., 2020). COIN provides step-level descriptions with temporal boundaries across 11,827 videos and 180 tasks (Tang et al., 2019). Those are step and action labels on video, the most dependable kind of retrofit.
Dense 3D labels usually come from the capture setup, not the labeling queue. EgoDex paired 829 hours of egocentric video with 3D hand and finger tracking recorded by the Apple Vision Pro that captured it (Hoque et al., 2025). Assembly101 used 8 static and 4 egocentric cameras and reports 18M 3D hand poses (Sener et al., 2022). An archive recorded from one ceiling camera has neither.
Feasibility checklist
Run these checks on a random sample before signing a statement of work. Thresholds are examples; set your own from the label type and model.
| Check | Example pass criterion | Evidence to request |
|---|---|---|
| Visibility of the target | Hands and manipulated object visible in at least 80% of sampled frames | 30 clips drawn by you, not curated by the vendor |
| Viewpoint stability | Camera position changes rarely within a clip | Camera type, mount and any relocation log |
| Frame rate and compression | Constant rate; no heavy compression artifacts on hands | ffprobe output per file |
| Label agreement | Two annotators agree on step boundaries within a tolerance you set | Double-labeled gold set with agreement report |
| Guideline fit | Labels map to your action taxonomy | Written guidelines with edge-case rulings |
| Throughput reality | Reject rate stated per label type | Pilot report with clips excluded and why |
| Rights to derive | Footage license permits derivative labels | License text, not a summary |
Hand labels deserve their own check. If you want 2D keypoints or hand-object contact, review the guidelines in hand-object interaction annotation against a handful of your own clips, since gloves and occlusion change what annotators can mark.
A gold set matters because it separates annotator disagreement from footage ambiguity. If trained annotators disagree on where a step ends, the footage or the taxonomy is the problem, and more annotators will not fix it. See annotation quality for task video and step segmentation annotation for the quality methods.
Illustrative annotation uplift scope
# Illustrative only - synthetic values for a pilot statement of work
pilot:
source: archive_A (400 clips; 3 sites; ceiling cameras plus some handheld)
sample: 30 clips drawn at random by the buyer
labels:
step_segments: start/end per procedure step, tolerance set by buyer
caption: one instruction sentence per step
outcome: completed | corrected | abandoned | not_visible
hand_visibility: per-second flag (left, right, both, none)
object_masks: key tools and containers only, every 5th frame
not_requested: 3D hand pose, contact, force, robot actions
qa:
gold_set: 10 clips double-labeled, agreement report required
acceptance: reject clips where hands are hidden for more than half the step
deliverables: JSONL labels, guideline document, rejected-clip list with reasons
Red flags
- The vendor quotes a flat price per hour without sampling the footage.
- Labels are promised that need views the archive does not contain, such as 3D hand pose from one overhead camera.
- No double labeling, or agreement reported only as a single accuracy number.
- Guidelines were written by the vendor without your taxonomy.
- Rejected clips are quietly dropped, which biases the labeled set toward easy, well-lit work.
- The annotation contract is silent on who owns the labels and what happens to the footage copy.
Rights and provenance
Annotation vendors work on a copy of the footage under the owner's permission. They can assign you the labels they create, typically as work-for-hire, but they hold no rights in the underlying recordings. The footage owner still has to be able to license the video, and the people in it must have been recorded under a notice that fits this use. Ask who produced each recording, whether any is vendor-made or third-party content, and what the vendor may do with the copy. Redaction does not settle these questions; see redaction in demonstration video for what it changes about training value, and worker consent and identifiable people for the consent questions. This is not legal advice.
Document what you end up with. A datasheet written in the style of Datasheets for Datasets, covering collection process, composition and recommended uses, records which labels are human, which are model-generated and which clips were excluded.
Questions to ask annotation providers
- Can you label a sample I draw before we agree on scope?
- How do you handle clips where the target hand or object is hidden?
- What agreement did you measure on comparable step or mask labels?
- Who writes and versions the guidelines, and how are edge cases ruled on?
- Are any labels produced by a model first and corrected by people? Which ones?
- What share of the sample did you exclude, and by what rule?
- Where is the footage stored during labeling, and who can access it?
- What do you retain after delivery?
Discuss your requirements
If you hold or are evaluating a footage archive, describe the labels and downstream use you need to SourceX. SourceX reviews the requirement and follows up about potential sources; annotation scope is considered case by case, and a request does not guarantee a match. The demonstration hub and the humanoid training data guide map the surrounding decisions. The catalog page on expert annotations and labels describes how labeled material is typically specified.
Describe the workplace demonstration data your robotics project needs
Sources
- Rescaling Egocentric Vision — EPIC-KITCHENS-100 narration-first annotation pipeline and kitchen scale.
- COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis — COIN step-level descriptions with temporal boundaries.
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video — EgoDex pairs 829 hours of egocentric video with device-recorded 3D hand tracking.
- Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities — Assembly101 multi-camera capture and 3D hand poses.
- Datasheets for Datasets — Datasheets for datasets as documentation practice.