Short answer
Hand-object interaction annotation records, at key frames or every frame, where each hand is, which object it touches, when contact starts and ends, the grasp type, and which object is the active tool or target. A specification should fix hand and object classes, the contact rule, a grasp vocabulary, the sampling rate and how occlusion and gloves are marked, then test agreement on a gold set before scaling.
What these labels are for
Hand-object interaction (HOI) labels teach a perception model to find hands, tell left from right, detect contact, pick out the object being manipulated, recognize the grasp and relate a tool to its target. Robotics teams use them to train and evaluate those perception components, to build object-centric representations, and as an input to hand-pose and retargeting pipelines when combined with 3D data.
This page covers spatial labels. When steps begin and end is a temporal question, specified in step segmentation and task boundary annotation. Masks and tracks for every object in the scene are covered in object tracks and segmentation masks, and metric 3D hands in 3D hand pose data from workplace recordings.
Label layers
| Layer | Geometry | Typical sampling | What it supports |
|---|---|---|---|
| Hand box and side | One box per hand, left or right | Key frames plus interpolation | Hand detection and tracking |
| 2D hand keypoints | Wrist and finger joints | Sampled frames | Pose priors, grasp recognition |
| Contact state | Per hand: free, touching, holding, unknown | Every transition | Contact prediction, grasp phases |
| Active object | Box or mask of the object in hand | Transitions plus key frames | Active-object detection |
| Object identity | Class or instance ID from a fixed list | Per active object | Links to product and part data |
| Grasp type | Class from a fixed vocabulary | Once per grasp | Grasp recognition, pre-shape priors |
| Tool-target relation | Tool ID, target ID, verb | Per tool use | Tool-use understanding |
| Visibility and PPE | Visible, partial, hidden; glove type | Per segment | Filtering and robustness analysis |
Choose the layers your models consume. Keypoints and masks cost far more per frame than boxes and contact states.
Published precedents
MECCANO, recorded in an industrial-like assembly setting, labels temporal action segments and active-object bounding boxes and defines an egocentric human-object interaction detection task (Ragusa et al., 2020). HA-ViD provides 3,222 multi-view assembly videos with 96K temporal labels and 2M spatial labels (Zheng et al., 2023), a reminder that spatial labels can far outnumber temporal ones. IKEA ASM adds object segmentation and human pose to furniture-assembly video (Ben-Shabat et al., 2020), and Assembly101 includes 18M 3D hand poses alongside its action segments (Sener et al., 2022).
When 2D labels are not enough, 3D ground truth comes from instrumented capture. HOT3D obtained 3D poses of hands and objects from optical-marker motion capture and released hand annotations in UmeTrack and MANO formats (Banerjee et al., 2024). Third-person HOI images are also useful on their own: affordance grounding has been learned from exocentric interaction images and transferred to egocentric views of objects (Luo et al., 2022).
The contact rule
Write contact down as a visible test, not an intuition:
- Touching — the hand visibly meets the object, but the object does not move with it.
- Holding — the object moves rigidly with the hand, or is clearly enclosed by it.
- Free — no contact with any annotated object.
- Unknown — the contact region is hidden. Annotators must use this rather than guess.
Label the frame of each transition, and set a tolerance in frames for agreement checks. For tools, record two relations: hand holds tool, and tool contacts target.
The grasp vocabulary
Start from a published classification such as the GRASP taxonomy, which merges earlier taxonomies into 33 static one-handed grasp types, reducible to 17 general grasps if object shape and size are ignored (Feix et al., 2016). Then collapse it to the classes annotators can reliably tell apart at your resolution and that matter to your robot hand.
An illustrative reduced workplace set might contain power wrap, precision pinch, lateral pinch, tripod, hook, flat-palm push and tool grip. Fine distinctions that collapse in low-resolution or gloved footage only add noise. Report a confusion matrix from the gold set before freezing the list.
Illustrative annotation records
{
"_comment": "Illustrative only — synthetic IDs and values",
"clip_id": "clip_site024_0217",
"frame": 3810,
"hands": [
{"side": "right", "box": [612, 404, 781, 560], "visibility": "visible",
"glove": "nitrile", "contact": "holding", "active_object": "obj_0031",
"grasp_type": "precision_pinch"},
{"side": "left", "box": [402, 455, 540, 590], "visibility": "partial",
"glove": "nitrile", "contact": "touching", "active_object": "obj_0007",
"grasp_type": null}
],
"objects": [
{"object_id": "obj_0031", "class": "connector_housing", "box": [690, 470, 742, 515]},
{"object_id": "obj_0007", "class": "wire_harness", "box": [380, 500, 610, 640]}
],
"events": [{"type": "contact_start", "hand": "right", "object": "obj_0031", "frame": 3794}],
"annotator_id": "ann_12",
"guideline_version": "hoi_guide_v1.1"
}
Sampling strategy
Annotate key frames at a fixed interval plus every contact transition, and interpolate hand and object boxes between them; CVAT, for example, supports interpolation for video annotation (CVAT.ai). Interpolation drifts during fast motion, so require human review of interpolated stretches around grasps and releases.
Model pre-labeling is reasonable if it is disclosed. Record which frames were pre-labeled and measure how often reviewers correct them; a low correction rate on difficult frames is a warning sign, not a success.
Gloves, tools and occlusion
Workplace hands are often gloved, which changes appearance and hides finger joints; mark glove type on every hand, and accept that keypoints may be impossible on thick gloves. The effects on tracking and transfer are covered in gloved hands and PPE in workplace video. Mark the other hand, the object and the worker's body as occluders where they hide contact, so that "unknown" contact can be traced to a cause.
Quality and acceptance
| Check | Example pass criterion (buyer sets) |
|---|---|
| Hand box accuracy | Mean IoU against the gold set above threshold |
| Side assignment | Left/right accuracy above threshold |
| Contact timing | Transitions within a set number of frames of gold |
| Grasp-type agreement | Chance-corrected agreement per class above threshold |
| Active-object identity | Correct object ID on sampled contacts |
| Use of "unknown" | Rate consistent with measured occlusion, not used as a shortcut |
Audit procedures and metric definitions for video labels are covered in measuring annotation quality in task video.
Limitations
- 2D boxes and keypoints carry no depth; contact is inferred from appearance, and no force is measured.
- Human grasp types do not map one-to-one onto robot hands with fewer degrees of freedom.
- Existing footage often lacks the resolution for reliable finger keypoints.
- Labels describe what an object looks like when handled, not how it behaves; the distinction is explained in object recognition vs manipulation data.
- Detailed hand data can raise biometric-privacy questions in some jurisdictions; review them with counsel. This is not legal advice.
Questions to ask annotation vendors
- Which layers will you label, and at what sampling rate for each?
- What contact rule will annotators apply, and how is hidden contact recorded?
- Which grasp vocabulary do you propose, and what confusion did the gold set show between classes?
- Do you pre-label with models, and how are corrections tracked and reported?
- How do you handle gloved hands, tools held in the hand and two-handed grasps?
- How are object identities linked to our object list or the supplier's catalog?
- What share of frames is double-annotated, and what agreement did you reach on the last comparable project?
- Which export formats do you support for boxes, masks, keypoints and events?
Discuss your requirements
If you need HOI labels on footage you plan to license or commission, tell SourceX which layers, objects and permitted uses you need. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case. SourceX reviews the requirement and follows up about potential sources, and a request does not guarantee a match. Related capture and labeling topics are on the workplace task demonstration hub and the humanoid training data guide, and the catalog page on expert annotations and labels describes labeled business assets.
Explore sourcing and licensing options with SourceX
Sources
- The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain — active-object boxes and egocentric HOI detection in an industrial-like setting.
- HA-ViD: A Human Assembly Video Dataset for Comprehensive Assembly Knowledge Understanding — relative volume of spatial and temporal labels.
- The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose — object segmentation and pose labels in assembly video.
- Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities — 3D hand poses alongside action segments.
- HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos — 3D hand and object ground truth from motion capture.
- Learning Affordance Grounding from Exocentric Images — affordance learning from exocentric interaction images.
- CVAT (Computer Vision Annotation Tool) — video annotation with interpolation.
- The GRASP Taxonomy of Human Grasp Types — a published grasp classification (33 types, 17 general grasps) used as a starting vocabulary.