Tool Affordances and Grasp Annotations

What this specification is for

Knowing that an image contains a "screwdriver" is not enough to use it. A manipulation system needs to know which part is held, which part acts on the work, which part is dangerous and where a trigger, switch or guard sits. Those are functional labels, and they rarely exist in image collections built for object recognition. Annotation vendors can add them to images or video you already hold or have licensed.

This page is a template you adapt. It assumes the underlying images or recordings come from elsewhere, such as the sources on tool identification image data or hand tool use demonstrations. Annotation is the work-for-hire layer; the rights in the pictures remain a separate question. See the maintenance hub and the humanoid data sourcing guide for the wider cluster.

What published datasets annotate

Three published works show different layers of the same idea.

HANDAL contains 308k annotated image frames from 2.2k videos of 212 real-world objects in 17 categories, focused on robotics-ready manipulable hardware and kitchen tools such as pliers, utensils and screwdrivers. Annotations include 6-DoF category-level pose and scale, handle affordance segmentation and 3D reconstructed meshes, gathered from a single off-the-shelf camera with semi-automated processing (Guo et al., 2023).

TACO contains 2.5K bimanual motion sequences across tool-action-object combinations with 12 third-person views plus one egocentric view, hand-object 3D meshes, 2D segmentation and behavior triplet annotations (Liu et al., 2024). It links what the tool is, what is done and what it is done to.

AGD20K, introduced with work on affordance grounding from exocentric images, has over 20K images across 36 affordance categories, and transfers affordance knowledge from human-object interaction images to egocentric object images (Luo et al., 2022).

A buyer can mix these layers: segmentation of functional parts, pose of the tool, and the action it affords. Which layers you need depends on the model and robot.

What annotation can and cannot do

Annotation can add functional part masks, 3D regions, poses, grasp candidates, tool-action-object triplets, captions and quality flags to images, scans or video. It can make a recognition dataset useful for affordance learning and grasp planning.

It cannot create the underlying data, supply rights to it, or provide robot-specific information. A human-drawn grasp label is not a validated robot grasp: it does not encode the gripper's geometry, friction or the object's mass. Contact forces and motion come from demonstrations or robot interaction, covered in object recognition versus manipulation data. Image-level affordance labels also do not tell how hard to grip a trigger or how a tool behaves in motion.

Illustrative annotation specification

# Illustrative only - fields and values are examples to adapt
spec_version: 0.1
scope:
  tool_classes: [screwdriver, pliers, wrench_open_end, hammer, cordless_drill, utility_knife]
  input_assets: images (RGB, optional depth) and short video clips supplied by buyer
object_level:
  instance_id: required
  class: from controlled list
  mask: instance segmentation
  pose: category-level 6DoF + scale (where depth or multi-view available)
  mesh_ref: optional, buyer-supplied scan id
functional_parts:       # one polygon or mask per visible part
  - handle
  - grip_zone            # subset of handle intended for hand placement
  - working_end          # tip, jaws, head, blade
  - trigger_or_switch
  - guard_or_shield
  - hazard_edge
  - attachment_point     # bit chuck, socket drive, blade release
part_attributes:
  visibility: visible | partly_occluded | not_visible
  material_hint: metal | rubber | plastic | wood
affordances:             # many-to-many with parts
  labels: [hold, squeeze, press, twist, strike, cut, pry, hang]
  evidence: image_only | from_demonstration_video
grasp_annotation:
  hand_pose_ref: optional (if hand tracking supplied)
  grasp_type: power | precision | hook | pinch | other (document the taxonomy used)
  candidate_regions: list of part ids
  notes: free text (e.g., "gloved hand covers trigger")
action_triplets:
  tool: id; action: from verb list; target_object: id
quality:
  annotator_id: pseudonymous
  confidence: 1-3
  flags: [ambiguous_part, motion_blur, label_uncertain]

Guidelines the vendor must write

A specification is only the schema. The guideline document does the work. Request that annotation vendors deliver, before production, a written guide that includes:

• A definition and picture for each part label on each tool class, with edge cases (a hammer's neck, a drill's side handle).
• Rules for occlusion and for parts that are not visible, including whether to infer them.
• The grasp taxonomy, with examples for each type and for gloved hands.
• A decision tree for ambiguous cases and an escalation route to a domain reviewer.
• Handling of worn, modified or improvised tools.
• A tool list with model names so annotators learn real parts, which matters for power tools and measuring instruments.

Gold sets, audits and agreement

• Gold set. A small number of images annotated by a domain expert and used to train and test annotators. Size and tolerance are yours to set.
• Double annotation. A sampled portion annotated twice, with agreement measured by mask overlap for segmentation and by label match for categorical fields. Report the metric you chose and its threshold.
• Audit loop. Reviewers check a rolling sample; failures return to annotators with reasons; guideline updates are logged by version.
• Drift check. Re-test annotators on the gold set at intervals.

Acceptance tests

1. Every delivered image passes schema validation, with no missing required fields.
2. Agreement on the double-annotated sample meets the thresholds in your contract.
3. A reviewer you designate finds the sampled error rate under your limit, per part class.
4. Class and part coverage matches the scope table, including rare parts.
5. Held-out evaluation images are annotated to the same standard and kept separate from training from the start.
6. The delivery includes the guideline version, annotator pool statistics without personal data, and a datasheet describing composition and known gaps, in the style proposed in Datasheets for Datasets.

Choosing which layers to buy

Not every project needs every field. A grasp-planning team may want hand-independent functional regions and mesh references; a vision-language team may want part names and short descriptions of how each part is used; a policy-learning team may want action triplets from video. Specify the minimum set first, price a pilot batch against the gold set, and expand only if agreement is acceptable. Tool-selection reasoning, such as which tool suits which task, is a knowledge asset and sits with tool selection knowledge, while storage layouts are covered by tool storage and shadow boards.

Rights and provenance

Annotation vendors produce labels under work-for-hire or similar terms, which gives a cleaner position on the labels than on the images. The images still belong to whoever made or licensed them, and may show brands, workers' hands or private sites. Diligence questions include: does the license to the images allow annotation by third parties and use of the labeled result for training; who owns derivative masks and meshes; can annotators view the data under the confidentiality terms; and are tool brands or trademarks an issue. This is not legal advice.

Limitations and preparation

Affordance labels reflect annotator judgment, and categories like "grip zone" can vary with the hand and task. Single-view annotation misses hidden parts. Labels drawn from static images may not match how the tool is used in action, so combine them with demonstrations when function matters. Calibration, depth and pose from a single camera need validation before use as geometry.

Questions to ask annotation vendors

1. Can you show a written guideline for functional parts, with edge cases?
2. How do you train and test annotators, and what is the gold-set process?
3. Which agreement metric will you report, and on what sample?
4. How do you treat occluded or invisible parts?
5. What tooling is used for 3D or pose labels, and what accuracy has it shown?
6. Who reviews domain-specific tool classes?
7. How are guideline changes versioned and applied retroactively?
8. Where is data stored and who can view it during annotation?

Discuss your requirements

If you have images or video that need functional labels, or need to find images to label, describe the tool classes, label fields and permitted uses to SourceX. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case. A request does not guarantee a match. See also the catalog entry on expert annotations and labels and the tool pose tracking page.

Discuss your humanoid training-data requirements with SourceX

Sources

• HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructions — handle affordance, pose and mesh annotations for hardware and kitchen tools.
• TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding — tool-action-object triplets with multi-view capture.
• Learning Affordance Grounding from Exocentric Images — affordance categories across images.
• Datasheets for Datasets — dataset documentation.