Affordance and Part Labels for Workplace Objects

The buyer's problem

A humanoid in a kitchen, workshop or laboratory will often meet objects it has not seen before. Category recognition says "pan"; the useful question is which part to hold, which to avoid and what each part allows. Affordance labels give a model that second layer. The risk is vagueness: "graspable" means different things to different annotators, and an affordance for a human hand may not be one for a gripper.

This page is a labeling specification for workplace objects. It builds on image sets that already exist and adds part-level function. For grasp poses on top of those regions, see grasp annotations for products. For mechanical parts that move, see articulated object models.

What published affordance datasets contain

AGD20K contains over 20,000 images across 36 affordance categories and was built to learn affordance grounding by transferring from exocentric images of people using objects to egocentric views of the objects themselves (Luo et al., 2022). It shows that affordance regions can be learned from images of people interacting with objects, which matters for buyers because such interaction images are easier to collect than robot data.

OakInk is organized as two knowledge bases. Oak holds affordance annotations on 1,800 common household objects, and Ink holds recorded human interactions with 100 objects transferred to virtual counterparts; the dataset covers about 50,000 affordance-aware, intent-oriented hand-object interactions and benchmarks pose estimation and grasp generation (Yang et al., 2022). The notable design choice is intent: the same object is held differently to use, to hand over or to move.

HANDAL focuses on manipulable hardware and kitchen tools. It reports 308,000 annotated frames from 2,200 videos of 212 real-world objects in 17 categories, with 6-DoF category-level pose, handle affordance segmentation and reconstructed meshes, captured with a single off-the-shelf camera and semi-automated annotation (Guo et al., 2023). That capture style is close to what a business could do with a handheld camera, which is why it is a useful template for workplace objects.

Illustrative label specification

# Illustrative: synthetic taxonomy; replace with your own closed vocabulary
object_class: saucepan
parts:
  - {part: body,   function: contain,    contact_ok: avoid_when_hot}
  - {part: handle, function: grasp_hold, contact_ok: yes}
  - {part: lid,    function: cover,      contact_ok: yes}
  - {part: lid_knob, function: grasp_lift, contact_ok: yes}
  - {part: spout_rim, function: pour,    contact_ok: no}
affordance_vocabulary: [grasp_hold, grasp_lift, push, pull, rotate, press, pour, support, hang]
label_geometry: [instance_mask, part_mask, 3d_region]
annotation_rules:
  one_region_may_have_multiple_affordances: true
  occluded_parts: label_visible_portion_only
  actor: [human_hand, parallel_jaw, suction]   # affordances can differ by actor
qa: {double_annotation_share: 0.2, adjudication: senior_annotator}

Design decisions to settle before buying

Closed vocabulary and actor. List the action terms and define each with a positive and a negative example. Decide whether the label means "a person would do this here" or "this robot hand could do this here". They are not the same.

Part taxonomy. Choose a hierarchy that reaches the level you act on. A handle may need to split into grip area and attachment point. Align class names with your recognition taxonomy; the product taxonomies page explains how classification systems are structured.

Geometry. Image masks are cheapest. 3D regions on meshes transfer across views but require models. Keypoints work for tool tips and handles. The segmentation masks for products page covers mask quality.

Context-dependent function. The same region of a mug serves holding, hanging and pouring. Store several labels per region instead of forcing one.

Intent. If intent matters (use versus handover), record it as in intent-oriented interaction datasets, not just the region.

Source data and routes

Existing images from manufacturers and retailers give clean views of parts but little handling context. Operating businesses, such as kitchens, workshops and laboratories, hold images and video showing how objects are held in practice. Annotation providers add the labels. See kitchen equipment and utensils and equipment controls and interfaces for two common object families, and object-in-hand images for views showing contact. Commissioned capture is relevant when you need specific objects held in specific ways.

Acceptance criteria

• Inter-annotator agreement reported per affordance, with region overlap as the metric you set. Rare affordances usually score lowest.
• Vocabulary conformance. No labels outside the closed set.
• Part coverage. Every instance has all taxonomy parts labeled or marked occluded.
• Held-out objects. Test classes and instances absent from training, so generalization to new objects can be measured.
• Actor consistency. Labels do not mix human-hand and gripper meaning.

Piloting the labeling guidelines

Run a small pilot round with two or three annotators on a mixed sample of objects before the full job. Collect the cases where they disagree and write each resolution into the guidelines with an image. Typical disputes include whether a lid edge counts as a grasp region, whether a hot surface is an affordance at all, and whether a hanging loop counts as a handle. Re-run the pilot on fresh images with the updated guidelines and compare agreement. This is cheaper than finding inconsistent labels after delivery, and it gives you a record to attach to the dataset documentation, in the spirit of datasheets for datasets.

What the labels cannot support

A region label does not specify an approach direction, grip aperture, contact force or the timing of a motion. It does not say whether the part is hot, wet, loose or locked. Two similar handles can differ in strength. Affordance labels narrow the search for a grasp; they do not replace grasp annotation, physical properties or robot interaction data. They are also appearance-based, so a model may fail on an unusual instance with an unfamiliar shape.

Permissions and provenance

Images showing hands also show people. Ask whether consent covers training use, and whether annotated derivatives are covered by the supplier's grant. Brand-marked equipment images carry trademark and photo rights. This is not legal advice.

Questions to ask suppliers

1. What is the affordance vocabulary and who defined it?
2. Is the label meant for human hands or for a specific gripper?
3. How are multiple affordances on one region represented?
4. What agreement was measured between annotators, and on what share?
5. What proportion of images show the object in use versus at rest?
6. How are occluded parts handled?
7. Are 3D models or only images provided?
8. Can unseen-object splits be reproduced from your metadata?

Discuss your requirements

If you can list the object classes, parts and actions your robot must handle, describe the requirement to SourceX. SourceX reviews it, follows up about potential US sources, and checks the data and licensing permissions; it can also arrange annotation, scoped case by case. A request does not guarantee a match. Start from the humanoid training data guide or the product and object data hub. The robotics and embodied AI use case page describes the wider context.

Discuss your humanoid training-data requirements with SourceX

Sources

• Learning Affordance Grounding from Exocentric Images — AGD20K: over 20K images from 36 affordance categories, exocentric to egocentric transfer.
• OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction — OakInk: affordance annotations on 1,800 household objects and about 50,000 hand-object interactions.
• HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructions — HANDAL: 212 objects in 17 categories with handle affordance segmentation and 6-DoF pose.
• Datasheets for Datasets — Datasheets for datasets as documentation model for annotation guidelines.