Short answer
A grasp annotation dataset for products attaches gripper-specific grasp poses and outcome labels to images, depth or 3D models: pose in a stated frame, gripper type, approach direction, quality score and how the label was obtained. Specify the gripper, label origin (analytic, simulated or real trials) and a held-out object list, because grasp labels do not transfer across end effectors.
The buyer's problem
Recognition data tells a robot what an item is. Grasp labels tell it where and how to hold it, and they are tied to a hand. A pose that works for a parallel-jaw gripper is wrong for a suction cup and may be unreachable for a humanoid hand. Buyers who ask for "grasp annotations" often receive labels for a gripper they do not own, produced by a method they cannot audit.
This page is a specification you can adapt. It follows from the distinction in object recognition data versus manipulation training data, and its hand-versus-gripper dimension is expanded in object data for hands versus grippers.
How published grasp datasets are built
Three label origins appear in the literature, and each has different trust properties.
Analytic labels on models. Dex-Net 2.0 generated 6.7 million synthetic point clouds, grasps and analytic grasp metrics from thousands of 3D models, trained a grasp-quality network on them and reports 93% success on eight known adversarial objects over more than 1,000 trials on a physical robot, and 99% precision on 40 novel objects (Mahler et al., 2017). The labels are only as realistic as the models and contact assumptions.
Simulated outcomes. ACRONYM labeled 17.7 million parallel-jaw grasps over 8,872 objects in 262 categories by running physics simulation (Eppner et al., 2020). The scale is large, and the outcome depends on simulator settings that the buyer should be able to inspect.
Annotation on real scenes. GraspNet-1Billion provides 97,280 RGB-D images with over one billion grasp poses and an analytic evaluation system (Fang et al., 2020). Real images carry sensor noise and clutter, while the labels still derive from an analytic procedure, not from real attempts on every pose.
MetaGraspNet adds a gripper-type dimension: it provides parallel-jaw and vacuum grasp labels, with 217,000 synthetic RGB-D images over 82 article types and more than 2,300 real annotated images in five difficulty levels (Gilles et al., 2022). It is a good model for requiring one label set per gripper.
Illustrative specification
# Illustrative: synthetic values; adapt to your end effector and sensors
dataset_spec:
gripper_profiles:
- id: pj_A # parallel-jaw, max_opening_mm: 85, finger_width_mm: 12
- id: vac_B # single suction cup, diameter_mm: 30
input_modalities: [rgb, depth, mesh] # mesh optional
coordinate_frame: camera_optical # also supply extrinsics to robot base
grasp_label:
pose: {position_m: [x,y,z], orientation_quat: [qx,qy,qz,qw]}
approach_vector: [x,y,z]
gripper_id: pj_A
width_or_seal: 0.043 # jaw opening in m, or seal quality 0-1
quality_score: 0.71 # define scale and source
label_origin: simulation # analytic | simulation | real_trial | human_annotated
outcome: success # success | slip | collision | unreachable | untested
n_trials: 5 # if real_trial
scene:
clutter_level: moderate # isolated | moderate | dense
object_ids: [obj_0102, obj_0455]
split: {train: sites_001_to_008, test_novel_objects: obj_9000_series}
license: training_and_evaluation_no_redistribution
Field-level guidance
Gripper profile. Attach a drawing or CAD of each gripper used, or require labels for yours. Without geometry, collisions between fingers and neighbors are unknowable.
Frames and calibration. State the coordinate frame, intrinsics and the camera-to-base transform if labels are to be executed. Grasp poses in an object frame need accurate 6D pose; see 6D pose annotation.
Label origin. This is the field to protect. A buyer who wants to train on simulated grasps and test on real ones needs the origin recorded per label.
Outcome semantics. Define success: lifted and held for a stated time, or lifted and moved. Separate "untested" from "failed".
Object scope. Products vary in shape. The manufacturer 3D models and CAD page covers when geometry comes from the maker. The physical property annotations page covers mass and friction that analytic labels depend on.
Choosing labels per gripper type
Parallel-jaw. Labels are a pose plus an opening width, with a notion of antipodal contact. They suit boxes, cylinders and handles but need clearance for both fingers, which clutter removes.
Suction. Labels are seal locations on flat or gently curved surfaces, scored by seal quality. They fail on porous, textured or perforated surfaces and on small faces. Surface information from material attributes is a useful input.
Multi-finger hands. Labels need a hand model and a grasp type taxonomy, finger contact points or joint configurations. Few commercial image sets provide these; expect to commission or adapt them. Region labels from affordance and part labels can restrict where contact is allowed.
Whole-body or two-handed holds. Large or awkward objects need pairs of contacts and balance information. Images and meshes alone are weak here.
State the gripper mix in the specification and ask for one label set per gripper, as the vacuum and parallel-jaw labels in published bin-picking data illustrate.
Acceptance tests
- Physical spot check. Execute a random sample of labeled grasps on your gripper and report the success rate per label origin.
- Collision audit. Replay labels against a mesh with your gripper geometry and count interpenetrations.
- Split integrity. Confirm the test objects share no instances or near-duplicate meshes with training.
- Coverage report. Distribution of grasp positions per object, so labels are not clustered on one face.
- Reachability filter. For a humanoid, filter by arm and body constraints; labels that ignore reach overstate usable data.
Limitations
Grasp labels describe a static moment. They do not capture approach motion, regrasping, in-hand adjustment, or how a stack reacts to contact. Annotations from images alone assume the geometry is visible; transparent and reflective items break that, see transparent and reflective objects. Label sets built for tabletop picking may not represent shelf lips or tight containers. If your robot has a multi-finger hand, parallel-jaw labels give at most a starting prior.
Sourcing
Annotation by specialist providers is the common route: the buyer or a supplier holds images and models, and annotators or simulation pipelines add grasp labels. Retailer and manufacturer image sets supply the objects; they do not supply the grasps. The rights check covers the images and the model files, and any annotation tool outputs. This is not legal advice.
Questions to ask suppliers
- Which gripper types do the labels cover, and is geometry provided?
- Is each label analytic, simulated, annotated by hand or based on a real trial?
- What simulator, friction and mass settings were used, if simulated?
- How is success defined, and are failures recorded?
- What are the sensor, calibration and frame conventions?
- How were train and test objects separated?
- Can you provide a small sample to execute on our hardware before a full purchase?
- Who holds rights in the images, meshes and labels?
Discuss your requirements
If you can describe your gripper, object list and sensor setup, describe the requirement to SourceX. SourceX reviews the request and follows up about potential sources for the images or models; it can also arrange annotation and engage specialist capture partners, scoped case by case. A request does not guarantee a match. Start from the humanoid training data guide or the product and object data hub, and see the robotics and embodied AI use case.
Discuss your humanoid training-data requirements with SourceX
Sources
- GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping — GraspNet-1Billion: RGB-D images of cluttered scenes with over one billion grasp poses and an analytic evaluation system.
- ACRONYM: A Large-Scale Grasp Dataset Based on Simulation — ACRONYM: 17.7M simulated parallel-jaw grasps labelled by physics simulation.
- Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics — Dex-Net 2.0: synthetic grasps with analytic metrics and 93% success on 8 known adversarial objects in 1,000+ trials.
- MetaGraspNet: A Large-Scale Benchmark Dataset for Scene-Aware Ambidextrous Bin Picking via Physics-based Metaverse Synthesis — MetaGraspNet: parallel-jaw plus vacuum grasp labels.