Short answer
Object recognition data teaches a robot what an object is and where it is: labels, boxes, masks, attributes, 6D pose and 3D shape. Manipulation training data teaches what to do with it: where to grasp, how to move, and how the object responds to contact. Recognition data is far easier to source from businesses, but it never shows how an object behaves when handled, so a humanoid needs both.
Why the distinction drives budgets
Object data is the most accessible category in this section. Retailers, brands and distributors hold product images, attributes, dimensions and sometimes 3D models, and licensing them is comparatively simple. It is tempting to assume a large product-image set also helps a robot pick those products up. It helps the robot find them; it says nothing about holding them.
The reverse mistake is also common: manipulation datasets often carry thin object labels, so a policy trained on them may grasp well and still pick the wrong item. Decide which question is your bottleneck before you buy.
A ladder from recognition to manipulation
| Level | Question it answers | Typical data | What it adds for manipulation |
|---|---|---|---|
| 1. Identity | What is it? Which SKU? | Labeled images, boxes, masks | Tells a policy which object to act on |
| 2. Attributes and language | What is it called, made of, sized? | Catalog attributes and descriptions | Hints about material and size; language grounding |
| 3. Pose and shape | Exactly where, and how oriented? | 6D pose labels, CAD or scanned meshes | Geometry for grasp and motion planning |
| 4. Grasp hypotheses | Where could a hand or gripper hold it? | Analytic or simulated grasp labels | Candidate grasps, mostly for parallel-jaw grippers |
| 5. Physical properties | How heavy, stiff, slippery, fragile? | Measured weights, materials, stiffness | Limits for force, speed and grasp choice |
| 6. Interaction | What happens when it is handled? | Robot or human demonstrations with object state | The motion and the object's response |
Levels 1 to 3 are recognition. Level 6 is manipulation. Levels 4 and 5 sit between: they inform manipulation but contain no executed motion.
Public datasets mark each rung. COCO labels about 2.5 million instances across 91 categories in 328,000 images (Lin et al., 2014), and Products-10K labels 10,000 SKU-level products (Bai et al., 2020). ABO pairs catalog images and up to 18 metadata attributes per product with 3D models (Collins et al., 2021). BOP benchmarks 6D pose estimation of rigid objects from a single RGB-D image across eight datasets in a unified format (Hodan et al., 2018).
The YCB object set supplies physical objects that vary in shape, size, texture, weight and rigidity, with physical properties and models for benchmarking manipulation (Calli et al., 2015). At the top, Open X-Embodiment pools more than 1 million robot trajectories covering 527 skills (Open X-Embodiment Collaboration, 2023).
Why grasp labels are not manipulation data
Grasp datasets look like manipulation data but behave like perception data. GraspNet-1Billion provides 97,280 RGB-D images with over one billion grasp poses and an analytic evaluation system (Fang et al., 2020). ACRONYM labels 17.7 million simulated parallel-jaw grasps on 8,872 objects by physics simulation (Eppner et al., 2020), and Dex-Net 2.0 trained grasp-quality models on 6.7 million synthetic point clouds with analytic grasp metrics (Mahler et al., 2017).
These datasets score where to close a gripper. They do not record the approach, re-grasps, in-hand adjustment, slip, deformation, articulation, or what the object does after lift-off. Many target parallel-jaw or suction grippers; a multi-fingered humanoid hand needs different labels, as object data for humanoid hands vs grippers explains. The field-level spec for adding grasp labels to product data is on grasp annotations for products.
How deployed systems separate the layers
Some large robot models keep embodied reasoning and control as separate components. Gemini Robotics pairs a vision-language-action model that controls robots with Gemini Robotics-ER, an embodied reasoning model that handles pointing, grasp and trajectory prediction, multi-view correspondence and 3D boxes (Google DeepMind, 2025). Recognition data can strengthen the reasoning layer; the control layer still learns from robot data.
Context matters too. ARMBench collected images, videos and metadata for more than 235,000 pick-and-place activities on more than 190,000 unique objects in a warehouse workcell, and defines segmentation, identification and defect-detection tasks on them (Mitash et al., 2023). Those are still recognition tasks, but the images look like what a working robot sees, which catalog photography rarely does.
Illustrative: one object, two records
# Illustrative only — the same synthetic product described for recognition and for manipulation
recognition_record:
sku: SKU-000-4471 (1 L detergent bottle, synthetic)
images: 6 catalog views + 40 in-situ shelf crops
labels: category, brand-free instance ID, box, mask
attributes: {material: HDPE, color: blue, fill: liquid}
geometry: CAD mesh, 6D pose per in-situ image
physical: {weight_g: 1100 (listed), dims_mm: [95, 70, 260]}
manipulation_episode:
episode_id: ep_site03_000912
embodiment: humanoid_X right hand (or tracked human hand)
observations: head + wrist RGB-D at 30 Hz, joint states
actions: end-effector deltas + finger commands at control frequency
events: approach, grasp at handle, lift, slip at 2.4 s, regrasp, place in tote
object_state: pose track; liquid slosh visible; cap orientation
outcome: success after one regrasp
The recognition record can be licensed from a catalog holder and a photo set. The manipulation episode has to be recorded, with a robot, a teleoperation rig or a tracked human.
Choosing what to buy
| Symptom in your robot | Buy | Not enough on its own |
|---|---|---|
| Picks the wrong item among similar SKUs | Instance-level images of your SKUs, in situ | Category labels |
| Finds the item, but grasps fail | Pose, 3D models and grasp labels for your hand | Catalog photos |
| Grasp holds, but items slip or crush | Physical properties and demonstrations with force | Grasp labels |
| Task needs opening, folding or inserting | Demonstrations or robot trajectories | Any recognition data |
| Cannot resolve "the blue one on the left" | Attributes and referring expressions | Category labels |
Two pages refine the first two rows: instance vs category recognition and physical property annotations. For the action side, human demonstrations vs teleoperated robot demonstrations compares the sources of interaction data.
Limitations
- Domain gap. Studio product photos differ from cluttered, occluded, badly lit views from a robot's cameras.
- Simulation assumptions. Simulated grasp labels inherit the friction, mass and mesh quality of the models used.
- Rigid-object bias. Most pose and grasp datasets cover rigid objects; deformables, liquids and articulated items are thin.
- License limits. Many public recognition and grasp datasets restrict commercial use; check before training a product model.
- No behavior. No recognition dataset, however large, records how an object moves, bends or breaks under contact.
Questions to ask suppliers
- Are images studio packshots, in-situ photos, or both, and in what proportion per SKU?
- Do you hold 3D models or CAD for the products, and how do they relate to the photographed items?
- Are weights and dimensions measured or copied from supplier data?
- Do attributes include material, fragility, or other properties relevant to handling?
- Are any images or videos of people or machines handling the products?
- How are packaging variants and design changes tracked over time?
- Who owns each image: you, the brand, or a photographer?
Discuss your requirements
If you know which rung of the ladder is your bottleneck, tell SourceX which objects, labels and handling data you need. SourceX looks for US businesses that hold suitable object data, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. It reviews the request and follows up about potential sources; a request does not guarantee a match.
For supply on the recognition side, start with product catalog data for robot perception, the product and object data hub or the humanoid training data guide. The catalog's robotics and embodied AI use case lists related record types.
Discuss your humanoid training-data requirements with SourceX
Sources
- Microsoft COCO: Common Objects in Context — category-level recognition labels.
- Products-10K: A Large-scale Product Recognition Dataset — SKU-level product recognition.
- ABO: Dataset and Benchmarks for Real-World 3D Object Understanding — catalog attributes paired with 3D models.
- BOP: Benchmark for 6D Object Pose Estimation — 6D pose benchmarking.
- GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping — grasp poses on RGB-D scenes.
- ACRONYM: A Large-Scale Grasp Dataset Based on Simulation — simulated grasp outcomes.
- Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics — synthetic grasp-quality data.
- Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols — physical object set with properties.
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models — manipulation as robot trajectories.
- Gemini Robotics: Bringing AI into the Physical World — embodied reasoning separated from control.
- ARMBench: An Object-centric Benchmark Dataset for Robotic Manipulation — recognition tasks on data captured during warehouse picking.