Short answer
Object-in-hand images show items while they are being held, with fingers and palms occluding part of the object and the pose changing from frame to frame. Few business archives contain them, so they usually come from purpose-built capture with chosen viewpoints and labels. They support in-hand recognition and pose estimation, but they contain no robot actions or contact forces unless the capture is instrumented.
Why held objects need their own data
Catalog photos and warehouse stills show objects at rest, on a background, with every surface exposed. Once an object is in a hand, three things change. Fingers hide parts of it. Its pose rotates quickly and unpredictably. And the hand itself becomes part of the image, with skin, gloves, sleeves and motion blur.
A perception system that recognizes a bottle on a shelf may fail on the same bottle held by its neck. For a humanoid that checks what it has picked up, confirms it grasped the right item or hands it to a person, in-hand recognition is a separate capability. That is the distinct intent of this page: a collection brief for held-object imagery. For the contrast with broader recognition data, see object recognition data vs manipulation training data.
What published work shows
Two research datasets set the pattern for hand-object capture.
DexYCB recorded 582K RGB-D frames from 1,000 sequences in which 10 subjects grasped 20 YCB-Video objects, captured from 8 views, and benchmarked 2D object and keypoint detection, 6D object pose estimation and 3D hand pose estimation (Chao et al., 2021). Multiple views let annotations be propagated and verified.
HOI4D is egocentric: about 2.4M RGB-D frames over about 4,000 sequences, with 800 object instances in 16 categories across 610 indoor rooms, and annotations that include panoptic and motion segmentation, 3D hand pose, category-level object pose and hand action labels (Liu et al., 2022).
Both are research collections with tabletop or household objects. Neither guarantees coverage of a buyer's workplace items, gloves or tools. They are useful as design references for what a purpose-built capture needs to record.
Why most of this data has to be collected
Business archives rarely contain it. Training videos may show hands, but often at low resolution and without object labels. Warehouse cameras typically see parcels, not item details. Catalog photography excludes hands. A buyer therefore usually plans for commissioned collection, designed with chosen viewpoints, sensors and labels. SourceX describes a separate physical-world path for new recordings of hands-on work coordinated with businesses, where each opportunity has its own requirements. The consent and access rules are covered in worker consent and identifiable people.
What this data can support and what it cannot
It can support:
- Recognizing an object while partially occluded by a hand.
- 6D pose estimation and tracking of held objects.
- Hand-pose and hand-object contact estimation.
- Verification steps, such as confirming the item in hand matches the task.
- Evaluation under occlusion, rotation and motion blur.
It cannot support:
- Control. Images do not contain joint states, commanded actions or contact forces. A humanoid needs robot, teleoperation or instrumented-device data to learn grasping and handover. Human hands differ from robot hands in degrees of freedom and strength; see object data for hands vs grippers.
- Grasp success. A human grasp does not show what a robot gripper would hold.
- Object properties. Weight, slip and compliance are not visible.
- Coverage you did not ask for. Gloved hands, uncommon tools or unusual lighting appear only if the capture plan includes them.
Illustrative capture specification
The specification is Illustrative. Parameters are examples a buyer would set; none describes an available dataset.
# Illustrative only - values are examples, not offers
collection: held_object_images_v0
objects: 40 workplace items across 6 categories # buyer sets list
participants: 8 demonstrators, mixed hand sizes, gloved and ungloved
viewpoints:
fixed_cameras: 4 (front, left, right, top)
head_mounted: 1 egocentric
sensors: RGB-D synchronized; calibration file per session
actions_per_object: [pick_up, rotate_inspect, hand_over, put_down]
lighting: normal, low, mixed
labels:
object_mask_per_frame: true
object_pose_6d: for rigid items
hand_pose_3d: tracked or annotated
grasp_type: categorical
occlusion_ratio: estimated per frame
split_policy: hold out by object and by participant
consent: recorded per participant; face handling stated
Acceptance criteria you can set
- Label accuracy. Review a sample of masks and poses against the images.
- Occlusion coverage. The distribution of occlusion levels is reported, not just the average.
- Calibration. Intrinsics and extrinsics are delivered and checked.
- Synchronization. Frame timing across cameras is documented.
- Held-out design. Test objects and participants do not appear in training.
- Metadata. Session, participant pseudonym, lighting and object ID per frame.
Design choices that change the value of the capture
Several decisions made before anyone records a frame determine what the images can be used for.
Who holds the object. Human demonstrators are easy to recruit and produce natural grasps, but a robot gripper occludes the object differently from fingers. If your target is a robot hand or gripper, ask whether any frames can be captured with the robot itself, or with a handheld gripper device, so the occlusion pattern matches.
What the camera sees. An egocentric head-mounted view approximates what a humanoid's head camera sees in front of its hands. Fixed cameras give cleaner multi-view labels but a different perspective. Many programs use both and keep the pairing in the metadata.
How tasks are scripted. A free-form pick-and-look sequence yields more varied poses than a fixed task script, but harder-to-compare sequences. Decide whether you want a controlled set of grasp types or natural variety, and say so in the brief.
Whether the objects are yours. If the items come from a workplace, the supplier needs permission to photograph them, and branded items bring their own rights questions. See product image rights.
Questions to ask suppliers
- How many viewpoints are captured at once, and how is calibration verified?
- How were hand and object pose labels produced, and what is their estimated error?
- Which objects come from the workplace and which are generic stand-ins?
- Are gloves, sleeves and different hand sizes represented?
- How is consent recorded, and how are faces handled in the egocentric views?
- Can capture be repeated at a customer site with its own objects?
- What is the documented split policy for objects and participants?
- What does the license permit for training, evaluation and redistribution?
Permissions and provenance
Hand images may reveal identity through skin marks, tattoos, rings and watches, and egocentric footage can capture faces and screens. Written consent for each participant, a clear scope of use and agreement from the site are the starting points. Blurring or cropping reduces exposure but does not resolve consent. See worker consent and identifiable people.
Labels built on this capture connect to neighboring specifications: 6D pose annotation for held-object pose, grasp annotations for products for grasp labels, object state recognition for open, closed or filled states, cluttered scenes for recognition for the contrast with scenes without hands, and packshot vs in-situ images for why studio photos do not transfer.
Discuss your requirements
If you need held-object imagery with specific viewpoints, objects and labels, describe the objects, sensors, labels and permitted uses to SourceX. SourceX reviews the requirement, looks for US businesses that hold related material, and can scope a commissioned collection if nothing existing fits. Annotation and specialist capture partners can be arranged case by case, and a request does not guarantee a match. Read the objects hub and the humanoid training data guide for neighboring topics, and the catalog entry for first-person video of manual work.
Describe the physical-world collection your robotics project needs to SourceX
Sources
- DexYCB: A Benchmark for Capturing Hand Grasping of Objects — multi-view hand-grasping capture and benchmarks.
- HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction — egocentric hand-object interaction annotations.