Short answer
Human video can support dexterous manipulation as visual priors and, when hands are tracked in 3D, as hand-motion references. In-hand reorientation is where it is weakest: fingers occlude each other and the object, contact and force are invisible, and human and robot hands differ. Buyers should expect to add instrumented capture or robot data and should test video on their own tasks.
Why in-hand manipulation is the hard case
Picking up a box and placing it needs the arm to go to the right place. In-hand manipulation is different: a person rolls a screw between fingertips, flips a card, rotates a bottle cap, shifts a tool from a power grip to a precision grip. The hand changes the object's pose while it stays in the hand. Most of the information that matters is contact: which fingertip pushes, how hard, and where the object slides.
Camera video sees the outside of that process. This page sets out what published work supports, what you can and cannot recover from footage, and what a specification for dexterous data should contain. For the general limits of human video, see what human demonstration video can and cannot do.
Task anatomy: a regrasp and reorientation
Consider "rotate a screwdriver in-hand so the tip points to the screw, then seat it":
- Acquire. Grasp the handle from a tray with a power grip.
- Reposition. Use thumb and index finger to roll the handle until the tip faces down.
- Finger gait. Release and re-place one or two fingers while others hold, changing the contact set.
- Align. Make fine pose corrections against the screw head, relying on touch.
- Seat and turn. Apply axial force and torque.
Steps 2 to 4 are the in-hand part. Hand and object are partly or fully hidden from any outside camera, the contact set changes several times per second, and the signal that decides success is force and slip. Variants include regrasping a pen, reorienting a connector and flipping a part for inspection.
What published work shows
Large egocentric collections now pair video with tracked hands. EgoDex recorded 829 hours of egocentric video with paired 3D hand and finger tracking, using Apple Vision Pro, across 194 tabletop tasks with household objects (Hoque et al., 2025). Its tasks are tabletop and household, so a buyer targeting industrial parts must check how far they carry over.
Instrumented capture gives richer ground truth. ARCTIC provides 2.1 million video frames with 3D hand and object meshes and dynamic contact information, recorded from eight allocentric views plus one egocentric view with motion capture as ground truth, for bimanual articulation of objects such as scissors and laptops (Fan et al., 2023). HOI4D offers about 2.4 million RGB-D egocentric frames over about 4,000 sequences with 3D hand pose, object pose and segmentation annotations (Liu et al., 2022).
Learning from this data is usually indirect. VideoDex used visual, action and physical priors from human internet video for a robot arm with a dexterous hand (Shaw et al., 2022). DexCap built a portable hand motion-capture rig combining SLAM and electromagnetic-field tracking, used it to learn policies for six dexterous tasks, and notes a session limit of about 40 minutes from power and no force sensing (Wang et al., 2024).
What video can and cannot recover
| Information needed | In RGB video | With 3D hand tracking | Needs |
|---|---|---|---|
| Hand location and rough grasp type | Often | Yes | Clear view |
| Finger joint angles during occlusion | Unreliable | Estimated, with confidence | Multi-view or glove |
| Object pose while in-hand | Unreliable | Partly, if tracked | Object markers or depth |
| Contact points and contact order | Rarely | Partly, from geometry | Tactile sensing or simulation |
| Normal force, friction, slip | No | No | Force or tactile sensors |
| Robot hand joint commands | No | No | Retargeting plus robot data |
The practical rule is that visible kinematics can be tracked, hidden kinematics are estimated, and forces are absent. Published rigs such as ARCTIC use motion capture to improve the hidden part, which footage from workplaces will not have. For tracked-hand data in general, see 3D hand pose data; for tasks where touch decides the outcome, see when video is not enough.
Why the human-to-robot step is harder here
Human hands have a different number of degrees of freedom, compliant skin and a thumb that opposes the fingers in ways many robot hands do not. A regrasp that relies on a rolling fingertip may have no counterpart on a robot hand with fewer joints or rigid fingertips. Tracked human hand poses must be retargeted, and retargeting can break contact; see retargeting human hand demonstrations. Humanoid hands also differ across platforms, so a dataset should state which hand model its retargeting assumes. The wider issue is covered in the embodiment gap.
Illustrative collection specification
# Illustrative only - values are examples, not an available dataset
task_family: in_hand_reorientation_v0
objects: 12 hand-held items (screwdriver, pen, 3 connector types, cap, card, ...)
capture:
egocentric: RGB-D head-mounted, 30 fps, known intrinsics
allocentric: 2 fixed RGB cameras, hardware-synced, calibrated
hand_tracking: glove or optical markers on a subset (about 20% of takes)
object_tracking: fiducials on object, removed in a no-marker pass
labels:
hand_pose_3d: 21 keypoints per hand, confidence per keypoint
contact_events: start/end per finger per object (annotated)
grasp_phase: acquire | reposition | finger_gait | align | seat
outcome: success | slipped | dropped | regrasp_needed
metadata: participant_id (pseudonymous), hand_size_band, handedness, glove_type
excluded: forces, tactile signals, robot actions
The marker and no-marker pass pair matters: markers help ground truth but change appearance, and a model trained only on marked footage may fail without them.
Acceptance criteria
- Visibility. Report the share of frames where each fingertip is visible from at least one camera, per task.
- Tracking error. Require a validation subset against motion capture, with error per finger.
- Contact labels. Measure agreement between annotators on contact start and end times.
- Outcome diversity. Include slips and drops, not only successful rotations; see failure and recovery demonstrations.
- Held-out objects. Keep some objects and participants out of training.
Deciding what to buy for a dexterous program
Match the data to the claim you want to make. If the goal is a visual encoder that recognizes hands and objects in contact, broad egocentric video with hand tracking is a reasonable buy. If the goal is a reference for finger motion that a retargeting step can use, you need calibrated multi-view or glove-based tracking, ideally with a no-glove pass. If the goal is a policy that performs the reorientation, no purchase of human video alone gets there: you need robot or teleoperated hand data on the target hand, and the human data plays a supporting role. Write this into the requirement so a supplier does not offer tabletop household footage as a substitute for industrial in-hand work. Ask for the object list and the share of clips in which the object is held for more than a few seconds, since many "manipulation" clips are mostly transport.
Questions to ask suppliers
- What tracking method produced the 3D hand pose, and what is its validated error?
- How is the pose labeled when a finger is hidden?
- Are objects tracked, and are markers visible in the video?
- Is any force, pressure or tactile signal captured, even for a subset?
- Which hand sizes, glove types and handedness are represented?
- How many distinct objects and grasp types appear?
- Were participants instructed to perform in a way that suits cameras?
- Are failed attempts retained with labels?
- What permissions cover close-up hand imagery, which can include distinguishing marks?
Discuss your requirements
If your program depends on dexterous or in-hand data, describe the objects, hand-tracking needs and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources; where no existing asset fits, SourceX can scope a commissioned collection. A request does not guarantee a match. For wider context see the humanoid training data guide and the workplace task demonstration hub. Two-handed work is covered in bimanual task demonstrations, and hand-object interaction annotation covers contact labeling.
Discuss your humanoid training-data requirements with SourceX
Sources
- EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video — scale and structure of egocentric video with hand tracking.
- ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation — mocap-grounded hand and object meshes with contact.
- VideoDex: Learning Dexterity from Internet Videos — priors from human video for a dexterous robot hand.
- DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation — portable hand mocap and its limits.
- HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction — RGB-D egocentric annotations.