Retargeting Human Hand Demonstrations to Robot Hands

The buyer's problem

Hand data is easier to collect from people than from robots. Dexterous robot hands are less widely available and hard to teleoperate, while people can be tracked with wearable or camera-based systems. The question is how far human hand data goes once it has to drive a robot hand. This page covers use-side limits. How hand pose is captured and labeled is covered in 3D hand pose data, and hand-object annotation in hand-object interaction annotation.

What published work did

• DexMV built a pipeline that extracts 3D hand and object poses from human videos, then uses demonstration translation to convert human motion into robot demonstrations for a simulated multi-finger hand; the demonstrations improved imitation learning over reinforcement learning alone (Qin et al., 2021).
• VideoDex used visual, action and physical priors from human internet video datasets and evaluated on a robot arm with a dexterous hand (Shaw et al., 2022).
• DexCap is a portable hand motion-capture system that combines SLAM with electromagnetic-field tracking and records 3D scene observations; its DexIL method learns dexterous policies from the mocap data, evaluated on six dexterous tasks, with a session limit of about 40 minutes from power and no force sensing (Wang et al., 2024).
• EgoDex collected 829 hours of egocentric video with an Apple Vision Pro, paired with 3D hand and finger tracking, over 194 tabletop tasks with household objects, and released it publicly (Hoque et al., 2025).

These show that tracked human hands are used for dexterous learning. Each study has a specific robot hand, task set and tracking method, and none says how much hand data a different program needs.

Where retargeting loses information

The embodiment gap for the whole body is covered in the human-to-humanoid embodiment gap. Hand retargeting is the same problem at finer scale.

Common retargeting approaches

Buyers need not choose an algorithm, but should know what the data must support.

1. Fingertip or keypoint matching. Map human fingertip positions to robot fingertip targets by optimization. Needs reliable fingertip positions in a known frame.
2. Joint-angle mapping. Map human joints to robot joints with scaling. Needs a hand model fitted per person; with different joint counts the mapping is partial.
3. Object-centric goals. Retarget the object trajectory and let a robot planner or learned policy find the hand motion. Needs accurate object poses.
4. Learned residual correction. Use human reference with a policy trained in simulation or on robot data to correct errors. Needs robot-side data.
5. Pseudo-actions. Infer actions from video where no hand model exists, then co-train with robot data.

The first two need 3D hand tracking and the third needs accurate object poses; the last two need robot-side data. In the published work above, human hand data is used alongside simulation or robot data rather than replacing it.

What to require in a hand-data package

• 3D hand pose per frame, with a stated hand model and coordinate frame.
• Per-joint or per-frame confidence and occlusion flags.
• Calibrated cameras and timestamps; see calibration requirements.
• Object identity, pose or masks where the task involves manipulation.
• Task and step labels and a success flag.
• Information on gloves, rings, sleeves and skin variation.
• The tracking method and its known failure modes.
• Documentation of any smoothing or filtering already applied.

For capture protocols, see head-mounted camera capture specification and in-hand manipulation from video.

Illustrative acceptance checks

# Illustrative - buyer sets every threshold
hand_pose_acceptance:
  coverage:
    frames_with_valid_both_hands_pct_min: "set per task"
  confidence:
    per_joint_confidence_field: required
    low_confidence_frames_flagged: true
  geometry:
    bone_length_consistency_check: "within a tolerance you set across frames"
    fingertip_to_object_distance_during_grasp_check: "sample review"
  tracking_audit:
    human_reviewed_sample_frames: "set per batch"
    reprojection_check_against_manual_keypoints: true
  filtering: {smoothing_applied: "document method and cutoff"}
  retarget_feasibility:
    run_sample_through_target_hand_ik: true
    report_infeasible_pct: true

What hand data cannot support

Human hand data does not include robot hand commands, tendon or motor limits, or the contact forces needed to hold an object. It cannot confirm that a retargeted motion is stable on your hand. Egocentric tabletop sets like EgoDex cover household objects on tables, which may not match industrial tasks or humanoid hand scale. Gloved hands, tool occlusion and fast motion degrade tracking. Tactile and wrist force data are separate; see tactile sensing data. Workplace video alone, without tracking, offers weaker support: see when video is not enough.

Sourcing implications

Existing business footage seldom includes calibrated 3D hand tracking. Commissioned capture with a tracked rig or a depth-capable head-mounted device is the usual route, and a specialist provider can supply hardware and QA. Compare with existing workplace footage vs commissioned recording. Performer releases are needed, because hand geometry may count as biometric data in some places; this is not legal advice, so put the question to counsel and see biometric privacy and workplace video.

Questions to ask suppliers

1. Which hand model and tracking method produced the 3D pose, and what is its measured error?
2. Is confidence or occlusion reported per joint and frame?
3. Which cameras and calibration records accompany each clip?
4. Are object poses or masks included for the manipulated items?
5. How were gloves, sleeves and tools handled in tracking?
6. Has any smoothing, retiming or retargeting been applied, and can raw tracks be delivered?
7. How many distinct people contributed, and what variation in hand size exists?
8. What share of frames is excluded for tracking failure?
9. Do releases cover biometric-type hand data and derivative uses?
10. Can you test a sample against our robot hand's kinematics before delivery?

Discuss your requirements

If your robot hand will learn from human demonstrations, describe the hand, tasks and tracking needs to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates license and delivery; if no existing asset fits, SourceX can scope a commissioned collection. A request does not guarantee a match. Start at the humanoid training data guide or the workplace task demonstration hub; the catalog entry for first-person video of skilled manual work describes a related dataset category.

Discuss your humanoid training-data requirements with SourceX

Sources

• DexMV: Imitation Learning for Dexterous Manipulation from Human Videos — DexMV extracts 3D hand and object poses from video and translates them into robot demonstrations in simulation.
• VideoDex: Learning Dexterity from Internet Videos — VideoDex uses priors from internet human video for a dexterous hand.
• DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation — DexCap hand mocap system and its limits.
• EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video — EgoDex egocentric video with paired 3D hand and finger tracking.