3D Hand Pose Data From Workplace Recordings

Why dexterity teams ask for 3D hands

Boxes and 2D keypoints say where a hand appears in an image. Learning across embodiments needs more: where the hand is in space, how each finger is bent, in metric units and in a frame tied to the camera or the world. That is what lets human motion share a state-action space with a robot.

Published work shows the pattern:

• Humanoid Policy ~ Human Policy collected hand and finger 3D poses with consumer VR devices on tasks aligned with the robot's, mapped human and humanoid data into a unified state-action space, and used roughly 27,000 human demonstrations against roughly 1,500 humanoid demonstrations (Qiu et al., 2025).
• EgoMimic co-trained one policy on tracked human data and robot data and, in its experiments, found an extra hour of human hand data more valuable than an extra hour of robot data (Kareer et al., 2024).
• DexMV extracted 3D hand and object poses from human videos and translated them into demonstrations for a simulated multi-finger hand (Qin et al., 2021); VideoDex drew visual, action and physical priors from human internet video for a robot arm with a dexterous hand (Shaw et al., 2022).
• DexCap recorded hand motion with a portable rig combining SLAM and electromagnetic-field tracking and trained dexterous robot policies from it (Wang et al., 2024).

Each of these still relies on robot data or a simulated robot hand. Published results report 3D hand data helping alongside robot data, not replacing it.

Representations you may be offered

MANO was learned from about 1,000 high-resolution scans of 31 subjects and attached to a body model as SMPL+H (Romero et al., 2017); its model files are offered free for research, so confirm terms before commercial use. SMPL-X unifies body, face and articulated hands and can be fitted from a single RGB image (Pavlakos et al., 2019).

Capture routes compared

Scale on the headset route is real: EgoDex recorded 829 hours of egocentric video with paired 3D hand and finger tracking (Hoque et al., 2025). A fixed multi-camera rig produced 18M 3D hand poses in Assembly101 (Sener et al., 2022). At the accuracy end, HOT3D used optical-marker motion capture for hands and objects across 833+ minutes, 19 subjects and 33 objects, with hand annotations in UmeTrack and MANO formats (Banerjee et al., 2024). DexCap reports a session limit of about 40 minutes from power, which shapes how a workplace shift can be recorded.

Ordinary workplace archives sit in the first row. Monocular estimates from uncalibrated, edited training films can supply coarse priors, but rarely the accuracy co-training expects.

What to measure before you accept 3D hand data

• Frame and units. Poses in camera or world coordinates, in meters, with a documented axis convention and camera poses per frame.
• Accuracy. Mean per-joint position error against a higher-accuracy reference on a validation subset, such as sessions recorded with markers; you set the threshold.
• Temporal stability. Jitter measured as frame-to-frame acceleration; dropouts flagged rather than interpolated silently.
• Anatomical plausibility. Consistent bone lengths per demonstrator; joint angles within human limits.
• Hand-object consistency. No fingers passing through objects, which requires object geometry and pose.
• Coverage. Both hands, the grasp types your tasks need, and the objects in your deployment.

Illustrative per-frame record

{
  "_comment": "Illustrative only — synthetic IDs, values and field names",
  "clip_id": "clip_site017_0412",
  "timestamp_ns": 1234567890123,
  "frame": "world",
  "units": "meters",
  "camera_pose": {"t": [0.12, 1.48, 0.33], "q": [0.99, 0.01, -0.12, 0.02]},
  "hands": {
    "right": {"source": "headset_tracking", "confidence": 0.91,
              "mano": {"pose": "[pose parameters]", "shape": "[shape parameters]"},
              "keypoints_xyz": "[joint array]", "glove": "none"},
    "left": {"source": "headset_tracking", "confidence": 0.12,
             "status": "dropout_flagged"}
  },
  "object_poses": [{"object_id": "obj_0031", "t": [0.41, 0.92, 0.18], "q": [1, 0, 0, 0]}]
}

Hands without objects tell only half the story

A hand trajectory without the pose of the object it holds says little about manipulation; DexMV needed both to produce robot demonstrations. Plan object pose, or at least object identity and masks, alongside hand pose. The 2D labeling layer is specified in hand-object interaction annotation.

Workplace conditions that degrade 3D hands

Gloves hide finger joints and change hand appearance, so tracking degrades or fails; see gloved hands and PPE in workplace video. Tools held in the hand, reflective metal, fast motion at production pace and poor lighting all add error. Headsets may not fit with hard hats or face shields, and markers or instrumented gloves can change how people grasp.

Limitations

• No forces or touch. Contact-rich tasks need more than hand kinematics.
• Human hands have more degrees of freedom and different dimensions from most robot hands; mapping loses information, as explained in retargeting human hand demonstrations to robot hands and the human-to-humanoid embodiment gap.
• In-hand reorientation is where these limits are sharpest; see human video for dexterous in-hand manipulation.
• Whole-body motion data is a different specialist purchase, covered in motion capture data for humanoid motion.
• Hand geometry can count as biometric data in some jurisdictions; review this with counsel using biometric privacy laws and workplace video. This is not legal advice.

Questions to ask suppliers

1. How were the 3D hand poses produced: device tracking, multi-view fitting, markers or monocular estimation?
2. In which coordinate frame and units are poses delivered, and are camera poses included?
3. What accuracy was measured, against what reference, and on which subset?
4. How are dropouts, low-confidence frames and gloved hands flagged?
5. Are object poses or identities delivered alongside hand poses?
6. Which hand model or format is used, and what license applies to it?
7. Are calibration files and raw streams available for re-processing?
8. Were demonstrators told that hand geometry would be captured, and how was their consent recorded?

Discuss your requirements

If your program needs 3D hands for co-training or retargeting, describe the tasks, accuracy needs and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources; if no existing asset fits, SourceX can scope a commissioned collection. A request does not guarantee a match. Related pages sit on the workplace task demonstration hub and the humanoid training data guide, and the catalog entry for first-person video of skilled manual work describes a related asset type.

Discuss your humanoid training-data requirements with SourceX

Sources

• Humanoid Policy ~ Human Policy — 3D hand poses from VR devices as cross-embodiment humanoid data.
• EgoMimic: Scaling Imitation Learning via Egocentric Video — co-training with tracked human hand data.
• DexMV: Imitation Learning for Dexterous Manipulation from Human Videos — hand and object poses translated into robot demonstrations.
• VideoDex: Learning Dexterity from Internet Videos — priors for dexterous control from human video.
• DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation — portable hand motion capture and its session limits.
• EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video — headset video with paired 3D hand tracking at scale.
• HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos — marker-based 3D hand and object ground truth.
• Embodied Hands: Modeling and Capturing Hands and Bodies Together — the MANO hand model.
• Expressive Body Capture: 3D Hands, Face, and Body from a Single Image — the SMPL-X body model with hands.
• Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities — 3D hand poses from a multi-camera rig.