Full-Body Human Pose and Motion From Workplace Recordings

The buyer's problem

Hand-centric demonstration data covers what the fingers do. Humanoid programs also need to know how a body moves while it works: how a person bends to a low shelf, shifts weight while carrying a tote, turns in a narrow aisle or steps onto a ladder. Operating businesses record this kind of movement constantly, but the footage was shot to document work, not to measure posture.

Before buying, decide what the motion data is for. A reference-motion library for a whole-body controller, a text-to-motion model and a task-understanding model all consume "pose" differently, and each puts different demands on the capture. This page covers what pose estimated from workplace video can support, where it breaks, and what to write into a specification. For the physical differences between a person and a robot, see the human-to-humanoid embodiment gap.

Three ways whole-body motion data is produced

Published pipelines show how the first row is used. SMPL-X is a body model with articulated hands and an expressive face, and SMPLify-X fits it to a single RGB image (Pavlakos et al., 2019). Humanoid-X reports more than 20 million humanoid poses with text motion descriptions, derived from human videos through a pipeline of video mining, captioning, human-to-humanoid retargeting and policy learning (Mao et al., 2024). Those poses are retargeted estimates, so their quality is bounded by the estimation step.

Marker-based archives show the measured end. AMASS unifies 15 optical marker-based motion-capture datasets under the SMPL body model, with more than 40 hours, over 300 subjects and over 11,000 motions (Mahmood et al., 2019). Nymeria pairs full-body motion ground truth with egocentric Project Aria recordings: 300 hours of daily activity across 1,200 recordings, 264 participants and 50 locations (Ma et al., 2024). Both are staged or instrumented collections, which is the standard a video-only estimate is judged against.

What whole-body motion data can support

• Reference motion for whole-body controllers. HumanPlus trained a low-level policy in simulation with reinforcement learning on 40 hours of human motion data, then shadowed a person's body and hands from a single RGB camera on a custom 33-DoF humanoid (Fu et al., 2024). The motion data shaped how the robot moves; task skills still came from separate demonstrations, up to 40 per task in that study.
• Motion-language models. Pose sequences with text descriptions support text-to-motion and motion captioning, as the Humanoid-X pipeline does.
• Ergonomic and posture analysis. Reach, bend and carry postures can be measured for task design, with caveats on accuracy. Stairs and ladders need their own coverage; see workplace movement on stairs and ladders.
• Task understanding. Body pose is a cue for activity recognition, though it is rarely sufficient alone.

What it cannot support

• Robot actions. Pose is not a joint command. Retargeting to a humanoid with different link lengths, joint limits and balance constraints is an extra step that can produce infeasible motion, such as foot skating or self-collision, which must be filtered or corrected.
• Forces and contact. Carrying a 10 kg load and a 1 kg load can look similar in pose. Balance and effort need force plates, load cells or instrumented tools.
• Precision manipulation. Whole-body estimators place hands coarsely. Finger work needs dedicated hand tracking; see 3D hand pose data and retargeting human hand demonstrations.
• Reliable ground truth in clutter. Racking, machinery, PPE and occluding colleagues degrade monocular estimates. Report confidence per joint rather than trusting every frame.

Limitations specific to workplace footage

Existing footage is usually shot from ceiling-mounted or high wall-mounted cameras with wide lenses. These give strong lens distortion, small subjects and steep viewing angles, all of which hurt pose estimation. Bulky clothing, high-visibility vests and harnesses hide joint positions. Workers are often partly outside the frame, behind equipment or in a group. Frame rate and compression matter too: a 10 fps archive cannot represent a fast lift or a step cleanly.

Egocentric footage has its own problem. A head-mounted camera sees the wearer's hands but not their body, so body pose must come from camera motion, IMUs or a second device, as in Nymeria's observer device. See body-worn IMU demonstrations and multi-view synchronized task recording.

Illustrative specification: pose-annotated task clips

# Illustrative only - values are examples a buyer would set, not an available dataset
deliverable: workplace_body_motion_v0
capture:
  cameras: 3 fixed RGB, 1080p, 30 fps, hardware-synchronized
  calibration: intrinsics + extrinsics per session, checkerboard log included
  reference_subset: 5% of sessions also captured with inertial suit
body_model: SMPL-X (neutral), shape fixed per participant_id
per_frame:
  joints_3d: 24 body joints + root translation, metres, world frame z-up
  confidence: per joint, 0-1
  visibility: per joint, visible | occluded | out_of_frame
per_clip:
  task_id, site_id, participant_id (pseudonymous), height_cm_band
  ground_plane: fitted plane or floor markers
validation:
  mean_per_joint_error_vs_reference: reported per task, per joint group
  foot_skate_metric: reported per clip
rights: training and internal evaluation; no redistribution

Change the numbers to your architecture. A team feeding a retargeting pipeline may need a world-frame trajectory with a fitted ground plane; a team training a motion-language model may accept lower accuracy and want more diversity and better text.

Acceptance checks to define up front

1. Reference validation. Require a subset captured with a measured method and report joint error per body part, not one average.
2. Temporal plausibility. Measure jitter, foot sliding and bone-length drift per clip, and set thresholds you will reject on.
3. Coverage by posture. Count clips by action type (reach high, reach low, carry, turn, step) rather than total hours.
4. Frame-level flags. Require visibility and confidence so low-quality frames can be masked.
5. Held-out evaluation. Hold out participants and sites, not random clips, as in demonstration dataset splits.

Choosing between estimation and instrumented capture

The decision is mostly about what error you can tolerate downstream. If poses feed a retargeting pipeline that a physics-based controller then tracks, small joint errors may be absorbed, but systematic errors such as a consistent forward lean or shortened stride are copied into the robot's reference motion. If poses feed a text-to-motion or activity model, jitter matters less than label diversity. A reasonable procurement pattern is to buy a large estimated set for breadth and a small instrumented subset for validation, then report how the estimator behaves on the subset. Ask for the error to be broken down by task, because lifting from the floor and walking an aisle fail in different ways. Treat any single "average error" figure with suspicion: it hides which joints and which postures are wrong.

Questions to ask suppliers

1. Which estimator and body model produced the pose, and which version?
2. Was any part of the recording validated against marker-based or inertial capture?
3. What are the camera heights, angles, lens types and frame rates?
4. How are occluded or out-of-frame joints marked?
5. Can clips be traced to task, site and participant without exposing identity?
6. Did participants consent to body-shape and motion data being used for model training?
7. Were the recordings staged for capture, and did that change how people moved?
8. What is the plan for retargeting-friendly output, such as world-frame coordinates and a ground plane?
9. Is there a motion-capture reference option for a subset? See motion capture for humanoid motion.

Body shape and movement can identify individuals, so permissions need care; see worker consent and identifiable people. Blurring a face does not resolve questions about who consented to what.

Discuss your requirements

If your program needs whole-body motion from real work settings, describe the tasks, body-pose requirements and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources; its role is sourcing, rights review, licensing and delivery coordination, and specialist capture partners can be engaged case by case. A request does not guarantee a match. For the wider map, see the humanoid training data guide and the workplace task demonstration hub. If no existing asset fits, SourceX can scope a commissioned collection of new recordings.

Discuss your humanoid training-data requirements with SourceX

Sources

• Expressive Body Capture: 3D Hands, Face, and Body from a Single Image — SMPL-X and fitting from one RGB image.
• Learning from Massive Human Videos for Universal Humanoid Pose Control — video-to-humanoid pose pipeline and Humanoid-X.
• HumanPlus: Humanoid Shadowing and Imitation from Humans — motion data for a low-level policy; shadowing from RGB.
• Nymeria: A Massive Collection of Egocentric Multi-modal Human Motion in the Wild — full-body ground truth recorded in the wild.
• AMASS: Archive of Motion Capture as Surface Shapes — marker-based mocap unified under SMPL.