RGB-D and Depth Capture for Human Task Demonstrations

When depth changes the purchase

Color video shows what happened. Depth adds where things were. For a perception engineer building hand-object pose estimators, that is often the difference between a usable and an unusable dataset: monocular video can be lifted to 3D by a model, but a model's depth guess is not ground truth, and the errors correlate with exactly the cases you care about, such as hands partly hidden behind objects.

Depth is not free. It changes the sensor, the calibration burden, the storage footprint and the failure modes. Decide whether your model consumes depth at training, at inference or only for labeling. If depth is used only to produce labels, you may be able to capture it for a calibration subset rather than every take.

Published reference points

• HOI4D provides about 2.4 million RGB-D egocentric frames over about 4,000 sequences, with 800 object instances in 16 categories across 610 indoor rooms, and annotations for panoptic and motion segmentation, 3D hand pose, category-level object pose and hand action (Liu et al., 2022).
• DexYCB records 10 subjects grasping 20 YCB-Video objects from 8 views, 582K RGB-D frames in 1,000 sequences, and benchmarks 2D detection, 6D object pose and 3D hand pose (Chao et al., 2021).
• ARCTIC adds motion-capture ground truth for bimanual manipulation with eight allocentric views and one egocentric view, 2.1 million frames, 3D hand and object meshes and dynamic contact information (Fan et al., 2023).
• DROID is a robot dataset, not a human one, but its rig shows the convention of stereo cameras with calibration, depth and language per episode (Khazatsky et al., 2024).

Note that the ARCTIC-style contact information comes from a motion-capture setup, not a standard RGB-D sensor. Contact is not recoverable from depth alone.

What RGB-D human data can and cannot support

Can support

• 3D hand pose (see 3D hand pose data) and hand-object relations (see hand-object interaction annotation), with the caveat that estimates are labels, not measurements, unless a mocap or glove system produced them.
• Object pose tracking and segmentation, particularly for rigid objects; see object tracks and masks.
• Scene geometry for reachability and clearance estimates.
• Evaluating monocular depth or pose models against a sensor reference.

Cannot support

• Forces, tactile state or whether a grasp is secure.
• Robot joint states or actions. Depth does not change this; see human demonstration video for robot learning.
• Reliable geometry on difficult materials (below).
• A match to a humanoid's depth sensor. Humanoids use varied depth hardware; unit-to-unit differences in noise and range matter for transfer.

Depth sensor families and trade-offs

The table is general guidance, not measured performance; check each point on sample depth maps from your own task and setting.

Choice depends on working distance. A tabletop task at 40 cm and a warehouse aisle at 4 m call for different hardware. Ask a collection team to justify the sensor against the task's working distance and object materials, with sample depth maps.

Failure modes to look for

Treat these as checks to run on samples, not as measured rates for any sensor.

• Reflective and transparent surfaces: check whether glassware, polished metal, shrink wrap and clear containers return missing or wrong depth.
• Dark and absorbing surfaces: check whether black plastics and fabrics leave holes.
• Sunlight and strong infrared sources: check whether active sensors wash out near windows or outdoors.
• Thin structures: check wires, utensils and fingers for flying pixels at edges.
• Motion: check whether fast hand movements smear, especially with rolling-shutter or sequential-capture sensors.
• Interference: where several active sensors face the same scene, check whether they degrade each other.
• Misalignment: check whether depth registered to color shows halos at object edges.

Calibration and storage

Specify intrinsics for the depth and color sensors, extrinsics between them, and the depth unit and scale. Ask whether depth is delivered registered to color or in its native frame, and whether raw and filtered versions are both kept. Storage differs by representation: lossless 16-bit depth PNG or compressed video-codec depth saves space but can introduce artifacts; point clouds are heavier. Plan this with storage and transfer planning for video. When several depth cameras share a scene, add multi-view synchronization.

Illustrative RGB-D specification

# Illustrative only - synthetic values the buyer must adapt
sensor:
  type: active stereo or ToF (buyer to choose by working distance)
  working_range_m: 0.3-1.5          # example for tabletop tasks
  color: 1080p, 30 fps
  depth: 720p, 30 fps, 16-bit, unit = mm
alignment:
  depth_registered_to_color: yes
  intrinsics_extrinsics: json per sensor per day
invalid_pixels:
  encoding: 0 = no return
  report: per-take fraction of invalid pixels on hands and target objects
delivery:
  raw_depth: lossless
  filtered_depth: optional, filter documented
  point_clouds: optional
labels: [task_id, step_segments, hand_visibility, object_ids, outcome]
acceptance:
  max_invalid_pixels_on_hands: buyer-set share
  depth_error_check: flat target at known distance, buyer-set tolerance

Acceptance tests to run on a pilot

1. Measure a flat target at several known distances; compare to reported depth.
2. Capture the task's worst materials (glossy, clear, dark) and review the depth maps.
3. Check color-depth alignment by overlaying object edges.
4. Count frames where hands have a valid depth mask during contact.
5. Estimate bytes per task-hour for raw delivery.

Permissions and provenance

Depth sensors capture geometry of faces and bodies as well as scenes. Diligence questions: does the consent cover depth and 3D data, as distinct from video? Does the supplier treat depth-based hand or body shape as biometric-type data in its jurisdiction? Hand geometry used to identify an individual could fall into regulated categories in some places. This is not legal advice; ask the supplier for its analysis. Combine with worker consent and identifiable people, and with object recognition vs manipulation data when product geometry is the goal.

Questions to ask collection teams

1. Which depth sensor do you propose, and why for our working distance?
2. Can you show depth maps on glossy, clear and dark objects from our task?
3. Is depth registered to color, and how is that verified?
4. What share of pixels on hands and target objects is invalid in a sample?
5. How are sensors calibrated, and how often?
6. What is raw data volume per task-hour?
7. How does the consent form cover depth and 3D reconstruction of people?

Discuss your requirements

If your model needs depth, describe the sensor, working range and task requirements to SourceX. If no existing asset fits, SourceX can scope a commissioned collection of new recordings; SourceX's role is sourcing, rights review, licensing and delivery coordination, with specialist capture partners engaged case by case. A request does not guarantee a match. Return to the humanoid training data guide or the workplace task demonstration hub; see also the catalog page on robotics and embodied AI datasets.

Describe the workplace demonstration data your robotics project needs

Sources

• HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction — RGB-D egocentric frames with 3D hand and object annotations.
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — stereo rig with depth and calibration per episode.
• ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation — mocap ground truth, multi-view and contact information.
• DexYCB: A Benchmark for Capturing Hand Grasping of Objects — multi-view RGB-D grasping benchmark.