Handheld Gripper Demonstrations (UMI-Style Collection)

The buyer's problem

Teleoperation produces robot-native data but needs a robot at every site, which limits where and how fast data can be collected. Human video is easier to scale but contains no robot actions. A handheld device attempts a compromise: a person demonstrates with a tool that has the robot's gripper geometry and sensors, so the recording approximates what a robot wrist camera would see and encodes gripper motion as a trajectory. Buyers ask whether this is a viable way to collect at business sites, and what to specify.

What the UMI work reports

The Universal Manipulation Interface (UMI) is a hand-held gripper for in-the-wild demonstration collection. Its only sensor is a wrist-mounted GoPro with a fisheye lens, plus side mirrors to provide extra viewpoints, and it uses IMU-aided SLAM to track the gripper's pose; the authors report that the resulting policies are hardware-agnostic and deployable across multiple robot platforms, with roughly 70 percent success in out-of-distribution tests (Chi et al., 2024). That is one design evaluated on a set of tasks; it does not establish performance on humanoid hands or whole-body skills.

For contrast, DROID used teleoperation with a Meta Quest 2 controller on a Franka Panda arm with a Robotiq gripper, two exterior stereo cameras and a wrist stereo camera, at 15 Hz control, across 564 scenes and 86 tasks collected by 50 collectors; the same rig was replicated at 13 institutions (Khazatsky et al., 2024). The comparison shows the trade: DROID needed a replicated robot rig, UMI-style collection needs a portable device and a way to estimate poses.

What handheld collection captures

Fit with humanoid programs

Policies trained from handheld data predict end-effector motion for a particular gripper form. Humanoids add bimanual coordination, a torso, a floating base and multi-fingered hands. A parallel-jaw handheld device can supply data for the grasp-and-place layer of a humanoid only if the humanoid has, or can emulate, a compatible end-effector, and if inverse kinematics, reach limits and balance are handled elsewhere. Dexterous hands require a hand-mapping approach, not a gripper; see retargeting human hand demonstrations and 3D hand pose data.

This page does not decide the broader question of which demonstration type to buy; that is in human demonstrations vs teleoperated robot demonstrations. For a related, body-worn view, see wrist-camera human demonstrations.

Where it fits in a sourcing plan

Handheld data is captured with a purpose-built device by trained demonstrators; it is not something business footage archives contain. It suits tasks where a gripper-sized end-effector does the work and where a site cannot host a robot: a kitchen during service, a small shop, a room that is being turned over. It is a poor fit for tasks that depend on the robot's own body, for heavy loads and for dexterous in-hand work. A realistic plan often combines routes: handheld demonstrations for breadth across sites and objects, then a smaller set of robot-collected episodes on the target platform to ground actions and test transfer. Decide this mix with your architecture in mind, and reserve some demonstrators or sites for evaluation rather than training.

Specification points

• Gripper match. Finger geometry, stroke and compliance should match the target end-effector, or the policy sees a different scene geometry.
• Camera. Lens model, field of view, mounting and exposure; the wrist view must resemble the robot's wrist view. Calibration records are covered in calibration requirements.
• Pose tracking. SLAM or other tracking method, scale handling, failure flags, and a way to measure drift against a reference.
• Action definition. Absolute or delta, frame, and rate; see action space and control frequency.
• Timing. Camera, IMU and gripper-state alignment; see sensor synchronization requirements.
• Demonstrator guidance. Tasks, speed limits and instructions about avoiding poses the robot cannot reach.
• Environment logging. Site, object IDs, lighting and any hazards.

Illustrative collection record

# Illustrative - synthetic values
device: {gripper_model: "handheld_v2", camera: "wrist_fisheye", imu: true, mirrors: true}
episode:
  id: hh_000087
  site: site_017
  operator: operator_A12
  task: "load_dishwasher_rack"
  duration_s: 38.4
streams: [wrist_video, imu, gripper_width, camera_pose_est]
tracking:
  method: "visual-inertial SLAM"
  failure_flags: ["lost_track_frames"]
  scale_check: "known-size target at start"
labels: {outcome: success, instruction: "place plate in rack"}
quality: {pose_drift_check: pass, blur_frames_pct: 3.1}

Acceptance checks

1. Tracking quality. Share of frames with valid pose; flagged losses are excluded or reported.
2. Drift check. Return to a start target at the end of the episode and measure displacement against a limit you set.
3. Gripper state fidelity. Sensed width versus visual estimate on a sample.
4. View parity. Wrist-view images resemble target robot's field of view and mount.
5. Reachability. Sample trajectories run through your kinematic model to flag unreachable poses.
6. Demonstrator coverage. Counts of sites, operators and object types, and a split plan; see splitting demonstration data by site and operator.

What handheld data cannot support

It does not include a robot's joint trajectories, so whole-body and arm-specific behavior come from elsewhere. Demonstrators may move faster, smoother or in ways the robot cannot reproduce. Contact force is missing unless instrumented. Heavy tasks and tasks near hazards are awkward to demonstrate with a handheld device, and bimanual tasks need two devices with synchronized clocks and a shared reference frame. Business-process context, such as why a step happens, is not in the recording. Teleoperated collection, covered in teleoperation data collection requirements, remains the route when you need the robot's own states, and the gap between human and humanoid bodies is explained in the human-to-humanoid embodiment gap. Tracking failures in featureless, reflective or fast-motion scenes can silently corrupt actions, so the quality flags matter.

Permissions and site checks

The device records the workplace through a wrist camera, so people, screens and products may appear. This is not legal advice. Ask how notices and worker permissions are handled, whether the site owner approved capture, and whether the recording scope is limited to the task area. See worker consent and identifiable people and recording protocol on live work sites.

Questions to ask suppliers

1. Which gripper design and camera do you use, and does it match our end-effector?
2. How is gripper pose estimated, and what drift or failure rates do you measure?
3. How is gripper opening recorded?
4. Are calibration files provided per device and per session?
5. How many demonstrators and sites, and how are they trained?
6. How do you prevent demonstrations the robot cannot reproduce?
7. What fraction of episodes is rejected for tracking or blur, and how is that reported?
8. Which formats and action conventions are delivered?
9. How are bystanders, screens and branded items handled at the site?
10. Are devices and software available for us to reproduce collection ourselves?

Discuss your requirements

If you are considering robot-free demonstration capture, describe the tasks, gripper and sites to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates license and delivery; if no existing asset fits, SourceX can scope a commissioned collection. A request does not guarantee a match. Start at the humanoid training data guide or the procurement hub.

Discuss your humanoid training-data requirements with SourceX

Sources

• Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots — UMI handheld gripper with wrist-mounted camera, IMU-aided SLAM and hardware-agnostic policies.
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — DROID as a robot-teleoperation comparison point with fixed rig and calibration.