Short answer
Tasks whose success depends on how hard, how long or how precisely something is pressed, twisted, inserted or held need force, torque or tactile data as well as video. Connector seating, threaded fastening, press fits, wiping under pressure, cable routing and fragile-item handling are typical. Video shows where the hands go; it cannot measure contact forces. Plan instrumented capture, force-sensing teleoperation or specialist tactile collection for these tasks.
Why video stops at the point of contact
Human task video is valuable for structure, approach and outcome, as set out in what human demonstration video can and cannot do. At the moment of contact, four things go wrong:
- Force is invisible. A light touch and a firm press can look identical on camera. So can a screw driven to specification and one stripped.
- The contact region is hidden. The hand, the tool and the part itself cover the place where force is applied.
- Success can be invisible. A connector that is half-seated, a seal that is not compressed or a fastener that is under-torqued often looks finished.
- Deformation is tiny. The compression of a gasket or the stretch of a cable rarely spans more than a few pixels.
Even when human forces are measured, they do not transfer directly. A humanoid hand has different compliance, friction, strength and sensing from a human hand, which is why the human-to-humanoid embodiment gap bites hardest in contact-rich work.
Triage: which tasks need more than video
| Task feature | Workplace examples | What video alone gives | What to add |
|---|---|---|---|
| Tight-tolerance insertion | Connector seating, press-fit bushings, fixture pins | Approach path, alignment, final state | Wrist force/torque during insertion; a labeled seating event |
| Threaded fastening to spec | Screws, bolts, caps, hose clamps | Tool choice, sequence, hand placement | Tool torque and angle logs; force/torque at the tool |
| Sustained surface contact | Wiping, sanding, polishing, applying film or labels | Stroke path and coverage | Normal-force profile; before-and-after surface check |
| Deformables under tension | Cable routing, bagging, peeling liners | Sequence and grasp points | Grip or tension force; fingertip tactile signals |
| Grip-sensitive objects | Glassware, produce, wet or oily parts | Grasp type and approach | Grip force; slip events from tactile sensing |
| Pushing and pulling with resistance | Loaded carts, stiff doors, heavy drawers | Posture and path | Handle force, or robot joint torque via teleoperation |
If a task appears in this table, budget for at least one stream beyond RGB. Twisting and regrasping inside the hand is a special case covered in human video for dexterous in-hand manipulation.
What published datasets add beyond video
RH20T shows the pairing pattern most buyers end up with: more than 110,000 contact-rich robot sequences, each with visual, force, audio and action data and each paired with a human demonstration video and a language description (Fang et al., 2023). The human video describes the task; the robot sequence carries the forces and actions.
Portable capture devices recover motion, but not necessarily force. The Universal Manipulation Interface uses a hand-held gripper whose only sensor is a wrist-mounted fisheye camera, with IMU-aided SLAM to track the gripper (Chi et al., 2024). DexCap captures hand motion with SLAM and electromagnetic-field tracking for dexterous policy learning, but it has no force sensing (Wang et al., 2024). If force matters, ask for a sensor to be added rather than assuming the device records it.
Robot demonstrations carry something human video lacks even without a force sensor. Visuomotor methods such as Diffusion Policy learn from the robot's camera images paired with the robot's own recorded actions (Chi et al., 2023); because those actions were executed on the robot's body, contact behavior is at least consistent with the embodiment. Audio is a cheap supplementary cue: REFLECT summarizes multisensory robot data, including audio and robot states, to explain failures (Liu et al., 2023). In your own capture, clicks, scrapes and impacts can help time contact events.
Sensing options, from least to most robot-specific
| Option | What it measures | Match to your robot | Typical source |
|---|---|---|---|
| Video plus annotated contact events and outcomes | When contact starts and ends; success or failure | None | Existing footage plus annotation |
| Synchronized audio | Clicks, scraping, impacts | None, but useful timing | Commissioned capture |
| Instrumented tools and fixtures | Tool torque, clamp or press force | Partial, at the tool | Commissioned capture with logging tools |
| Handheld gripper with a force/torque sensor | Gripper pose, opening, wrench at the device | Close if the gripper resembles your end effector | Specialist capture providers |
| Force-sensing teleoperation | Robot actions, joint states, wrist wrench or joint torque | Exact, for that robot | Specialist robot-data collectors |
| Tactile sensing on robot fingers | Contact location, pressure, slip | Exact, for that sensor and hand | Specialist labs |
Each option has its own sourcing page: force/torque data for contact-rich tasks, tactile sensing data and handheld gripper demonstrations. Where the boundary between human and robot data falls is covered in human demonstrations vs teleoperated robot demonstrations.
What video still contributes to contact-rich tasks
Keep buying video for the parts it does well: step order, tool selection, approach trajectories, pre-grasp hand shape, visual success checks, failure examples and language. A common split is broad human video for task understanding plus a smaller, instrumented or robot-collected set for the contact phases.
Illustrative specification for a contact-rich capture
# Illustrative only — example values a buyer would replace; not an available dataset
task: connector_seating_station_09 # synthetic ID
streams:
video: wrist camera + one fixed side camera, constant frame rate
wrench: 6-axis force/torque at the tool flange; rate set from your controller
gripper: opening width and commanded state
audio: mono contact microphone near the fixture
events (labeled): approach_start, first_contact, seated, release
outcome_check: physical pull test logged pass/fail; not judged from video
calibration:
ft_bias: zeroed at session start and after every tool change
gravity_compensation: tool mass and center of mass recorded
sensor_frame: extrinsics from sensor to tool tip documented
sync: shared clock; tap test at start and end to measure video-to-force offset
excluded: fingertip tactile (not instrumented in this example)
Acceptance checks for force and tactile streams
- Zero drift between the start and end of a session stays within a limit you set.
- Saturated or dropped samples are flagged, not silently clipped or interpolated.
- The sensor frame, units and sign conventions are documented.
- A tap or impact visible in video lines up with the force trace within your sync tolerance; sensor synchronization requirements explains how to verify this.
- Outcome labels come from a physical check where one exists.
Limitations and extra preparation
- Forces measured on a human, a hand tool or a different gripper describe that system, not your humanoid; expect to re-collect or adapt.
- Instrumented tools change weight, grip and sometimes technique.
- Tactile signals are specific to the sensor design, so data from one tactile sensor may not carry over to another.
- A force/torque sensor at the wrist measures the net wrench, not fingertip contact.
- Specialist supply is scarce, and volumes are small compared with video.
Questions to ask suppliers
- Which force, torque or tactile channels are recorded, at what sampling rate and in which reference frame?
- How are bias and gravity compensation handled, and how often is the sensor re-zeroed?
- How are force streams synchronized with video, and what offset did the tap test measure?
- Does the capture device resemble your robot's end effector, or will the data need translating?
- Are task outcomes verified physically, such as by a pull test or torque audit, or judged visually?
- Are failed and partial insertions included with their force traces?
- How are sensor saturation and dropouts flagged in the delivery?
- Can raw logs be delivered in an open format alongside any processed version?
Discuss your requirements
If your task list includes insertions, fastening, pressure-controlled contact or fragile items, describe the contact phases and the sensing you need to SourceX. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also engage specialist capture partners, scoped case by case. SourceX reviews the requirement and follows up about potential sources, and a request does not guarantee a match. For other routes, see the workplace task demonstration hub and the humanoid training data guide; sensor logs held by businesses are described under sensor and IoT data.
Discuss your humanoid training-data requirements with SourceX
Sources
- RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot — contact-rich robot sequences with force, audio and actions paired with human video.
- Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots — hand-held gripper that records motion from a wrist camera.
- DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation — portable hand motion capture without force sensing.
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion — visuomotor policies trained on robot observations and actions.
- REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction — audio and robot-state streams used to explain failures.