Short answer
Instrumented tools add force, torque and pose sensing to the tools people already use, so a recording shows how hard and in which direction a task was performed, not only what it looked like. It needs hardware, calibration, synchronization and per-episode QA from a specialist collection provider, and the forces describe a human grip, not a robot's.
Why instrument the tool
Video of someone tightening a fastener or scraping a gasket shows motion but not effort. Whether a bolt was seated, a seal was crushed or a blade was angled correctly is often decided by force that no camera sees. Robot data can include force, as the RH20T dataset does with its 110,000-plus contact-rich sequences carrying visual, force, audio and action data, each paired with a human demonstration video and a language description (Fang et al., 2023). But robot collection is slow and tied to one embodiment.
Handheld capture sits between the two. It uses a person's skill and speed, and the sensing is mounted on the tool, not the robot. DexCap shows the portable-capture idea for hands: a rig that tracks hand motion with SLAM and electromagnetic-field tracking, with a session limited to about 40 minutes by power and, notably, no force sensing (Wang et al., 2024). An instrumented tool fills part of that gap. It is a specialist build, so a buyer will generally need to commission it from a provider that already has the hardware; the tradeoffs against other routes are covered in human demonstrations vs teleoperated robot demonstrations.
What it can and cannot capture
| Quantity | Instrumented tool can capture | Notes |
|---|---|---|
| Force and torque at the tool | Yes | Depends on sensor placement and range |
| Tool pose and motion | Yes, with tracking | Needs external or inertial tracking; see tool pose tracking |
| Hand posture | Only with added tracking | Separate glove or camera system |
| Grip force on the handle | Only with handle sensors | Often omitted |
| Robot joint commands | No | Human data; no robot action space |
| Force a robot would need | No | Different hand, compliance and friction |
Illustrative specification
This is a starting template, with synthetic values, for a request to a specialist provider. Adjust every value to your architecture and task.
# Illustrative only - values are synthetic examples, to be set by the buyer
capture_id: instrumented_tool_v0
tool: handheld_driver_class_A (buyer-specified), 3 interchangeable tips
sensing:
force_torque: 6-axis sensor between handle and tool body; range set from expected loads
sample_rate_hz: buyer-set, high enough to resolve contact transients
tool_pose: 6-DoF, optical or inertial, logged at the same clock
video: 2 fixed + 1 head-mounted, constant fps, shared timecode
calibration:
per_session: zero offset with tool unloaded in two orientations
gravity_compensation: tool mass and center of mass logged
extrinsics: camera-to-tool transform recorded at session start
episodes:
structure: start_pose, approach, contact, task, release
labels: [task_id, fastener_id, material, outcome, operator_id]
metadata: [site_id, bench_height, lighting, tool_serial, firmware]
qa:
drift_check: unloaded reading after each episode
saturation_flag: any sample at sensor limit
sync_check: clap or LED event visible in video and logged signal
Parameters the buyer must decide
- Sensor position. A sensor between handle and tool body measures the loads the person applies, but adds mass and changes balance. A sensor near the tip is closer to the interaction but is more exposed to damage.
- Force range and resolution. Choose from expected loads, with margin, and record the sensor's range with the data.
- Synchronization. Force streams and video must share a clock or a measured offset. Ask how offset is estimated and checked per episode.
- Frame definitions. Every wrench needs a stated frame (sensor frame, tool frame, tip frame) and a stated sign convention.
- Task and material coverage. Torque on a fastener depends on thread, material and lubrication; log these.
Requirements for fastener tasks overlap with torque specs and fastening procedures and, in manufacturing, screw-driving and torque-controller data.
Acceptance tests
Define tests before the first shoot, and have the provider report results for every session:
- Zero-offset drift stays within a buyer-set bound across an episode, measured by an unloaded reading.
- Synchronization error between force events and video events stays below a buyer-set bound on a sampled set of episodes.
- Calibration traceability: each session links to a calibration record and sensor serial.
- Saturation rate is reported, and saturated episodes are flagged.
- Label audit: a sample of episodes checked for task, fastener and outcome labels by a second reviewer.
- Coverage report by operator, tool tip, material and site, with a held-out split defined in advance.
Limitations and additional preparation
The forces are a person's. A human wrist, grip and tool compliance shape the signal, and a robot hand will not reproduce them. The data can serve as a target for force profiles, reward shaping, or evaluation of what "enough" looks like; it does not amount to robot actions. Using it for control requires retargeting or co-training with robot data, and published work reports such human data as helping, not replacing, robot data.
Instrumentation changes behavior. A heavier, cabled tool feels different, and people adapt. Record a comparison set with the unmodified tool to quantify the shift, and keep the cable and sensor out of the camera's view where possible.
Business context is thin. A commissioned bench session shows controlled tasks, not the shop-floor interruptions and awkward access of real repair. Combine it with hand tool use demonstrations recorded in working conditions. Operators need to consent to recording and to the use of their body motion and voice data; see worker consent and identifiable people.
Where instrumented capture fits among other routes
Instrumented tools suit questions about effort: how much torque seats this fastener, how much pressure a scraper needs, how force changes as a part seats. They suit them better than video alone, and better than robot teleoperation when the aim is a clean record of skilled human technique.
They suit other questions less well. If the goal is end-to-end behavior on your robot, teleoperated or robot-recorded data matches your action space directly. If the goal is breadth over many workplaces, existing video is cheaper. A common arrangement combines them: a smaller instrumented set provides force targets and calibration, while video from working sites provides breadth. Define how the two will be linked, for instance through shared task IDs and fastener types, before collection starts.
Plan for a pilot of a few sessions with a go/no-go review against the acceptance tests. Pilots expose problems in mounting, drift and labeling cheaply, and they let the provider and host business confirm access and consent arrangements.
Instrumented capture can also extend to drivers and drills; see power tool use demonstrations for what plain video of those tools does and does not show.
Questions to ask suppliers
- What sensor model, range and resolution will be used, and where is it mounted?
- How are force, pose and video synchronized, and how is the offset verified per episode?
- What is the calibration procedure, and how often is it repeated?
- How is gravity and tool mass compensated, and what frames and sign conventions are used?
- Can you provide a comparison set with the same tasks using the unmodified tool?
- How many operators will be recorded, and what are their experience levels?
- How are failed attempts and saturated episodes handled?
- Who owns the tool design and the raw signal files, and what do operators consent to?
- What host site is used, and does the host business need to approve recording?
Discuss your requirements
If you want a commissioned force-and-video capture of tool use, describe the tools, tasks and sensing you need to SourceX. SourceX reviews the requirement and follows up about potential sources; if no existing asset fits, SourceX can scope a commissioned collection. A request does not guarantee a match. For wider context see the maintenance data hub, the humanoid training data guide, and the catalog entry for sensor and IoT data.
Discuss your humanoid training-data requirements with SourceX
Sources
- RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot — contact-rich robot sequences with force data paired with human demonstration video.
- DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation — portable hand motion capture with no force sensing and a session limit from power.