Tool Pose Tracking Annotation in Video

Why tool trajectories matter

A person's hand is a poor stand-in for a robot's end effector, because humans grasp the same tool many ways. The tool, however, is a rigid object whose motion matters regardless of who holds it. A screwdriver's tip path, a wrench's rotation about a fastener axis and a scraper's sweep are properties of the tool in the world. Tracking them gives an embodiment-neutral description of what the demonstration did, which is why trajectory labels on human video are of interest to teams studying cross-embodiment transfer.

A tool track does not by itself make video into robot training data. Mapping a tool trajectory to arm and hand joint commands needs retargeting, a kinematic model of the robot and, for contact-rich work, force information. See what human demonstration video can and cannot do. This page is about specifying the annotation, not about whether to buy it; neighboring tasks are mapped in the maintenance cluster hub. The tool affordances and grasp annotations page covers where on the tool to grasp.

Reference points from published work

Two public efforts illustrate the range of what is feasible, at very different cost.

HANDAL annotates real-world manipulable tools and hardware with category-level 6-DoF pose and scale, handle affordance segmentation and 3D reconstructed meshes. The authors report 308k annotated frames from 2.2k videos of 212 objects in 17 categories, collected from a single off-the-shelf camera with semi-automated processing (Guo et al., 2023). This route is cheap to capture but is about static or lightly moved objects, not busy task video.

TACO targets bimanual tool-action-object understanding. It contains 2.5K motion sequences across tool-action-object combinations, captured with 12 third-person views plus one egocentric view, and provides hand-object 3D meshes, 2D segmentation and behavior-triplet annotations, using multi-view sensing combined with optical motion capture (Liu et al., 2024). That is a laboratory-grade pipeline, which shows what precision is possible when the capture is designed for it.

Neither dataset was recorded in workplaces doing maintenance, so treat them as annotation and capture references. A buyer working with ordinary workplace video should expect less precision and set acceptance thresholds accordingly.

Choosing a tracking approach

Choose the cheapest approach that meets the decision you need to make. If you only want step segmentation, keypoints suffice. If you want to compare a human tool path with a robot's, you need real calibration and a metric scale.

Illustrative annotation specification

This specification is synthetic. Replace each value with your own requirements; none is a recommended minimum.

# Illustrative only
video:
  views: [egocentric, fixed_1]
  fps: buyer_defined
  intrinsics: per-camera calibration file or declared unknown
  extrinsics: rig_calibration_or_none
tools:
  classes: [screwdriver_A, wrench_B, scraper_C]  # synthetic
  models: provided_mesh | category_template | none
per_frame_labels:
  tool_pose: {rotation: quaternion, translation: meters, frame: camera}
  tool_keypoints: [tip, axis_end, handle_end]
  visibility: visible | partial | occluded | out_of_frame
  hand_contact: {left: bool, right: bool}
  contact_with_workpiece: bool
  confidence: float 0-1
per_clip_labels:
  tool_id, step_label, operator_id (pseudonymous), outcome
quality:
  double_label_fraction: buyer_defined
  gold_set: 50 frames per tool class   # arbitrary example
  disagreement_log: required
delivery: [json, video_overlay_samples, annotation_guideline_pdf]

The gold set size above is an arbitrary example, not a minimum.

Acceptance tests

Define these before the supplier starts and apply them to a pilot batch.

• Reprojection check. Project annotated poses onto the frame and measure agreement with an independent reviewer on a sampled subset. Set the tolerance in pixels or millimeters based on your decision, not on a vendor default.
• Temporal consistency. Look for jitter or flips that a moving rigid tool cannot produce. Rotation ambiguity about symmetric axes is a common source.
• Occlusion honesty. Frames where the tool is hidden should be marked as hidden, not filled with a guess. Ask what share of frames each visibility class represents.
• Inter-annotator agreement. Request double-labeled clips and a report of where reviewers differed.
• Scale check. If metric values are claimed, verify against a known dimension in the scene.
• Held-out evaluation. Keep a subset of sites, operators or tools out of any model training, and label it by the same protocol.

Limits: what tool tracks do not provide

• No joint commands, no force. A tool pose says where the tool went, not what torque was applied or how it was held. Tracks support recognition and pose estimation; training manipulation control still needs robot or teleoperation data.
• Hidden at the crucial moment. Contact, engagement and cutting often happen where cameras cannot see.
• Monocular depth ambiguity. Single-camera pose has scale and depth uncertainty unless calibrated or anchored.
• Per-tool effort. Mesh-based pose needs a model per tool; category-level methods lose precision.
• Rights in the underlying video. Annotation does not create rights. A partner can label video the buyer is entitled to use, but cannot grant permission for footage it does not control. This is not legal advice.
• Embodiment. The path a human takes to move a tool may not be reachable by a humanoid.

Questions to ask suppliers

1. Which pose representation do you propose, and in which coordinate frame?
2. How are tool models obtained, and who owns them?
3. How do you label occlusion, and what share of frames is typically hidden?
4. What camera calibration do you require, and what happens when it is missing?
5. What reviewer process and agreement metrics do you apply?
6. Can you show a pilot batch on my footage before committing to a full run?
7. How are hand and tool identities kept consistent when tools are swapped?
8. What happens to the source video and my guidelines after delivery?
9. Who retains rights in the labels?

Discuss your requirements

If you are specifying tool tracking labels, describe the video, tool classes and annotation outputs to SourceX. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation and engage specialist capture partners, scoped case by case. A request does not guarantee a match, and SourceX follows up about potential sources. See the humanoid training data guide for the wider map and the hand tool use demonstrations page for the underlying video, and consider instrumented tools if you need direct force and motion measurement. The catalog category for expert annotations and labels is related. For constrained access settings see tool use in tight spaces.

Describe the workplace demonstration data your robotics project needs

Sources

• HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructions — category-level pose, affordance and mesh annotation from a single camera.
• TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding — multi-view bimanual tool-action-object capture with hand-object meshes and behavior triplets.