Tool Selection Knowledge: Matching Tools to Fasteners and Tasks

Choosing the tool is a separate step

Before a technician loosens a fastener, a decision has already been made: a hex key or a Torx bit, which size, whether a ratchet fits or only a short driver will clear the surrounding frame, whether a torque tool is required. A robot that can grasp a screwdriver but picks the wrong one fails before it starts. Selecting a tool is a reasoning step over fastener type, size, access, material and the procedure's requirements.

This page covers sourcing the knowledge that supports that decision. Recognizing tool images is covered in tool identification image data, and using a tool in hand tool use demonstrations. It sits in the maintenance cluster of the humanoid data guide.

Where tool-selection knowledge lives

Workshops and documentation teams hold the first four. The last two are usually built. Documentation produced to the S1000D specification stores procedural data in stand-alone data modules, each identified by a unique data module code in a common source database (ASD, S1000D); where a supplier uses it, a tool requirement can be read from a module and joined to a task rather than scraped from prose. See S1000D and ATA documentation for access questions.

What an annotation layer adds

Raw documents list tools but seldom explain choices. Annotation turns them into a mapping a model can learn from. Published object datasets show the style of labeling involved. TACO annotates bimanual sequences as tool-action-object triplets, with 2.5K motion sequences, 12 third-person views plus one egocentric view, hand-object 3D meshes and 2D segmentation (Liu et al., 2024). HANDAL labels 212 real-world objects in 17 categories, including hardware such as pliers and screwdrivers, with category-level pose and handle affordance annotations (Guo et al., 2023).

Those datasets concern tool use and object pose. They do not label why a tool was chosen for a given fastener, so a buyer needing selection reasoning typically commissions that annotation layer over documents or footage the supplier already holds. Annotation work done for a buyer is usually work-for-hire and carries low rights risk of its own, but it gives no rights in the underlying documents or footage. Those must be licensed from the holder.

Illustrative mapping record

# Illustrative only - synthetic values, not an available dataset
mapping_id: tsel_0213
task: {name: remove_access_panel, procedure_ref: proc_0058, step: 3}
fastener: {drive: torx, size: T25, count: 8, head: pan, location: top_edge}
access: {clearance_mm: 40, constraint: "adjacent duct limits full swing"}
tool_choice:
  primary: {type: driver_handle_plus_bit, bit: T25, length_class: standard}
  acceptable_alternatives: [{type: ratchet_bit_holder, condition: "clearance above 60 mm"}]
  disallowed: [{type: impact_driver, reason: "panel material damage"}]
  torque_requirement: {required: false}
evidence: {source_doc: doc_0412, image_ref: img_9021, annotator: ann_04, review: second_pass}

Adapt the schema to the model you train. A text-only planner needs fastener, access and choice. A visual system needs image references with tool and fastener boxes, and a robot-in-the-loop evaluation needs tool inventory that matches what the robot can carry.

What this data cannot support

• Tool handling. Knowing a T25 bit is needed does not teach grasp, alignment, engagement or the feel of a stripped head. See tool affordances and grasp annotations and power tool use demonstrations.
• Force and torque. A required torque value is a number in a document, not a measurement. See torque specs and fastening procedures.
• Robot constraints. Choices made by a human technician assume human hands, reach and strength. A humanoid may have a different tool set and different clearance limits; selection labels need re-checking against the robot's end effector.
• Unseen equipment. Mappings are specific to fastener and equipment types in the source. Coverage of one workshop does not extend to another trade.
• Tacit judgment. Experienced technicians substitute tools in ways no document records. Documentation-only data misses those substitutions.

Quality and acceptance checks

• Check that each mapping names tool type and size, not just a family such as "screwdriver".
• Check that mappings carry access context. A choice that holds in open space may be wrong in a tight enclosure.
• Sample and re-label to measure annotator agreement, and record the guideline version.
• Keep tool names in a controlled vocabulary, with a list of synonyms ("spanner" and "wrench").
• Hold out whole equipment types or workshops for evaluation rather than random rows.

Using the mapping in a robot system

A tool-selection layer usually sits between a task planner and a manipulation policy. The planner reads the procedure step and the fastener, the mapping suggests the tool, and the robot then fetches it or confirms it is already attached. This has two data consequences.

First, the mapping must be expressed in the robot's tool vocabulary. If the humanoid carries a fixed set of end-effector attachments, human choices that depend on a wrench set it does not have need either substitution rules or exclusion. Second, an evaluation should test selection separately from execution, so a wrong choice is not hidden by a good grasp or the reverse.

Where fastener and part data come from catalogs, exploded views and parts catalogs can supply part identifiers and drive types to anchor the mapping. Selection data stays text-and-image knowledge; execution quality has to be established with demonstration or robot data.

Questions to ask suppliers

1. Which trades and equipment types do the job cards or procedures cover, and how many distinct fastener types appear?
2. Are tools recorded by type and size, or only by family name?
3. Is the mapping taken from documents, observed use, or technician judgment, and how is each marked?
4. Do records include alternatives and disallowed tools, or only the preferred one?
5. If documentation uses a structured standard, can tool requirements be extracted from the data modules?
6. Who wrote the documents, and can the supplier license them for training use?
7. What guidelines, gold sets and audits govern the annotation, and what agreement was measured?
8. Do records include images of the fastener and tool in situ, and under what lighting and viewpoint?
9. Which parts come from customer-owned equipment, and what restrictions follow?

Discuss your requirements

If you need tool-selection data for a defined set of tasks, describe the tasks, fastener types and formats to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. SourceX can arrange annotation, scoped case by case. A request does not guarantee a match, and nothing is contracted until a supplier agrees. The catalog entry for expert annotations and labels describes the related record type.

Explore sourcing and licensing options with SourceX

Sources

• TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding — Published bimanual tool-action-object dataset showing how tool-action-object triplets are annotated.
• S1000D International specification for technical publications using a common source database — Specification whose data modules carry procedural data that can name required tools.
• HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructions — Published object-pose dataset covering hand tools such as pliers and screwdrivers.