Short answer
Fetching and returning tools from shadow boards, drawers and tool cribs is a plausible early humanoid task, and workshops and plants may hold images and records of how tools are stored. Still images support recognition and layout understanding; retrieval itself needs video of reach, grasp and return, and robot control needs robot or teleoperation data rather than footage alone.
Why this task is worth specifying carefully
Tool retrieval looks small. A technician asks for a torque wrench, someone walks to the wall, takes it off a hook and brings it back. For a humanoid it is a composite of perception, navigation, reach and grasp, with a closing step that is easy to overlook: returning the tool to the right place so the next person finds it. A shadow board is a storage layout that makes that last step checkable, because each tool has an outlined position and an empty outline signals a missing tool.
That structure helps data collection. The board gives a natural label for "which tool, which slot", and a before-and-after image pair makes retrieval and return verifiable without a person judging every clip. It also creates failure modes worth capturing: tools hung in the wrong outline, tools stacked or tangled, tools partly hidden behind others, and outlines that no longer match the tools.
Task anatomy
The steps below are a working breakdown for planning, not a standard sequence.
- Locate the storage point: wall board, drawer, cabinet, crib window or cart.
- Identify the requested tool among similar ones (several wrenches of adjacent size, or near-identical screwdrivers).
- Reach past obstacles, sometimes at shoulder height or below the waist.
- Grasp and lift from a hook, clip, magnet, foam cutout or drawer tray.
- Carry the tool to the task, possibly handing it to a person.
- Return the tool and confirm it sits in its slot.
Variations matter more than the happy path. Foam cutouts hold tools with friction, so lifting needs a particular angle. Magnetic strips hold with force that varies by tool. Hooks may require a specific orientation. A crib window means the tool is handed over by an attendant, with checkout recorded in a system the robot would not see.
What the data can support
| Data type | What it helps with | Where it stops |
|---|---|---|
| Still images of boards and drawers | Layout recognition, slot occupancy, tool identification | No motion or contact information |
| Before/after image pairs | Retrieval and return verification labels | Does not show how the tool was moved |
| Egocentric or multi-view video of retrieval | Step understanding, reach path, where hands approach | No joint states or forces |
| Checkout records from a crib | Which tools are requested together, for which jobs | No visual or motion content |
| Tool dimensions and part numbers | Matching a request to a visual instance | No behavior under handling |
For object-level labels, HANDAL offers a useful point of comparison. It reports 308k annotated frames from 2.2k videos of 212 real-world objects in 17 categories, focused on hardware and kitchen tools, with 6-DoF category-level pose, handle affordance segmentation and reconstructed meshes (Guo et al., 2023). It does not cover shadow boards or storage fixtures, so treat it only as a reference for how to capture and label graspable tool geometry. The related tool affordances and grasp annotations page covers that labeling step, and tool identification image data covers recognition.
Suppliers and what they may hold
Machine shops, maintenance workshops, plants with tool cribs, aviation and automotive service operations and training facilities may hold relevant material. Typical forms are 5S or lean audit photos of boards, inventory counts of crib items, tool-control checklists and photos of mis-stored tools. Photos taken for audits often show the full board but at a single angle, and checklists record whether a tool was present, not how it was fetched.
Video of retrieval is rarer and usually incidental: a wide camera over a work cell, a training clip. Where it exists, the board may be out of frame or at an oblique angle. Commissioned recording can be designed: cameras fixed on the board plus a head-height or wrist view, repeated retrievals across tool types, return events and deliberate misplacements. For the general trade-off, see existing footage versus commissioned recording.
Tool selection, as opposed to retrieval, is a knowledge question: which tool does this job call for. That belongs to tool-selection knowledge and draws on procedures and work orders, not storage imagery.
Capture parameters the buyer sets
- Viewpoints. A fixed frontal view of the whole board, a closer view of the slots, and a view from the retriever's perspective.
- Lighting. Workshop lighting often shines off metal; ask for samples at the actual times of day.
- Resolution. Enough to read outline labels and tool markings at the distance used.
- Variety. Multiple boards, several tool sets and different states of tidiness.
- Return events. Both correct returns and mis-returns, labeled as such.
- Calibration. If a robot will later reuse the footage for geometry, fixed-camera intrinsics and extrinsics must be recorded.
Label fields could include slot ID, tool class, occupied/empty state, tool-in-wrong-slot flag, occlusion level and event type (retrieve, return, handover). Condition information, such as worn or damaged tools, is a separate topic in tool condition recognition.
Illustrative package
# Illustrative only - synthetic values, not an available dataset
package: tool_retrieval_v0
sites: [site_017, site_021]
storage_types: [shadow_board_wall, drawer_foam, crib_window]
images:
board_state: before/after pairs per retrieval event
video:
views: [board_frontal_fixed, retriever_head_mounted]
sync: shared_clock
labels:
slot_id, tool_class, occupancy, wrong_slot_flag, occlusion_level
events: [retrieve, return, handover, misplacement]
metadata: board_layout_id, lighting_note, camera_extrinsics_file
excluded: joint states, grasp forces, robot actions
rights: training and internal evaluation; staff pseudonymized
What this data cannot support
- Robot grasping and carrying. Learning to lift a tool from a foam cutout or a magnetic strip needs robot or teleoperation data. Images, tool catalogs and recognition labels alone do not teach how a gripper should approach, and see object recognition versus manipulation data.
- Contact behavior. How much force releases a tool from magnetic hold is invisible in video.
- Reasoning about need. Knowing which tool a task requires is a separate knowledge problem.
- Inventory accuracy. Audit photos record a moment, and the board may have been tidied beforehand.
- Generalization to other boards. A model trained on one outline style may misread another.
Keep evaluation boards separate from training boards. If both come from the same site with the same layout, the evaluation tests memorization of a wall, not retrieval.
Permissions and provenance
Storage photographs can capture employee badges, whiteboards, customer parts or proprietary tooling designs. Ask who owns the images, who may appear in them and whether any customer-specific or export-controlled items are visible. Blurring does not by itself settle permission questions, and it may remove content the model needs. This is not legal advice.
Any measurement of how often individual workers visit the crib is a separate monitoring question; keep the evaluation of robot perception apart from it.
Questions to ask suppliers
- What storage types are in use (shadow boards, drawers, foam inserts, magnetic strips, cribs), and how many boards of each?
- Are there before/after images or only single snapshots?
- Does any video show retrieval and return, and from which viewpoint?
- Are tool IDs, slot IDs or crib checkout records available to link to images?
- How are mis-stored or missing tools documented?
- Can recording be scheduled with fixed cameras and a consistent lighting setup?
- Which tools or fixtures are proprietary, customer-owned or restricted from imaging?
- Who appears in the images or video, and what notice did they receive?
- Who can license the material and under what terms?
Discuss your requirements
If you are defining a tool-retrieval task, describe the storage types, tools and permitted uses to SourceX. SourceX reviews the requirement, investigates whether US businesses hold suitable material, checks the data and the supplier's licensing permissions, and follows up about potential sources. If no existing asset fits, SourceX can scope a commissioned collection of new recordings. A request does not guarantee a match. For orientation, see the humanoid training data guide and the maintenance data hub; the hand tool use demonstrations page covers what happens after the tool is fetched.
Describe the workplace demonstration data your robotics project needs
Sources
- HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructions — tool pose, affordance and mesh annotations; a capture and labeling reference only, not a storage dataset.