Privacy and preparation
What a physical AI tool change dataset teaches that repeated picks cannot
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A repeated pick shows one grasp done many times. A tool change shows a sequence with preconditions, a state change and a check: park the old tool, take the new one, confirm it is seated, update offsets, resume. Recorded across tools, reasons, starting states and outcomes, including retries and interventions, tool-change video teaches ordering, verification and recovery that pick footage cannot, provided each clip carries its energy-control status and worker consent.
Video and sensors
Physical AI tool change
Key takeaways
- A pick varies the object; a tool change varies the state of the machine, the tool and the job, which is the part models find hard.
- Record a tool change only when it alters tooling, constraints or the next step; quick-swap gripper fingers that change nothing downstream are closer to picks.
- The task variation matrix lists nine axes; an empty cell is a finding to report, not a gap to hide.
- Every clip needs an outcome label (clean, retry, intervention, abort) and its sub-step boundaries, or the recovery value is lost.
- Tool changes are often servicing under OSHA's lockout/tagout rule, so record which energy-control regime applied and never stage an unsafe change for the camera.
- Hands and faces appear in most tool-change footage; consent, notice and redaction decisions travel with each clip.
What can tool-change demonstrations add beyond repeated pick actions?
They add sequence, state and verification. A pick-and-place clip usually answers one question, where to grasp this object, and repeating it a thousand times mostly varies the object and its pose. A tool change answers several linked questions: is the current tool free to release, where does it go, which tool comes next, did the new one seat and lock, and what has to be updated before work resumes.
Each of those steps has a visible precondition and a way to fail. On a robotic changer from one vendor, ATI, the robot-side master plate locks to the tool plate with pneumatic locking mechanisms and utilities such as air and signals pass to the tooling only once the plates are coupled. A manual change on a CNC spindle, a press brake or a filling line has the same shape with a person doing the steps: clear the work area, release, swap, seat, check, record.
That structure is what pick datasets lack. A model trained on picks learns where things go; a model trained on tool changes can learn that steps happen in order, that a check gates the next step, and what to do when the check fails. The rest of this page is about recording that value instead of losing it in an unlabeled video archive.
Which tool changes are worth recording, and which are just more picks?
Keep a tool change when it alters the tooling, the constraints or the next step of the job. Drop or downgrade it when nothing downstream depends on it. The test is simple: if you deleted the clip's last ten seconds, would a viewer still know whether the change succeeded and what happened next? If the answer is no, the clip needs its outcome recorded before it is worth anything.
Manual and robotic changes are both worth keeping, but do not mix them in one count. A person changing an insert and a robot docking a gripper share the release, seat and verify pattern, yet the signals differ: a robot leaves controller events, while a person leaves only what the camera and the work order show. Report them as separate rows so a buyer can see how much of each exists.
| Change type | What it records beyond a pick | Typical failure worth capturing | Keep or downgrade |
|---|---|---|---|
| Robot automatic tool changer swap (gripper to screwdriver, welder to probe) | Docking approach, lock confirmation, utility handoff, new tool frame | Misaligned approach, lock not confirmed, tool left in wrong stand position | Keep, with controller log references |
| Manual CNC tool or insert change | Release, seating, offset or length update, first-part check | Wrong offset entered, insert seated loosely, chip build-up on taper | Keep, with the offset and first-part result |
| Changeover between jobs (dies, nozzles, fixtures) | Multi-step plan with parts staged, verified and signed off | Missing part found mid-change, setup sheet out of date | Keep, split into sub-steps |
| Breakage-driven change | Detecting the fault, stopping safely, recovering the job | Broken tool not noticed until scrap appears | Keep with priority; usually the rarest cell |
| Quick-swap gripper fingers with no downstream change | Little beyond a grasp and a place | Dropped finger set | Downgrade; treat as pick footage |
The task variation matrix for tool use video
The matrix is the asset to fill before export. Each row is a way tool changes differ in real operations; the levels are what a buyer would expect to see represented; the last column is what a clip must carry for that row to count. Fill it with counts from your own footage, one count per level, and keep the zeros visible.
Use it twice: once to decide what to keep from footage you already have, and once as a collection plan for the gaps. Nine rows is deliberate. Fewer misses the reasons and outcomes that make tool changes useful; many more turns the matrix into a taxonomy nobody fills.
| Variation axis | Levels to cover | How to record it | Minimum evidence per clip |
|---|---|---|---|
| Tool pairing | From-tool and to-tool, by family and specific ID | Tool IDs from the tool crib, holder or changer stand position | Both IDs, not just the incoming tool |
| Reason for change | Scheduled, wear limit, breakage, job changeover, trial | Reason code chosen by the operator or taken from the work order | One code; free text optional |
| Starting state | Tool in holder, stuck or seized, wrong pocket, missing | Annotator label from the first seconds of the clip | Label plus timestamp |
| Who performs it | Robot, operator, operator assisting robot | Cell type and role code, never a name | Role code and cell ID |
| Energy-control regime | Full lockout, minor servicing with alternative protection, robot cell interlock | Supervisor confirmation per clip or per procedure version | Regime code and procedure reference |
| Viewpoint and visibility | Overhead, wrist or head camera, side; occluded or clear | Camera ID with mount, intrinsics file, occlusion flag | Camera ID and sync source |
| Workpiece context | Part family, material, first part after change | Job or routing reference, tokenized | Job reference and first-part result |
| Outcome | Clean, retry, intervention, abort | Sub-step boundaries plus final status | Outcome code and end timestamp |
| Worker visibility | Hands only, face visible, badge or screen visible | Redaction status and consent reference | Visibility flags and consent ID |
The per-clip input dictionary that makes the matrix checkable
A matrix built from guesses is worse than none, so the counts must come from fields stored with each clip. Robot learning teams commonly receive data in episode-and-step formats. RLDS, a format from Google Research, stores each episode as a sequence of steps with is_first and is_last flags, an optional is_terminal flag, and user-defined episode metadata. It treats an episode that ends without a terminal step as truncated, and it requires every step in a dataset to carry the same fields.
At a glance
The per-clip input dictionary that makes the matrix checkable
Those two rules matter for tool changes. A clip that stops before the lock is confirmed is a truncated episode, not a failure, and should be labeled that way. And if one camera or log is missing for some clips, the field still has to exist with an explicit missing value. The dictionary below is a minimum that maps cleanly onto episode metadata and per-step fields.
- episode_id: stable, unique across your sites, no personal names.
- task_text: a short instruction, for example swap the 10 mm end mill for the 6 mm ball mill in cell 2.
- tool_from and tool_to: crib or stand IDs.
- reason_code, starting_state, regime_code and procedure_ref, as in the matrix.
- substeps: ordered list of release, store, select, approach, couple or seat, verify, update offsets, resume, each with start and end times.
- cameras: ID, mount, intrinsics file reference and the clock each was synced to.
- logs: controller, PLC or robot program references with the same clock, or not recorded.
- outcome: clean, retry, intervention or abort, plus truncated if the clip ends early.
- intervention: who stepped in by role code, at which sub-step and why.
- visibility and consent: hands, face, badge and screen flags, redaction status and consent ID.
Where should annotators put sub-step boundaries on a manual change?
At the moments when something becomes true that was not true before, not at fixed time intervals. Released means the old tool is no longer held by the spindle, chuck or changer; seated means the new tool is in its working position; verified means the operator or controller has checked it, by a touch-off, a probe cycle, a lock signal or a first-part measurement. Those state changes are what a model needs to align with, and they are visible on video even when hands obscure the detail.
Write the boundary rules down before annotation starts and test them on ten clips with two annotators. Where they disagree by more than a second or two, the rule is ambiguous, not the annotators careless. Typical fixes are naming the camera that decides each boundary and saying what to do when the event is occluded: mark it occluded with the last visible time, rather than guessing.
Robot changes are easier because the controller log usually records lock and unlock commands. Even then, keep the video boundary and the log time side by side. A gap between the commanded lock and the visible seating is exactly the kind of evidence a recovery model learns from, and it disappears if one source silently overrides the other.
Which changeovers and recoveries does your footage never show?
Usually the reasons and the failures. Footage recorded for training or audit tends to show scheduled changes done well, because that is when someone remembered to start the camera. Breakage-driven changes, seized tools, wrong-pocket picks and retries happen at bad moments and are rarely on video, yet they are the cells that teach recovery.
Large research datasets show why breadth is not the same thing. DROID reports about 76,000 demonstration trajectories across 564 scenes and 86 tasks, and Open X-Embodiment pools data from 22 robots demonstrating 527 skills. Counts like these describe how many scenes and skills a collection spans. They do not tell you whether a specific changeover, its seized-tool variant and its recovery appear at all. Your matrix answers that question for your own footage, which is why buyers ask for it.
Run three checks before you call coverage adequate. Count clips per outcome level and per reason; any level at zero is reported as zero observed. Check that sub-step boundaries exist for at least the clips with retries or interventions, since those are the ones a model learns recovery from. And sample a few clean clips to confirm they really were clean, because a retry that happened off camera turns a failure into a false success.
Which assumption changes the decision in a worked shop example?
The example below is illustrative. Every count is invented to show the method and does not describe a real company. A job shop with three CNC cells and one collaborative robot with an automatic changer reviews two weeks of overhead and wrist-camera footage and finds 40 tool changes with usable views.
Filled in, the matrix shows 31 scheduled or wear-limit changes, 7 job changeovers, 2 breakage-driven changes and no seized-tool starts. Outcomes are 34 clean, 4 retries, 2 interventions and no aborts. Three clips are truncated before the first-part check. Faces are visible in 9 clips, all from one camera angle.
The assumption that changes the decision is whether the buyer wants recovery or routine. If the goal is routine sequencing, the 37 clips with complete sub-steps are a reasonable pilot. If the goal is recovery, 6 retry or intervention clips and 2 breakage changes are too thin to promise anything, and the honest output is a coverage report that says so, plus a plan to switch cameras on automatically when a breakage alarm fires. Uncertainty is shown as counts per cell and an explicit zero observed, never as a smoothed percentage that hides how few examples sit behind it.
Which tool changes need an energy-control note before anyone films them?
Most of them. OSHA's lockout/tagout standard defines servicing and maintenance to include making adjustments or tool changes where an employee may be exposed to unexpected energization or startup. Minor tool changes during normal production fall outside the standard only when they are routine, repetitive and integral to production and alternative measures give effective protection. In a 2002 interpretation letter, OSHA treated changing carriers and tools on a packaging machine as servicing and maintenance unless an exception applied.
For a dataset, that means two practical rules. First, record which regime the change followed, by procedure reference, so a buyer never trains on footage that shows a hazardous shortcut as normal behavior. Second, never set up a tool change for the camera that departs from the written procedure. If existing footage shows a guard bypassed or a lockout skipped, hold it back and route it to your safety lead; it is not training data. Which regime applies to a given machine is a call for your safety function, not for the data team.
Hands, faces and badges: whose consent a tool-change clip needs
Assume a person is visible unless the clip proves otherwise. Manual changes show hands at close range, often faces and badges, and sometimes a screen with a name or a work order. Record visibility flags per clip and decide redaction, consent and notice before export, not after a buyer asks.
State law can turn a hand into more than a hand. Illinois's Biometric Information Privacy Act defines a biometric identifier to include a scan of hand or face geometry, while excluding photographs. Whether a given use of footage involves such a scan is a legal question for counsel, but it is a good reason to document how video will be processed, where workers are located and what notice they received.
Keep the distinctions straight. Owning the cameras does not settle whether workers agreed to training use, and blurring faces does not by itself create a right to license the footage. Access, privacy permission, ownership and contractual licensing rights are separate questions, and each clip should point to the record that answers each one.
Offering tool-change footage through SourceX
If the matrix shows real coverage, the footage may be worth licensing. For a manufacturer, SourceX handles rights review, contracting, preparation and delivery of a licensed video dataset built from selected footage, and the company approves the scope and signs before any of it moves. Large video archives stay in your own storage or ship on encrypted drives, because SourceX does not host multi-terabyte datasets.
The starting point is a short company assessment that asks about your operations and records, with nothing uploaded. Bring the filled matrix: it tells a reviewer within minutes whether the footage holds routine sequences, recovery behavior, or both.
Frequently asked questions
Is a tool change dataset just a longer pick-and-place dataset?
No. A pick varies the object and its pose, while a tool change varies the state of the tool, the machine and the job, with checks that gate each next step. The useful signal is in the ordering, the verification and the recoveries, which pick footage rarely shows.
How many tool-change clips does a buyer need?
There is no fixed number; it depends on whether the goal is routine sequencing or recovery. A coverage report with counts per reason, starting state and outcome is more persuasive than a total, because it shows where the examples are thin. Report empty cells as zero observed.
Do failed or retried tool changes have value?
Yes, often more than clean ones, as long as the sub-steps and the outcome are labeled. A retry shows what a failed check looks like and what the operator or robot did next. Clips that show unsafe shortcuts are the exception and should be held back for safety review.
Should we film new tool changes specifically for AI training?
Only within your written procedures and with notice and consent in place. Scheduling cameras around changeovers or triggering them on breakage alarms can fill gaps in the matrix. Never alter a procedure to make a better shot.
What metadata matters most if we can only add a little?
Tool from and to, reason code, outcome, sub-step boundaries and the energy-control regime. Without these, a buyer cannot tell a routine change from a recovery or a safe change from a shortcut. Camera sync and consent references come next.
Sources
- OSHA's lockout/tagout standard defines servicing and maintenance to include making adjustments or tool changes where employees may be exposed to unexpected energization, and excludes minor tool changes during normal production only under stated conditions. Source
- A 2002 OSHA interpretation letter treated changing carriers and tools on a packaging machine as servicing and maintenance unless an exception applied. Source
- ATI's QC-1310 manual describes the master plate locking to the tool plate with pneumatic locking mechanisms and utilities passing to the tooling once coupled. Source
- RLDS stores episodes of steps with is_first, is_last and optional is_terminal flags, treats non-terminal endings as truncation and requires all steps in a dataset to share fields. Source
- The DROID paper reports about 76k demonstration trajectories across 564 scenes and 86 tasks. Source
- The Open X-Embodiment paper reports data from 22 robots across 21 institutions demonstrating 527 skills. Source
- Illinois's Biometric Information Privacy Act defines a biometric identifier to include a scan of hand or face geometry and excludes photographs. Source
See if your company qualifies
A short company assessment. No data uploads are needed.