Skip to content

Privacy and preparation

Physical AI maintenance demonstration data: which exceptions to capture

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Physical AI maintenance demonstrations should capture the exceptions that clean demos leave out: seized fasteners, wrong parts, stored energy found at verification, blocked access, revised diagnoses, failed tests with rework and stop-work calls. Each one needs a consenting worker, synchronized cameras, a linked work order that confirms the exception and an outcome field, all set in a written collection brief before recording starts.

Video and sensors

Physical AI maintenance

This page covers physical AI maintenance demonstration data: which exceptions to capture.Illustrative

Key takeaways

  • The value of maintenance footage is in deviations and recoveries, so set minimum counts for each exception type before collection starts.
  • Every exception on video should be confirmed by a second record such as a parts transaction, revised failure code or rework labor entry.
  • Energy isolation and verification are part of the episode under OSHA's lockout/tagout definition of servicing and maintenance, not setup to trim.
  • Worker consent, company ownership of the footage and the right to license it are three separate questions with separate sign-offs.
  • Footage of unsafe practice is withheld from a demonstration set and routed to safety review, not relabeled as a negative example.
  • The dataset card should name what the cameras could not see: off-frame work, tacit cues, force data and long-term repair outcomes.

Which maintenance exceptions should physical AI demonstrations capture?

Capture the moments where the job stops going to plan: a fastener that will not break loose, a part that does not match the work order, stored energy found during lockout verification, blocked access, a diagnosis revised halfway through, a functional test that fails and forces rework, and a stop-work call. A demonstration set of only clean, start-to-finish repairs teaches a system the happy path and nothing about recovery.

Each exception is only useful if the footage is tied to evidence that it really happened and to what the technician did next. That means a consented, task-level episode with synchronized cameras, the linked work order, and an outcome field that says whether the equipment returned to service. The brief below is how a site specifies that package before anyone presses record.

This page is written for the maintenance, operations and safety leads who would host such a collection, and for the robotics teams who would scope it with them. It assumes the work is real production maintenance, not a staged task in a lab.

Why maintenance work breaks the clean-demo pattern

Maintenance is a poor fit for the tidy pick-and-place loops that dominate many robot demonstrations, because the task is defined by deviation. The equipment arrived broken or due, the technician has to find out what state it is in, and the plan changes as evidence appears. The interesting behavior is the judgment between steps, not the steps themselves.

The regulation that governs much of this work already treats it as distinct from production. OSHA's lockout/tagout standard defines servicing and maintenance to include activities such as adjusting, inspecting, modifying, lubricating, cleaning, unjamming and tool changes where a worker could be exposed to unexpected energization or release of stored energy. For a demonstration dataset, that means energy isolation, verification and release are part of the episode, not overhead to be trimmed from it.

Maintenance also already has a data vocabulary. ISO 14224, written for the petroleum, petrochemical and natural gas industries, sets out how to collect and exchange reliability and maintenance data, including equipment data, failure data such as cause and consequence, and maintenance data such as actions, resources and downtime. Even outside those industries, its split between the equipment, the failure and the maintenance action is a sensible way to link a video episode to the record that explains it.

The consented maintenance episode collection brief

The brief is a one-page agreement between the site and the collection team, filled in per task family and signed before recording starts. Each row names what must be decided, who owns the decision, and what the filled field looks like. Example values are illustrative and should be replaced with the site's own.

The consented maintenance episode collection brief
Brief fieldWhat it must stateWho signs offIllustrative entry
Task family and boundaryThe maintenance task, its first and last observable step, and what is out of scopeMaintenance leadGearbox replacement on an accumulation conveyor, from lockout applied to first loaded run; motor rewinds excluded
Exception targetsWhich exception types the set must contain and a minimum count of each before collection closesMaintenance lead and robotics leadSeized fasteners, wrong part, failed post-repair test, access blocked; at least 5 episodes each (illustrative)
Consent and withdrawalWho is asked, in writing, how to decline without consequence, and how a worker withdraws or reviews footage laterHR and counselOpt-in form per technician; withdrawal by request to the site coordinator; review window before delivery
Worker visibilityWhich cameras may show faces, voices or badges, and what is blurred or muted before deliveryPrivacy reviewerHead-mounted view only; fixed camera framed below shoulder height; audio kept for narration, names bleeped
Safety constraintsWhere cameras and staff may stand, and the rule that the energy-control procedure always overrides filmingSafety managerNo camera mounting on guarded equipment; recording pauses if lockout is challenged
Camera and sensor metadataDevice, mount, resolution, frame rate, clock source and any tool or torque data capturedCollection engineerHead camera and two fixed cameras at 30 frames per second, one clock source, torque wrench log
Synchronization methodHow streams share a time base and how drift is checked per episodeCollection engineerVisible clap or light flash at start and end; offset checked against tool log
Record linkageThe work order, asset ID and failure or maintenance codes the episode joins toPlannerCMMS work order number, asset tag, failure code and completion note
Outcome and intervention fieldsHow success, rework, supervisor takeovers and stop-work calls are recordedMaintenance leadOutcome: returned to service, returned with rework, or not completed; intervention flag with reason
Rights and retentionWho owns the footage and records, what may be licensed, and when raw footage is deletedOwner or counselCompany owns footage; licensing scope set in the agreement; raw clips deleted after delivery check

An exception catalog: what to film, and the record that proves it happened

An exception belongs in the set only when a second record confirms it. Video alone cannot tell a reviewer whether a technician paused because a bolt was seized or because the phone rang, so each exception type below is paired with the system record that proves it and a capture rule for the crew.

An exception catalog: what to film, and the record that proves it happened
ExceptionHow it shows on videoRecord that confirms itCapture rule
Seized or damaged fastenerRepeated attempts, tool change, penetrating oil or heatWork order note or extra labor timeKeep filming through every attempt; do not cut to the successful one
Wrong or missing partComparison of old and new part, trip away from the assetParts issue and return transactionsNarrate the mismatch on camera; log the time the technician leaves the frame
Stored energy found at verificationTry-start or gauge check shows the asset is not at zero energyLockout log or permit annotationStop and follow the procedure; mark the episode for safety review before use
Access blocked or reach limitedGuard removal, ladder, body repositioning, second personWork order labor and crew fieldsAdd a wider fixed view when the head camera loses the hands
Diagnosis revised mid-taskTechnician abandons the planned repair for a different oneRevised failure code or second work orderRecord a short spoken reason at the moment the plan changes
Failed functional test and reworkTest run, fault, disassembly againTest result and rework labor on the same orderTreat the rework as part of the same episode, not a new one
Supervisor or senior takeoverHands change, coaching voice, tool handed overCrew change on the work orderSet the intervention flag with the reason and the time it happened
Stop-work callTask halted, area clearedSafety log entryEnd the episode as not completed; never stage a resolution for the camera

Where an episode starts and stops, and how the streams line up

An episode should start at the first step a learner would need to imitate and end at a recorded outcome; when it ends before that, mark it as truncated rather than complete. The RLDS format used for many robot learning datasets makes the same distinction: each step carries is_first and is_last flags, an optional is_terminal flag, and an episode that ends on a step that is not terminal is treated as truncated. It also suggests an invalid flag for episodes that are incomplete, which is a reasonable home for clips cut short by a camera fault.

At a glance

Where an episode starts and stops, and how the streams line up

Multiple cameras and tool logs only help if they share a time base. The MCAP container format, common in robotics logging, records a log_time for when each message was recorded and a publish_time for when it was published, set to the log time when no separate value exists. Write down which clock each device used and which time field your export carries, so the reviewer can tell a real delay from a clock offset.

Many large robot datasets now store episodes inside shared files and rebuild each one from metadata. LeRobotDataset v3.0, for example, keeps states, actions and timestamps in Parquet, encodes camera frames as MP4, and records frame rates and episode start and end offsets in metadata. Whatever format the buyer asks for, the brief should keep the boundary decision and the reason for it in a human-readable field, not only in an index offset.

  • Episode start: the first lockout or isolation step, or the first diagnostic action if no isolation applies.
  • Episode end: the first loaded run after repair, a recorded failed test with no further work that shift, or a stop-work call.
  • Truncation reasons to record: shift change, parts wait over the shift, camera battery or storage failure, worker withdrawal.
  • Per-episode metadata: camera list with mounts, frame rates, clock source, sync check result, work order and asset IDs, outcome, intervention flag.

Who agreed to be filmed, and who decides what stays visible

Consent for this kind of collection is a decision each worker makes, and it is separate from the company's ownership of the equipment and records and from any right to license the result. A company can own its work orders and its cameras and still need a person's agreement, under state law, a labor agreement or its own policy, before footage of that person is used to train someone else's system.

Public research collections show what careful practice looks like. The Ego4D paper describes collecting first-person video from consenting participants and applying de-identification where relevant, with guidance that camera wearers could ask questions, withdraw at any time, and review and redact their own video. The Ego-Exo4D paper names bicycle repair among its skilled activity domains and adds spoken commentary from experts who were not the people being filmed, which shows that the recording and the explanation of a repair can come from different people.

Build the same options into the brief. Offer a head-mounted view that rarely shows the wearer's face, frame fixed cameras to the work rather than the room, decide how voices and names are handled before delivery, and give coworkers who walk through a shot a way to be removed. Contractors and visiting vendor technicians need their own conversation; they may be covered by a different employer's policy.

Illustrative brief: gearbox swaps on a conveyor line

The figures in this example are illustrative, chosen to show how the brief is filled in, not drawn from any real site. A distribution center plans 60 gearbox replacement episodes on its accumulation conveyors over one quarter, with 4 technicians who have opted in and 2 who have not.

After the first month, the collection engineer compares the episode log with the exception targets. Clean replacements make up most of the set; there are several seized fasteners and failed tests but no wrong-part episodes, because the parts room happened to be well stocked. Rather than staging a wrong part, the team extends the window and asks planners to flag the next mismatched issue so the crew can record it as it occurs.

One episode shows a technician finding residual pneumatic pressure at verification and bleeding it before continuing. The procedure was followed, so the episode stays in the set, flagged for safety review. A second episode shows a step skipped before the technician caught it. That clip is withheld from the demonstration set entirely and sent to the safety manager: a recording of unsafe practice should never be offered as behavior to imitate, even with a corrective label.

Two technicians who declined consent still did gearbox swaps during the period. Their episodes are simply not recorded; the fixed cameras are switched off while they work, and their absence is noted in the dataset card as a limit on technician variety.

Off-camera work and recoveries the episodes will still miss

Even a well-run collection misses the parts of maintenance that happen away from the asset or inside the technician's head. Name these gaps in the dataset card so a buyer does not mistake their absence for absence in the real work.

At a glance

Off-camera work and recoveries the episodes will still miss

  • Off-frame work: trips to the parts room, calls to the equipment maker, searching manuals. Log the start and end times and the reason, even when there is no footage.
  • Tacit cues: smell, sound, vibration and feel that drive a diagnosis. A short spoken note at decision points, or later commentary from a senior technician, captures some of it.
  • Tool and force data: torque, pressure and electrical readings that video cannot show. Capture the tool log where the tool records it, and say when it does not.
  • Rare conditions: night shifts, extreme temperatures, peak volume weeks and older equipment variants. Record which ones the set includes rather than implying broad coverage.
  • Long-horizon outcomes: whether the repair held for weeks. Join the episode to later work orders on the same asset and report how many have that follow-up.
  • Team tasks: two-person lifts and handoffs between shifts. Decide whether these are single episodes with two actors or excluded, and say which.

Signals that the collection should pause, shrink or go to a specialist

Some findings mean the collection plan itself has to change, not just a clip. Treat each of these as a reason to pause and bring in the named reviewer before more footage is recorded.

  • Filming changes the work: crews slow down, skip steps or perform for the camera. Pause and ask the maintenance lead whether the episodes still represent normal practice.
  • A camera position interferes with guarding, egress or energy control. Stop and have the safety manager reposition or remove it.
  • Workers raise concerns about discipline or performance monitoring. Bring in HR and counsel before continuing, and confirm in writing that footage will not be used for evaluation unless the policy says so.
  • Footage captures customer sites, other companies' equipment or confidential processes. Narrow the scope to assets the company controls, or get the other party's written permission.
  • The exception targets cannot be met without staging. Report the shortfall instead of recreating failures; staged exceptions are a different dataset and must be labeled as such.
  • A buyer asks for close-up face or voice data that the brief did not cover. Treat it as a new collection with new consent, not an extension.

Bringing maintenance episodes to a licensing conversation

A maintenance archive that follows this brief is far easier to license because the scope, consent records and linkage are written down before anyone discusses terms. For companies that choose to license such footage and work records, SourceX handles the rights review, contracting, preparation and delivery that turn them into an AI dataset.

The first step is a short company assessment that needs no footage or uploads. If a deal goes ahead, the company approves the scope and terms and signs before anything is delivered. Video collections tend to be large; deliveries of that size stay in the company's own storage or ship on encrypted drives, because SourceX does not host multi-terabyte datasets.

Frequently asked questions

What is physical AI maintenance demonstration data?

It is recorded maintenance work, usually video plus tool and system logs, packaged as task-level episodes that a robot or embodied model can learn from or be evaluated on. Useful sets link each episode to its work order and outcome and include the exceptions and recoveries, not only clean repairs.

Do workers have to consent to being filmed for robot training?

In practice, plan on asking each worker for written, revocable consent and on letting them decline without consequence. Whether consent is legally required depends on state law, labor agreements and the company's own policies, so counsel should review the specific facts before collection starts.

Can we stage exceptions if real ones are rare?

Staged failures can be useful, but they are a different kind of data and must be labeled as staged in the dataset card. Mixing them silently with real episodes misleads anyone measuring how a system handles real exceptions; it is usually better to extend the collection window and report a shortfall.

Which format should maintenance episodes be delivered in?

That depends on the buyer. Robotics teams commonly use formats such as RLDS, MCAP or LeRobotDataset, which differ in how they store steps, timestamps and episode boundaries. Keep the boundary decision, sync method and outcome in readable metadata so the episode survives conversion between them.

What should happen to a clip that shows a safety violation?

Remove it from the demonstration set and send it to the safety manager under the company's normal process. A dataset meant to teach behavior should not contain unsafe practice presented as an example, even with a corrective label.

Sources

  • OSHA's lockout/tagout standard defines servicing and maintenance to include adjusting, inspecting, lubricating, cleaning, unjamming and tool changes where workers could be exposed to unexpected energization or release of stored energy. Source
  • ISO 14224:2016 sets out how to collect and exchange reliability and maintenance data for equipment in the petroleum, petrochemical and natural gas industries, including equipment, failure and maintenance data. Source
  • The Ego4D paper describes first-person video collected from consenting participants with de-identification where relevant, and guidance that camera wearers could withdraw and review and redact their own video. Source
  • The Ego-Exo4D paper names bike repair among its skilled activity domains and adds spoken commentary from experts who are distinct from the participants filmed. Source
  • RLDS steps carry is_first and is_last flags and an optional is_terminal flag; an episode ending on a non-terminal step is truncated, and an optional invalid flag marks incomplete episodes. Source
  • The MCAP specification records a log_time for when a message was recorded and a publish_time for when it was published, set to the log time if not available. Source
  • LeRobotDataset v3.0 stores states, actions and timestamps in Parquet, camera frames as MP4, and frame rates and episode start and end offsets in metadata. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies