Skip to content

Privacy and preparation

Robotics failure recovery video: filming the fix, not just the perfect run

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Workplace demonstration videos should capture the whole episode around a mistake: normal work, the moment it goes wrong, how the worker notices, the recovery back to a good state, the correction and the final outcome. Plan which natural mistakes to film, get written consent before any camera runs, never stage unsafe failures, and time-stamp each phase so robotics teams can train on the recovery itself.

Video and sensors

Robotics failure recovery

This page covers robotics failure recovery video: filming the fix, not just the perfect run.Illustrative

Key takeaways

  • Film continuous episodes and mark six phases: nominal work, deviation, detection, recovery, correction and outcome.
  • Robot learning teams want recovery data because policies trained only on clean demonstrations fail in states they never saw.
  • Choose frequent, low-hazard mistakes your systems already log, and never stage unsafe failures.
  • Get a separate written release for AI use before recording, with withdrawal and a no-discipline promise.
  • Link each episode to an outcome record and document coverage gaps instead of hiding them.

How should workplace demonstration videos capture recovery from mistakes?

Film continuous episodes, not trimmed highlights, and mark six phases inside each one: nominal work, deviation onset, detection, recovery to a good state, correction, and the outcome. The recovery footage is only useful when a model can see the state that went wrong and the actions that fixed it, so the cut must start before the slip and end after the result is confirmed.

Everything else in a robotics failure recovery video plan follows from that. Decide in advance which mistakes are worth capturing, get the filmed workers' written agreement before recording, collect natural errors rather than staging risky ones, and link each episode to a record from your own systems that proves how it ended. A business that does this turns ordinary rework into footage most robot datasets lack.

Why robot learning teams want the stumble as well as the clean run

Because a policy trained only on successful demonstrations has never seen the states it lands in after a small error. The LeRobot documentation on human-in-the-loop collection states the problem directly: standard behavioral cloning trains on successful demonstrations only, and during deployment small errors compound and push the robot into states never seen in training.

Robot researchers fill that gap with interventions. HG-DAgger, published at ICRA 2019, lets a human expert take control from a learned policy when needed and learns from the data generated while the expert is in control. Sirius has robots handle most of the work while human operators monitor and intervene in challenging situations, then reuses that deployment data in training. LeRobot's guide describes the same loop and, following later work called RaC, splits each intervention into recovery, which brings the robot back to a good state, and correction, which demonstrates the right behavior from there.

A business filming its own people is in a different position: the person recovering is the worker, not a teleoperator. The value is similar. Footage of a packer re-gripping a sliding carton, or an assembler backing out a misthreaded fastener, shows the off-nominal states and recovery strategies that clean demonstrations never contain. Whether a buyer can use human recovery video directly or as a reference for its own collection is a buyer decision, so describe what you have accurately rather than promising a training result.

Which recovery events in your operation are worth filming

Film recoverable, low-hazard mistakes that happen naturally and repeat often enough to collect in volume. Rare dramatic failures make poor training data and raise safety and liability questions; frequent small deviations are the useful core. The examples below are illustrative.

  • Prefer events your systems already log, such as rescans, rework codes or quality holds, so the outcome can be verified.
  • Treat a near miss involving people, vehicles or energized equipment as a safety matter, not a data opportunity.
Which recovery events in your operation are worth filming
Event familyIllustrative exampleWhat makes the clip usefulExclude or never stage
Grip slip and regraspA polybag slides out of a packer's hand and is picked up againHand pose before and after, object deformationDrops of heavy, sharp or hazardous items
Wrong item and swapA picker scans the wrong size and returns it to the binScan event marks the error and the fixErrors created by mislabeling stock on purpose without disclosure
Orientation errorA part is seated upside down, removed and reseatedVisible fit failure and the corrective motionForcing parts in ways that damage tooling
Jam or misfeed clearanceA carton jams in a taper and is cleared by handRecovery steps on routine, guarded equipmentAny clearance that requires lockout or bypassing a guard
Routing or placement errorA cable is routed over the wrong clip and reroutedFine manipulation during correctionLive electrical work
Interrupted task and restartA worker is called away mid-pack and resumesRe-establishing state after a pauseInterruptions caused by injuries or incidents
Abandoned attemptA damaged item is set aside instead of packedShows when stopping is the right actionRecording customers or visitors who cause the interruption

What the consented recovery event collection plan includes

The plan is a ten-part document the site lead, a safety owner and the person responsible for privacy sign before any recording. It records decisions, not intentions, so a buyer or reviewer can check the footage against it later. Use the table as a template and keep the completed version with the dataset.

At a glance

What the consented recovery event collection plan includes

What the consented recovery event collection plan includes
Plan sectionDecision to makeRecord to keepOwner
Task boundaryWhich station, task and start and end conditions define an episodeTask card with start and end triggersOperations lead
Recovery event listWhich deviations count, using the event families aboveEvent definitions with examples and exclusionsOperations lead
Consent and noticeWho is filmed, what they agree to, how they withdrawSigned releases, notice text, withdrawal logPrivacy owner and counsel
Worker visibilityHands only, upper body or full view; faces in or out of frameCamera placement diagrams per stationPrivacy owner
Bystanders and client materialHow others in frame and customer-owned products or labels are handledMasking rules and client permissionsPrivacy owner and account manager
Camera and metadataCameras, frame rate, resolution, mounting, calibrationCamera manifest with serials and settingsTechnical lead
SynchronizationHow video aligns with scans, PLC events or quality recordsClock source and measured offset per sessionTechnical lead
Variation quotaShifts, lighting, product variants and operators to coverCoverage matrix filled per weekOperations lead
Phase labels and outcomeHow the six phases and the final result are markedLabel guide and outcome source per episodeAnnotation lead
Stop rulesWhen recording pauses for safety, withdrawal or incidentsStop log with reason and timeSafety owner

Treat consent for an AI dataset as its own agreement, separate from any workplace monitoring notice or security camera policy. A worker who knows the dock is under CCTV has not agreed to have their recovery motions licensed to a robotics developer.

The release should name the purpose, the kinds of recipients, how long footage is kept, what is masked, whether voices are recorded, and how a worker can withdraw and what happens to footage already delivered. Make participation optional, with no effect on pay, scheduling or evaluation, and keep a refusal list that camera operators check every shift. Footage of a worker's mistakes is sensitive in a particular way: agree in writing that it will not be used for discipline or performance review.

Some state laws add formal steps. Illinois's Biometric Information Privacy Act requires a private entity, before it collects a biometric identifier, to inform the person in writing of the specific purpose and length of term and to receive a written release. Whether plain video creates biometric identifiers depends on how it is processed, which counsel should decide for the specific sites and states. Large research collections show what careful practice looks like; the Ego4D authors describe collecting egocentric video from consenting participants with de-identification procedures where relevant.

Unionized sites, temporary staff from agencies and workers of a client company on your floor each need their own path. Settle those before the first session, because consent gathered after recording is much harder to defend.

Turning raw footage into recovery episodes a robotics team can load

Package each recording as episodes with step-level timing and episode-level metadata, in a layout that common robot data tools already understand. Recovery footage that arrives as long unlabeled files usually needs to be re-cut by the buyer, and the phase boundaries are the part they cannot reconstruct.

At a glance

Turning raw footage into recovery episodes a robotics team can load

Two public formats show the expected structure. RLDS stores episodes as sequences of steps with mandatory is_first and is_last flags and an optional is_terminal flag; an episode that ends without a terminal step is treated as truncated. LeRobot's v3 format keeps states, actions and timestamps in Parquet, encodes camera frames as MP4 sharded per camera, and records per-episode metadata, tasks and offsets. You do not need to deliver in either format, but your metadata should map onto them cleanly.

Record the phases as time spans inside the continuous episode rather than as separate clips, the same way LeRobot's human-in-the-loop guide keeps autonomous and human-controlled segments in one continuous trajectory. Then attach the outcome from a system of record: the scan confirmation, the quality pass, the rework ticket or the scrap entry.

  • Episode fields: site, station, task id, operator pseudonym, shift, camera ids, start and end times, consent record id.
  • Phase spans: nominal, deviation onset, detection, recovery, correction, outcome, each with start and end timestamps.
  • Event fields: event family, cause if known, whether it was natural or a declared seeded variation.
  • Outcome fields: result (completed, completed after rework, abandoned), source system, record id, date read.

A worked plan for an illustrative pick-and-pack line

This example is illustrative; the counts show the shape of a plan, not a real collection. A fulfillment company decides to film recovery events at three packing stations over four weeks.

  • Scope: three stations, two shifts, eight packers who signed releases out of fourteen asked; overhead and side cameras show hands, items and the carton, with faces out of frame.
  • Event list: grip slips, wrong-size item swaps, carton tape misfeeds, and interrupted packs. Jams that need a guard opened are excluded and the camera keeps running only until the operator reaches for the guard.
  • Natural rate check: the first week yields recoveries in about 6% of episodes (illustrative), too few for wrong-item swaps. The team adds a declared seeded variation, mixing two similar sizes in one bin on two days, and flags those episodes as seeded.
  • Synchronization: station scans are written with the warehouse system clock; the team logs each camera's offset against that clock at the start of every session and checks it at the end.
  • Outcome link: every episode carries the order scan and, where present, the rework or return record from the following two weeks.
  • Package: continuous episodes with phase spans, a coverage matrix, the consent log, and a withdrawals list showing one packer who left the study in week three and whose footage was removed.

What operating variation or recovery behavior is missing from the footage?

Check coverage against the plan's variation quota every week, not at the end. The usual gaps are predictable: one experienced operator produces most recoveries, night shifts and seasonal products are thin, and almost every recovery uses the same strategy.

Run a short gap review with these questions. Do recoveries come from several operators with different experience levels and handedness? Are there episodes where the mistake was noticed late, or never, and the outcome record caught it? Is there at least some abandoned work, where stopping was correct? Do lighting, packaging variants and station layouts match what the robot will face? If a buyer cares about a robot recovering from its own errors, are your human recoveries visible enough in hand and object detail to be informative?

Write the answers into the dataset documentation. A buyer can work with known gaps; unexplained gaps make the whole set harder to trust.

When to pause recording or shrink the plan

Pause when safety, consent or ownership questions arise mid-collection, and narrow the scope rather than pressing on. The cost of a paused week is small next to footage that cannot be licensed.

At a glance

When to pause recording or shrink the plan

  • A worker withdraws, or a new worker joins a station without a signed release.
  • Customers, visitors, drivers or another company's staff appear in frame in ways the masking plan does not cover.
  • Client-owned products, labels or packing instructions are visible and the client has not agreed.
  • A recovery event turns into an incident with injury, equipment damage or a guard bypass.
  • Camera clocks drift beyond what the synchronization plan allows, so phases no longer line up with system events.
  • Reviewers cannot agree on phase boundaries for an event family; fix the label guide before labeling more.

Offering consented recovery footage through SourceX

If your operation already produces the kind of footage this plan describes, or could, SourceX can assess whether it fits buyer needs. The first step is a short company assessment, and no data uploads are needed to see whether you qualify.

Nothing is delivered until your company has approved the scope and terms and signed the agreement. Large video collections do not need to be uploaded to SourceX; they stay in your own storage or ship on encrypted drives.

Frequently asked questions

Should a business stage mistakes for robot training video?

Only safe, declared variations, such as mixing similar items in a bin, and only when natural errors are too rare. Never stage failures that put people or equipment at risk, and label every seeded episode so buyers can separate it from natural recoveries.

What is the difference between recovery and correction in demonstration data?

Recovery brings the task back to a good state after something goes wrong, such as re-gripping a slipping item. Correction then demonstrates the right behavior from that state. Robot learning guides such as LeRobot's human-in-the-loop documentation separate the two.

Do workers have to consent to being filmed for AI training if the site already has cameras?

Treat it as a separate permission. A security or monitoring notice does not cover licensing footage of someone's work to a robotics developer, and some states impose written notice and release steps for biometric data. Counsel should review the release for each site.

How long should a recovery episode be?

Long enough to include normal work before the deviation and the confirmed outcome after the correction. Keep the continuous recording and mark phase spans inside it rather than cutting short clips around the error.

Can we film hands only to protect worker privacy?

Often, yes, and hands and objects are what most manipulation work needs. Hands-only framing still records identifiable details such as tattoos, rings and voices, so keep the consent process and masking rules in place.

Sources

  • LeRobot's human-in-the-loop guide says behavioral cloning on successful demonstrations alone lets small errors compound into unseen states, records autonomous and human segments in one continuous trajectory, and splits interventions into recovery and correction following RaC. Source
  • LeRobot dataset v3 stores states, actions and timestamps in Parquet, camera frames as MP4 sharded per camera, and per-episode metadata with tasks and offsets. Source
  • RLDS episodes are sequences of steps with mandatory is_first and is_last flags and an optional is_terminal flag; an episode ending without a terminal step is truncated. Source
  • HG-DAgger is an interactive imitation learning method in which a human expert takes control from the learned policy when needed and learning uses data generated while the expert is in control. Source
  • Sirius has partially autonomous robots handle most decisions while human operators monitor and intervene in challenging situations, with deployment data passed to policy training. Source
  • Illinois 740 ILCS 14/15(b) requires a private entity to inform the person in writing of the specific purpose and length of term and to receive a written release before collecting a biometric identifier. Source
  • The Ego4D authors describe collecting egocentric video from consenting participants with robust de-identification procedures where relevant. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies