Short answer
Failed attempts and recoveries show what happens when a task goes wrong: the slip, the correction, the retry. Published robot datasets now include labeled failure demonstrations, and human assembly datasets include unscripted mistakes. Specify them as a labeled share of a collection, with cause, detection point and recovery action, and ask suppliers whether their footage was edited to remove errors.
The buyer's problem
Most workplace footage is the happy path. Businesses keep the clean runs, trainers cut the stumbles, and people who know they are on camera slow down and avoid errors. A policy trained on that never sees a dropped item, a jammed drawer or a misread label, and it has no example of what to do next.
Recovery behavior is the part of a skill that distinguishes a demo from a deployable behavior. It is also scarce, because it is rare, awkward to record and sometimes sensitive: an error on camera can be an incident report. This page covers how to specify failure and recovery data. The classification of failures for inspection and detection models is covered in robot manipulation failure datasets and in human procedural mistake videos; this page is about recovery for policy learning.
What published datasets show
RoboMIND contains 107,000 demonstration trajectories over 479 tasks and four embodiments, including a humanoid with dual dexterous hands, and adds about 5,000 real-world failure demonstrations annotated with failure causes (Wu et al., 2024). That is a robot-teleoperation dataset, so its failures include robot states; failures in human video lack those.
On the human side, Assembly101 records 4,321 videos of people assembling and disassembling 101 toy vehicles with 8 static and 4 egocentric cameras, unscripted and with mistakes, and introduces mistake detection as a task (Sener et al., 2022). HoloAssist is a published egocentric human-interaction dataset that may be relevant to interventions and corrections (Wang et al., 2023).
The quality of demonstrations matters in its own right: results in offline learning depend on dataset quality, and datasets in that study ranged from worse to better operators (Mandlekar et al., 2021). Failures are not "bad demonstrations" to discard, but they must be labeled so they are not imitated as success.
Anatomy of a failure and recovery episode
Take "pull a tray from a rack and place it on a cart":
- Approach and grasp. The tray edge catches the rail.
- Slip. The tray tilts and one corner drops.
- Detection. The person notices from visual and tactile cues, within a fraction of a second.
- Correction. They stop, re-grasp with both hands, lift clear of the rail.
- Resume. The task continues and finishes.
Each stage is a label. The failure cause (rail contact), the detection cue (visual tilt, felt resistance), the recovery action (re-grasp with the second hand) and the final outcome are all separate fields. Video shows the first and last well; tactile detection cues and effort are invisible.
What failure and recovery data can and cannot support
It can support failure-detection models, recovery-aware policy training when combined with robot data, evaluation slices that test robustness, and language descriptions of what went wrong.
It cannot provide robot actions for the recovery. A human's re-grasp is a human strategy; a humanoid with different hands may need a different one. Human mistakes also differ from robot mistakes: people rarely misestimate depth by centimeters or lose calibration. A dataset of human slips therefore informs perception and task structure more than low-level control. Robot failures are collected with robots, and a robot teleoperation or autonomous run is the more direct source for that.
Three sourcing routes for failures
| Route | How failures arise | Strength | Weakness |
|---|---|---|---|
| Existing footage | Natural errors left in the archive | Realistic frequency and causes | Often edited out; rarely labeled |
| Commissioned natural | Long unscripted sessions with errors kept | Real errors, with cause labels | Many hours per failure; costs time and money |
| Commissioned induced | Instructed perturbations (slippery item, blocked path) | Controlled coverage | Staged behavior may differ from real errors |
Combine routes, and label which is which. Never present induced failures as natural ones.
Illustrative specification
# Illustrative only - values are examples a buyer would set
collection: failure_recovery_v0
tasks: 8 workplace tasks; per task at least one failure type induced and one natural
episode_labels:
episode_type: nominal | failure_recovered | failure_unrecovered | aborted
origin: natural | induced
failure_cause: [slip, drop, collision, wrong_object, misalignment, blocked_path, tool_failure]
failure_onset_t: seconds from start
detection_t: seconds (operator reaction)
detection_cue: visual | auditory | tactile_inferred | external_alert
recovery_actions: [regrasp, reposition, retry, ask_help, tool_change]
recovery_complete_t: seconds
human_intervention: none | coworker | supervisor
quotas:
nominal: unspecified
failure_episodes: reported per task, per cause, per site
safety: induced failures reviewed by site safety lead before recording
exclusions: incidents involving injury or near-miss; handled outside the dataset
Acceptance criteria
- Failure episodes are labeled with onset, cause and outcome, and a sample is double-checked.
- Origin (natural vs induced) is present on every failure clip.
- Report counts per task and cause; do not accept a bare total of failures.
- Unrecovered failures and aborts are included, not only neat recoveries.
- Onset-to-recovery timestamps are consistent with the video on a sampled audit.
- Held-out evaluation clips include failure types absent from training; see demonstration dataset splits and rare events in demonstrations.
Handling failure footage responsibly
Failure data can show a worker at their worst moment, a safety lapse or a quality escape the business would rather not circulate. Agree in advance what is in scope. Many suppliers will want to exclude incidents with injuries, equipment damage or customer-visible errors, and may need legal or safety review before releasing others. Buyers should accept reasonable exclusions but ask for the exclusion rule, so you know what is missing. For induced failures, agree who can approve the scenario, that the site's safety lead has reviewed it, and that participants are told which attempts are staged. Label staged and natural failures differently; models that learn from staged errors may mis-estimate how errors really look.
Questions to ask suppliers
- Was the footage edited, trimmed or curated, and what was removed?
- Do retained recordings include fumbles, drops, restarts and corrections?
- Are failures labeled with cause, and by whom?
- Were any failures induced, and how were workers kept safe?
- How does the supplier treat recordings of incidents, near misses or injuries?
- Are coworkers or supervisors who intervened visible, and have they been covered by permissions?
- How many failures exist per task and per site?
- Could errors in the footage reveal confidential quality or compliance problems that the business needs to review first?
Footage of errors can raise sensitivities for workers and employers. See worker consent and identifiable people and safety behaviors in demonstrations.
Discuss your requirements
If recovery behavior is part of your requirement, describe the tasks, failure types and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match. Where no existing asset fits, SourceX can scope a commissioned collection. Start from the humanoid training data guide, the workplace task demonstration hub, and see success and outcome labels and expert vs novice demonstrations for related labeling choices.
Discuss your humanoid training-data requirements with SourceX
Sources
- RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation — labeled real-world failure demonstrations.
- HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World — egocentric task guidance dataset.
- Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities — unscripted assembly with mistakes and mistake detection.
- What Matters in Learning from Offline Human Demonstrations for Robot Manipulation — dependence on demonstration quality.