Short answer
Robot failure datasets contain episodes where a defined robot embodiment tried a task and did not succeed, with synchronized observations, actions and a label for what went wrong. They train failure detectors and explanations and support evaluation. They come mainly from robotics labs, deployed fleets and specialist data providers, and rarely from ordinary business footage, which shows people, not robots.
The buyer's problem
Successful demonstrations are what most robot datasets collect, because that is what policy training asks for. A team that wants a model to notice that a grasp slipped, explain why, and propose a retry needs the opposite: episodes that went wrong, in enough variety to cover the causes that matter.
Failure data is scarce for a simple reason. Labs discard bad runs, fleets log them inconsistently, and annotating them requires someone who knows what should have happened. This guide covers where failure episodes come from, what a usable episode contains, and how to evaluate an offer.
What published work shows
REFLECT builds hierarchical summaries of multisensory robot data (RGB-D, audio and robot states) that a language model queries to explain failures and plan corrections, and it introduces RoboFail, a dataset of failed executions in simulation and the real world (Liu et al., 2023). The point for buyers is the input list: failure explanation used more than video.
AHA is an open vision-language model for detecting and reasoning over manipulation failures. It trains with FailGen, which procedurally perturbs successful simulated demonstrations to produce failure trajectories, and it generalizes to real-world failure datasets; the authors report it outperforming GPT-4o in-context learning by 10.3% (Duan et al., 2024). Simulated failures are therefore one route, but whether they transfer to your hardware is something to test, not assume.
RoboMIND collected 107k demonstration trajectories across four embodiments, including a humanoid with dual dexterous hands, and includes 5k real-world failure demonstrations annotated with causes (Wu et al., 2024). It shows the scale at which a lab can collect failures alongside successes. These are numbers for those studies, not targets for yours.
What the data can support - and what it cannot
| Use | Supported? | Condition |
|---|---|---|
| Failure detection on the same embodiment | Yes | Matched sensors and task family |
| Failure explanation and recovery planning | Yes | Cause labels and language descriptions |
| Evaluating policies on known failure modes | Yes | Held out by object, site or time |
| Transfer to a different robot | Partly | Observation and action spaces differ |
| Business-process failure context at scale | No | Robot episodes rarely carry workflow context |
| Human mistakes in a workflow | No | See human procedural mistake videos |
Robot failure data is robot-specific. It is tied to a hardware platform, its calibration, its control frequency and its action space. A failure on a parallel-jaw gripper tells you little about a five-finger hand. See robot trajectory dataset specification for the fields a buyer needs.
Supply routes
Robotics labs and academic groups hold failure runs from experiments. Quality varies; episodes are often unlabeled or labeled for the lab's own purposes, and licensing may be unclear.
Operators of deployed robots log interventions, faults and aborted tasks. These show real operating conditions, but formats are proprietary, and logs may contain customer, site and employee information.
Specialist robot-data providers (see also failure and recovery demonstrations and service task failure examples) can collect failure episodes to a specification on a named embodiment, including deliberate and induced failures. This is the route when you need coverage you can define. See teleoperation data collection requirements.
Simulation pipelines generate failures by perturbing successful runs, as in FailGen. They scale cheaply and support pretraining of detectors; real failures are needed to validate them.
SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also engage specialist capture partners, scoped case by case. Whether suppliers with suitable robot episodes exist among US businesses is something a request would investigate, not something to assume.
Illustrative episode package
# Illustrative only - synthetic values, not an available dataset
package: manipulation_failures_v0
embodiment: single-arm, parallel-jaw gripper (robot_model_X)
episodes: failure and matched success for the same task
streams:
rgb: 2 cameras, 30 Hz, intrinsics + extrinsics provided
depth: optional
proprioception: joint positions, gripper width, 100 Hz
actions: end-effector delta, 10 Hz, logged commands
force_torque: wrist sensor, if present
audio: optional
sync: single clock, drift report per episode
labels:
outcome: success | failure | recovered
failure_phase: approach | grasp | transport | place | release
failure_cause: slip | misalignment | collision | perception_error | planning_error | other
failure_onset_time: seconds from start
language_explanation: one sentence, annotator-written
induced: true | false # deliberate vs natural
split_keys: object_id, scene_id, session_id
rights: training and evaluation; no redistribution
The induced field matters. Deliberately induced failures are easy to collect but can differ in character from natural ones; keep both identifiable. Define cause categories with the help of failure taxonomies for embodied tasks.
Why matched successes and onset times matter
A failure detector learns from contrast. If a package contains only failed episodes, the model can learn that "this scene looks like our failure set" instead of what went wrong. Matched successful attempts at the same task, with similar objects and scenes, force the model to attend to the actual difference.
Onset time matters for the same reason. An episode-level "failed" tag cannot tell a model whether the grasp slipped at second four or the object was dropped at second nine, and recovery planning depends on that moment. Ask for the time at which the failure began and, where recovery was attempted, the time of the retry.
Evaluating an offer
- Check that failure labels are tied to a time, not only an episode.
- Check that matched successes exist, since detectors need contrast.
- Verify the distribution of causes, and whether one cause dominates.
- Confirm sensors, calibration and synchronization are documented for every episode.
- Reserve a held-out set by object or scene before any training.
Questions to ask suppliers
- Which robot, gripper and sensor suite produced the episodes, and can you license your target platform's data?
- Are failures natural, induced or generated in simulation, and are they marked?
- Who assigned cause labels, from what taxonomy, with what review?
- Are successful and failed attempts at the same task included?
- Is the failure onset time labeled?
- What calibration and clock-sync information accompanies each episode?
- Do logs contain people, site details or customer information that limits licensing?
- Is the same data already public, or licensed to others on an exclusive basis?
Discuss your requirements
If your detector or evaluation set depends on failure episodes from a specific embodiment, tell SourceX what the set needs to cover. SourceX reviews the requirement and follows up about potential sources, checks the data and the supplier's licensing permissions, and coordinates the license and delivery; a request does not guarantee a match. For wider context see the humanoid training data guide, the inspection and failure data hub, and the catalog page on robotics and embodied AI datasets or evaluation sets. Related pick-error data from warehouses is covered in pick errors and mispicks.
Tell SourceX what your evaluation set needs to cover.
Sources
- REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction — multisensory summaries and the RoboFail dataset.
- AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation — failure detection model and FailGen perturbation.
- RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation — multi-embodiment trajectories including real-world failures with causes.