Short answer
Failure reasoning data pairs a recorded failure, usually video plus robot state, with a language explanation of what went wrong, the likely cause and a correction. Published work trains and prompts models on such pairs to detect and explain manipulation failures. A buyer specifies the failure taxonomy, evidence the annotator may use, explanation format and review rules; the annotators supply language, not the underlying recordings or rights.
The buyer's problem
A policy that fails silently is more dangerous than one that fails visibly. Teams building humanoid and mobile manipulation systems increasingly want a reasoning layer that notices a failed grasp, says what happened and proposes a retry. That layer needs examples in which a failure is paired with an explanation a model can learn to produce or condition on.
Two things make this hard to buy. Failures are under-recorded, because most logging keeps successes and deletes the rest. And explanations are only as good as the evidence the annotator had: a person watching a clip with no force or state trace can name a visible cause (the cup slipped) but not a hidden one (the gripper closed late).
This page is the specification for the language-annotation layer. Where to find failure recordings themselves is covered in robot manipulation failure datasets, and the category scheme in failure taxonomies for embodied tasks.
What published work does with this data
REFLECT builds a hierarchical summary of multisensory robot data (RGB-D, audio, robot states) that a language model queries to explain a failure and plan a correction, and it introduces the RoboFail dataset of failed executions in simulation and the real world (Liu et al., 2023). AHA is a vision-language model for detecting and explaining manipulation failures in natural language. Its authors generate failure trajectories with FailGen, which perturbs successful simulated demonstrations, and report a 10.3% advantage over GPT-4o in-context learning (Duan et al., 2024).
Inner Monologue shows the closed-loop use: language models replan from success detection, scene description and human feedback expressed in natural language, without additional training (Huang et al., 2022). RoboMIND reports 5,000 real-world failure demonstrations with causes recorded alongside its trajectories (Wu et al., 2024).
These results show the data type is in active use. They do not tell a buyer how much explanation data a particular system needs, which depends on the model, the tasks and the embodiment.
What the data can support, and what it cannot
It can support: training or evaluating failure detectors that output text; reward or success-detection modules fed by language; retry and replanning policies that consume explanations; and evaluation of whether a model's explanation matches a human judgment.
It cannot support, on its own:
- Control. Explanations are not actions. A correction written as "regrasp from the handle" needs a policy that can execute it.
- Causal truth, where annotators had only video. Explanations are inferences unless tied to logged state, force or the operator's own report.
- Real-failure coverage, if failures were induced or simulated. Procedurally perturbed failures differ from the ones a deployed system produces.
- Rights in the underlying footage. Annotation providers can write labels; they cannot grant rights in recordings they did not make, and annotation work is typically work-for-hire only for the labels.
Illustrative record schema
Values are synthetic; fields are what a buyer would adapt.
{
"episode_id": "ep_00417",
"task": "place cup on tray",
"source": "real | simulated | perturbed",
"evidence_available": ["rgb_wrist", "rgb_third", "joint_state", "gripper_width"],
"failure_time_s": [4.2, 6.8],
"failure_type": "grasp_slip",
"visible_cause": "cup rotates in gripper during lift",
"likely_root_cause": "grasp point too close to rim",
"confidence": "medium",
"explanation_text": "The cup slipped because the fingers closed near the rim.",
"correction_text": "Re-approach and grasp lower on the body, then lift slowly.",
"correction_verified": false,
"annotator_id": "ann_07",
"reviewer_id": "rev_02"
}
Specification parameters
| Parameter | Buyer decision | Note |
|---|---|---|
| Failure taxonomy | Closed list, open text or both | Fix it before annotation starts |
| Evidence shown | Video only, or video plus state and force | Determines how deep a cause can be claimed |
| Explanation style | One sentence, structured fields or free text | Match what the model will emit |
| Correction | Required, optional or verified by retry | Unverified corrections are hypotheses |
| Span | Whole episode or time-stamped segments | Timing helps detectors |
| Language | Language and register | Include operator jargon where relevant |
| Split policy | By task, object, site or robot | Keep eval episodes held out |
Acceptance tests
Set thresholds yourself; these are the tests.
- Gold set. A subset annotated by experts with agreed answers; annotators are scored against it before production starts and during it.
- Cross-review. A second reviewer re-labels a sample; disagreements are adjudicated and logged. Methods for measuring this are discussed in inter-inspector agreement and label noise, which applies here to explanation labels as well.
- Evidence discipline. Reject explanations that cite a cause the available evidence could not show.
- Specificity check. Reject generic text such as "the robot failed to grip".
- Balance. Report counts by failure type, task and source (real, simulated, perturbed) so no single type dominates unseen.
- Held-out evaluation. Hold out episodes by site, object or robot, as the claim requires; never mix them back into training.
Limitations and additional preparation
Language annotation is expensive per episode and slow to scale, and annotators drift toward fluent but unsupported explanations. Guidelines, gold sets and periodic audits matter more than annotator count. Human failures differ from robot failures: human procedural mistake videos can teach what a mistake looks like but not how a specific robot fails. For service tasks, see service-task failure examples.
Maintenance and diagnostic text, such as diagnostic reasoning traces, offers a related style of cause-and-fix writing for equipment, but it is not a substitute for explanations of robot behavior.
Budgeting and sequencing
Run a pilot before a production order. Pick fifty or so episodes spanning the failure types you expect, have two annotators write explanations and corrections independently, and read them side by side. The pilot shows whether the taxonomy fits, whether the evidence given is enough and how much the annotators diverge. It also shows whether corrections are specific enough to act on or are generic advice.
Decide who verifies corrections. A correction can be marked as annotator opinion, as reviewed by a robotics engineer, or as tested by replaying a retry on a robot or in simulation. These carry different weight. A buyer building a replanning module will want at least a tested subset, because an unexecutable correction teaches the planner to propose actions that the robot cannot perform.
Finally, consider mixing sources. Simulated or perturbed failures give breadth cheaply, real failures give fidelity, and annotation effort can be concentrated on the real ones. Keep the source flag in the delivered schema so that evaluation can be reported separately for each.
Questions to ask suppliers
- What material do annotators see: video only, or synchronized state, force and operator notes?
- How were the failures obtained: logged in operation, induced on purpose or generated in simulation?
- What is the taxonomy, and who wrote and revised it?
- How are root causes distinguished from visible symptoms in the labels?
- What is the gold-set size, and how are annotators scored and retrained?
- Are corrections ever tested by retrying, or are they annotator opinion?
- Who owns the annotation output, and who must grant rights in the source recordings?
- Can labels be delivered in a machine-readable schema with episode IDs that map back to the source files?
Discuss your requirements
If you need failure recordings, explanation labels or both, describe the tasks, evidence and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources; annotation and specialist capture partners can be arranged case by case, and a request does not guarantee a match. For the surrounding map, see the humanoid training data guide and the inspection, defect and failure data hub. The catalog page on expert annotations and labels describes the label type.
Discuss your humanoid training-data requirements with SourceX
Sources
- REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction — multisensory summaries queried by a language model; RoboFail.
- AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation — failure detection and explanation; FailGen; reported 10.3% gain.
- Inner Monologue: Embodied Reasoning through Planning with Language Models — language feedback for closed-loop replanning.
- RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation — 5k real-world failure demonstrations with causes.