Short answer
A failure taxonomy for embodied tasks labels what went wrong at several levels: the task phase, the failure mode, the cause, the severity and whether recovery was possible. Define it before collection, because failure data tends to be scarce and costly to relabel. Keep failure modes observable, keep causes separate from modes, and test the scheme for annotator agreement before it is applied at scale.
Why failures need their own taxonomy
Product defect schemes describe a flaw in a thing. A robot or task failure describes an event in time: something that should have happened did not. Mixing the two produces labels that fit neither. "Dropped object" is an event with a before and after; "scratch" is a state of a surface. Embodied failure labels must say where in the task the failure arose, what was observed, what probably caused it, and what happened next.
A taxonomy also decides what a model can learn. Systems that detect and explain failures learn from labels written by someone. If the vocabulary conflates "gripper slipped" with "grasp planned on the wrong part", a model trained on it cannot tell the two apart, and neither can the evaluation. For defect labels on products rather than tasks, see defect taxonomy and label schema.
What published work labels
Published datasets show the range of failure labeling. REFLECT builds hierarchical summaries of multisensory robot data (RGB-D, audio and robot states) that an LLM can query to explain a failure and plan a correction, and it introduces the RoboFail dataset of failed executions in simulation and the real world (Liu et al., 2023). AHA is an open-source vision-language model that detects and explains manipulation failures in natural language; its FailGen procedure perturbs successful simulated demonstrations to produce failure trajectories, and the model generalizes to real-world failure datasets (Duan et al., 2024).
RoboMIND contains 107k demonstration trajectories on 479 tasks across four embodiments, including a humanoid with dual dexterous hands, and includes 5k real-world failure demonstrations with causes (Wu et al., 2024). Together these show failures recorded as trajectories with sensor streams and as language explanations. They do not agree on one taxonomy, which is a reason to define yours explicitly. For published sets and how to source them, see robot manipulation failure datasets.
Design principles
- Separate mode from cause. The mode is what was observed ("object slipped during lift"). The cause is an inference ("insufficient grip force", "wet surface"). Label both, and record how certain the cause is.
- Use observable definitions. Each mode needs a rule an annotator can check in the video or logs.
- Index by task phase. The same symptom means different things during approach, grasp, transport and placement.
- Keep severity and recovery separate. A harmless slip that was corrected is not the same as one that damaged the item.
- Plan for several causes. Allow primary and secondary cause labels, plus "unknown".
- Version the scheme. Record the taxonomy version on every label, since definitions will change.
Illustrative taxonomy schema
# Illustrative only - categories are examples a buyer would adapt to their embodiment and tasks
failure_record:
episode_id: ep_00431
task: "place tote on shelf"
phase: approach | grasp | lift | transport | place | release | tool_use | human_interaction
mode:
perception: [missed_detection, wrong_identity, pose_error, state_misread]
planning: [infeasible_plan, wrong_order, unsafe_path, goal_misunderstood]
contact: [slip, drop, collision, jam, excessive_force, missed_contact]
placement: [misaligned, unstable, wrong_location]
system: [timeout, controller_fault, comms_loss, estop]
human: [interference, ambiguous_instruction, handover_failure]
cause:
primary: sensor_noise | occlusion | object_property | model_error | environment | operator_input | unknown
secondary: optional
confidence: low | medium | high
severity: none | minor_damage | major_damage | safety_event
recovery: self_recovered | assisted | task_abandoned | not_attempted
evidence: [rgb_video, depth, joint_states, force_torque, audio, operator_note]
annotator_id: annotator_A07
taxonomy_version: v0.3
Acceptance tests for a labeled failure set
- Agreement. Two or more annotators label a shared sample; compute agreement at each level (phase, mode, cause), and expect cause agreement to be lower than mode agreement.
- Coverage. Every taxonomy category appears at least a few times, or is explicitly marked absent.
- Evidence match. Each label lists the modalities used; labels that depend on joint states or force data are only valid when those streams are present.
- Class balance. Report counts per mode; rare modes may require targeted collection.
- Success controls. Include successful episodes of the same tasks so detection models see both sides.
- Held-out design. Split by site, operator, object or time as the claim requires; see defect detection evaluation sets for parallel practice.
Where failure examples come from
| Source | What it offers | Taxonomy caution |
|---|---|---|
| Robot or teleoperation logs | Joint states, actions, sensors, failure timing | Failure labels often absent; must be added |
| Simulation with perturbation | Many failures with known cause | Cause is assigned by the generator, not discovered |
| Human procedural mistakes | Realistic errors in human work | Human errors differ from robot failures; see human procedural mistake videos |
| Service-task logs | Failed customer-facing tasks | See service task failure examples |
| Repair histories | Failed and repeated repairs | See difficult and failed repairs |
Limitations: what a taxonomy cannot do
A taxonomy is a label vocabulary, not data. It cannot create failure examples, and it cannot make a rare failure frequent. Labels attach to episodes that already exist, so the scheme can only describe what was recorded.
Causes are hypotheses. Without the right sensors, an annotator can only guess whether a drop came from slip, mistaken pose or a controller fault, and different annotators will guess differently. Labels derived from human video cannot be assumed to apply to a robot, because human and robot failure modes differ; the broader contrast is covered in the human-to-humanoid embodiment gap. A taxonomy defined for one embodiment may omit modes specific to another, such as finger-level slip in a multi-finger hand. Finally, failure labels support detection, explanation and evaluation; they do not by themselves teach recovery, which needs recovery demonstrations.
Adapting the taxonomy to your embodiment
Start from your robot's own failure log and trim the scheme to modes you can distinguish with your sensors. A wheeled mobile manipulator with a parallel gripper rarely needs finger-level slip categories; a multi-finger humanoid hand may need them, plus categories for bimanual coordination and balance. Add task-specific modes sparingly, and keep a catch-all "other" with a free-text field so rare events are recorded rather than forced into the wrong box. Review the "other" bucket monthly, and promote repeated entries to named modes with a version bump.
Questions to ask suppliers
- Which taxonomy and version do your failure labels use, and can I read the written definitions?
- Are failure modes and causes labeled separately?
- Which sensor streams were available to annotators when assigning causes?
- What was inter-annotator agreement at each level?
- How were failures collected: naturally occurring, induced or simulated?
- What share of episodes are failures versus successes, by task?
- Are recovery attempts and their outcomes recorded?
- Can the labels be mapped to my taxonomy, and who would do that work?
- How many embodiments, sites and operators does the failure set cover?
- Were evaluation episodes held out by site or operator?
Discuss your requirements
If you can describe the tasks, embodiments and failure types you need labeled, discuss your humanoid training-data requirements with SourceX. SourceX reviews the requirement and follows up about potential sources. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation and engage specialist capture partners, scoped case by case. A request does not guarantee a match.
Start from the humanoid training data guide or the inspection, defect and failure data hub. The robotics and embodied AI overview and expert annotations and labels pages are related. See also failure reasoning data.
Discuss your humanoid training-data requirements with SourceX
Sources
- REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction — multisensory failure summaries and the RoboFail dataset.
- AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation — failure detection and explanation; generated failure trajectories.
- RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation — real-world failure demonstrations with causes across four embodiments.