Short answer
Service-task failures such as spills, dropped items and broken ware are often not recorded, because operations may delete them or never film them. Sources include incident footage, staged recordings by trained staff, and annotated robot logs. Buyers should separate human mistakes from robot failures, label cause and consequence, and treat staged failures as a design choice with limits.
Why service failures differ from manufacturing defects
A manufacturing defect is a property of a part: a scratch, a void, a missing screw. A service-task failure is an event: a tray tilts, a glass slips, a bin lid catches, a mop pushes dirt rather than lifting it. It happens in time, depends on the object's behavior when handled and often ends in a consequence such as spilled liquid, a broken item or a stopped task.
That makes the data different. Defect imagery is static and plentiful in quality archives; failure examples are sequences, rarely retained and seldom labeled. Systems that serve, clean, restock or carry objects need such sequences to learn what going wrong looks like, to detect it early and to recover. For defect images of products, start instead from defect image datasets.
Types of failure and what they look like
| Failure | Typical onset | Visible signal | Consequence |
|---|---|---|---|
| Spill | Tilt or overfill while carrying | Liquid on item, tray or floor | Hazard, rework |
| Drop | Slip during grasp or transfer | Object leaves hand, impact | Breakage, loss |
| Breakage | Impact or excess force | Fragments, cracks | Safety and cleanup |
| Collision | Misjudged clearance | Contact with fixture or person | Damage, stop |
| Incomplete task | Interruption or error | Missing step, wrong placement | Redo |
| Hand-over failure | Release before the receiver grasps | Object falls between agents | Drop |
The consequences also create new states, such as liquid on a floor, which feed hazard detection; see environmental hazard images. Category design is covered in failure taxonomies for embodied tasks.
Where failure examples come from
Existing material. Incident video, security footage, near-miss reports and training clips can include real failures. They are scarce, sensitive and usually recorded from fixed cameras at poor angles for manipulation analysis. Records such as near-miss and incident reports help locate events but rarely contain usable video.
Staged recordings. Trained staff perform tasks and are asked to fail on purpose, or are asked to perform under conditions that raise failure probability (overfilled cups, wet hands, unstable stacks). This gives control over viewpoint, labeling and consent, but staged failures can be unnatural, and they cover only the failures someone thought of.
Robot logs. Failures from teleoperated or autonomous robots include robot state and actions. RoboMIND reports 5,000 real-world failure demonstrations with causes, alongside trajectories from four embodiments, one of them a humanoid with dual dexterous hands (Wu et al., 2024). These are published research data; business logs from service deployments are rare and tied to specific robots.
Human-assistant datasets. Egocentric human-interaction datasets built for AI assistants, such as HoloAssist, have been published (HoloAssist, 2023). Whether such a set contains the mistakes you care about is something to check: a general egocentric set is not a service-failure corpus, and confirming what it contains is part of due diligence. See human procedural mistake videos.
What this data can and cannot support
It can support failure detection, early warning from visual cues (tilt, slip, object leaving hand), evaluation of recovery behavior, and language explanations when paired with annotation, as described in failure reasoning data.
It cannot support, without more:
- Robot recovery policies from human footage. Human recovery does not map onto robot actions without retargeting or robot data; see the human-to-humanoid embodiment gap.
- Forces and slip physics. Video shows that a glass slipped, not the grip force or friction. Instrumented gloves or robot tactile sensing are needed for that.
- A representative failure rate. Staged and incident sets are selected, so frequencies in them say nothing about how often failures occur in operation.
- Behavior of your robot. A robot with different hands and gait fails differently.
Illustrative package
Values are synthetic placeholders.
| Component | Illustrative content |
|---|---|
| Clips | Pre-failure lead-in, failure event, aftermath |
| Viewpoints | Wrist or head-worn plus two fixed cameras |
| Task context | Task name, objects, container, liquid or solid |
| Failure label | Type, onset time, consequence, severity |
| Cause label | Visible cause and hypothesized root cause, kept separate |
| Staged flag | Natural, induced or scripted |
| Recovery | What the person did next, if anything |
| Consent record | Role, release scope, whether faces were captured |
Give each clip a matched success clip of the same task where possible, otherwise detectors learn the background rather than the failure.
Preparation, consent and limits
Staged failures need safety planning: nobody should be asked to risk injury, and breakable items should be low-value test items. Ergonomic and handling hazards are real in service work; OSHA's overview of ergonomics describes how forceful exertion, repetitive motion and awkward postures contribute to musculoskeletal disorders (OSHA), which is a reason to keep staged tasks short and non-strenuous.
Workers, guests and customers can appear in frame. Recording people at work raises consent, notice and workplace-policy questions, and anyone other than the performer may need separate handling; see worker consent and identifiable people in workplace recordings. Blurring faces does not settle whether the recording may be used for training and may remove cues a model needs.
Choosing a mix for your evaluation
For an evaluation set, the unit of coverage is the failure mode, not the clip. List the failures that matter for your tasks, such as liquid spills during carrying, drops during hand-over and breakage when placing on hard surfaces, and count how many distinct examples you have for each under different objects, containers and surfaces. A few examples repeated across many clips add little.
Match the objects. A set built around paper cups and plastic trays will not exercise glassware, hot containers or irregular items. If your deployment involves specific objects, ask the supplier whether those can be included, and whether the staged items represent them. For product-level object data, see the guidance on object recognition data versus manipulation training data, which explains why images of objects do not show how they behave when handled.
Keep an unseen portion. Hold out entire sites, people or object types when the claim concerns generalization, and keep those out of any training or tuning.
Questions to ask suppliers
- Are the failures real, induced or scripted, and is that recorded per clip?
- What viewpoints and frame rates were used around the failure moment?
- Do clips include the lead-up so early cues can be learned?
- Is there a matched success example for each task and setup?
- Who performed the tasks, how many people, and under what consent terms?
- Were guests, customers or bystanders captured, and how is that handled?
- How are cause and consequence labeled, and by whom?
- Can you provide a sample of clips with labels so I can test them against my evaluation design?
Discuss your requirements
If failure examples are what your service-task evaluation lacks, describe the tasks, failure types and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. Where no existing asset fits, SourceX can scope a commissioned collection; a request does not guarantee a match. See the humanoid training data guide and the inspection, defect and failure data hub. The page on robot manipulation failure datasets covers robot-side logs, and the catalog page on video recordings describes the media type.
Describe the workplace demonstration data your robotics project needs
Sources
- RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation — 5k real-world failure demonstrations with causes; four embodiments.
- HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World — egocentric human interaction data for assistants (existence and nature only).
- Ergonomics - Overview — ergonomic risk factors and musculoskeletal disorders.