Capturing Rare Events and Edge Cases in Workplace Demonstrations

The long-tail problem

Routine footage is dominated by the normal path: object where expected, tool works, hands unobstructed. The cases that stop a robot are rarer and more varied: a dropped item, a jammed latch, a missing part, a blocked path, a wet floor, a person walking through the workspace, a wrong object in the bin. A dataset with thousands of normal attempts and a handful of exceptions teaches little about the exceptions, and an evaluation set without them overstates reliability.

This page is about acquiring rare-event data. The broader treatment of mistakes and recovery behavior is in failure and recovery demonstrations; here the focus is how to find, stage and label events you cannot wait for.

Define the event list first

Start with a list built from operational knowledge, not from what footage happens to contain. Sources include incident logs, quality-escape reports, maintenance records, supervisor interviews, and the task's failure analysis. For each event, record:

Do not assume rarity estimates; measure them from a sample of actual footage when you can.

Four capture routes

1. Mine existing footage. Search archives for natural occurrences. Tools range from supplier logs (incident tickets, quality flags, work-order notes) to model-assisted search. Egocentric datasets built for interactive assistants, such as HoloAssist (HoloAssist, 2023), show the kind of recorded human-assistance interactions researchers study; check whether any such corpus covers your events rather than assuming it does. For corpora of daily-life video, scale helps: Ego4D spans 3,670 hours across 74 locations (Grauman et al., 2021), but a corpus that size gives no guarantee about any specific event, and you must search to find out. Mining is often the cheapest route but is limited by how often the event occurred and was kept.

2. Stage safe variants. Recreate events that can be reproduced without risk: a dropped item, an obstructed path, mislabeled containers, an out-of-place tool, a missing part, a jammed lid. Staged events need a script, a safe setup, and a label saying "staged". Never stage hazards that endanger people or equipment; skip them or use simulation.

3. Deliberate error execution. Ask demonstrators to make planned mistakes. Published procedural-activity research does this: CaptainCook4D records normal and intentionally erroneous executions of recipes (Peddi et al., 2023) and Assembly101 is unscripted and includes natural mistakes (Sener et al., 2022). Planned errors are controllable but may not match how real errors look.

4. Simulation or synthetic variants. For events that are hazardous or too costly to stage, simulation can supplement, with all the usual caveats about the gap to real footage. This page does not cover it further.

Robot datasets show the same concern from the other side. RoboMIND includes about 5,000 real-world failure demonstrations with failure causes, alongside 107,000 demonstration trajectories (Wu et al., 2024). That is robot teleoperation, not human video, but it shows the kind of labeling buyers may ask for: the outcome and its cause.

What rare-event data can and cannot support

Can support

• Evaluation sets that stress a model on conditions it must handle; see task understanding evaluation sets.
• Anomaly, error and outcome recognition from video, using success and outcome labels.
• Training for recovery behavior when paired with robot data on the same events.
• Safety review: which situations require stopping or asking for help.

Cannot support

• Rare-event robot control on its own. Human video of a recovery shows what a person does, not what a robot should command.
• Estimates of true event frequency, unless the sample was random and large.
• Coverage of hazards that cannot be staged.
• Generalization to events unlike the listed ones.

Quality and acceptance

• Each event clip carries: event name, trigger, cause, response, outcome and whether staged or natural.
• Clips include enough lead-in and aftermath to see cause and response, for example 10 seconds before and after, a buyer-set value.
• Staged and natural events are separable by metadata so you can train on one and evaluate on the other.
• Held-out events are split by site, operator or scenario type, not by random clip. See demonstration dataset splits.
• Near-misses are labeled as such.

Illustrative event capture plan

# Illustrative only - synthetic values
task: restock_shelf
events:
  - id: EV01 item_slips_during_grasp
    route: stage
    target_clips: buyer-set
    safety: spill-safe items only
  - id: EV02 shelf_slot_occupied
    route: mine + stage
    mining_source: supplier incident notes
  - id: EV03 customer_in_aisle
    route: mine
    note: subject to consent; no staged strangers
  - id: EV04 heavy_item_wrong_shelf
    route: skip   # hazard; consider simulation
labels: [event_id, cause, response, outcome, staged_or_natural]
split: hold out EV02 and EV03 by site for evaluation

Permissions and provenance

Rare events are often the sensitive ones: injuries, customer disputes, safety incidents, equipment failures. Footage kept for incident review or insurance may carry privacy, confidentiality or legal-hold constraints, and may show identifiable people at a bad moment. Ask who decides whether incident footage can be licensed, and whether people shown know. This is not legal advice. Redaction does not by itself resolve permission questions. See worker consent and identifiable people.

Questions to ask suppliers and collection teams

1. How do you identify rare events in existing footage, and what logs support the search?
2. Can you provide counts of each event type found in a sample, and how the sample was drawn?
3. Which events are staged, how, and what is the safety plan?
4. How are staged and natural events labeled?
5. Do clips include lead-in and aftermath?
6. Who owns incident footage, and is it releasable for training?
7. How do you label cause and recovery, and who reviews the labels?
8. Can you hold out events by site or operator?

Build the list with your demonstration coverage plan and choose between mined and staged capture with staged vs live operation recording. For selection from archives, see curating raw workplace video.

Discuss your requirements

If your evaluation or training plan depends on rare situations, describe the events, tasks and permitted uses to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery; where no existing asset fits, SourceX can scope a commissioned collection. A request does not guarantee a match. Start from the humanoid training data guide and the workplace task demonstration hub; the catalog page on evaluation sets covers evaluation data in general.

Tell SourceX what your evaluation set needs to cover

Sources

• Ego4D: Around the World in 3,000 Hours of Egocentric Video — scale and diversity of a broad daily-life corpus.
• RoboMIND — robot failure demonstrations with causes.
• HoloAssist — egocentric human interaction dataset for interactive assistants.
• CaptainCook4D — recordings of normal and intentionally erroneous procedure execution.
• Assembly101 — unscripted assembly with natural mistakes.