Human Procedural Mistake Videos for Error Detection

Why mistakes are a separate class of video

Routine footage shows the right way to do a job. Error detection needs the opposite: clips where a step is skipped, done out of order, done with the wrong part or left incomplete, labeled against a written procedure. A humanoid that monitors its own work or a coworker's can flag a wrong step only if it has seen what wrong looks like at the step level.

Two neighboring data classes are easy to confuse with this one. Product defects are flaws on parts, covered in the guide to sourcing defect image datasets. Robot execution failures, such as drops and collisions, come from robots and are covered in robot manipulation failure datasets. Human mistake video sits between them: it shows procedure-level wrongness produced by people.

What published datasets show

These are research designs on toy vehicles, kitchens and industrial-like tasks, not workplace archives. They show which structure to ask a supplier for.

• Assembly101 recorded 4,321 videos of people assembling and disassembling 101 toy vehicles with 8 static and 4 egocentric cameras, unscripted, so mistakes occur naturally, and it introduced a mistake-detection task (Sener et al., 2022). Unscripted work yields real errors but gives you no control over which ones appear; many views help resolve steps hidden from one camera.
• IndustReal includes procedural errors such as omissions and execution errors, many of them only in its validation and test splits, in assembly and maintenance procedures (Schoonbeek et al., 2023). Errors can be reserved for evaluation.
• CaptainCook4D holds 384 egocentric recordings (94.5 hours) in real kitchens, with both normal and intentionally erroneous executions, 5.3K step annotations and IRB-approved written consent (Peddi et al., 2023). Scripted errors inside real environments are possible, and consent can be designed in.
• EgoPER covers error types beyond ordering errors and trains its detection framework only on error-free videos (Lee et al., 2024). If your method learns from normal video, you can buy plenty of it, but you still need labeled errors to test the method.
• HoloAssist is an egocentric dataset of human interaction for interactive AI assistants in real-world settings (HoloAssist, 2023). Check its documentation for how, and whether, errors are annotated before relying on it.

The way these sets scripted or reserved errors is compared with live capture in staged vs live operation recording.

An error taxonomy to ask for

Taxonomies differ between datasets, so define yours against your own procedure. The table below is a starting point.

Natural mistakes versus staged ones

Natural mistakes are realistic, rare and unevenly distributed across performers. Staged errors are controlled and cheap to produce, but they can differ from real ones: pauses, exaggeration or errors made in a consistent, easy-to-spot way. A model can learn the cues of staging instead of the error. Flag provenance per clip and per error (natural, scripted or induced), report evaluation results separately for each, and keep some natural errors in the test set.

"Induced" means conditions designed to raise natural error rates, such as an unfamiliar product variant, an ambiguous instruction or a less experienced performer. Treat it as a design option to pilot, and check whether the errors that result resemble those on your floor. Never stage an error that creates a hazard; some errors can only be recorded on dummy parts or in simulation. Failures and recoveries more broadly are covered in failed attempts and recoveries in demonstration data and rare events in demonstrations.

Where mistake video may exist

Real errors are recorded where something else records them. Useful companions are deviation reports, error-proofing records and rework and repair operations; process-monitoring camera footage covers evaluating the video. If no existing asset fits, SourceX can scope a commissioned collection of new recordings.

What to specify

• The procedure: a written version with step IDs; see the procedure step schema and SOP-to-video step alignment.
• The error taxonomy: definitions, example clips and an unclear label.
• Per clip: pseudonymous performer, experience level, session, site, camera views, error provenance and procedure version.
• Per error: type, step, start and end time, expected versus observed, whether and when it was corrected, and severity.
• Views: an egocentric view plus a fixed view, synchronized, so hidden steps can be resolved.
• Completeness: failures and recoveries kept in the clip, not trimmed.
• Splits: by performer and session, with some sites and error types held out for testing.

Illustrative annotation record

# Illustrative only — synthetic procedure, identifiers and values
clip_id: clip_site017_s0412_0033
procedure: {id: PROC_0417, version: rev_C, steps: 14}
performer: {id: operator_A12, experience: novice}
views: [ego_head, exo_fixed_left]
error_provenance: scripted          # natural | scripted | induced
events:
  - {step: S05, type: omission, t_start: 71.2, t_end: 74.8,
     expected: "apply retaining clip", observed: "step not performed", corrected: false}
  - {step: S09, type: wrong_object, t_start: 140.5, t_end: 146.0,
     expected: "screw M4x12", observed: "screw M4x16", corrected: true, t_corrected: 171.3}
outcome: {final_state: incorrect, check: "inspector verdict; no functional test run"}
annotation: {primary: ann_03, second: ann_07, event_agreement: adjudicated}

Quality and acceptance criteria

1. The procedure and its version are recorded, and every error is expressed against a step ID.
2. A sample is double-annotated, with agreement and adjudication reported per error type; see annotation quality for task video.
3. Counts per error type are given by distinct performers and sessions, not clips. Ten clips of one performer's habit are one example.
4. Provenance is flagged, and natural and staged errors are reported separately.
5. Corrections and recoveries are retained, with timestamps.
6. Timestamp precision and view synchronization meet the tolerance you set.
7. The test split contains natural errors, and performers or sites not seen in training.
8. Consent covers the specific purpose, including retention and use of error footage.

What this data cannot support

Human mistake video has no robot actions, joint states or forces. It does not transfer directly to robot failures: a person skips a step, while a robot drops a part or collides, which is why failure taxonomies for embodied tasks exist separately. It shows that an error happened and when, but not why, unless performers narrate. It cannot reveal errors that are invisible, such as a missing torque or a hidden connector, without outcome checks or tool data; see when video is not enough. And "wrong" is defined by a procedure, so a model trained on one site's procedures needs checking against another's.

Permissions and consent

Footage of mistakes shows people performing poorly, and employers may treat it differently from routine recordings. Ask whether consents cover AI training and the retention of error footage, whether any footage is connected to performance management, discipline or incident investigations, and who else appears, such as supervisors and trainers. CaptainCook4D's written, IRB-approved consent is a design reference, not a template for your jurisdiction. Blurring or pseudonymizing identities does not resolve permission questions. See worker consent and identifiable people. This is not legal advice.

Questions to ask suppliers

1. Does any footage show real errors, and how were they identified: by reviewers, supervisors or rework records?
2. Were errors edited out of training or marketing versions of the footage?
3. Which procedure and version does each clip follow, and are step IDs available?
4. For staged errors, who scripted them, were performers told which error to make, and are staged clips flagged?
5. How many distinct performers and sessions sit behind each error type?
6. Are corrections recorded as they happened?
7. Were performers told the footage might be used for AI training, and can they withdraw?
8. Is any footage connected to incident investigations or performance reviews?

Discuss your requirements

If you need video of procedural errors for detection or evaluation, tell SourceX which procedures, error types and views your evaluation set needs to cover. SourceX looks for US businesses that hold suitable footage, checks the data and the supplier's licensing permissions, and coordinates pricing, allowed uses and delivery in a license. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation and engage specialist capture partners, scoped case by case. SourceX reviews each request and follows up about potential sources; a request does not guarantee a match.

See also the inspection, defect and failure hub, the humanoid training data guide, the page on assembly completeness verification, procedural reasoning evaluation sets and the catalog entry for first-person video of skilled manual work.

Tell SourceX what your evaluation set needs to cover

Sources

• Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities — unscripted assembly with natural mistakes and a mistake-detection task.
• IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting — procedural and execution errors concentrated in evaluation splits.
• CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities — intentionally erroneous executions in real kitchens with written consent.
• Error Detection in Egocentric Procedural Task Videos — varied error types and detection trained on error-free video.
• HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World — egocentric human interaction data for AI assistants.