AI uses for records
Physical AI near-miss data: turning close calls into robot test cases
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Near-miss records support physical AI evaluation by describing real situations where harm almost happened: the task, the hazard, the precursors, what stopped it and what small change would have caused injury. Annotated with a safety-reviewed framework, each report becomes a test case stating what an embodied system in the same role should have noticed and avoided. They test judgment, not motion, so they complement robot demonstrations.
Video and sensors
Physical AI near miss data
Key takeaways
- A near miss records the moment before harm, which is exactly what an evaluator wants a robot to recognize early.
- Reports describe hazards in words after the fact; they contain no trajectories, so use them as evaluation scenarios rather than demonstrations.
- Each case needs a precursor sequence, the change that would have caused harm, what intervened and a plain statement of what a robot in that role should have done.
- A safety professional, not the annotator, signs off the hazard mechanism and the expected avoidance behavior.
- Injury logs, privacy concern cases, disciplinary files and footage of identifiable workers without consent stay out of the set.
How can near-miss workplace records support physical AI evaluation?
They give an evaluator realistic hazard scenarios with known precursors and a known near outcome, which is the hardest material to stage. OSHA's template near-miss form describes a near miss as an event where nothing was damaged and no one was hurt, but where a slight shift in time or position could easily have caused damage or injury. That slight shift is the evaluation target: did the system see the situation developing, and would it have kept the margin?
OSHA's hazard identification guidance treats close calls alongside injuries as a clear indication of where hazards exist, and recommends investigating them for root causes. A company that has run a reporting program for years therefore holds a catalogue of its own hazard situations, written by the people who were there. For a team building a mobile manipulator or a humanoid for a warehouse, plant or kitchen, that catalogue answers a question simulation designers guess at: which situations actually go wrong in this kind of workplace.
The limit is equally clear. A report is a short narrative written after the event. It has no joint angles, no camera calibration and rarely a timestamp finer than the hour. It can tell an evaluator what to test, and give a ground truth for what should have been avoided, but it cannot teach a policy how to move. Treat near-miss data as scenario and judgment data that sits next to demonstrations, teleoperation and robot logs.
Which near-miss records are worth annotating, and which should stay out?
Annotate records that describe a task, a hazard mechanism and what stopped the harm; leave out records whose value is the injury or the person. Most safety systems mix several record types, and they differ sharply in evaluation value and in risk.
| Record type | What it gives an evaluator | What it lacks | Decision |
|---|---|---|---|
| Written near-miss report | Task, location type, hazard and what nearly happened, in the reporter's words | Precise timing, positions and often the root cause | Annotate; the core of the set |
| Investigation with root cause | Causal chain, contributing conditions and the reviewer's conclusion | Often written for a few serious events only | Annotate and link to its report |
| Corrective action record | The control the site chose: guard, procedure, layout or training | The event itself unless linked by ID | Link; it states the expected avoidance behavior |
| Safety observation or audit card | Unsafe conditions seen before anything happened | Any outcome or consequence | Keep as condition-only cases, labeled separately |
| Camera clip tied to a report | Real geometry, timing and the moment of intervention | Consent of visible workers in many archives | Annotate only with consent or effective redaction |
| OSHA 300 and 301 injury entries | Confirmation that harm occurred | Any near-miss content; adds injury and privacy detail | Exclude |
| Disciplinary or HR files about the event | Nothing the evaluator needs | Fairness and confidentiality for the worker | Exclude |
What should the safety-reviewed near-miss annotation framework include?
Each case needs twelve fields: enough to restate the situation for a robot, the change that would have caused harm, what intervened and the behavior an embodied system should show. The table below is the annotation record. Annotators fill most fields from the report; a safety reviewer owns the hazard mechanism, the near outcome and the expected behavior.
For the hazard mechanism, borrow an existing vocabulary rather than inventing one. The Bureau of Labor Statistics publishes the Occupational Injury and Illness Classification System manual, whose Event or Exposure structure comes with code titles, descriptions and selection rules. Using its event categories, such as struck by, caught in or fall to a lower level, lets cases be counted and balanced across hazard types and compared with a company's own safety statistics.
- Two further fields complete the record: robot relevance (yes, partial or no, with a reason when a case depends on a purely human act) and withdrawal status, so a case can be removed later without breaking the rest of the set.
- Keep every free-text field in role language. A reviewer should be able to read the whole record without learning who was involved.
| Field | Values or format | Filled by | Why an evaluator needs it |
|---|---|---|---|
| Case ID and source links | Stable case ID plus IDs of the report, investigation and corrective action | Data engineer | Keeps the chain auditable and lets cases be withdrawn |
| Task, phase and setting | Task name, phase (approach, handling, transfer, exit) and location type | Annotator | Places the case where a robot would be working |
| Hazard mechanism | Event category from the OIICS Event or Exposure structure plus the energy source | Safety reviewer | Allows balancing and comparison across hazard types |
| Actors and positions | Roles only (driver, picker, contractor) with relative positions | Annotator | States who was where without naming anyone |
| Precursor sequence | Ordered list of observable cues before the moment of risk | Annotator, checked by reviewer | Defines what a system should have perceived and when |
| Near outcome | The slight shift in time or position and the harm it would have caused | Safety reviewer | Turns the report into a scored test, not a story |
| What intervened | Person, guard, interlock, emergency stop, layout or chance | Annotator | Separates designed protection from luck |
| Expected embodied behavior | One or two sentences: stop, yield, slow, re-route, refuse, ask | Safety reviewer | The ground truth the system is graded against |
| Evidence level | Narrative only, narrative with photo, or narrative with synchronized video | Data engineer | Tells the evaluator how much the case can support |
| Review record | Reviewer role, date, agree or amend, open disagreement note | Safety reviewer | Shows the case was checked and by whom |
How does a safety reviewer sign off each annotated case?
In two passes: the annotator drafts the case from the records, then a qualified safety professional confirms or amends the hazard mechanism, the near outcome and the expected behavior before the case is accepted. The reviewer is checking safety logic, not grammar, so the review form should show the original report text next to the draft fields.
At a glance
How does a safety reviewer sign off each annotated case
The reviewer also protects the reporting program. OSHA's template near-miss policy tells employees that reporting will not lead to discipline except for serious misconduct, and the same template counts challenged safety barriers, such as a bypassed machine guard, as reportable events. Reports written under that promise should never come back as evidence against a worker, so the annotation rules forbid blame language and keep the reviewer focused on conditions and controls.
- Pass 1, annotator: fill the fields from the report, investigation and corrective action; flag anything inferred rather than stated.
- Pass 2, safety reviewer: confirm the event category, rewrite the near outcome if the annotator overstated or understated it, and write or approve the expected embodied behavior.
- Disagreement: when the reviewer and annotator differ on the mechanism or expected behavior, record both and send the case to a second reviewer; do not average them.
- Rejection: cases where the mechanism cannot be determined from the records are logged as rejected with a reason, not silently dropped, so the batch report shows what was excluded.
- Calibration: review a small shared batch together before the main run and write down how borderline cases are decided.
What variation and recovery behavior do near-miss records leave out?
They leave out the many times the same situation passed without incident, most of the recovery motion, and every event nobody reported. A reporting archive is a record of what people chose to write down, so busy shifts, contractors and small sites are often underrepresented, and a case set built only from reports will overweight memorable events.
Recovery is usually one line in a report, such as the driver braked or the picker stepped back. That line is valuable as the expected behavior, but it is not a demonstration. Where a camera clip exists, annotate the recovery as a separate segment with its own start and end, and note whether the clip was trimmed before the situation resolved.
If the cases will sit beside robot data, align them with robot episode conventions. Open X-Embodiment represents each of its datasets as a sequence of episodes in the RLDS format, where each step marks whether it is the first or last of an episode. RLDS also notes that an episode ending on a step that is not terminal has been truncated. A near-miss clip cut off before the outcome is truncated in exactly that sense, and labeling it so stops a model or evaluator from reading the cut as the end of the risk.
- Record the reporting rate context: which sites and shifts reported, so coverage gaps are visible.
- Add matched normal cases: the same task and location on an ordinary day, so the system is tested on not overreacting as well.
- Mark camera viewpoint, field of view and whether the hazard is visible at all in the frame.
- Note synchronization: whether clip time and report time agree, and by how much they differ when known.
Worked example: a forklift and pedestrian close call at a dock door
This example is illustrative. A distribution center reports that a picker stepped out from behind a pallet stack into a forklift travel lane at a dock door; the driver braked and the picker stepped back. The investigation names blind-corner visibility as the root cause, and the corrective action adds a convex mirror and a painted pedestrian zone.
Annotated, the case reads: task, pallet transfer in the dock approach phase; hazard mechanism, struck by a powered industrial vehicle; actors, driver and picker by role; precursors, stacked pallets blocking the line of sight, a pedestrian path crossing the lane, reversing alarm audible; near outcome, a slight shift in timing would have put the picker in contact with the forks; what intervened, the driver's braking; expected embodied behavior, an autonomous mobile robot or a humanoid moving loads in that aisle slows before the blind corner and yields until the crossing is visibly clear. Evidence level is narrative only, so the case tests scene reasoning, not perception.
In an illustrative batch of 400 reports, a team might find that roughly a quarter lack a recoverable task or mechanism and are rejected, that dock and aisle events dominate, and that only a small share have linked footage. Those proportions are placeholders; the point is that the batch report states them so a buyer sees coverage before using the set.
- Exception: a report that a worker skipped a lockout step before clearing a jam does not map directly to a robot doing the same task, because the robot would be under its own operating mode and stop rules. Keep it as partial relevance, with an expected behavior stated at the level of the system, such as refusing to reach into a cell that is not in a safe state, and let the safety reviewer decide.
- Exception: a report whose hazard was purely a human decision with no physical cue, such as working while fatigued, is marked not robot-relevant and excluded from the robot set.
Which privacy and consent limits decide what stays in the set?
Keep hazards, conditions and controls; remove people. The OSHA recordkeeping rule already recognizes that some injury cases are sensitive: for privacy concern cases, 29 CFR 1904.29 replaces the employee's name on the log with the words privacy case and lets the employer describe a case in less detail when the worker could otherwise be identified. That is one reason injury logs stay out of a near-miss evaluation set entirely.
Narratives can identify people even without names. A small crew, a rare task or a dated shift can point to one person, so generalize dates to a period, replace site names with location types and drop details that only matter for blame. Footage needs more care. The Ego4D project, for example, describes recording either in controlled settings with informed consent from all participants or in public spaces with faces and other personal information blurred. A workplace archive collected for safety monitoring usually had neither purpose in mind, so check consent, notices and any labor agreement before a clip enters the set.
Access to the safety system, permission under privacy law and the right to license are separate questions. Redaction reduces privacy risk but does not by itself give a company the right to license records, and counsel should review the specific facts, including any reports prepared at a lawyer's direction.
Which cases call for a smaller batch or a robotics safety engineer?
Narrow the batch when cases cannot be annotated honestly, and bring in a robotics safety engineer when expected behaviors start to look like safety requirements. An annotated case says what a system should have done in one situation; it is not a risk assessment and does not replace one. ISO 10218-1:2025 sets safety requirements for industrial robots, with requirements on modes and robot classification, and ISO 10218-2:2025 covers robot applications and cells. Those standards, applied by an engineer, govern how a deployed system is made safe; the case set only tests judgment.
- Narrow to one hazard family when most cases in other families are rejected for missing mechanisms.
- Narrow to narrative-only cases when footage consent cannot be confirmed for a site.
- Pause any case under a litigation hold, insurer claim or open regulatory inquiry until counsel clears it.
- Escalate to a robotics safety engineer when a buyer wants expected behaviors written as pass or fail criteria for a deployed robot.
- Escalate to a privacy reviewer when a site is small enough that role and date alone identify a person.
Can a company license its near-miss archive?
A company with years of near-miss reports, investigations and corrective actions can offer them as licensed scenario data, once rights, privacy and scope are settled. SourceX handles that path from rights review and contracting through preparation and delivery, turning selected records into licensed datasets for AI developers.
A safety or operations leader can learn whether the company qualifies through a brief questionnaire, with no reports sent, and nothing is delivered until the company has approved the scope and terms and signed, which leaves room to hold back sites, clips or record types the safety team is not comfortable sharing.
Frequently asked questions
Is a near miss the same as an incident?
No. OSHA's template form describes a near miss as an event where no one was hurt and nothing was damaged, but a slight change in time or position could have caused harm. Incidents involve actual harm and bring injury and privacy details that a robot evaluation set does not need.
Can near-miss reports train a robot directly?
Not as motion data. Reports describe situations in words after the fact and contain no trajectories or sensor streams. They are best used as evaluation scenarios and expected behaviors, next to demonstrations and robot logs that teach movement.
Do near-miss cases need video to be useful?
No. Narrative-only cases test whether a system reasons correctly about a described scene. Synchronized video adds perception testing, but only where visible workers consented or were effectively redacted, so many sets start with narratives.
Should OSHA 300 logs go into a near-miss dataset?
No. Injury logs record harm rather than near misses and include privacy concern cases that the recordkeeping rule protects. Keep the evaluation set to near-miss reports, investigations and corrective actions.
Who should review the annotations?
A qualified safety professional should confirm the hazard mechanism, the near outcome and the expected behavior for each case. A privacy reviewer checks identifiability, and a robotics safety engineer is needed if the expected behaviors will be used as acceptance criteria for a deployed robot.
Sources
- OSHA's template near-miss form defines a near miss as an event without damage or injury where a slight shift in time or position could easily have caused it. Source
- OSHA's template near-miss policy counts challenged safety barriers as reportable and says reporting does not lead to discipline except for serious misconduct. Source
- OSHA treats close calls and near misses as indicators of where hazards exist and recommends investigating them for root causes. Source
- The BLS OIICS manual provides an Event or Exposure code structure with titles, descriptions and selection rules. Source
- For privacy concern cases the OSHA log shows privacy case instead of the name, and employers may describe cases with less detail. Source
- RLDS steps mark the first and last step of an episode, and an episode ending on a non-terminal step is truncated. Source
- Open X-Embodiment represents each dataset as a sequence of episodes in the RLDS format. Source
- Ego4D describes recording with informed consent in controlled settings or blurring faces and personal information in public spaces. Source
- ISO 10218-1:2025 sets safety requirements for industrial robots, including mode requirements and robot classification. Source
- ISO 10218-2:2025 specifies safety requirements for industrial robot applications and robot cells, covering integration, commissioning, operation, maintenance and decommissioning. Source
See if your company qualifies
A short company assessment. No data uploads are needed.