AI uses for records
World model scenario coverage: which operating conditions are missing?
By SourceX Editorial · Updated
9 sections
Short answer
List the operating conditions the model must handle before looking at the data, using an operational design domain style taxonomy: layout, environment, load, equipment state, people, actions and time. Then map every episode to those conditions and mark each cell as observed with consequences, observed without them, absent but possible, impossible, or unobservable. The absent-but-possible and unobservable cells are the coverage gaps to fill or disclose.
Video and sensors
World model scenario
Key takeaways
- Define the condition space before looking at the data, or you can only describe what is there.
- Operational design domain taxonomies from automated driving give a usable structure for other operations.
- Tag conditions from records with units and timestamps, not from impressions of video.
- An empty cell may be absent, unrecorded, impossible or unobservable; only some are real gaps.
- Observed conditions without linked actions and consequences teach states, not dynamics.
- Cells outside the deployment domain can stay empty if the documentation says so.
How do you identify operating conditions absent from a world model dataset?
Write down the conditions first and test the data against them second. A team that starts from the data can only describe what is there; a team that starts from a list of conditions can see what is missing. The list becomes a grid, each episode is tagged with the conditions it shows, and the empty cells are the answer.
The method has four steps. Define the condition space from the operation's own constraints and records. Tag episodes against it using logged values, not impressions from video. Classify every cell by what kind of evidence it holds, separating true absences from logging failures and physically impossible combinations. Then check whether the conditions you do have come with the actions and consequences a world model needs to learn from.
The output is a condition coverage inventory: one table that a buyer can read to see what a dataset can and cannot teach, and that a supplier can use to decide what to collect or disclose.
Why is scenario coverage a list of conditions rather than a count of hours?
Because a world model is judged on how well it predicts what happens next under a given condition and action, and hours of footage say nothing about which conditions they contain. A thousand hours of normal weekday operation can cover fewer conditions than fifty hours that include a power outage, a peak season and a short-staffed shift.
World models are built to forecast consequences. The DreamerV3 paper describes a world model that predicts future outcomes of potential actions, with a separate critic and actor learning from those predictions. A model of that kind can only forecast consequences for conditions it has some evidence about, so the coverage question is about conditions and the actions taken under them, not volume.
Safety assessment work for automated driving already frames coverage this way. A 2024 paper on coverage metrics for scenario databases asks whether collected scenarios cover the relevant aspects of the operational design domain and whether they capture the important situations present in the driving data, and proposes metrics to answer both questions quantitatively. The same two questions apply to an operating business: does the data span the conditions that matter, and does it include the situations that actually occur.
Defining the condition space with an operational design domain
Use the operational design domain idea from automated driving, adapted to your operation. An operational design domain names the operating conditions a system is designed to function under, and published taxonomies give it a hierarchy you can borrow.
ISO 34503:2023 specifies a hierarchical taxonomy for describing the operating conditions of an automated driving system, with static and dynamic attributes. BSI's PAS 1883:2025 is a guide to implementing that taxonomy, with practical examples such as how to describe spatial and temporal aspects of wind and rainfall. A warehouse, plant or field operation is not a road, but the structure transfers: fixed layout, environment, things that move, and how they change over time.
Research on machine learning datasets has made the link explicit. A 2024 paper on building a dataset compliant with an ML-based system's operational design domain proposes translating high-level operating constraints into properties that can be checked on individual data items, producing verifiable data quality requirements. That translation step is what turns a list of conditions into an inventory you can test.
Keep the taxonomy small enough to maintain. Seven or eight top-level dimensions, each with a handful of values defined by thresholds in the source systems, is usually enough for a first version. Version the taxonomy itself, because a change in a threshold or a new value changes what every cell means.
| Dimension | Examples of values | Typical record source | Common blind spot |
|---|---|---|---|
| Layout and fixed assets | Zone, aisle type, door, station, equipment model | Facility drawings, asset register | Layout changes not reflected in old episodes |
| Environment | Temperature band, lighting, noise, floor condition, weather | Building management system, weather logs | Sensors installed after collection began |
| Load and capacity | Throughput band, queue length, occupancy, staffing level | WMS or MES counts, timekeeping | Peak periods trimmed from archives |
| Equipment state | Normal, degraded, under maintenance, alarm active | CMMS work orders, PLC alarms | Degraded states logged only when someone files a ticket |
| People and other agents | Number present, roles, vehicles, contractors | Access control, shift rosters, telematics | Visitors and contractors missing from rosters |
| Actions and interventions | Routine action, override, manual takeover, stop | Control logs, operator entries | Overrides done verbally and never logged |
| Time structure | Shift, day type, season, changeover window | Calendars, production schedules | Holiday and shutdown periods absent |
The condition coverage inventory
The inventory has one row per condition cell that matters, not per possible combination. Combinations explode quickly, so start with single dimensions, then add the pairs your domain expert says change behavior, such as high load with degraded equipment. The illustrative columns below show what each row records.
Every value should come from a record with units and a timestamp, not from someone watching footage and judging the conditions. Where a dimension has no record, say so in the evidence column; that is itself a finding.
Choosing combinations is where domain knowledge earns its place. Ask the people who run the operation which situations change what they do: a full yard with a door down, a cold snap during a staffing shortage, a changeover during peak demand. Each named situation becomes a combined cell. Combinations nobody can name a reason for can wait; listing every cross product produces thousands of empty cells that hide the few that matter.
Count episodes per cell, and also count distinct days or events behind them. Nine episodes from one afternoon are one occurrence seen nine times, not nine independent examples, and a buyer needs to know the difference before deciding a cell is covered.
| Inventory column | What it holds | Illustrative entry | Check before accepting |
|---|---|---|---|
| Condition cell | One value or a named combination of values | Dock door 4, ambient below freezing, high inbound volume | Values defined with units and thresholds |
| Evidence source | System and field that establish the condition | BMS temperature sensor; WMS receipts per hour | Source existed for the whole period |
| Episodes observed | Count of episodes tagged with the cell | 9 episodes | Tagged from records, not from video review |
| Actions present | Which actions appear under the condition | Routine unload; 2 manual overrides | Overrides logged with time and actor |
| Consequences linked | Outcome records joined within a stated window | Dwell time, damage reports within 24 hours | Window stated and applied to every episode |
| Cell status | Observed with consequences, observed only, absent, impossible or unobservable | Observed only | Status rule written down and applied uniformly |
| Plausibility failures | Episodes dropped for impossible or contradictory values | 1 dropped: door open while dock leveler stowed | Reason recorded for every drop |
| Disclosure | What the buyer is told about the cell | Thin cell; consequences missing for 4 of 9 episodes | Stated plainly in the dataset documentation |
Absent, unrecorded or impossible: which kind of gap is it?
An empty cell can mean four different things, and only one of them is a real coverage gap. Before you report a condition as missing, decide whether it never happened, happened but was not recorded, cannot happen, or cannot be seen with the sensors in place.
Source errors look like rare events and need to be ruled out the other way round as well. A temperature spike to an impossible value is a sensor fault, not an extreme condition, and a sudden drop in throughput may be a logging outage rather than a slow day. Check sensor health records and log completeness for the period before tagging an episode with an unusual condition.
Unrecorded conditions are found by asking, not by querying. Shift supervisors, maintenance technicians and safety staff usually know about events that never reached a system: a door propped open during a heat wave, a manual workaround during an outage, a week when a scanner was down. Interview them with the condition list in hand and note each answer in the inventory.
Impossible states should be checked against written constraints rather than intuition. Interlocks, rated capacities, safety procedures and physical limits define what cannot happen, and any episode that contradicts them points to a recording problem. Keeping that list next to the inventory makes the plausibility check repeatable when new data arrives.
| Cell status | What it means | How to test it | What to do |
|---|---|---|---|
| Absent but possible | The condition can occur and did not in the collected period | Domain expert confirms it is possible; records show no occurrence | Disclose; collect more or source elsewhere |
| Unrecorded | The condition happened, but no system captured it | Shift notes or interviews mention it; no field records it | Disclose as unmeasurable; add logging if collection continues |
| Impossible | Physical or procedural constraints rule it out | Interlocks, capacity limits or safety rules prevent it | Mark as excluded by design; drop episodes that claim it |
| Unobservable | It may occur, but the sensors cannot see it | Camera or sensor placement cannot cover the location or variable | Disclose; do not infer it from proxies without saying so |
| Source error | A record shows the condition, but the value is wrong | Sensor health log, contradictory fields, units mismatch | Drop or correct the episode and log the reason |
Do the conditions you observed carry their consequences?
Often only partly. A condition that appears without the action taken under it and the consequence that followed teaches a world model what the state looks like, not what happens next, so give those cells their own status.
Check each observed cell for three links: the state at the start, the action or intervention taken, and the consequence within a stated window. Record where an episode ended and why. The RLDS format makes the same distinction for robot learning data: each step marks whether it is first, last or terminal, and an episode that ends on a step that is not terminal is treated as truncated. A truncated episode in a rare condition is common, because recording often stops when something goes wrong, and it should be counted separately from a complete one.
Causal claims need care. An episode that shows high load, an override and a later delay does not show that the override caused the delay. The inventory records what was observed together; it does not assert cause, and the dataset documentation should say so.
Set the consequence window per outcome, not once for the whole dataset. Dwell time settles within hours; product damage may surface days later when a customer reports it; equipment wear may only show up at the next inspection. Write the window beside each consequence type in the inventory and apply it to every episode, so a cell is not marked complete simply because its episodes were recorded early.
Illustrative worked example: dock operations at a cold storage warehouse
This example is illustrative; the site, counts and windows are invented to show the method. A buyer wants a world model of dock operations at a cold storage warehouse, from camera footage joined to warehouse management, building management and maintenance records over twelve months.
The domain expert lists seven dimensions and eleven combinations that change behavior. Tagging from records puts most episodes in the expected cells: normal staffing, doors cycling, ambient temperature in its usual band. The inventory then shows the gaps. Freezer door failures appear in maintenance tickets four times, but cameras caught only one of them. A power outage occurred once, on a weekend when recording was paused, so its cell is unrecorded rather than absent. High inbound volume with one dock door out of service has six episodes, but consequence records exist for only two, because damage reports were filed on paper that month.
Two cells turn out to be impossible. The site's interlocks prevent a door from opening while the leveler is stowed, so three episodes tagged that way are traced to a logging fault and dropped. Overnight receiving never happens at this site, so that cell is excluded by design rather than reported as a gap.
The final inventory gives the buyer a clear request: more episodes of door failures with cameras running, consequence records for the high-volume cells, and a statement that outages are unrecorded. The supplier gets a short list of logging fixes if collection continues.
One cell needed a judgment call. Ambient temperature below freezing at the dock was recorded by a sensor installed in month five, so the first four months could not be tagged for that dimension. The team marked those months as unmeasurable for temperature rather than assuming normal conditions, which reduced the tagged episode count for every cold-weather cell but kept the inventory honest.
When a missing condition is fine, and when to stop
A missing condition is fine when the model will never be deployed under it. If a site does not operate overnight, a world model for that site does not need overnight episodes; mark the cell as outside the operating domain and say so. The gap matters only when the deployment domain includes it.
Stop or narrow the scope when the condition space cannot be defined with the people available, when key dimensions have no record at all, or when most cells in the deployment domain are unrecorded or unobservable. A world model claim built on that data would rest on proxies. Narrow to the zone, period or process where records are complete, and bring in a world model researcher and an operating domain expert when constraints, interlocks or sensor placement are disputed.
Rights belong in the inventory too. Each source system may have a different owner and terms, and footage of people needs its own permission review. Note in the evidence column which sources are cleared for the intended use.
Putting the inventory into a data request
Send the condition list with the request, not after the first delivery. Name the dimensions, the cells that matter for deployment, the consequence window and the status rule, and ask for the completed inventory as a deliverable alongside the data.
SourceX sources each dataset from U.S. businesses when a team asks rather than keeping stock, so the condition inventory can shape the request from day one. Its data request form takes scope, volume, language, format, timing and licensing needs, and the condition list belongs under scope. No footage or log leaves a supplier before it approves the scope and terms and signs.
Frequently asked questions
What is scenario coverage for a world model?
It is the extent to which a dataset contains the operating conditions a world model must handle, together with the actions taken and the consequences that followed. It is measured against a defined list of conditions, cell by cell, rather than as total hours or episodes.
How is a condition coverage inventory different from a slice coverage matrix?
A slice matrix usually compares a dataset's attributes with production traffic for any model. A condition coverage inventory is built for world models: it starts from the operating domain, classifies each empty cell as absent, unrecorded, impossible or unobservable, and records whether each observed cell has its actions and consequences linked.
Can conditions be tagged from video alone?
Only for what the camera can see, and even then judgment varies between reviewers. Tag from records with units and timestamps wherever they exist, and use video to confirm. Conditions such as temperature, equipment state or staffing are usually invisible or ambiguous on video.
Do automated driving standards apply to warehouses or plants?
Not directly. ISO 34503 and PAS 1883 are written for automated driving systems. Their structure of static and dynamic attributes is a useful template for listing conditions in other operations, but the dimensions and values must come from the operation itself.
What should be done with impossible episodes?
Trace them to their cause, which is usually a logging fault, unit mismatch or clock error. Drop or correct each one and record the reason, so a buyer can see that the data was checked against the site's real constraints.
Sources
- DreamerV3 describes a world model that predicts future outcomes of potential actions, alongside a critic and an actor. Source
- Coverage metrics for scenario databases test whether scenarios cover the relevant aspects of the operational design domain and capture the important situations in the source data. Source
- ISO 34503:2023 specifies a hierarchical taxonomy for describing the operating conditions of an automated driving system's operational design domain. Source
- PAS 1883:2025 is a guide to implementing BS ISO 34503, with practical examples such as spatial and temporal aspects of wind and rainfall. Source
- Translating high-level ODD constraints into item-level properties yields verifiable data quality requirements for an ML dataset. Source
- RLDS marks first, last and terminal steps and treats an episode that ends on a non-terminal step as truncated. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.