Skip to content

AI uses for records

World model scenario coverage: which operating conditions are missing?

By SourceX Editorial · Updated

9 sections

Short answer

List the operating conditions the model must handle before looking at the data, using an operational design domain style taxonomy: layout, environment, load, equipment state, people, actions and time. Then map every episode to those conditions and mark each cell as observed with consequences, observed without them, absent but possible, impossible, or unobservable. The absent-but-possible and unobservable cells are the coverage gaps to fill or disclose.

Video and sensors

World model scenario

This page covers world model scenario coverage: which operating conditions are missing.Illustrative

Key takeaways

  • Define the condition space before looking at the data, or you can only describe what is there.
  • Operational design domain taxonomies from automated driving give a usable structure for other operations.
  • Tag conditions from records with units and timestamps, not from impressions of video.
  • An empty cell may be absent, unrecorded, impossible or unobservable; only some are real gaps.
  • Observed conditions without linked actions and consequences teach states, not dynamics.
  • Cells outside the deployment domain can stay empty if the documentation says so.

How do you identify operating conditions absent from a world model dataset?

Write down the conditions first and test the data against them second. A team that starts from the data can only describe what is there; a team that starts from a list of conditions can see what is missing. The list becomes a grid, each episode is tagged with the conditions it shows, and the empty cells are the answer.

The method has four steps. Define the condition space from the operation's own constraints and records. Tag episodes against it using logged values, not impressions from video. Classify every cell by what kind of evidence it holds, separating true absences from logging failures and physically impossible combinations. Then check whether the conditions you do have come with the actions and consequences a world model needs to learn from.

The output is a condition coverage inventory: one table that a buyer can read to see what a dataset can and cannot teach, and that a supplier can use to decide what to collect or disclose.

Why is scenario coverage a list of conditions rather than a count of hours?

Because a world model is judged on how well it predicts what happens next under a given condition and action, and hours of footage say nothing about which conditions they contain. A thousand hours of normal weekday operation can cover fewer conditions than fifty hours that include a power outage, a peak season and a short-staffed shift.

World models are built to forecast consequences. The DreamerV3 paper describes a world model that predicts future outcomes of potential actions, with a separate critic and actor learning from those predictions. A model of that kind can only forecast consequences for conditions it has some evidence about, so the coverage question is about conditions and the actions taken under them, not volume.

Safety assessment work for automated driving already frames coverage this way. A 2024 paper on coverage metrics for scenario databases asks whether collected scenarios cover the relevant aspects of the operational design domain and whether they capture the important situations present in the driving data, and proposes metrics to answer both questions quantitatively. The same two questions apply to an operating business: does the data span the conditions that matter, and does it include the situations that actually occur.

Defining the condition space with an operational design domain

Use the operational design domain idea from automated driving, adapted to your operation. An operational design domain names the operating conditions a system is designed to function under, and published taxonomies give it a hierarchy you can borrow.

ISO 34503:2023 specifies a hierarchical taxonomy for describing the operating conditions of an automated driving system, with static and dynamic attributes. BSI's PAS 1883:2025 is a guide to implementing that taxonomy, with practical examples such as how to describe spatial and temporal aspects of wind and rainfall. A warehouse, plant or field operation is not a road, but the structure transfers: fixed layout, environment, things that move, and how they change over time.

Research on machine learning datasets has made the link explicit. A 2024 paper on building a dataset compliant with an ML-based system's operational design domain proposes translating high-level operating constraints into properties that can be checked on individual data items, producing verifiable data quality requirements. That translation step is what turns a list of conditions into an inventory you can test.

Keep the taxonomy small enough to maintain. Seven or eight top-level dimensions, each with a handful of values defined by thresholds in the source systems, is usually enough for a first version. Version the taxonomy itself, because a change in a threshold or a new value changes what every cell means.

Defining the condition space with an operational design domain
DimensionExamples of valuesTypical record sourceCommon blind spot
Layout and fixed assetsZone, aisle type, door, station, equipment modelFacility drawings, asset registerLayout changes not reflected in old episodes
EnvironmentTemperature band, lighting, noise, floor condition, weatherBuilding management system, weather logsSensors installed after collection began
Load and capacityThroughput band, queue length, occupancy, staffing levelWMS or MES counts, timekeepingPeak periods trimmed from archives
Equipment stateNormal, degraded, under maintenance, alarm activeCMMS work orders, PLC alarmsDegraded states logged only when someone files a ticket
People and other agentsNumber present, roles, vehicles, contractorsAccess control, shift rosters, telematicsVisitors and contractors missing from rosters
Actions and interventionsRoutine action, override, manual takeover, stopControl logs, operator entriesOverrides done verbally and never logged
Time structureShift, day type, season, changeover windowCalendars, production schedulesHoliday and shutdown periods absent

The condition coverage inventory

The inventory has one row per condition cell that matters, not per possible combination. Combinations explode quickly, so start with single dimensions, then add the pairs your domain expert says change behavior, such as high load with degraded equipment. The illustrative columns below show what each row records.

Every value should come from a record with units and a timestamp, not from someone watching footage and judging the conditions. Where a dimension has no record, say so in the evidence column; that is itself a finding.

Choosing combinations is where domain knowledge earns its place. Ask the people who run the operation which situations change what they do: a full yard with a door down, a cold snap during a staffing shortage, a changeover during peak demand. Each named situation becomes a combined cell. Combinations nobody can name a reason for can wait; listing every cross product produces thousands of empty cells that hide the few that matter.

Count episodes per cell, and also count distinct days or events behind them. Nine episodes from one afternoon are one occurrence seen nine times, not nine independent examples, and a buyer needs to know the difference before deciding a cell is covered.

The condition coverage inventory
Inventory columnWhat it holdsIllustrative entryCheck before accepting
Condition cellOne value or a named combination of valuesDock door 4, ambient below freezing, high inbound volumeValues defined with units and thresholds
Evidence sourceSystem and field that establish the conditionBMS temperature sensor; WMS receipts per hourSource existed for the whole period
Episodes observedCount of episodes tagged with the cell9 episodesTagged from records, not from video review
Actions presentWhich actions appear under the conditionRoutine unload; 2 manual overridesOverrides logged with time and actor
Consequences linkedOutcome records joined within a stated windowDwell time, damage reports within 24 hoursWindow stated and applied to every episode
Cell statusObserved with consequences, observed only, absent, impossible or unobservableObserved onlyStatus rule written down and applied uniformly
Plausibility failuresEpisodes dropped for impossible or contradictory values1 dropped: door open while dock leveler stowedReason recorded for every drop
DisclosureWhat the buyer is told about the cellThin cell; consequences missing for 4 of 9 episodesStated plainly in the dataset documentation

Absent, unrecorded or impossible: which kind of gap is it?

An empty cell can mean four different things, and only one of them is a real coverage gap. Before you report a condition as missing, decide whether it never happened, happened but was not recorded, cannot happen, or cannot be seen with the sensors in place.

Source errors look like rare events and need to be ruled out the other way round as well. A temperature spike to an impossible value is a sensor fault, not an extreme condition, and a sudden drop in throughput may be a logging outage rather than a slow day. Check sensor health records and log completeness for the period before tagging an episode with an unusual condition.

Unrecorded conditions are found by asking, not by querying. Shift supervisors, maintenance technicians and safety staff usually know about events that never reached a system: a door propped open during a heat wave, a manual workaround during an outage, a week when a scanner was down. Interview them with the condition list in hand and note each answer in the inventory.

Impossible states should be checked against written constraints rather than intuition. Interlocks, rated capacities, safety procedures and physical limits define what cannot happen, and any episode that contradicts them points to a recording problem. Keeping that list next to the inventory makes the plausibility check repeatable when new data arrives.

Absent, unrecorded or impossible: which kind of gap is it?
Cell statusWhat it meansHow to test itWhat to do
Absent but possibleThe condition can occur and did not in the collected periodDomain expert confirms it is possible; records show no occurrenceDisclose; collect more or source elsewhere
UnrecordedThe condition happened, but no system captured itShift notes or interviews mention it; no field records itDisclose as unmeasurable; add logging if collection continues
ImpossiblePhysical or procedural constraints rule it outInterlocks, capacity limits or safety rules prevent itMark as excluded by design; drop episodes that claim it
UnobservableIt may occur, but the sensors cannot see itCamera or sensor placement cannot cover the location or variableDisclose; do not infer it from proxies without saying so
Source errorA record shows the condition, but the value is wrongSensor health log, contradictory fields, units mismatchDrop or correct the episode and log the reason

Do the conditions you observed carry their consequences?

Often only partly. A condition that appears without the action taken under it and the consequence that followed teaches a world model what the state looks like, not what happens next, so give those cells their own status.

Check each observed cell for three links: the state at the start, the action or intervention taken, and the consequence within a stated window. Record where an episode ended and why. The RLDS format makes the same distinction for robot learning data: each step marks whether it is first, last or terminal, and an episode that ends on a step that is not terminal is treated as truncated. A truncated episode in a rare condition is common, because recording often stops when something goes wrong, and it should be counted separately from a complete one.

Causal claims need care. An episode that shows high load, an override and a later delay does not show that the override caused the delay. The inventory records what was observed together; it does not assert cause, and the dataset documentation should say so.

Set the consequence window per outcome, not once for the whole dataset. Dwell time settles within hours; product damage may surface days later when a customer reports it; equipment wear may only show up at the next inspection. Write the window beside each consequence type in the inventory and apply it to every episode, so a cell is not marked complete simply because its episodes were recorded early.

Illustrative worked example: dock operations at a cold storage warehouse

This example is illustrative; the site, counts and windows are invented to show the method. A buyer wants a world model of dock operations at a cold storage warehouse, from camera footage joined to warehouse management, building management and maintenance records over twelve months.

The domain expert lists seven dimensions and eleven combinations that change behavior. Tagging from records puts most episodes in the expected cells: normal staffing, doors cycling, ambient temperature in its usual band. The inventory then shows the gaps. Freezer door failures appear in maintenance tickets four times, but cameras caught only one of them. A power outage occurred once, on a weekend when recording was paused, so its cell is unrecorded rather than absent. High inbound volume with one dock door out of service has six episodes, but consequence records exist for only two, because damage reports were filed on paper that month.

Two cells turn out to be impossible. The site's interlocks prevent a door from opening while the leveler is stowed, so three episodes tagged that way are traced to a logging fault and dropped. Overnight receiving never happens at this site, so that cell is excluded by design rather than reported as a gap.

The final inventory gives the buyer a clear request: more episodes of door failures with cameras running, consequence records for the high-volume cells, and a statement that outages are unrecorded. The supplier gets a short list of logging fixes if collection continues.

One cell needed a judgment call. Ambient temperature below freezing at the dock was recorded by a sensor installed in month five, so the first four months could not be tagged for that dimension. The team marked those months as unmeasurable for temperature rather than assuming normal conditions, which reduced the tagged episode count for every cold-weather cell but kept the inventory honest.

When a missing condition is fine, and when to stop

A missing condition is fine when the model will never be deployed under it. If a site does not operate overnight, a world model for that site does not need overnight episodes; mark the cell as outside the operating domain and say so. The gap matters only when the deployment domain includes it.

Stop or narrow the scope when the condition space cannot be defined with the people available, when key dimensions have no record at all, or when most cells in the deployment domain are unrecorded or unobservable. A world model claim built on that data would rest on proxies. Narrow to the zone, period or process where records are complete, and bring in a world model researcher and an operating domain expert when constraints, interlocks or sensor placement are disputed.

Rights belong in the inventory too. Each source system may have a different owner and terms, and footage of people needs its own permission review. Note in the evidence column which sources are cleared for the intended use.

Putting the inventory into a data request

Send the condition list with the request, not after the first delivery. Name the dimensions, the cells that matter for deployment, the consequence window and the status rule, and ask for the completed inventory as a deliverable alongside the data.

SourceX sources each dataset from U.S. businesses when a team asks rather than keeping stock, so the condition inventory can shape the request from day one. Its data request form takes scope, volume, language, format, timing and licensing needs, and the condition list belongs under scope. No footage or log leaves a supplier before it approves the scope and terms and signs.

Frequently asked questions

What is scenario coverage for a world model?

It is the extent to which a dataset contains the operating conditions a world model must handle, together with the actions taken and the consequences that followed. It is measured against a defined list of conditions, cell by cell, rather than as total hours or episodes.

How is a condition coverage inventory different from a slice coverage matrix?

A slice matrix usually compares a dataset's attributes with production traffic for any model. A condition coverage inventory is built for world models: it starts from the operating domain, classifies each empty cell as absent, unrecorded, impossible or unobservable, and records whether each observed cell has its actions and consequences linked.

Can conditions be tagged from video alone?

Only for what the camera can see, and even then judgment varies between reviewers. Tag from records with units and timestamps wherever they exist, and use video to confirm. Conditions such as temperature, equipment state or staffing are usually invisible or ambiguous on video.

Do automated driving standards apply to warehouses or plants?

Not directly. ISO 34503 and PAS 1883 are written for automated driving systems. Their structure of static and dynamic attributes is a useful template for listing conditions in other operations, but the dimensions and values must come from the operation itself.

What should be done with impossible episodes?

Trace them to their cause, which is usually a logging fault, unit mismatch or clock error. Drop or correct each one and record the reason, so a buyer can see that the data was checked against the site's real constraints.

Sources

  • DreamerV3 describes a world model that predicts future outcomes of potential actions, alongside a critic and an actor. Source
  • Coverage metrics for scenario databases test whether scenarios cover the relevant aspects of the operational design domain and capture the important situations in the source data. Source
  • ISO 34503:2023 specifies a hierarchical taxonomy for describing the operating conditions of an automated driving system's operational design domain. Source
  • PAS 1883:2025 is a guide to implementing BS ISO 34503, with practical examples such as spatial and temporal aspects of wind and rainfall. Source
  • Translating high-level ODD constraints into item-level properties yields verifiable data quality requirements for an ML dataset. Source
  • RLDS marks first, last and terminal steps and treats an episode that ends on a non-terminal step as truncated. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies