Skip to content

Privacy and preparation

World model rare event coverage: sampling the unusual operating day

By SourceX Editorial · Updated

9 sections

Short answer

Include unusual operating days on purpose, not by luck. Define the day or shift as the sampling unit, stratify days by operating records (saturation, outages, interventions, extremes, process changes, incidents), oversample rare strata with recorded inclusion probabilities, and link each day's observations to its actions and matured consequences. Then require a coverage report showing counts per stratum, zero-observed strata with their exposure, plausibility failures and exclusions.

Video and sensors

World model rare event

This page covers world model rare event coverage: sampling the unusual operating day.Illustrative

Key takeaways

  • A uniform sample of days mostly teaches a world model what ordinary days look like; consequences change on the rare ones.
  • Define unusual days from operating records, such as capacity, downtime, interventions and incidents, not from how footage looks.
  • Oversample rare strata, but store each day's inclusion probability so the ordinary distribution can be restored by weighting.
  • Rule out sensor faults, missing observations and impossible states before calling something a rare event.
  • Report zero-observed strata with the number of days examined, never as an empty row or a smoothed percentage.
  • A coverage report is the artifact to require from a supplier; without it, rare-event claims cannot be checked.

How do you include unusual operating days in world model datasets?

Treat unusual days as a sampling design problem. Pick a unit, usually an operating day or a shift at one site, build a frame of every unit in the history, label each with strata derived from operating records, and then draw a sample that deliberately over-represents the rare strata while recording the probability with which each unit was drawn. Every selected day keeps its full sequence of observations, actions and consequences, not a highlight clip.

Two things make this different from adding edge cases to a classifier's training set. A world model predicts what happens next given what was done, so the value of an unusual day is the change in consequences, which often unfolds over hours. And because the model also has to predict ordinary days well, the dataset must let you restore the true mix, which is what the recorded inclusion probabilities are for.

The output a buyer should ask for is a coverage report: what the population was, how days were classified, how many of each kind exist, how many were sampled and delivered, which strata were never observed, and what was excluded and why. The rest of this page builds that report.

Why does a uniform sample of days leave a world model blind to consequences?

Because rare days are rare, and a random sample of a modest size will contain few or none of them. In DreamerV3, a widely cited world model agent, the model learns from experience to predict the future outcomes of potential actions, and behavior is trained in imagination on those predictions. If the experience never included a jammed conveyor at peak volume, the model has nothing to imagine from when it meets one.

The exposure needed to observe rare outcomes grows quickly. RAND's 2016 Driving to Safety report concluded that showing autonomous vehicle safety through test driving alone would take hundreds of millions, and sometimes hundreds of billions, of miles, because fatal and injury crashes are so infrequent per mile. An operating site has the same arithmetic on a smaller scale: if a dock overflow happens a few times a year, a sample of thirty random days is likely to contain none.

Teams that care about the tail curate for it. Waymo's WOD-E2E benchmark, for example, contains 4,021 driving segments, about 12 hours, selected for long-tail scenarios that the paper says occur with a frequency below 0.03% in daily driving. The lesson for operating data is the method, not the domain: decide which rare situations matter, find them in the records, and select them deliberately.

Which days count as unusual, and who decides?

Operating records decide, and the definitions are written down before anyone looks at footage. Classifying days by how interesting the video looks favors visually dramatic events and misses quiet ones, like a day when a sorter ran at reduced speed and every downstream queue grew. Safety engineering makes a similar distinction: ISO 21448 addresses risk from functional insufficiencies, where nothing broke but the function was not adequate for the situation, which is exactly the kind of day a visual skim misses.

The frame below is a starting set of strata for a physical operation such as a warehouse, plant or yard. Each stratum has a definition a person can compute from records, and a note on where consequence evidence lives. A day can belong to several strata; record all of them and assign a primary stratum for sampling.

Which days count as unusual, and who decides?
StratumDefinition from recordsExample signalsWhere consequences are recorded
Capacity saturationThroughput or queue above a set percentile of the site's history for a sustained periodWMS task backlog, dock door occupancy, buffer full eventsLate shipments, overtime, carrier detention
Equipment outage or degradationUnplanned downtime or reduced-speed mode on a key assetCMMS work orders, PLC fault codes, speed setpoint changesRerouted work, missed cutoffs, maintenance closeout
Intervention-heavyManual overrides or robot assists above a set rate per hourOverride logs, teleoperation sessions, exception queuesResolution time, rework, error tickets
Environmental extremeTemperature, weather or lighting outside a set rangeSite sensors, weather records, facility alarmsDamage claims, slowed operations, closures
Process or layout changeFirst days after a new layout, product, shift pattern or software releaseChange tickets, slotting changes, release notesError rates and productivity in following weeks
Safety eventA day with an OSHA-recordable injury or a reported near missIncident reports, OSHA 300 log entriesRecordability, days away, corrective actions
Sensor or data degradationCamera offline, clock drift or log gaps above a thresholdHealth monitors, sync checksNot an operating consequence; tracked for exclusion
Ordinary baselineNone of the aboveNormal ranges on all signalsRoutine outcomes

The rare event sampling and coverage report

This is the asset to require, or to produce if you are assembling the dataset yourself. It is short enough to fit on two pages, and every part can be checked against delivered data. If a supplier cannot fill a part, the blank is itself useful information.

At a glance

The rare event sampling and coverage report

Keep the inclusion probability per day even when it feels like bookkeeping. With it, you can train on an oversampled mix and still weight evaluation, or a second training stage, back to the site's real distribution. Without it, the rare strata silently inflate how often the model thinks things go wrong.

  • Population and unit: sites, date range, and whether the unit is a day, a shift or a fixed window.
  • Stratum frame: the definitions above as used, with each threshold and the record that computes it.
  • Frame counts: how many units fall in each stratum across the full history, before sampling.
  • Sampling design: draws per stratum, the method, and the inclusion probability stored for every selected unit.
  • Delivered counts: units sampled, units delivered, and the reason for every unit sampled but not delivered.
  • Zero-observed strata: each stratum with no units, stated as zero observed in a stated number of days examined.
  • Consequence linkage: the share of delivered units whose consequences were joined, and the maturity window used.
  • Plausibility failures: units flagged for impossible states, clock problems or missing observations, and how each was resolved.
  • Exclusions for rights and privacy: counts withheld and the category of reason, without identifying details.
  • Versions: frame definitions, sensor and layout versions in force, and the report date.

Is it a rare day, a broken sensor or a gap in the logs?

Often, and it should be ruled out before a day is labeled rare. A spike in robot stop events can be a real obstruction or a failing sensor. A day with no recorded picks for two hours can be an outage or a log that stopped writing. Before stratifying, run plausibility checks on each candidate day: inventory that goes negative, an asset in two places at once, timestamps that run backward, units that change mid-stream, and counts that disagree between two systems that should match.

Separate three kinds of missing. A missing event never happened; a missing observation happened but was not captured, for example because a camera was occluded; a truncated sequence stopped being recorded before the outcome. Robot data formats already draw part of this line. RLDS, for instance, treats an episode that ends without a terminal step as truncated rather than finished. Apply the same rule to operating days: a day whose logs stop before close is truncated, not quiet.

Watch the curation step too. General-purpose video pipelines filter heavily; the Cosmos world foundation model platform, for example, describes splitting, filtering, annotation, deduplication and sharding, with filters for motion, visual quality, text overlay and video type. Filters like these are designed for overall quality, and they need a check that they do not discard unusual but valid footage, such as a dim loading bay during a power fault.

Finally, record interventions as actions, not noise. When a supervisor halts a line to prevent a jam, the consequence the model would have learned never occurs. That day still belongs in the sample, with the intervention recorded, because it shows the causal path that was cut.

How should a coverage report show strata with few or no days?

With counts and exposure, not percentages. Three sampled safety-event days out of a frame of five should read as 3 of 5, not 60%, because the percentage hides how little evidence there is. A stratum with nothing in it reads as zero observed in, say, 730 days examined, so a reader knows how much history was searched and can judge for themselves how rare the situation might be.

Do not fill empty strata with generated or simulated days and report them under the same heading. Simulation can be useful, but it belongs in a separate, labeled section of the dataset with its own provenance. Mixing the two makes the report claim coverage that the operating history does not contain.

Show how sensitive the counts are to the thresholds. If capacity saturation means the top 5% of backlog hours, say what the count becomes at the top 10% or top 2%. A stratum that swings from forty days to four under a small change in definition is a definition problem, and buyers should see that before they build evaluation sets on it. The same goes for the maturity window: report how many consequences were joined at the chosen window and how many more arrive if it is doubled.

Finally, keep the frame and the sample separate in the report. The frame counts say what the history contains; the sample counts say what was delivered. A buyer who sees both can tell the difference between a site where outages are genuinely rare and a delivery that simply left them out.

What does a filled report look like for one distribution center?

The following is illustrative. All counts are invented to show the method and do not describe a real site or a delivered dataset. A buyer asks for one year of operations from a single distribution center: synchronized overhead video, WMS tasks, conveyor PLC events, labor and maintenance records, with days as the unit.

The frame holds 362 operating days. Records classify 41 as capacity saturation, 17 as equipment outage, 23 as intervention-heavy, 6 as environmental extreme, 9 as process change, 4 as safety event, 12 as sensor degradation and the rest as baseline, with overlaps assigned a primary stratum. The design draws 30 baseline days at random and up to 6 days from each rare stratum, storing inclusion probabilities.

The report then shows that all 4 safety-event days were withheld for privacy review, so that stratum is delivered as zero with the reason stated. Two outage days were flagged as clock drift and resolved by re-syncing to the PLC clock. One saturation day was truncated when recording stopped mid-shift and is delivered with that flag. Consequence linkage reaches 57 of 60 delivered days, using a 14-day window for late-shipment outcomes. The exception worth noting is the sensor-degradation stratum: it was used only to exclude days, not delivered as an operating condition, and the report says so.

Linking each unusual day to states, actions and consequences

A rare day without its consequences is a curiosity, not training data. Join observations to actions on one clock, with declared units and coordinate frames, and attach state snapshots such as inventory positions, queue lengths and asset modes at regular intervals. Record capacity constraints in force that day, because a world model cannot learn that a full buffer blocks upstream work unless the buffer's limit is known.

Consequences mature on different timescales. A jam shows up in minutes, a missed carrier cutoff in hours, a damage claim in weeks. Set a maturity window per consequence type, join what is known by the window's end, and mark the rest as not yet matured rather than as no consequence.

Incidents need care. OSHA treats a work injury as recordable when it leads to death, days away, restricted work or transfer, medical treatment beyond first aid or loss of consciousness, or is a significant diagnosed injury. Those records are strong consequence evidence and also contain health information about identifiable people, which is why many suppliers will withhold or heavily reduce them, as in the example above.

Asking SourceX for unusual-day coverage

A buyer who needs unusual operating days can make the coverage report part of the request itself, because SourceX sources datasets from U.S. businesses on request instead of holding them in stock. Share the scope, volume, format, timing and licensing needs, and name the strata you care about, the unit and the consequence window. Long, consistent operating histories make rare days findable, and a smaller set can still qualify when the situations it captures are rare. The supplying company approves the scope and signs before anything is delivered, which is also when the withheld categories are settled.

Frequently asked questions

What is rare event coverage in a world model dataset?

It is the extent to which the dataset contains the unusual operating states a model will face, together with the actions taken and the consequences that followed. It is measured per stratum of unusual day, with counts and the amount of history searched, rather than as a single percentage.

Doesn't oversampling rare days bias the model?

It changes the training mix, which is the point, but it should not be hidden. Store each day's inclusion probability so evaluation, or a later training stage, can be weighted back to the site's real distribution. Without the weights, a model may overestimate how often things go wrong.

Can simulation replace rare operating days?

Simulation can add scenarios that never happened, but it cannot show how a real site responded. Keep simulated or generated days in a separate, labeled part of the dataset with their own provenance, and never count them toward coverage of observed strata.

How many years of history are needed to find rare days?

It depends on how often each situation occurs. A situation that happens a few times a year needs several years of consistent records to yield a handful of examples, so ask suppliers for frame counts per stratum across the full history before agreeing on a sample size.

What if a supplier withholds incident days?

That is common and often appropriate, because incident records contain health information about identifiable people. The coverage report should still show the stratum, the count withheld and the category of reason, so you know the gap exists and can plan evaluation around it.

Sources

  • DreamerV3 learns a world model that predicts future outcomes of potential actions and trains behavior in imagination. Source
  • RAND's 2016 Driving to Safety report concluded that hundreds of millions, sometimes hundreds of billions, of test miles would be needed for clear statistical evidence of autonomous vehicle safety. Source
  • WOD-E2E contains 4,021 driving segments, about 12 hours, curated for long-tail scenarios occurring below 0.03% of daily driving. Source
  • The Cosmos platform's video curation pipeline splits, filters, annotates, deduplicates and shards video, filtering on motion, visual quality, text overlay and video type. Source
  • ISO 21448 addresses the absence of unreasonable risk from hazards caused by functional insufficiencies of the intended functionality. Source
  • OSHA's general recording criteria list death, days away, restricted work or transfer, medical treatment beyond first aid, loss of consciousness and significant diagnosed injuries as recordable outcomes. Source
  • RLDS treats an episode that ends on a non-terminal step as truncated. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies