Skip to content

Systems and records

World model simulation gap audit: simulated site versus real site

By SourceX Editorial · Updated

10 sections

Short answer

To measure the gap between a simulated operating site and a real one, initialize the simulator from an observed state, replay the interventions that actually happened over held-out periods, and compare simulated and observed state variables, event order and consequences declared in advance. Rule out clock, sensor and logging errors first, then report each remaining gap with its uncertainty. This guide gives the method and reports no results.

Video and sensors

World model simulation gap

This page covers world model simulation gap audit: simulated site versus real site.Illustrative

Key takeaways

  • A simulation gap audit replays real starting states and real interventions over held-out windows, then compares outcomes on measures fixed in advance.
  • Audit the observed record first: clock drift, unit errors, unlogged interventions and blind spots often look like simulator gaps.
  • Use several measures together: state error, distribution distance, event order, consequences and the rate of impossible simulated states.
  • A gap is real only if it persists across replications and held-out windows; resample whole windows, not single events.
  • For simulators randomized on purpose, test whether reality falls inside the randomized range rather than matching one nominal run.

How do you measure the gap between a simulated operating site and a real one?

Start the simulator from an observed state of the real site, replay the interventions that actually happened over a period the model was not built from, and compare simulated and observed outcomes on state variables, event order and consequences you declared in advance. Report each gap with its uncertainty, and only after ruling out errors in the observed record itself. This page describes the method; it reports no results.

That is the core of a world model simulation gap audit, whether the simulator is a discrete-event model of a warehouse, a physics simulator of a work cell or a learned world model that predicts the next state of a site. In each case the question is the same: given what really happened at the site, does the simulated site end up where the real one did, for the reasons the real one did?

Simulation practitioners have a long-standing name for part of this. Robert Sargent's verification and validation tutorials for the Winter Simulation Conference describe historical data validation, where part of the collected data is used to build the model and the remaining data is used to test whether the model behaves as the system does, and they describe driving the model with recorded traces. The protocol below extends that idea to the state, action and consequence structure a world model needs.

Is the observed record good enough to serve as the reference?

Audit the observed record before you audit the simulator. Apparent simulation gaps can trace back to clock drift, unit mismatches, unlogged interventions or sensors that could not see part of the site, and none of those are the simulator's fault.

Is the observed record good enough to serve as the reference?
CheckQuestion to answerGap it can fake
Time and unitsAre all sources on one clock and timezone, and are units declared per field?Events that appear early or late, or rates off by a constant factor
Sensor provenanceWhich device or system produced each value, and when was it calibrated?Drift that looks like a model bias
State constraintsDo observed states respect known capacities, such as dock doors, racks and staffing?Observed states that are themselves impossible
InterventionsIs every human override, reroute and stoppage logged with a time?Divergence that started with an unrecorded decision
Occlusions and coverageWhich zones or periods could the sensors not observe?Missing events read as events that never happened
Capacity changesWere equipment outages, shift changes or layout changes recorded?Throughput gaps caused by a different real site
RightsIs there a documented basis to use these records for model evaluation?No faked gap, but a dataset that cannot be used
Consequence evidenceAre outcomes such as late shipments or stockouts confirmed in a system of record?Consequences inferred from the same signals being tested

What must the comparison protocol fix before the first run?

Fix every element of the comparison before running it, write it down and date it. The table is the protocol; each row names what to decide and what the audit produces at that step.

  • Pin run settings tightly. Isaac Lab's documentation, for example, notes that its simulation is reproducible for rigid bodies and articulations on the same hardware and software version, but that results can vary across hardware because of floating point precision, and that the underlying physics engine does not ensure determinism for scenes with non-rigid bodies such as cloth.
  • Decide in step 5 whether you are testing the site model or the site model plus a policy. Replaying observed interventions tests the former; letting the simulator choose actions tests both, and the two should be reported separately.
What must the comparison protocol fix before the first run?
StepWhat to fix in advanceWhat the audit produces
1. Question and intended useThe decision the simulator will support and the outputs that matter for itA one-paragraph scope that limits which gaps count
2. Population and samplingSites, zones and calendar windows, with held-out windows chosen before anyone looks at resultsA sampling frame and the list of held-out windows
3. State variablesEach variable, its unit, its observed source and its simulated counterpartA variable dictionary that both sides map to
4. InitializationHow the simulator is set to the observed state at each window startA logged initial state per window
5. Intervention replayWhich logged actions are replayed exactly and which are left to the simulator's policyAn intervention trace per window
6. Run settingsSimulator version, seeds, time step, hardware and number of replicationsA run manifest that allows exact reruns
7. Comparison measuresMetrics per variable, event sequence and consequence, with their unitsA metrics specification
8. UncertaintyReplications, resampling scheme and interval methodIntervals around every reported gap
9. Acceptance thresholdsThe gap per variable that is acceptable for the intended useA pass, fail or inconclusive call per variable
10. LimitsConditions the audit cannot speak to, such as seasons or layouts not sampledA stated domain of applicability

Which measures compare a simulated trajectory with an observed one?

Use several kinds of measure, because a simulator can match averages while getting order and consequences wrong. Each measure answers a different question about the gap.

Which measures compare a simulated trajectory with an observed one?
MeasureWhat it comparesGood forCaution
State error over timeSimulated and observed values of a variable at matched timestampsQueue lengths, inventory positions, work in progressSmall timing offsets inflate the error; also report with a tolerance window
Distribution distanceSimulated and observed distributions of a quantity, such as dwell timesWhether the variability, not just the mean, is rightThe two-sample Kolmogorov-Smirnov test assumes independent samples, which consecutive operating data rarely are
Wasserstein-1 distanceHow much probability mass must move, and how far, to turn one distribution into the otherA gap expressed in the variable's own unitsNot a test by itself; pair it with resampled intervals
Event sequence agreementOrder and timing of key events, such as arrivals, starts and completionsCausal structure and bottleneck behaviorRequires a shared event vocabulary on both sides
Consequence agreementOutcomes such as late shipments, stockouts or missed cut-offsThe outputs the business actually decides onRare consequences need long windows or several sites
Impossible-state rateShare of simulated states that violate known constraintsCatching simulators that fit numbers through impossible statesConstraints must come from the site, not from the model

Is it the simulator, or a bad record, a blind spot or an impossible state?

Before reporting a gap as a simulator error, test the three alternative explanations in turn. Each has a distinct signature, and the audit should record which one was ruled out and how.

  • Only the residual gap, after these checks, is attributable to the simulator. Report the excluded windows and why, so a reader can see how much of the record survived.
  • NASA's standard for models and simulations lists input pedigree among the factors for judging the credibility of results, alongside verification, validation and uncertainty. That framing is useful here: a gap is only as credible as the pedigree of the observed record behind it.
Is it the simulator, or a bad record, a blind spot or an impossible state?
SymptomLikely explanationHow to test itClassify as
Divergence starts at one timestamp and growsAn unlogged intervention or outage at that timeCheck shift notes, maintenance and exception logs for that windowMissing intervention
Observed series has flat lines or sudden stepsSensor dropout or a backfilled valueCompare against a second source, such as scan events versus countsSource error
Simulated activity in a zone the real record shows as idleThe zone was not observed in that periodMap sensor coverage by zone and timeMissing observation
Simulated values exceed physical or staffing limitsSimulator dynamics or bad parametersRun the impossible-state check on every replicationSimulator error
Gap appears only on certain daysA condition outside the sampled domain, such as a holiday scheduleTag windows by condition and compareOut of domain
Gap disappears when one input is correctedInput pedigree problemTrace that input from its source through every transformationSource error

How many runs, windows and sites before a gap counts as real?

Treat one simulated run against one observed window as an anecdote. A gap counts as real when it persists across replications of a stochastic simulator, across held-out windows and, where the intended use spans them, across sites.

Run enough replications per window that the simulated distribution is stable, and compute intervals by resampling whole windows, such as days or shifts, rather than individual events, because events inside a window are correlated. Report every gap with its interval and the number of windows behind it.

Sargent's tutorials make a point worth repeating in any audit report: a model that passes at a number of experimental conditions is not thereby shown valid across its entire domain of intended applicability. State the domain the audit sampled, and do not let a pass in one season, layout or volume band stand in for the others.

When the intended use is a decision with high consequences, add a sensitivity check: perturb the inputs with the largest pedigree uncertainty within their plausible range and see whether the pass or fail call changes.

How would an illustrative held-out week at a cross-dock be replayed and checked?

The following example is illustrative and describes the method only; it reports no measured gaps. A team has a discrete-event simulator of a cross-dock with an illustrative 12 dock doors and wants to use it to test staffing plans.

They hold out one week of operations, initialize the simulator each morning from the observed trailer queue and door states, replay the observed trailer arrival times and door assignments, and leave unloading times to the simulator's sampled distributions. They declare four variables in advance: trailer dwell time, door utilization by hour, outbound departures by cut-off, and the count of late departures.

  • The exception in this example: one day falls on a holiday schedule that the simulator was never built for. The team reports it as out of domain rather than averaging it in.
  • The next step after this audit would be a second held-out week in a different volume band, because a pass in one week supports staffing decisions only in conditions like that week.
How would an illustrative held-out week at a cross-dock be replayed and checked?
Check during the auditWhat the team looked forHow the finding would be classified
Clock alignmentYard system and warehouse system timestamps compared for the same trailer eventsSource error if offset; corrected before comparison
Door outage on one dayMaintenance log entry showing a door out of service for part of a shiftMissing intervention; replayed and the day rerun
Dwell time distributionWasserstein-1 distance between simulated and observed dwell times, with resampled intervals by daySimulator gap only if it persists after the two fixes above
Late departuresCount per day in each replication versus the transport system's recordConsequence agreement; inconclusive if the counts are too small
Impossible statesAny replication with more trailers at doors than doors existSimulator error regardless of other measures

How should the test change when the simulator was randomized on purpose?

If the simulator was deliberately randomized, comparing the real site with one nominal simulated run is the wrong test. Ask instead whether the observed behavior falls inside the range the randomized simulations cover, and how often.

Domain randomization is the reason. Tobin and colleagues trained an object detector only on simulated images with randomized rendering, including non-realistic textures, and reported that it transferred to real images. Isaac Lab's actuator documentation similarly notes that randomizing actuator delay between resets can model transport delay during sim-to-real training. In both cases the simulator is meant to span reality, not to copy it.

For these simulators, add a coverage measure per variable: the share of observed windows whose values fall within the central range of the randomized ensemble. Low coverage says the randomization ranges miss the real site; very wide ranges with full coverage say the simulator may be too vague to support fine decisions. Report both.

What gets pre-registered, and when should the audit pause?

Write and date these items before the first comparison run, and change them only with a logged amendment. Then stop or narrow the audit if any of the listed conditions appear.

At a glance

What gets pre-registered, and when should the audit pause

  • Pre-register: the question and intended use, sites and held-out windows, variable dictionary with units, initialization and intervention replay rules, run manifest, metrics, interval method, thresholds and stated limits.
  • Pre-register: the triage rules for source error, missing observations, impossible states and out-of-domain windows, and how excluded windows are reported.
  • Pause if more than a small share of windows must be excluded for source problems; the record needs repair before the simulator can be judged.
  • Pause if the gap depends mainly on decisions people made that were never logged; the audit is now about capturing interventions, not about the simulator.
  • Bring in an operations expert for the site when observed states look impossible, because either the constraint list or the record is wrong.
  • Bring in a world model researcher when the gap appears only under the simulator's own policy, since that points to the policy or the learned dynamics rather than the site model.
  • Narrow the claim when the intended use spans seasons, layouts or sites the audit did not sample.

Where do observed operating records for the comparison come from?

An audit like this depends on observed operating records with states, interventions and consequences linked on one clock, which usually come from the companies running the sites rather than from public datasets.

A simulation team can describe the sites, windows and linked records it needs in SourceX's request-data form, with the scope, volume, language, format, timing and licensing needs, and SourceX then looks for them at U.S. businesses rather than drawing on a stock of site data. As a general guide, useful histories run two or more years of consistent records and tens of thousands of records or more in one system, though rare operations can qualify with less, judged deal by deal. The operator holding the records approves the scope and terms and signs before anything moves.

Frequently asked questions

What is a sim-to-real gap for an operating site?

It is the difference between what a simulator predicts and what the real site does when both start from the same state and receive the same interventions. It should be measured per variable and per consequence, with uncertainty, after errors in the observed record are ruled out.

Is a digital twin validation the same as a world model simulation gap audit?

They share the method. A digital twin validation typically checks a hand-built model against site data, while a world model audit may also cover learned dynamics; both use held-out observed windows, replayed interventions and pre-declared measures.

Can I use a Kolmogorov-Smirnov test to compare simulated and observed data?

It can describe how two distributions differ, but its p-value assumes independent samples, which consecutive operating data usually are not. Use it descriptively or on resampled windows, and pair it with a distance in the variable's units.

How much observed data does a simulation gap audit need?

Enough held-out windows to cover the conditions the simulator will be used for, with separate windows for each season, layout or volume band that matters. One week supports conclusions about weeks like it, not about the year.

Should interventions be replayed or left to the simulator?

Replay them to test the site model on its own, and let the simulator choose them only when you also want to test the policy. Report the two cases separately so a policy problem is not blamed on the site model.

Sources

  • Sargent's simulation validation tutorial describes historical data validation, where part of the data builds the model and the rest tests it, trace-driven runs, and notes that passing several conditions does not show validity across the whole intended domain. Source
  • NASA's standard for models and simulations assesses credibility with factors including verification, validation, input pedigree, results uncertainty and results robustness. Source
  • SciPy's ks_2samp compares two independent samples assumed drawn from continuous distributions, and wasserstein_distance computes the Wasserstein-1 distance between two one-dimensional distributions. Source
  • Isaac Lab documents that simulation results are identical on the same hardware and version for rigid bodies and articulations, can vary across hardware, and are not deterministic for non-rigid bodies. Source
  • Isaac Lab's actuator documentation notes that randomizing actuator delay between resets can model transport delay during sim-to-real training. Source
  • Tobin and colleagues trained an object detector only on simulated images with randomized, non-realistic textures and reported transfer to real-world images. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies