AI uses for records
World model counterfactual validity: testing business what-if outcomes
By SourceX Editorial · Updated
9 sections
Short answer
To test whether a business counterfactual has plausible consequences, have domain experts score each what-if rollout against a fixed rubric: is the intervention precise and feasible, does the starting state match real records, are capacity, timing and unit constraints respected, is the causal path named, and do analogous observed episodes support the direction and size of change. Then triage every failure as model error, source error or genuine uncertainty.
Video and sensors
World model counterfactual
Key takeaways
- A fluent or photorealistic rollout says little about whether its consequences would really follow.
- Score each counterfactual on ten criteria with written anchors, and require evidence for every score above zero.
- Mix real observed continuations into the panel's queue, blinded, to check that the experts and the rubric recognize reality.
- Before blaming the model, rule out clock and unit errors, actions taken outside the system and states that were never possible.
- Mark counterfactuals that rely on actions the business never took as outside support rather than pass or fail.
How do you test whether a business counterfactual has plausible consequences?
Break the rollout into a starting state, one intervention and a chain of consequences, then have people who run that operation score each link against written criteria and the records that bear on it. A counterfactual is the question DoWhy's documentation puts as: an outcome was observed under one setting, so what would have happened had that setting been different? A world model asked what if we had added a second shift answers that question in the form of a predicted sequence of states.
That makes the test harder than checking a forecast. DoWhy places interventions and counterfactuals on the second and third rungs of Pearl's ladder of causation, and notes that estimating counterfactuals requires stronger assumptions than generating interventional samples. No record will ever show the alternative history directly, so validity cannot be measured against ground truth. It can only be judged against constraints the operation cannot break and against observed episodes that resemble the intervention.
The practical test therefore has three parts: a rubric that turns expert judgment into comparable scores, a panel protocol that checks the experts themselves, and a triage step that decides whether a bad rollout reflects the model, the source data or a question nobody can answer with the evidence available.
Why does a convincing rollout prove so little?
Because models can learn to produce the look of an operation without its rules. The Physics-IQ benchmark, built from real videos filmed under controlled conditions, found that physical understanding in the video generation models it tested was limited and unrelated to how visually realistic their output was. The same split appears in business rollouts: a predicted week of warehouse states can read like a real operations report and still ship more orders than the docks could load.
There is also a feedback risk. In the original World Models paper, Ha and Schmidhuber trained agents inside a learned model of an environment and described how an agent could exploit the model's imperfections, finding behavior that worked in the learned model but not in reality. A planning system that searches over business counterfactuals will find the same kind of gaps: the cheapest plan is often the one the model gets most wrong.
World foundation model work makes the stakes plain. NVIDIA's Cosmos paper describes a world foundation model as a digital twin of the world that Physical AI systems can learn from, then post-trained into customized world models for specific uses. When a model is used to choose actions, the validity of its counterfactuals is the product being bought, not a side metric.
What should the domain expert counterfactual validity rubric include?
Ten criteria, each scored 0, 1 or 2 with written anchors and a required piece of evidence for any score above zero. Criteria one to three check that the question is well posed; four to seven check the consequences; eight to ten check how far the evidence reaches. A rollout passes only when no criterion scores zero, which stops a strong total from hiding one impossible step.
| Criterion | Question the expert answers | Anchors: 0, 1, 2 | Evidence required |
|---|---|---|---|
| 1. Intervention specified | Is exactly one change stated, with what, where, when and for how long? | 0 vague; 1 missing duration or scope; 2 fully specified | The intervention text as given to the model |
| 2. Intervention feasible | Could the business actually make this change in that state? | 0 impossible; 1 feasible with unstated preconditions; 2 feasible as stated | Policy, contract or equipment record showing the option existed |
| 3. Starting state real | Does the starting state match the records for that date and site? | 0 contradicts records; 1 partly matches; 2 matches | Source extract for the starting date |
| 4. Hard constraints kept | Are capacity, inventory balance, lead times, units and calendar respected at every step? | 0 any violation; 1 borderline; 2 none | Constraint list with limits and their source |
| 5. Causal path named | Can the expert trace how the change reaches each consequence? | 0 no path; 1 partial; 2 complete | Expert's written path, step by step |
| 6. Direction of effect | Do consequences move the way the operation's mechanics imply? | 0 wrong direction; 1 mixed; 2 consistent | Mechanism note plus any analog episode |
| 7. Side effects and timing | Are knock-on effects present and do they arrive with realistic delays? | 0 missing or instant; 1 some; 2 complete and timed | Downstream processes and their usual delays |
| 8. Magnitude supported | Is the size of change within what similar observed episodes showed? | 0 far outside; 1 at the edge; 2 within; or not assessable | IDs of analog episodes, or a note that none exist |
| 9. Off-system factors | Would people or events outside the records change the outcome? | 0 dominant and ignored; 1 acknowledged; 2 not material | Interview note or known manual workaround |
| 10. Uncertainty stated | Does the rollout's stated confidence match the evidence behind it? | 0 overconfident; 1 partly; 2 matched | Model's uncertainty output against criteria 8 and 9 |
How should an expert panel score rollouts without fooling itself?
Use at least two experts who score independently, blind to which model produced each rollout, and hide real observed continuations in the same queue. Those observed controls are periods where the change actually happened, such as a week when a dock door really was out of service, written in the same format as the model's output. If the panel scores real history as implausible, the problem is the rubric or the panel, not the model.
Measure agreement before trusting totals. Cohen's kappa expresses agreement between two raters corrected for the agreement expected by chance, and scikit-learn's implementation supports linear and quadratic weights, which suit ordinal scores like 0, 1 and 2 because they count a 0 against a 1 as a smaller disagreement than a 0 against a 2. Decide the agreement level and pass rule before scoring, not after.
- Calibrate on a shared practice batch, write down how borderline cases were settled and add those notes to the anchors.
- Rotate the order of rollouts so no model always appears first or last.
- Require a written causal path for criterion 5 even when the score is 2; it is the most useful artifact for the model team.
- Send any criterion where experts differ by two points to a third expert and keep all three scores.
- Version the rubric and record which version scored each rollout, because anchors will change after calibration.
Is the implausible rollout the model's fault, the data's, or nobody's?
Assign every failed criterion an origin before reporting it, because the remedy differs. A model trained on logs that recorded shipping before picking will predict the same; that is a source error, and retraining on the same records will not fix it. A model that violates a capacity limit that the records always respected has a model error. A question about an action the business never took has no answer in the data at all.
| Failure origin | Typical signature | Check | Action |
|---|---|---|---|
| Source error | Rollout repeats an impossible order of events seen in the records | Search source data for the same pattern and its cause | Fix or flag the source; rescore after retraining |
| Missing observation | Outcome depends on a phone call, expedite or manual override not logged | Ask operators how the gap was handled in practice | Add the factor to criterion 9; request off-system notes |
| Impossible state in training data | Starting state passes the model but fails criterion 3 | Compare with an independent system of record | Exclude the state from training and evaluation |
| Model error | Hard constraint broken although the records never break it | Confirm the constraint holds across the source period | Report to the model team as a defect |
| Causal uncertainty | No analog episodes; experts cannot agree on direction | Count episodes resembling the intervention | Score criterion 8 as not assessable; report outside support |
| Rubric ambiguity | Experts split on the same evidence | Review anchors against the disputed case | Revise anchors, version the rubric, rescore the batch |
Which negative controls catch a world model that invents effects?
Run interventions that should change nothing, and changes that should not alter the answer, and treat any movement as a defect. DoWhy's refutation guide describes two families of negative control: invariant transformations, which change the data in ways that should not change an estimate, such as subsampling, and nullifying transformations, such as a placebo treatment, after which the true effect is zero. Both translate into tests a business expert can design.
At a glance
Which negative controls catch a world model that invents effects
- Placebo intervention: change something with no mechanism to affect the outcome, such as a truck's label color or a cosmetic field, and expect no change in consequences.
- Irrelevant site: apply the intervention to a site that does not share the affected resource and expect the original site to be unaffected.
- Subsample stability: rerun the same counterfactual from several starting dates with similar states and expect similar consequences.
- Reversal: undo the intervention partway through and expect the operation to return toward its baseline with realistic delays.
- Dose check: a larger version of the same change should move consequences in the same direction, unless a known constraint caps it.
Worked example: a dock door closed for two weeks at a regional warehouse
This example is illustrative. A buyer's world model, trained on warehouse management and transportation records, is asked what would have happened had one of six outbound dock doors at a regional warehouse been closed for two weeks in March.
The rollout predicts slightly longer trailer dwell and no missed shipments. Two experts score it. Criteria 1 to 3 score 2: the change is specified, closures happen for repairs, and the starting state matches the records. Criterion 4 scores 0 for one expert, who notes that the rollout loads more trailers per door per shift than the site's own records ever show. Criterion 7 scores 1: the rollout raises dwell but never pushes late pickups into the next day's carrier schedule, which operators say is what happens. Criterion 8 is not assessable for the full two weeks, because the records contain only one closure of three days.
Triage assigns criterion 4 to model error, because the records consistently respect the per-door limit, and criterion 8 to causal uncertainty. The observed control, the real three-day closure, was scored plausible by both experts, so the panel is calibrated. The report to the model team names one defect and one question outside support.
The exception is a counterfactual about an action with no precedent, such as switching to a carrier the business has never used. Do not force a pass or fail. Score criteria 1 to 7 as usual, mark 8 as not assessable and state in the report that the consequence rests on the model's assumptions alone.
What should happen when the rubric cannot settle a rollout?
Stop scoring a class of counterfactuals when the evidence cannot support any verdict, and narrow the question until it can. A panel that keeps scoring unanswerable questions produces numbers that look like validation and are not.
- Stop when observed controls fail repeatedly: the panel or the records are not ready.
- Narrow to shorter horizons when experts agree on the first days of a rollout but not on later weeks.
- Narrow to interventions with analog episodes when most of criterion 8 is not assessable.
- Escalate to a causal inference specialist when the buyer wants effect sizes rather than plausibility judgments.
- Escalate to the source company's operators when a recurring failure points to an off-system practice nobody on the panel knows.
What should a buyer ask for when sourcing records to test counterfactuals?
Ask for the records that let the rubric be applied, not only the records the model trains on: periods when the intervention of interest really happened, the decision rules in force and their change dates, documented capacity limits, and notes on actions taken outside the system. Without them, criteria 2, 4, 8 and 9 cannot be scored.
Records that let a panel score counterfactuals are sourced through SourceX on request from U.S. businesses, not drawn from stock, and a buyer's request can describe the scope, volume, format, timing and licensing terms needed for an evaluation set like this. Before any of those records move, the business that holds them approves the scope and terms and signs.
Frequently asked questions
What is counterfactual validity for a world model?
It is the degree to which a model's what-if predictions respect how the real operation works: feasible interventions, real starting states, unbroken constraints and consequences supported by similar observed episodes. It cannot be measured against ground truth, because the alternative history never happened, so it is judged with a rubric and controls.
Is a counterfactual the same as predicting the effect of an intervention?
Not quite. An interventional question asks what happens on average if a change is made; a counterfactual asks what would have happened in a specific observed situation had something been different. DoWhy's documentation notes that counterfactuals require stronger assumptions.
Why mix real history into the experts' queue?
Real observed continuations, presented blind in the same format, test the panel and the rubric. If experts score reality as implausible, the anchors or the experts need calibration before any model verdict can be trusted.
Can historical records prove a counterfactual is right?
No. Records can show that a rollout breaks a constraint or contradicts similar observed episodes, which is strong evidence that it is wrong. They can support plausibility, but they cannot confirm what an untaken action would have caused.
Who should sit on the panel?
People who run the operation being modeled, such as a warehouse manager, planner or dispatcher, plus someone who knows the source systems well enough to spot logging artifacts. A causal inference specialist helps when the questions move from plausibility to effect sizes.
Sources
- DoWhy defines a counterfactual as what would have happened to an observed outcome under a different setting, and notes it needs stronger assumptions than interventional samples. Source
- DoWhy places interventions and counterfactuals on rungs 2 and 3 of Pearl's ladder of causation. Source
- DoWhy's negative control refutations use invariant transformations and nullifying transformations such as a placebo treatment. Source
- Cohen's kappa measures chance-corrected agreement between two annotators, with optional linear or quadratic weights. Source
- Ha and Schmidhuber trained agents inside a learned world model and described agents exploiting the model's imperfections. Source
- The Physics-IQ study found physical understanding in tested video models was limited and unrelated to visual realism. Source
- The Cosmos paper describes a world foundation model as a digital twin Physical AI systems can learn from, post-trained into customized world models. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.