AI uses for records
World model dataset pilot design: what to prove before collecting more
By SourceX Editorial · Updated
10 sections
Short answer
A small world model data pilot should establish that observations, actions and following states join on one clock in known units, that consequences are recorded, that actions vary beyond the routine path, that source errors and gaps are measurable, and that the records may be used. It should also test one written hypothesis against a baseline, with success and failure thresholds fixed before any results are seen.
Video and sensors
World model dataset pilot
Key takeaways
- A world model pilot proves the joins between observation, action and consequence before anyone pays for volume.
- Write a testable hypothesis with a target, horizon, conditioning actions, baseline and thresholds fixed in advance.
- Clock meaning, units and coordinate conventions fail more pilots than model choice; test them on day one.
- Label every episode terminal, truncated or invalid so a model does not learn from recording artifacts.
- A flat result calls for checking source errors, gaps, impossible states and unlogged interventions before giving up on the source.
What should a small world model data pilot establish before larger collection?
A pilot should prove five things on a few days of records from one operating area: that observations, actions and the states that follow can be joined on one clock in known units; that the consequences you want a model to predict are actually recorded; that the actions in the data cover more than the routine path; that source errors, gaps and impossible states are rare enough to measure and filter; and that the records may be used for the intended training. It should also test one stated learning hypothesis against a baseline.
If any of the first four fail, more volume will not fix them. A year of footage with no action log is still passive video, and a year of sensor data on two unsynchronized clocks is still a year of guesses about cause and effect. The pilot exists to find those faults while they are cheap.
The rest of this guide gives a pilot hypothesis and acceptance specification that a buyer and a data owner can agree before collection starts, the time, unit and episode checks that fail most often, and a worked example of a pilot that passed with an important exception.
Why collecting everything first rarely produces world model data
A world model learns how a scene changes in response to actions. In the original World Models paper, Ha and Schmidhuber describe a generative model that learns a compressed spatial and temporal representation of an environment, with a variational autoencoder compressing each frame and a recurrent network predicting what comes next. The training signal is the pairing of what was seen, what was done and what happened after.
Operational records are rarely collected with that pairing in mind. Cameras record continuously, control systems log commands on their own clock, and outcomes sit in a third system such as a warehouse or maintenance application. Collecting all of it before checking the joins tends to produce a large archive that cannot answer the basic question of which action preceded which change.
Scale does not remove the need for curation. NVIDIA's Cosmos platform paper describes a video pipeline that splits, filters, annotates, deduplicates and shards footage before training. A pilot applies the same discipline at small scale and adds what curation of public video cannot supply: the action and consequence records held by the operator.
How to write a pilot hypothesis a single operating area can test
A good hypothesis names one operating domain, one prediction target, the actions that condition it and a measurable improvement over a baseline. Vague goals such as 'see whether the data is useful' produce pilots that cannot fail and therefore teach nothing.
At a glance
How to write a pilot hypothesis a single operating area can test
Write the hypothesis in this form before any data moves:
Checklist · 6 items
- Domain: the physical area and the equipment or process in scope, with what is excluded.
- Target: the state the model should predict, at what horizon, for example the position of an item a few seconds after a diverter command.
- Conditioning actions: the commands or human actions logged with time and value.
- Baseline: the comparison, such as a model that ignores actions or a simple persistence forecast.
- Success criterion: the held-out improvement over baseline that would justify more collection, fixed before results are seen.
- Failure criterion: the result that would stop or narrow the program.
0 of 6 ticked. Tick each item as you go.
What should the pilot hypothesis and acceptance specification include?
The specification below is the asset to agree between the AI team and the data owner. Each row has an acceptance test that can be run on the pilot delivery. The example column describes an illustrative conveyor sortation pilot, not a real dataset.
| Area | What the spec states | Example (illustrative) | Acceptance test | Fails when |
|---|---|---|---|---|
| Hypothesis | Target, horizon, conditioning actions, baseline and success threshold | Predict parcel position at a short horizon given diverter commands, against a no-action baseline | Threshold written and signed before delivery | Threshold set after looking at results |
| Scope | Area, equipment, shifts and exclusions | One sortation line, two shifts, maintenance windows excluded | Delivery covers only the stated scope | Footage or logs from outside the scope appear |
| Time | Clock source per stream, timestamp meaning and allowed offset | Cameras and controller on one time server; offset under one frame | Known events align across streams within the offset | Offset unknown or drifting |
| Units and frames | Unit for every numeric field and the coordinate convention | Meters, seconds, radians; right-handed frame per the site map | Every field has a unit in the data dictionary | Unitless numbers or mixed conventions |
| Actions | Which actions are logged, with value and actor | Diverter command, belt speed setpoint, manual jam clearance | Each action has time, value and actor or source | Human interventions present in video but absent from logs |
| Consequences | Which outcomes are recorded and how they are verified | Scanned destination chute and exception reason | Outcome joins to the action episode it follows | Outcomes recorded only as daily totals |
| Coverage | Expected range of conditions and rare events | Peak and off-peak volume, jams, mis-sorts | Each listed condition appears at least a few times | Only routine operation present |
| Episodes | Start and end rules, terminal versus truncated | One parcel from induction to chute | Every episode flagged terminal, truncated or invalid | Episodes cut at file boundaries without a flag |
| Rights and people | Basis for using footage and logs, people in frame, client goods | Workers notified; client labels obscured | Written confirmation from the data owner and reviewer | Any stream with no recorded basis |
| Documentation | Datasheet covering motivation, composition and collection | Short datasheet delivered with the pilot | Reviewer can answer the datasheet questions from it | Collection process undocumented |
Time, units and coordinate frames: the joins that break first
Most failed pilots fail on time. Decide which clock each stream uses and what its timestamps mean before the first file is exported. The MCAP container format, widely used for robot logs, distinguishes a message's log_time, when it was recorded, from its publish_time, when it was published, and defines timestamps as nanoseconds since an epoch the user must understand, such as Unix time or robot boot time. If a delivery mixes boot-relative and wall-clock times without saying so, every join is wrong.
Units and frames are the second trap. ROS REP 103 standardizes on SI units, including meters, seconds and radians, requires right-handed coordinate systems, and sets body axes as x forward, y left and z up, with an optical frame variant for cameras. Whatever convention the site uses, the specification should name it, and the pilot should include a known motion, such as a measured item moving a known distance, that proves the convention was applied consistently.
Run both checks on day one of the pilot, not after training. A one-frame offset or a flipped axis produces a model that looks plausible and learns the wrong causality.
Where does an episode end: terminal, truncated or invalid?
World model training needs episodes whose ends mean something. The RLDS format makes the distinction explicit: each step carries flags for is_first, is_last and is_terminal, and an episode whose last step is not terminal is treated as truncated. Episodes can also be marked invalid so they are discarded at read time.
Operational records need the same three-way label. A parcel that reaches its chute ends in a terminal state. A recording that stops at a file rollover or a camera reboot is truncated, and a model should not learn that parcels vanish at the end of a file. An episode with a known logging fault is invalid. The pilot specification should state which events produce each label, and the acceptance test should count episodes with no label at all.
Telling a weak learning signal apart from broken records
When a pilot model fails to beat its baseline, check the records before concluding the data has no value. Each failure mode below has a quick diagnostic that can be run on the pilot itself.
| Possible cause | What it looks like | Diagnostic on the pilot | What to change |
|---|---|---|---|
| Source error | Outcomes that contradict the video, such as a scanned chute the parcel never reached | Sample episodes and compare the video against the logged outcome | Fix the join or drop the faulty source field |
| Missing observations | Gaps where the object leaves the frame or a sensor drops out | Share of episodes with gaps longer than the prediction horizon | Add a camera, shorten the horizon or mark gaps explicitly |
| Impossible states | Objects jumping between positions or speeds beyond physical limits | Physics checks on positions and velocities | Repair timestamps or units before training |
| Unlogged interventions | People changing outcomes without a logged action | Video review of episodes with unusual outcomes | Log the intervention or exclude the period |
| Too little variation | Actions almost always take the same value | Distribution of each action value in the pilot | Collect during periods with more variation before scaling |
| Model or horizon mismatch | Records check out but the target is unpredictable at that horizon | Compare results across several horizons | Revise the hypothesis rather than the data |
Worked example (illustrative): a sortation line pilot that passed with one exception
This example is invented to show how the specification is used; all counts and durations are illustrative. A parcel distribution center offers two weeks of overhead video from one sortation line, controller logs of diverter commands and the scan records showing which chute each parcel reached.
On day one the alignment test fails: controller timestamps run several seconds behind the cameras. The site traces it to a controller that was not synchronized with the time server, fixes the setting and re-exports the logs with a corrected offset. The unit check passes after the data dictionary is updated to state that belt speed is logged in meters per second.
The coverage test shows jams in only a small share of episodes, enough to keep, but too few for a rare-event target. The team keeps the original hypothesis, which concerns routine diversion, and records jam prediction as out of scope for this pilot.
The exception: during manual jam clearance, a worker moves parcels by hand and no action is logged. Those periods would teach a model that parcels move without cause. The specification is amended so manual clearance periods are marked from the video and excluded, and the full collection request adds a manual-mode flag from the controller.
Stop, narrow or scale: reading the pilot result
Scale when the acceptance tests pass, the hypothesis beats its baseline on held-out episodes and the coverage gaps are understood. The full request can then reuse the pilot specification almost unchanged, which is the main reason to write it carefully.
Narrow when the records are sound but one area is weak: drop a prediction target the sensors cannot see, shorten the horizon, or limit the scope to the shifts and equipment where actions are fully logged.
Stop, or bring in a domain expert, when consequences cannot be verified from the records, when interventions cannot be separated from automated actions, or when the people or client goods in the footage have no clear basis for use. An operator who knows the process can often explain an impossible state in minutes, and counsel should review any stream whose permissions are unclear before more is collected.
Requesting pilot-sized operational records through SourceX
A pilot can be requested at pilot size. SourceX sources datasets on request from U.S. businesses rather than holding them in stock, and the request can pin down the pilot's scope, volume, format, timing and licensing needs as precisely as this specification does. Multi-day video archives can stay in the data owner's storage or ship on encrypted drives, and the owner approves the scope and terms and signs before any footage or log is delivered.
Frequently asked questions
How much data does a world model data pilot need?
Enough episodes to run every acceptance test and to hold out a test set the model never sees, from one operating area. The right amount depends on how often the actions and rare events you care about occur, so set coverage targets per condition rather than a fixed number of hours.
Can passive video be used for a world model pilot?
Passive video can support pretraining of visual representations, but it cannot test whether a model learns the effect of actions. If the hypothesis is action-conditioned, the pilot needs logged actions with times and values, joined to the video on a known clock.
What is the difference between a truncated and a terminal episode?
A terminal episode ends because the process reached a real end state, such as a parcel arriving at its chute. A truncated episode ends for an unrelated reason, such as a file rollover or camera restart. Labeling the difference stops a model from learning that objects disappear when a recording stops.
Who should sign off the pilot specification?
The AI team owns the hypothesis and the success threshold, an operator or process expert confirms what the records mean, and the data owner confirms scope and permissions. Writing all three into one document avoids a pilot that passes technically but cannot be scaled.
What if the pilot model does not beat the baseline?
Check the records first: clock offsets, unit errors, missing observations and unlogged interventions often explain a flat result. If the records pass those checks, revise the hypothesis, for example by changing the horizon or target, before deciding the source has no value.
Sources
- Ha and Schmidhuber describe a world model that learns a compressed spatial and temporal representation of an environment, using a variational autoencoder for frames and a recurrent network that predicts future latent vectors. Source
- NVIDIA's Cosmos platform paper describes a video curation pipeline of splitting, filtering, annotation, deduplication and sharding. Source
- The MCAP specification defines log_time as when a message was recorded and publish_time as when it was published, with timestamps in nanoseconds since a user-understood epoch such as Unix time or robot boot time. Source
- ROS REP 103 standardizes on SI units such as meters, seconds and radians, right-handed coordinate frames and x forward, y left, z up body axes, with an optical frame convention for cameras. Source
- RLDS steps carry is_first, is_last and is_terminal flags; an episode ending on a non-terminal step is truncated, and episodes can be flagged invalid to be discarded at read time. Source
- Datasheets for Datasets proposes documentation organized into sections including motivation, composition and collection process. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.