Skip to content

Systems and records

Warehouse world model dataset: what to join to the camera footage

By SourceX Editorial · Updated

10 sections

Short answer

Camera footage shows what a warehouse looked like; a world model also needs what was done and what resulted. Join each clip to timed actions (WMS tasks, robot and forklift commands), state snapshots (locations, inventory, robot poses), business events, constraints such as layout and capacity, and matured outcomes, with one clock, declared units and coordinate frames, provenance, occlusion flags and rights.

Video and sensors

Warehouse world model

This page covers warehouse world model dataset: what to join to the camera footage.Illustrative

Key takeaways

  • Join video to five kinds of evidence: actions, states, events, constraints and matured outcomes.
  • Declare clock source, epoch, time zone offset, units and coordinate frame for every stream before joining.
  • Keep event corrections and voided scans; dropping them teaches a model events that never happened.
  • Label discrepancies as source error, missing observation, impossible state or causal uncertainty instead of deleting them.
  • Record coverage gaps and rights per clip, especially for client inventory and worker data.

What evidence should complement camera footage in a warehouse world model dataset?

Five kinds of evidence, joined to the video on a shared clock and coordinate frame: actions, states, events, constraints and outcomes. Actions say who or what acted, such as a pick task released by the warehouse management system or a drive command sent to an autonomous mobile robot. States record what was true at a moment, such as inventory at a location or a robot's pose. Events are business records of change, constraints are the rules the space obeys, and outcomes say how the work turned out.

Each joined record also needs provenance (which sensor or system produced it), an occlusion and coverage flag (what the cameras could not see), and a rights note. Without those, a warehouse world model dataset cannot tell a model's error from a data error, which is the question every evaluation eventually asks.

Why camera footage alone leaves a warehouse world model guessing

Because a world model is asked to predict how the world changes in response to something, and in a warehouse most of the somethings are invisible in pixels. The Cosmos paper from NVIDIA formalizes a visual world model as predicting the next observation from past observations and a perturbation to the world, and notes that post-training prompts can be action commands, trajectories or instructions. The earlier World Models paper by Ha and Schmidhuber describes learning a compressed spatial and temporal representation of an environment. Both framings need the perturbation, not just the frames.

In a warehouse the perturbations are spread across many actors and systems. A tote arrives because a replenishment task was released; a forklift stops because a pedestrian zone was triggered; a pallet is staged at the wrong door because an order was re-waved. From video alone a model learns that totes often appear before pickers move, not why. Footage also misses state that matters: the count inside a closed carton, whether a location is reserved, whether an order was canceled while a picker was walking to it.

There is a second reason. Real operations contain data errors, late scans and unrecorded actions. A dataset that joins video to system records exposes those discrepancies, and labeling them is what lets a buyer separate model failure from source failure.

The joined scene, event and outcome schema, table by table

The schema has ten linked tables keyed by site, a shared time base and stable identifiers for clips, actors and locations. Field names are suggestions; the relationships and checks are the part to keep. Every table carries source system, source record id and the time it was read.

At a glance

The joined scene, event and outcome schema, table by table

The joined scene, event and outcome schema, table by table
TableKey fieldsPurposeValidation check
scene_clipclip_id, camera_id, start_ns, end_ns, fps, site_idVideo segments on the shared time baseNo gaps or overlaps beyond the declared tolerance per camera
sensor_channelchannel_id, sensor type, frame_id, units, clock sourceWhat each stream measures and in which frameUnits declared; frame exists in the frame tree
frame_treeparent, child, transform, valid_from, valid_toCamera, robot and site coordinate framesEvery camera has a path to the site map frame
actoractor_id, type (worker pseudonym, AMR, forklift), capabilitiesWho can act and howPseudonyms only; no names or badge numbers
actionaction_id, actor_id, command or task, issued_ns, sourcePerturbations: tasks, commands, overridesIssued time precedes the motion it explains
state_snapshotentity_id, attribute, value, valid_ns, sourceLocation contents, reservations, robot pose, door statusNo entity in two places; capacity respected
eventevent_id, event_time, time zone offset, record_time, business step, disposition, corrects_event_idBusiness record of changerecord_time not earlier than event_time; corrections link back
constraintconstraint_id, type (layout, capacity, speed zone, rule), valid rangeRules the space obeysVersioned by date; matches layout at clip time
outcomeoutcome_id, linked action or order, result, matured_onWhat finally happenedRead after the maturation window closed
coverage_flagclip_id, region or entity, reason (occluded, out of view, sensor down)What could not be observedEvery event without visual support has a flag or a discrepancy label

Time, units and coordinate frames: the joins that break first

Time zones, epochs, units and frames cause more failed joins than any modeling choice, so declare them per stream before joining anything. Most warehouses run cameras, the WMS, robot fleet software and forklift telematics on separate clocks.

Robot logs make the problem visible. The MCAP container format records each message with a log_time, when it was recorded, and a publish_time, when it was published, and defines timestamps as nanoseconds since a user-understood epoch such as Unix time or robot boot time. A dataset that mixes boot-time and Unix-time streams without saying so will join events to the wrong frames. GS1's EPCIS 2.0 event schema requires an eventTime and an eventTimeZoneOffset on every event and allows a separate recordTime, which is the right pattern for business events: when it happened, in which offset, and when the system learned about it.

Units and frames need the same discipline. ROS REP 103 standardizes on SI units such as meters and seconds. REP 105 distinguishes the map frame, which should not drift but can jump in discrete steps when localization corrects, from the odom frame, which is continuous but drifts without bound. A robot pose that jumps across a rack in one frame may be a localization correction rather than a collision, and the dataset must say which frame each pose uses.

  • Declare clock source, epoch and measured offset for every stream and session.
  • Store event time, time zone offset and record time separately for business events.
  • Use SI units and record the frame for every pose; keep transforms with validity periods.
  • Measure synchronization against a shared visible signal, such as a scan beep or a light, at session start and end.

Taking events and outcomes from warehouse systems

Use the business events your systems already record, mapped to a shared vocabulary, and keep the corrections. The WMS and transport systems know which pick, put-away or load was intended and confirmed; that is the action and event layer the camera cannot supply.

EPCIS gives a ready vocabulary. Its business step values include receiving, storing, picking, packing, loading and shipping, and its disposition values include in_progress, damaged and returned. Its errorDeclaration structure records when an earlier event was declared in error, with a reason and links to corrective event ids. That last feature matters for world models: a scan that was later voided is not evidence of a state change, and a dataset that drops corrections teaches a model a world that never existed.

Outcomes need time to mature. A pick confirmed at 10:02 can become a short shipment at the dock, a damage claim a week later, or a return. Record the maturation window you used and read outcomes after it closes, or mark them provisional.

A valid worked record: one pallet put-away

The record below is illustrative; identifiers and times are invented to show the joins, not taken from any warehouse. It passes every check in the schema.

At a glance

A valid worked record: one pallet put-away

  • Action: put-away task T-1187 released by the WMS to forklift actor FL-04 at 14:03:10.200 local, offset -05:00, target location A-12-03.
  • Scene: clips C-221 (aisle 12 camera) and C-222 (dock camera) cover 14:02:40 to 14:06:00 on the site clock; measured camera offset 40 ms at session start and 55 ms at end.
  • State before: A-12-03 empty, capacity one pallet; pallet P-7781 at staging lane S2.
  • Motion: forklift telematics track in the site map frame, meters, from S2 to aisle 12; no frame jumps.
  • Event: location scan of P-7781 into A-12-03 at 14:05:21, record time 14:05:22, business step storing.
  • State after: A-12-03 holds P-7781; S2 empty.
  • Coverage: C-221 shows the placement; rack upright occludes the label scan for 2 seconds, flagged.
  • Outcome: P-7781 picked complete for an order four days later; no damage or count discrepancy within the 14-day maturation window.

A failing record and what the validator should report

The same structure with three faults shows why the checks exist. In this illustrative case the WMS confirms P-7781 into A-12-03 at 14:04:30, but the aisle camera shows the forklift arriving at 14:05:15; the location already holds another pallet in the state table; and the forklift pose is logged in the odom frame with no transform to the map frame for that session.

A good validator does not just reject the record. It labels each discrepancy with a likely type so a buyer can choose whether to train on it, hold it for evaluation, or drop it.

A failing record and what the validator should report
FindingCheck that firedLikely typeHandling
Confirmation 45 seconds before arrival on cameraEvent time precedes supporting motionSource error: scan made early from a handheldKeep video and motion; mark the event time unreliable
Location holds two pallets at onceCapacity constraint violatedImpossible state: earlier move not recordedSearch for a missing event; label as missing observation if none found
Pose in odom frame without transformFrame path to map frame missingPackaging errorReject until the transform is supplied
Label scan occluded for 2 secondsEvent without visual supportCoverage gap, not an errorKeep with coverage flag
Two forklifts near the location at the timeMultiple candidate actors for one changeCausal uncertaintyKeep for evaluation only; do not use as a labeled action

Could source error, missing observations or impossible states explain the result?

Ask that question before blaming the model, and make the dataset able to answer it. When a world model mispredicts where a pallet will be, the cause may be the model, a wrong system record, an event nobody logged, or a state that could not physically exist.

The discrepancy labels from the validator make that diagnosis possible. Report their rates per site and per period in the dataset documentation, alongside the coverage share of each camera. A buyer comparing two warehouse sources should compare those rates as carefully as video hours, because a source with many unexplained impossible states will teach a model the wrong physics of the building.

Keep the uncertain cases rather than deleting them. Episodes with causal uncertainty are poor training targets but useful evaluation items, since a model should express uncertainty where the data itself does.

People, client inventory and other rights in the joined record

Joining video to system records raises the sensitivity of both. Worker pseudonyms become linkable to shifts and productivity when combined with task logs, so replace badge ids with dataset-specific pseudonyms, drop names, and consider whether productivity fields are needed at all.

Third-party logistics operators handle inventory and orders that belong to their clients, and client contracts may limit how order and SKU data can be used. Product labels and customer addresses visible on cartons are also personal or client data. Confirm, for each source, who can grant rights to the footage and to each joined system, and record it in the rights field of every clip.

Requesting a joined warehouse dataset through SourceX

A joined warehouse set like this would be sourced through SourceX from an operating U.S. business on request, because nothing is kept in stock. A request describes scope, volume, language, format, timing and licensing needs, and for this kind of data it can name the tables above, the maturation window and the discrepancy labels you expect.

The supplying company approves the scope and terms and signs before any data is delivered, which is where questions about client inventory and worker data are settled.

Frequently asked questions

What is a warehouse world model?

It is a model that learns how a warehouse changes over time and in response to actions, such as how inventory, robots and people move after a task is released. It can be used to predict, simulate or plan, and it needs data that pairs observations with the actions that caused change.

Is WMS data enough without video?

No. WMS events record intended and confirmed business changes but miss the physical process, such as paths, congestion and handling. Video and robot telemetry supply that. The value is in joining them on a shared clock and frame.

How should timestamps be stored for joining video and warehouse events?

Store the event time, its time zone offset and the time the system recorded it, and declare each stream's clock source and epoch. Measure the offset between cameras and systems at the start and end of each session.

Should records with data errors be removed from the dataset?

Label them instead. Discrepancies such as early scans, missing events and impossible states tell a buyer how reliable a source is, and uncertain episodes make useful evaluation items even when they are poor training targets.

What is the difference between the map and odom frames in robot data?

In ROS conventions, the map frame should not drift but can jump when localization corrects, while the odom frame is smooth but drifts over time. A dataset must say which one each pose uses so jumps are not mistaken for real motion.

Sources

  • MCAP records each message with a log_time and a publish_time, and its timestamps are uint64 nanoseconds since a user-understood epoch such as Unix time or robot boot time. Source
  • ROS REP 103 standardizes on SI units, with meters for length and seconds for time. Source
  • ROS REP 105 defines the odom frame as continuous but drifting and the map frame as not significantly drifting but able to change in discrete jumps. Source
  • The EPCIS 2.0 JSON schema requires eventTime and eventTimeZoneOffset on events, allows recordTime, lists business steps such as receiving, storing, picking, packing, loading and shipping and dispositions such as in_progress, damaged and returned, and defines an errorDeclaration with reason and correctiveEventIDs. Source
  • The Cosmos paper defines a visual world model as predicting the next observation from past observations and a perturbation, with post-training prompts that can be action commands, trajectories or instructions. Source
  • Ha and Schmidhuber's World Models paper describes a generative model that learns a compressed spatial and temporal representation of an environment. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies