Systems and records
World model sensor provenance: which sensor produced this fact?
By SourceX Editorial · Updated
10 sections
Short answer
Every physical AI observation should carry the sensor that produced it, the calibration in force at that moment, the coordinate frame, the clock and units, and the processing steps that turned the raw reading into a fact. World model sensor provenance at that level lets a team tell a real state change from a miscalibrated camera, a dropped message or a derivation error, and reject records that cannot be traced.
Video and sensors
World model sensor
Key takeaways
- Provenance has to attach to each observation, not to the dataset as a whole, because calibration, clocks and processing change during a recording.
- Reuse existing conventions: ROS headers and camera info, REP 103 and 105 frames and units, nuScenes-style calibration tables and MCAP timestamps.
- Derived facts, such as an item dropped at a given time, need a lineage chain back to the readings and the processing run that produced them.
- A validator should fail records with zeroed calibration, unknown frames, clock gaps or impossible state jumps, and report which check failed.
- Separate three explanations for a surprising record: source error, a missing observation, or a real but rare event.
- Rights travel with provenance: the manifest should say who owned each sensor and its readings.
What sensor provenance belongs with physical AI observations?
Each observation needs six things attached: which physical sensor produced it, which calibration was in force, which coordinate frame it is expressed in, which clock stamped it and in what units, which processing steps changed it, and who owned the sensor and the data. Without those, a world model learns from numbers it cannot place in space or time.
The point is attribution at the level of a single fact. When a model predicts that a pallet will slide and the logged data says it did not, the team has to be able to ask which camera saw the pallet, whether that camera had been bumped since its last calibration, whether its clock agreed with the forklift's, and whether a tracking step in between could have lost the object.
Why world model sensor provenance has to travel with each observation
A world model learns how states change in response to actions, so a calibration error or a clock offset does not look like noise to it. It looks like physics. To take hypothetical cases, a camera that drifted a few degrees makes objects appear to move when nothing moved, and a clock offset of a fraction of a second between an arm and a camera makes contact appear to happen before the arm arrives.
Dataset-level documentation cannot catch this, because calibration and timing change during a recording. A camera is recalibrated after maintenance, a lidar is remounted, a clock resynchronizes after a network drop. If the provenance lives in one readme, every observation after the change inherits the wrong values silently.
Per-observation provenance also lets a buyer filter instead of discard. When one sensor's calibration is suspect for a week, the team can drop that sensor's readings for that week and keep everything else, provided each reading names its sensor and calibration version.
Which conventions should the manifest reuse instead of inventing?
Most of the vocabulary already exists in robotics tooling, and reusing it makes a supplier's manifest readable by a buyer's loaders without translation.
In ROS, the standard message header carries a two-part timestamp in seconds and nanoseconds and a frame_id naming the coordinate frame the data belongs to. The CameraInfo message adds the distortion model and the D, K, R and P matrices, states that these stay fixed until the camera is recalibrated, says the header stamp should be the image acquisition time, and treats zeroed matrices as an uncalibrated camera.
REP 103 standardizes SI units, right-handed frames and a body convention of x forward, y left and z up, with a separate optical-frame convention for cameras. REP 105 distinguishes an odom frame that drifts but stays continuous from a map frame that does not drift significantly but can jump. Both matter for provenance, because a pose in the wrong frame is a wrong fact.
The nuScenes schema shows how to keep calibration as data: a calibrated_sensor table gives each sensor's translation and rotation relative to the vehicle body frame plus camera intrinsics, every sample_data row points to its calibrated sensor, its ego pose and a timestamp, and a log table records the vehicle, capture date and location. The MCAP container records both a log_time and a publish_time per message, requires publish_time to fall back to log_time when unknown, and supports key-value metadata records where a manifest can live inside the file.
The sensor calibration and lineage manifest
The manifest below is the asset. Each row is a field group that every observation, or every message stream with a declared validity window, must carry. The validation column is the test a buyer runs on delivery.
| Field group | Fields | Purpose | Validation check |
|---|---|---|---|
| Sensor identity | sensor_id, model, serial, firmware version, modality | Names the physical device, not just a topic | Every observation resolves to exactly one sensor record |
| Mounting and extrinsics | parent_frame, translation in meters, rotation quaternion, valid_from, valid_to | Places the sensor relative to the body or site frame | Validity windows do not overlap and cover the whole recording |
| Intrinsics and model | calibration_id, distortion model, K, D, R, P or modality equivalent, method, residual | Lets a reader project or undistort the reading | No zeroed matrices; calibration_id in force at observation time |
| Frame | frame_id, frame convention, odom or map or site | Says what space the numbers live in | frame_id exists in the frame tree for that timestamp |
| Time | acquisition stamp, log time, clock source, sync method, measured offset | Orders readings across sensors | Monotonic per sensor; offset within the buyer's tolerance |
| Units and ranges | units per field, valid range, saturation flag | Stops silent unit and range errors | Values inside declared range or flagged |
| Gaps and occlusions | dropped message count, occlusion flag, sensor fault code | Separates no event from not observed | Gaps declared, not inferred from silence |
| Processing lineage | run_id, job name and version, input observation IDs, parameters | Traces derived facts to readings | Every derived record lists inputs that exist |
| Operating context | site, layout version, shift, interventions | Explains state constraints and human actions | Layout version known for every observation |
| Ownership and rights | sensor owner, data controller, license scope, people-in-view flag | Shows who may license each reading | No observation without an owner and a scope |
Tracing a derived fact back to the readings that produced it
Most of what a world model trains on is derived, so the manifest has to link derived facts to raw readings through the processing that made them. An object track, a contact event or a fill level is the output of a job that read several sensors and applied a model or a rule.
The W3C PROV ontology gives the shape: entities such as readings and facts, activities such as a tracking run, and agents such as a person or system that is responsible, with derivation linking one entity to another through an activity. OpenLineage gives a practical record for the activity: each run carries a globally unique runId, the job and its input and output datasets are identified by namespace and name, and run events mark start, completion or failure.
In practice, every derived record carries the run_id of the job that produced it, the job version and the IDs of its input observations. When a buyer finds a wrong fact, the chain answers whether the sensor, the calibration or the job version was at fault, and which other facts came from the same run.
What checks should the validator run on delivery?
A validator turns the manifest into pass or fail results a team can reproduce. Run it on a sample before accepting a delivery and on the full set before training. Each check should name the observation IDs that failed, not just a count.
At a glance
What checks should the validator run on delivery
Keep the validator itself versioned and deliver its output with the data. A buyer who reruns the same validator version on the same files should get the same failures, and when a check changes, for example a tighter clock tolerance, the change log explains why a record that passed last month now fails. Failures are not automatically grounds for rejection; a small, documented share of readings with a known calibration gap can be excluded while the rest of the delivery is accepted.
- Resolution: every observation's sensor_id, calibration_id and frame_id resolve to records valid at its timestamp.
- Calibration presence: no camera intrinsics are all zeros; every calibration has a method and a residual.
- Window coverage: extrinsic and calibration validity windows have no gaps or overlaps across the recording.
- Clock order: timestamps are monotonic per sensor, and cross-sensor offsets stay within the tolerance the buyer sets.
- Units and range: values sit inside declared ranges, and saturated readings are flagged rather than clipped silently.
- Frame consistency: poses in a map frame are not compared with poses in an odom frame without a recorded transform.
- State plausibility: no object or robot moves faster or further between consecutive readings than the declared limits allow.
- Lineage integrity: every derived record lists a run_id and input IDs that exist in the delivery.
- Rights completeness: every observation has an owner and a license scope, and people-in-view flags match consent records.
A valid record and a failing record, compared field by field
Both records below are illustrative and describe no real company or dataset. They show one overhead camera observation at a packing station. The valid record passes every check; the failing record looks complete at a glance but fails five of the validator's checks across six fields.
| Field | Valid record (illustrative) | Failing record (illustrative) | Check result for failing record |
|---|---|---|---|
| sensor_id and calibration_id | cam-04, calib-04-v3 valid from the last remount | cam-04, calib-04-v2 whose window ended before this reading | Fails resolution: calibration not in force |
| Intrinsics | Plumb bob model, nonzero K and D, residual recorded | K copied as all zeros from a template | Fails calibration presence |
| frame_id | cam-04 optical frame under the site frame | camera, a name absent from the frame tree | Fails resolution: unknown frame |
| Time | Acquisition stamp, site clock, measured offset recorded | Log time only, no clock source | Fails clock order: offset unknown |
| Derived fact | Carton placed, run_id and job version listed with three input IDs | Carton placed, run_id listed with one input ID that was purged | Fails lineage integrity |
| Rights | Owner and scope recorded, no person in view | Owner recorded, scope blank | Fails rights completeness |
Source error, missing observation or impossible state?
When a record surprises the model or the reviewer, decide which of three things happened before labeling it an interesting edge case. The manifest is what makes the distinction possible.
A source error means the sensor or its calibration was wrong: the validator usually shows a stale calibration, a clock offset or a saturated value. A missing observation means the event happened but no sensor recorded it: the gap or occlusion fields should declare it, and the fact should be marked unobserved rather than absent. An impossible state means the record contradicts physical or operating constraints, such as a carton in two places at once, and points to a frame mix-up or a derivation bug. Only after all three are ruled out is the record a genuine rare event, which is often the most valuable thing in the set.
Write the outcome back into the manifest as a review flag with the reviewer and date, so the next user does not repeat the investigation.
What to ask a supplier before accepting sensor data
Ask for the manifest before the data. A supplier that can produce sensor records, calibration history, a frame tree and clock documentation for a sample week can usually produce them for the full set; one that cannot will not find them later.
Ask who owned each sensor. In a warehouse or factory, cameras may belong to the site, robots to a vendor and telemetry to a platform provider, and the right to license each stream follows those contracts, not who holds the files. Ask whether people appear in view and what consent covers them. Ask which processing jobs ran before export and whether their versions were recorded, because a derived track without its job version cannot be reproduced.
Finally, ask for one week of raw readings alongside the processed export. Running the validator on both shows whether the supplier's own processing introduced the gaps, frame changes or unit conversions the manifest is meant to expose, and whether the processed facts can be regenerated from the raw stream with the declared job versions. A supplier that can only provide processed output should say so in the manifest, so the buyer can plan around the lost ability to re-derive facts.
Asking for provenance-complete sensor data through SourceX
Buyers can commission sensor data like this from SourceX, which looks for U.S. businesses that hold it once a request arrives instead of keeping datasets on hand. The request form covers scope, volume, language, format, timing and licensing, and the manifest above can be attached as the format requirement so the supplier knows from the start which fields must travel with each observation.
Large sensor deliveries typically stay in the supplying business's own storage or ship on encrypted drives, so plan the validator to run where the data sits.
Frequently asked questions
What is the difference between sensor provenance and data lineage?
Sensor provenance describes where a reading came from: the device, its calibration, its frame and its clock. Data lineage describes what happened to the reading afterwards: the jobs and versions that transformed it into derived facts. A world model dataset needs both, linked by observation IDs.
Is a calibration file per dataset enough?
Usually not. Calibration changes when sensors are remounted, serviced or recalibrated, so each observation should point to the calibration valid at its timestamp. A single file is only enough when the supplier can show nothing changed during the recording.
Which timestamp should be the source of truth?
Use the acquisition time from the sensor when it is available and record the clock that produced it, with the log time kept as a second field. MCAP, for example, separates log time from publish time. The manifest should state the clock source and the measured offset between clocks.
How do I mark an event that no sensor saw?
Declare it as unobserved using gap or occlusion fields rather than leaving a hole in the data. A model treating silence as no event will learn that things do not happen when they simply were not seen.
Do I need a new standard for this manifest?
No. The fields reuse ROS message conventions, REP frame and unit rules, nuScenes-style calibration tables, MCAP metadata and OpenLineage run records. The manifest is a checklist of which of those fields must be present and valid for every observation.
Sources
- The ROS standard header carries a seconds-and-nanoseconds timestamp and the frame_id the data is associated with. Source
- ROS CameraInfo holds the distortion model and D, K, R, P matrices, fixed until recalibration, with zeroed matrices meaning an uncalibrated camera. Source
- REP 103 standardizes SI units, right-handed frames and body axes of x forward, y left, z up, with a separate optical-frame convention for cameras. Source
- REP 105 defines an odom frame that drifts but is continuous and a map frame that does not drift significantly but can jump. Source
- The nuScenes schema stores sensor extrinsics and intrinsics in a calibrated_sensor table and links each reading to its calibration, ego pose and timestamp. Source
- MCAP records log_time and publish_time per message and supports key-value metadata records. Source
- W3C PROV-O models provenance as entities, activities and agents, with responsibility expressed through wasAssociatedWith and wasAttributedTo. Source
- OpenLineage run events carry a globally unique runId and identify jobs and datasets by namespace and name. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.