Systems and records
World model missing observations: documenting sensor and camera gaps
By SourceX Editorial · Updated
10 sections
Short answer
Record each missing camera or sensor event as its own row: stream, start and end on the recording clock, how it was detected, a gap type such as dropout, occlusion or unlogged action, the evidence for its cause, and whether another system confirms what changed. Then score the world model separately on fully observed intervals, gaps that other records resolve, and gaps nobody can resolve.
Video and sensors
World model missing
Key takeaways
- An unlabeled gap teaches a world model that objects can appear or vanish without a cause.
- Detect gaps mechanically first, from sequence counters, acquisition timestamps, diagnostics and nominal rates, then ask people to explain them.
- Seven gap types, from stream dropout to truncated episode, each corrupt a different part of what the model learns.
- Every gap row should cite cause evidence and name the records, if any, that confirm what happened inside it.
- Evaluate on separate slices and never treat interpolated frames as ground truth.
- When unrecovered gaps dominate the behavior you need, narrow the scope or collect again.
How do you document missing camera and sensor events in world model data?
Document a missing observation as its own record, not as an empty frame. Each gap gets a stream, a start and end on the recording clock, the way it was detected, a gap type, the evidence for its cause, and whether anything happened inside it that another source can confirm. The evaluation plan then scores the model separately on fully observed stretches, on gaps that other records can resolve, and on gaps nobody can resolve.
A world model learns how a scene changes when something acts on it. DreamerV3's documentation, for example, describes a world model that encodes sensory inputs and predicts future representations and rewards given actions. If a camera drops out for four seconds and a pallet appears afterward, an unlabeled gap teaches the model that pallets can materialize. A labeled gap teaches it that the state changed while nobody was watching.
This page gives the method in four parts: a taxonomy of the ways an event goes unobserved, signals for finding gaps in the recording itself, a field-level annotation schema, and an evaluation plan that keeps gaps from inflating or deflating scores. A worked example and the limits of the approach follow.
Seven ways an event goes unobserved, and why each needs its own label
Gap types matter because each one corrupts a different part of what the model learns. A sensor dropout hides the whole scene; an occlusion hides one object while the rest of the scene stays valid; an unlogged action leaves the camera running but removes the cause from the record. Lumping them together under a single missing flag throws away the information a researcher needs to decide what to mask.
Use the table below as the controlled vocabulary for the gap_type field. Add a code only when none of these fits, and record the new definition in the dataset card.
| Gap type | What happened | Evidence that shows it | What a model learns if it is left unlabeled |
|---|---|---|---|
| stream_dropout | A sensor or camera stopped producing data while the scene continued | Sequence counter jumps, frame interval far above the nominal rate, diagnostic status ERROR or STALE | Objects teleport, and state changes appear to have no cause |
| occlusion | The sensor ran, but the object of interest was hidden behind another object or person | Low visibility bin, few or no lidar points inside a tracked box, track continues on either side | Objects vanish and reappear, or occluding objects seem to destroy what they cover |
| out_of_view | The event happened outside every sensor's field of view | Object track leaves the frame boundary and re-enters later in a different state | Effects with no visible action, which pushes the model toward spurious causes |
| recorder_gap | Sensors ran, but the recorder or upload pipeline lost messages | Gaps in recorded log time that are absent from the device's own counters, missing chunks, failed checksums | Same as a dropout, though the root cause and fix sit in the logging stack |
| unlogged_action | A person or machine acted, and no system event or sensor captured the action itself | A state change in one source, such as a WMS move, with no matching action in another | That outcomes follow from whatever happened to be logged just before them |
| truncated_episode | The recording ended before the consequence of the last action played out | Recording end set by a time limit, shift end or storage cap rather than by the task finishing | That the final observed state is an end state, when the process actually continued |
| not_instrumented | The variable that drives the outcome was never measured at all | Domain expert names a driver, such as floor friction or operator fatigue, that has no sensor | Correlations with whatever proxy was recorded, presented as causes |
Which signals in the recording reveal a gap before anyone labels it?
Find gaps mechanically first, and only then ask people to explain them. The recording usually contains four signals that a validator can read without watching a single frame: per-message sequence numbers, two timestamps per message, the sensor's own health reports, and the nominal publishing rate.
At a glance
Which signals in the recording reveal a gap before anyone labels it
The MCAP recording format, used widely for robotics logs, gives each message an optional sequence counter intended to detect message gaps, and stores both the time a message was recorded and the time it was published. ROS 2's standard header carries a timestamp and a frame_id, and the Image message specifies that the timestamp should be the image's acquisition time, not the time it was saved. That distinction lets you tell a camera that stopped capturing from a recorder that stopped writing.
ROS diagnostic messages report each component at one of four levels, OK, WARN, ERROR or STALE, together with a hardware ID. A STALE or ERROR status overlapping a frame-interval spike is strong evidence of a true dropout; a spike with clean diagnostics points to the recorder. Keep every duration in seconds and every distance in meters, the SI convention ROS REP 103 sets, so gap lengths from different sensors can be compared without unit conversion.
- Sequence check: flag any jump in a stream's sequence counter greater than one.
- Interval check: flag any interval between acquisition timestamps above a set multiple of the nominal period; record the multiple you chose.
- Clock check: flag messages whose log time and acquisition or publish time diverge by more than the agreed tolerance.
- Health check: join diagnostic levels to each stream by hardware ID and attach the worst level seen during the gap.
- Coverage check: compare each stream's first and last timestamps with the episode boundaries to find streams that started late or stopped early.
The missing observation annotation schema, field by field
Store gaps in their own table keyed to the episode and stream, so the original sensor data stays untouched. Each row below is one field; the example column is illustrative and drawn from the warehouse case later on this page.
Two fields carry most of the value. cause_evidence forces the annotator to cite a record rather than a guess, and eval_treatment tells whoever builds the benchmark exactly how to handle the interval, so nobody has to rediscover the decision months later.
| Field | Allowed values or type | How to fill it | Illustrative example |
|---|---|---|---|
| gap_id and episode_id | Stable string identifiers | One gap_id per stream per interval; reuse episode_id from the recording | gap-0007 in episode aisle4-shift2 |
| stream_id and frame_id | Stream name and coordinate frame from the message header | Copy from the source header; never retype | cam_aisle4 in frame cam_aisle4_optical |
| start_s and end_s | Seconds on the recording clock, with clock name | Last good and first good acquisition timestamps around the gap | Gap from 1532.40 to 1536.55 on log time |
| detection_method | sequence, interval, clock, health, coverage, manual | Every check that fired, not only the first | interval plus health (STALE) |
| gap_type | Taxonomy code from the gap type table | One code; use manual review notes when two seem to apply | stream_dropout |
| cause_evidence | Free text plus record references | Name the diagnostic, log line or ticket that explains the gap, or write unknown | Diagnostic STALE on camera hardware ID; maintenance ticket raised the same shift |
| state_change_inside | yes, no, unknown | yes only when another source shows the state differs before and after | yes: pallet location changed in WMS |
| recovered_from | List of other sources and fields | Which records establish what happened inside the gap | WMS putaway confirmation, forklift telematics |
| inference_confidence | confirmed, probable, speculative | confirmed needs a timestamped record of the action; probable needs consistent before and after states | confirmed |
| eval_treatment | score, score_recovered_only, mask, exclude_episode | Set by the rules in the evaluation section, not by the annotator's preference | score_recovered_only |
Is it a broken sensor, an unseen event or an unknowable cause?
Decide which explanation the evidence supports before anyone fills the state_change_inside field. Three explanations compete for every surprising transition in world model data, and they call for different fixes: source error means the data is wrong, a missing observation means the data is incomplete, and causal uncertainty means the data is complete but cannot say why.
Work through the questions in order. Stop at the first one that the evidence answers, and record which question decided it.
- Did the sensor itself report a problem? A diagnostic ERROR or STALE level, a calibration change or a sequence jump inside the interval points to source error or a dropout. Label it, and treat any values recorded during the fault as untrustworthy, not merely missing.
- Does a physically impossible state appear? An object occupying two places, a speed no vehicle on site can reach, or a count that goes negative is a source error even when the stream looks healthy. Check units and frames first; a frame mismatch is the most common cause.
- Is the transition plausible but its cause unrecorded? If states before and after are both valid and no sensor fault fired, the event happened out of view or went unlogged. Search other systems for the action and record what you find in recovered_from.
- Can no source establish the cause? Then mark inference_confidence as speculative and let the evaluation mask the interval. Do not pick the most likely cause and label it as fact; a confident wrong label is worse for a world model than an honest gap.
- Did the episode end before the outcome? If the recording stopped for a reason outside the task, mark the episode truncated so nobody treats its last frame as an end state.
How should partial visibility be recorded for individual objects?
Record visibility per object, not only per stream, because an occluded object inside a healthy stream is the most common gap of all. The large driving datasets already solved a version of this problem, and their conventions are worth reusing.
nuScenes annotates each object instance with a visibility level, the fraction of the annotation visible across all six camera images, binned into 0-40%, 40-60%, 60-80% and 80-100%. Waymo's open dataset labels record the number of lidar points inside each box and a detection difficulty level. Neither convention says why an object was hard to see; both give an evaluator a field to stratify on.
For business world model data, where cameras watch shelves, docks, yards or job sites, a simple adaptation works: store a visibility bin per tracked object per keyframe, the number of sensors that could see it, and the occluding object's track ID when one is known. That last field lets you separate a forklift hiding a pallet from a pallet that was moved, which is exactly the distinction the model needs.
An evaluation plan that does not reward guessing inside gaps
Score the model separately on each kind of interval, and never treat an imputed or interpolated frame as ground truth. The plan below sets one slice per evidence level, so a reader can see whether a model is good at prediction or merely good at filling holes with plausible pixels.
Gymnasium's documentation draws a useful line here. It separates termination, an episode ending in a terminal state defined by the task, from truncation, an episode ended by an outside condition such as a time limit, and notes that the last state of a truncated episode is not a terminal state. Apply the same rule to recordings: a shift change or storage cap truncates an episode, and the evaluation should not score the model on predicting an end that never occurred.
- Pre-register the slices, the visibility bins and the interval multiple before looking at model output.
- Report the share of total duration and of episodes that falls in each slice next to every metric.
- Hold out whole episodes, not frames, so the model never sees the other half of a gap it is tested on.
| Evaluation slice | What it contains | How to score it | What a failure tells you |
|---|---|---|---|
| Fully observed | Intervals with healthy streams and visibility in the top bin | Standard prediction metrics against recorded observations | The model's core dynamics are weak; gaps are not the problem |
| Gap, recovered | Gaps where another source confirms the state change | Compare predicted post-gap state with the confirmed record, not with frames | The model cannot carry state through an unobserved interval |
| Gap, unrecovered | Gaps with no confirming record | Mask from accuracy metrics; report count and total duration only | Nothing about the model; the data cannot judge this interval |
| Synthetic mask | Fully observed intervals with frames deliberately removed | Predict across the mask and compare with the held-back frames | Robustness to gaps of a known length, under conditions you control |
| Partial visibility | Objects in lower visibility bins | Metrics stratified by visibility bin and sensor count | Whether errors concentrate where the sensors saw least |
| Truncated episodes | Final interval of episodes ended by outside conditions | Exclude the last interval from end-state metrics | A scoring bug, if these episodes change the headline number |
Worked example: a warehouse aisle camera that went stale for four seconds
This example is illustrative; the counts and durations are invented to show the method, not observed results. A buyer receives one shift of recordings from a warehouse aisle: one fixed camera, forklift telematics, and warehouse management system (WMS) task records.
The interval check flags the aisle camera: acquisition timestamps jump from 1532.40 to 1536.55 seconds, about forty times the nominal frame period. Diagnostics show the camera's level as STALE for the same interval, and a maintenance ticket from that shift mentions a loose cable. The annotator records a stream_dropout with the diagnostic and ticket as cause_evidence.
Before the gap, an empty slot is visible; after it, a pallet sits there. The WMS shows a putaway confirmation for that slot at 1534.9 seconds, and forklift telematics show the truck stopped at the aisle from 1533.1 to 1535.8 seconds. state_change_inside is yes, recovered_from lists both sources, inference_confidence is confirmed, and eval_treatment is score_recovered_only: the model is judged on whether it predicts a pallet in the slot after the gap, not on frames it never received.
The exception comes forty minutes later in the same illustrative shift. A carton disappears from a shelf during a 1.5-second occlusion by a passing worker, and no WMS pick or adjustment record explains it. The annotator marks gap_type occlusion, state_change_inside yes, inference_confidence speculative, and eval_treatment mask. It may have been an unrecorded pick, a fall or a mislabeled carton; the dataset cannot say, so the benchmark does not pretend to.
When gap labels stop being enough
Gap annotation fixes datasets where gaps are the exception. It does not rescue a recording where the gaps are the dataset. If unrecovered gaps cover a large share of the episodes that contain the behavior you care about, labeling them carefully only documents why the data cannot answer the question; narrow the scope to the streams and sites with healthy coverage, or collect again.
Bring in specialists at three points. A sensor or robotics engineer should confirm gap detection thresholds and clock alignment before annotation starts, because a wrong clock offset turns every join into a false gap. An operating domain expert should review unlogged_action and not_instrumented labels, since only someone who runs the process knows which actions never reach a system. A world model researcher should sign off the eval_treatment rules before the first model is scored.
Two limits apply to every recommendation on this page. Recovered states are only as good as the system that recorded them; a WMS confirmation shows the record was entered, not that the pallet was physically placed at that second. And people visible in camera footage raise privacy and consent questions that gap annotation does not touch; those need their own review.
Asking for gap-annotated recordings in a data request
When you source recordings from operating businesses, ask for the gap table as a deliverable alongside the sensor data, and name the taxonomy and eval_treatment values you expect. Ask for the raw diagnostics and the system records used for recovery, not only the annotator's conclusions, so your team can re-run the checks.
SourceX keeps no shelf of sensor recordings. When a team describes what it needs, SourceX goes looking for U.S. businesses that hold such footage, and the request form's fields for scope, volume, format, timing and licensing are where the gap schema and its eval_treatment values belong. A business that supplies footage signs off on scope and terms before a single file moves, and multi-terabyte recordings can remain on its own storage or travel on encrypted drives.
Frequently asked questions
Should missing sensor frames be interpolated before training a world model?
Not silently. If you interpolate, keep the original gap record, mark the interpolated frames as derived, and exclude them from evaluation ground truth. Otherwise the model is scored on how well it reproduces your interpolation method.
How is an occlusion different from a sensor dropout?
In a dropout the whole stream stops, so every object's state is unknown for the interval. In an occlusion the stream keeps running and only the hidden object is unknown, so the rest of the scene stays valid training data.
What should the missing observation annotation plan include?
A gap taxonomy, automated detection rules with their thresholds, a per-gap record with cause evidence and recovery sources, a per-object visibility field, and evaluation rules that say which intervals are scored, scored only against recovered records, or masked.
Can other business systems fill in what a camera missed?
Often, yes. Warehouse, maintenance and dispatch systems record actions with timestamps, which can confirm a state change inside a gap. Treat them as evidence that a record was entered, check their clocks against the sensors, and mark the confidence level.
How do I keep gap handling from biasing evaluation results?
Pre-register the slices and thresholds, hold out whole episodes, and report each metric with the share of duration that falls in each slice. If excluding truncated or unrecovered intervals changes the headline result, investigate before publishing it.
Sources
- DreamerV3's documentation describes a world model that encodes sensory inputs and predicts future representations and rewards given actions. Source
- The ROS 2 standard header carries a timestamp in seconds and nanoseconds and the coordinate frame the data is associated with. Source
- The ROS 2 Image message says its header timestamp should be the acquisition time of the image and its frame_id the camera's optical frame. Source
- ROS diagnostic status messages report a component's level as OK, WARN, ERROR or STALE, with a hardware ID and a message. Source
- The MCAP recording format stores an optional per-message sequence counter to detect message gaps, plus separate log and publish times. Source
- ROS REP 103 standardizes on SI units, with time measured in seconds and length in meters. Source
- nuScenes records how visible each annotated instance is across its six camera images, binned into 0-40%, 40-60%, 60-80% and 80-100%. Source
- Waymo Open Dataset labels carry the number of lidar points inside each box and a detection difficulty level. Source
- Gymnasium separates termination, a terminal state defined by the task, from truncation, an episode ended by a condition outside the task such as a time limit. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.