AI uses for records
Spatial reasoning operational datasets: tasks bound by real site rules
By SourceX Editorial · Updated
10 sections
Short answer
To ground spatial reasoning tasks in real operating constraints, build each task from one moment in a real operation: the layout snapshot with declared units and frames, the rules in force then (clearances, exit routes, capacity, time windows, holds), the action actually taken and what followed. Ask for a permitted route or placement, score feasibility before agreement with the record, and flag records that break the rules.
Video and sensors
Spatial reasoning
Key takeaways
- A spatial task is grounded when its correct answers are fixed by rules the site actually enforced at that moment, not by what looks plausible in an image.
- Every task carries a versioned constraint set, each rule tied to the record or document it came from.
- Score in two steps: is the answer permitted, then does it match or beat the recorded choice; the recorded choice is evidence, not the optimum.
- Declare units, coordinate frames and clocks in the task itself; a frame mix-up turns a permitted placement into a violation.
- When the record breaks a rule, decide whether it is source error, a missing observation or a real violation before the task enters a set.
How do you ground spatial reasoning tasks in real operating constraints?
You anchor each task to one recorded moment in a real operation and let that moment's rules decide which answers are allowed. A placement question is not "where would this pallet fit" but "given this layout, this load, these clearances and this capacity at 14:20 on this shift, which slots are permitted, and which did the team use."
That requires four linked pieces from the operator's own systems: a state (layout, occupancy, equipment positions), a constraint set (written rules and live limits such as holds or closed aisles), an action (the route or placement actually executed) and a consequence (what happened next: a clean pick, a relocation, a blocked aisle, a missed time window). A task without the constraint set tests guessing; a task without the consequence cannot tell a good permitted answer from a bad one.
The rest of this page specifies that task format, shows where each constraint comes from and covers the hard part: records that contradict the rules they were made under.
Why scene-only spatial benchmarks leave out the rules
Most spatial benchmarks ask about geometry a model can see, such as distance, size or relative position, and leave out the operating rules that decide what is allowed. VSI-Bench, for example, builds over 5,000 question-answer pairs from about 290 real indoor videos, and its authors identify spatial reasoning rather than perception as the main bottleneck for multimodal models.
Constraints change the problem. SSI-Bench, which asks about structures whose arrangement is fixed by geometry, topology and physical feasibility, reports the strongest closed-source model at 33.6 percent accuracy against 91.6 percent for humans on its 1,000 ranking questions. Operations add another layer those benchmarks do not carry: rules that are written down, change by shift and are enforced by people with consequences. Routing a forklift through an aisle that is open in the image but closed for a spill that morning is a wrong answer only the record can reveal.
Which operating constraints decide the answer
The constraints worth encoding are the ones a supervisor would cite when rejecting a move. Many already exist in written form. In U.S. general industry, OSHA requires safe clearances for aisles where mechanical handling equipment is used and stable, secure stacking; exit routes must stay free with no materials placed in them even temporarily; and sprinkler systems need at least 18 inches of vertical clearance to material below. Routing tools encode the same kind of limits: Google's OR-Tools treats vehicle load and time windows as dimensions the solver must keep within bounds.
These rules are examples of the type, not the full set. A site's own rules, its fire code edition, racking load ratings and customer handling requirements govern, and the task should cite whichever document the site actually follows.
| Constraint type | Typical operating record | How it is encoded in the task | Hard or soft |
|---|---|---|---|
| Aisle clearance for equipment | Facility layout drawing, equipment turning radius | Minimum free width per aisle segment, by equipment class | Hard |
| Exit routes | Emergency plan and marked egress paths | Polygons where no placement or staging is permitted at any time | Hard |
| Sprinkler clearance | Fire protection drawings, ceiling and head heights | Maximum load top height per zone | Hard |
| Slot capacity and weight | WMS location master, racking load ratings | Per-slot dimensions and maximum weight | Hard |
| Holds and closures | WMS holds, maintenance tickets, spill reports | Time-bounded exclusions on slots or aisle segments | Hard while active |
| Vehicle load | Vehicle or equipment master | Cumulative load limit along the route | Hard |
| Time windows | Dock appointments, order cut-offs | Earliest and latest arrival per stop | Hard or soft, as the site treats it |
| Handling preferences | Slotting rules, velocity class, product family | Ranked preferences that break ties among permitted answers | Soft |
The permitted routing and placement task specification
This is the asset: a field-level specification for one task record. A team can implement it as a JSON or Parquet schema as is. Each record answers one question at one moment; a shift with many moves becomes many records that share a layout snapshot.
At a glance
The permitted routing and placement task specification
| Field | Content | Required | Validation rule |
|---|---|---|---|
| task_id and task_type; as_of | Stable ID; route, placement or sequence. Timestamp of the decision moment with UTC offset | Yes | Unique; type from a fixed list. Falls inside the snapshot and constraint validity windows |
| layout_snapshot; occupancy_state | Reference to geometry: aisles, slots, zones, doors, with units and frame. Which slots and aisle segments are occupied or blocked at as_of | Yes | Units declared; frame named; snapshot version dated. Reconciles with WMS inventory within a stated tolerance |
| agent and object | Equipment class and load, or item dimensions and weight | Yes | Units declared; values within equipment and item master ranges |
| start and goal | Origin, destination or candidate slot list | Yes | All locations exist in the layout snapshot |
| constraint_set | Rules in force, each with type, parameters, source document and version | Yes | Every hard rule has a source; no rule without a validity window |
| permitted_answers | Set of routes or slots that satisfy every hard rule, computed by a checker | Yes | Recomputed by an independent checker; non-empty or flagged infeasible |
| recorded_action | Route or slot the operation actually used, from system logs | Yes | Timestamped; linked to a log event ID |
| consequence | What followed within a stated horizon: relocation, delay, blocked aisle, on-time pick | Yes | Horizon declared; outcome from a system event, not an assumption |
| evidence_flags; record_status | Occlusion, sensor gap, manual override or late scan, each naming the affected field. Consistent, source error, missing observation or real violation | Yes | Flags present when evidence is weak. Status set by the diagnosis rules below; real violations never used as positive examples |
| rights; provenance | Licensing scope, client inventory restrictions, security redactions. Source systems, export dates and transformation steps | Yes | Matches the license schedule for the source site. Every field traceable to a system and export |
Units, frames and clocks the task must declare
A placement that is permitted in meters can be a violation if a reader assumes feet, and a route that clears an aisle in one coordinate frame can clip a rack in another. Declare all three in every record rather than in a separate readme.
Robotics practice gives a usable default. ROS convention REP 103 standardizes on SI units, meters for length, and right-handed coordinate systems. REP 105 explains why the frame matters: a pose in the map frame should not drift but can jump when localization updates, while a pose in the odom frame is continuous but drifts without bound. For constraint checks against fixed geometry such as exit routes and rack faces, use a map-type frame tied to the layout drawing, and record the transform from any sensor frame.
Clocks are the third trap. WMS events, equipment telemetry and camera timestamps often come from different clocks and time zones. Store as_of with its UTC offset and record the measured offset between sources, so a hold that ended at 14:00 is not applied to a move at 13:58 on a fast clock.
Worked example: an illustrative pallet put-away task
This example is illustrative; the site, dimensions and outcome are invented to show the specification. A distribution center must put away one pallet, 1.2 m long, 1.0 m wide, 1.9 m tall and 700 kg, at 14:20. The task offers four candidate slots.
Slot A is high bay space where, after the pallet is placed, its top would sit about 30 cm below the sprinkler heads, inside the 18-inch clearance, so it is not permitted. Slot B is a floor location inside a marked exit route that staff sometimes use for overflow; it is not permitted at any time. Slot C fits by size but is rated for 500 kg, so it is not permitted. Slot D fits by size, weight and clearance, so it is permitted. The recorded action was slot D, and the consequence field shows the pallet was picked two days later without relocation.
A model that answers D passes both checks. A model that answers A has made a spatial error a scene-only task would not catch, because the slot looks open.
- permitted_answers: D
- recorded_action: D, log event linked
- consequence: picked after two days, no relocation, horizon seven days
- record_status: consistent
- evidence_flags: none
When the record breaks the rules: error, gap or real violation?
Some records will show a move the constraint set forbids: a pallet in an exit route, a truck at a dock after its window closed. Before using that record, decide which of three things it is, because each needs a different treatment. In an illustrative companion task from the same shift, the log showed a pallet placed in slot B; review found a late scan, because the pallet had been staged there for four minutes before moving to its real slot, so the record was marked source error.
| What you see | Likely explanation | Check | Treatment in the dataset |
|---|---|---|---|
| Placement in a forbidden zone with a later move | Late or wrong scan; source error | Compare scan times with equipment telemetry or video | Correct the record if evidence supports it; otherwise mark source error and exclude |
| Route through a closed aisle | Closure ended earlier than the ticket shows; missing observation | Find the reopening event or ask the site | Mark missing observation; exclude until resolved |
| Load above slot rating | Item master weight wrong | Check receiving weights and the item master history | Mark source error; fix the item master value in a new version |
| Impossible state, such as two pallets in one slot | Duplicate or merged records | Deduplicate on pallet and license plate IDs | Exclude both until reconciled |
| Confirmed forbidden move | Real violation | Evidence agrees across systems | Keep only as a negative example labeled violation; never as a positive |
Scoring: permitted first, recorded second
Score feasibility before anything else. An answer that breaks a hard rule fails regardless of how close it is to the recorded action. Among permitted answers, compare with the recorded action and its consequence: matching a choice that led to a clean outcome is good; differing from it is not automatically wrong, because operators make suboptimal but permitted choices every day.
For routing tasks, an optimization solver gives a useful reference. OR-Tools, for example, can compute a feasible route under capacity and time windows, which lets you report whether a model's route is permitted, how it compares with the solver's route and how it compares with the route the crew drove. Report the three numbers separately rather than blending them into one score.
Keep the checker independent of the task author. If the same code that wrote permitted_answers also grades responses, an error in a rule passes silently.
Where to narrow the task set or bring in a domain expert
Narrow the set when the constraint record is thinner than the task assumes. Not every site documents holds, closures or clearance heights well enough to build hard rules from them.
- Most hard rules cannot be tied to a source document: build feasibility-only tasks for the rules that can be, and drop the rest.
- Clearance or racking ratings come from memory rather than drawings: ask a facility engineer to confirm before any task depends on them.
- Video is the only evidence of the state: flag occlusions per task and expect a higher share of missing-observation records.
- Layouts reveal security features such as cash rooms or controlled-substance cages: redact those zones from the snapshot and exclude tasks that touch them.
- Client-owned inventory appears in occupancy data: confirm the licensing scope covers it or replace item identifiers with categories.
Requesting operating records for constraint-bound tasks
Tasks like these need records most operators never publish: layout versions, location masters, hold and closure tickets, equipment telemetry and the event logs that show what was actually done. SourceX keeps no stock of layout snapshots or hold tickets; it finds a U.S. operator holding them for each buyer request, and that request, covering scope, volume, format, timing and licensing needs, is where to name the constraint types and consequence horizon the tasks require.
A warehouse or yard operator that supplies such records signs off on scope and terms first, and the de-identified layouts and logs are then used only within what that agreement permits.
Frequently asked questions
What is a spatial reasoning operational dataset?
It is a set of spatial tasks, such as routing or placement, built from records of a real operation so that the correct answers are fixed by the rules and limits in force at that moment. Each task links a state, a constraint set, the action taken and its consequence.
Why not use the recorded action as the only correct answer?
Operators often make permitted but suboptimal choices, and records sometimes contain errors. Scoring against the full set of permitted answers first, then comparing with the recorded action and its outcome, separates rule-following from imitation.
Could source error, missing observations or impossible states explain a surprising result?
Yes, and they often do. Late scans, stale closure tickets and wrong item weights all produce records that appear to break rules. Diagnose each one against a second system before using it, and keep confirmed violations only as labeled negative examples.
Do OSHA rules define the constraints for every site?
No. They are examples of hard rules that already exist in writing for U.S. general industry. Each site's own rules, fire code edition, racking ratings and customer requirements govern, and the task should cite the document the site follows.
Which coordinate frame should placement tasks use?
Use a fixed, map-type frame tied to the layout drawing for checks against racks and exit routes, declare units in every record and store the transform from any sensor frame. A drifting odometry frame is fine for short motions but not for long-term geometry.
Sources
- VSI-Bench builds over 5,000 question-answer pairs from about 290 real indoor videos and identifies spatial reasoning as the main bottleneck for multimodal models. Source
- SSI-Bench tests reasoning in constraint-governed structures and reports the strongest closed-source model at 33.6 percent accuracy against 91.6 percent for humans. Source
- OSHA requires safe aisle clearances where mechanical handling equipment is used, clear aisles and stable, secure stacking. Source
- OSHA sets a minimum 18-inch vertical clearance between sprinklers and material below. Source
- OSHA requires exit routes to be free and unobstructed, with no materials placed in them even temporarily. Source
- OR-Tools enforces vehicle capacity through a dimension that tracks cumulative load along each route. Source
- OR-Tools enforces time windows through a time dimension that allows waiting slack. Source
- ROS REP 103 standardizes on SI units, meters for length, and right-handed coordinate systems. Source
- ROS REP 105 says map-frame poses should not drift but can jump, while odom-frame poses are continuous but drift without bound. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.