Object State Recognition: Open, Closed, Full, Empty, Clean, Dirty

Why states decide tasks

Much of workplace autonomy is deciding whether to act. A waste bin at 90% means empty it; a cooler door ajar means close it; a counter still streaked means wipe again; a guard left open means do not start the machine. Identity alone does not answer those questions. The robot needs the state of an object it already recognizes.

States play four roles in a humanoid stack: triggers that create tasks, preconditions that gate them, success checks that close them, and safety interlocks that forbid them. Language-model planners rely on the third; Inner Monologue, for example, replans using success detection as closed-loop feedback (Huang et al., 2022). Someone has to train the detector that answers. The written side of these conditions is covered in preconditions and postconditions; this page is about the images.

A working taxonomy of states

Equipment indicators are covered with buttons and switches on equipment controls and interfaces. Damage as a quality decision belongs to the inspection cluster.

What public data shows

Public datasets mostly label actions that change states rather than the states themselves.

• Template labels. Something-Something defines 174 classes as caption templates with "something" standing in for the object noun (Goyal et al., 2017). Many of its templates describe state changes, such as opening, closing or emptying, independent of object identity.
• Kitchen verbs. EPIC-KITCHENS-100 annotates 90K actions in head-mounted video from 45 kitchens, starting from pause-and-talk narration (Damen et al., 2020). Verbs like open, wash and pour imply transitions, but before and after states need extra labels.
• Assembly states. IndustReal benchmarks assembly state detection alongside action and procedure step recognition (Schoonbeek et al., 2023), one of the few public tasks that labels the state directly.
• Articulated geometry. SAPIEN builds a simulator around articulated objects, with part detection and motion recognition benchmarks (Xiang et al., 2020). It describes how parts move between states, not what each state looks like in a real workplace; see articulated object models.
• Logical conditions. MVTec LOCO separates structural anomalies from logical ones, such as an object missing or in an invalid location (Bergmann et al., 2022).

The gap for buyers is clear: workplace-specific states, defined to a business's standard, from many sites.

Where businesses keep state data

Many of these images exist because a checklist or app asked for proof. Photo evidence of step completion covers that record type, and checklists as success criteria covers the written definitions that give the photos their labels. In video, the moment a state changes can be marked as an outcome; see success and outcome labels.

Defining states so labels agree

States are where annotators disagree most, because most of them are thresholds on something continuous.

1. Tie each state to the site's standard. "Clean" means what the cleaning standard says, with reference photos, not what the annotator thinks.
2. Use few discrete values with boundary examples. Show images just inside and just outside each value.
3. Allow "uncertain" and "not visible." Forcing a call on an occluded bin poisons the label.
4. Scope states to object classes. A drawer has open, ajar and closed; a dispenser has fill bands.
5. Label transitions in video with timestamps, so before and after frames are paired automatically.
6. Record viewpoint. Fill level read from above differs from fill level read from the side.

Illustrative state label specification

# Illustrative only — synthetic classes, thresholds and records
state_sets:
  waste_bin:
    fill: {empty: "<10% visible capacity", partial: "10-80%", full: ">80%", overflowing: "contents above rim"}
  cooler_door:
    articulation: {closed: "door flush with frame", ajar: "visible gap", open: ">30 deg"}
  counter_surface:
    condition: {clean: "matches reference REF_CLEAN_03", soiled: "visible residue", wet: "visible film or droplets"}
uncertain_values: [uncertain, not_visible]
record:
  image_id: img_site031_bin_00412
  object: waste_bin (instance bin_031_07)
  state: {fill: full}
  standard_ref: site_031 janitorial standard, rev 4
  source: before photo from cleaning-round app; paired after photo img_site031_bin_00413
  viewpoint: oblique_above
  annotator_agreement: 2_of_2

What this data cannot support

• Changing the state. A photo of a closed door says nothing about how to close it, how hard to push or how to tell it has latched. That is demonstration and robot data.
• States you cannot see. Latched versus closed, a container's fill behind opaque walls, or residue that only shows under raking light may need touch, sound, weight or another sensor.
• Other sites' standards. A model trained on one operator's "clean" may not match another's.
• Rare states. Overflowing bins and spills are scarce in routine audit photos and may need deliberate collection.

Quality and acceptance criteria

1. Written definitions and reference images exist for every state value.
2. Agreement. Double-labeled samples agree above a rate you set, reported per state family.
3. Balance. Counts per value are reported; rare values are flagged.
4. Pairs. Before and after images are linked where the source app recorded both.
5. Splits. Sites and dates are held out, and the same physical object never spans splits.
6. Viewpoint coverage. Counts by viewpoint band are reported.

Questions to ask suppliers

1. What prompts staff to take these photos, and is a photo taken on pass as well as fail?
2. Are before and after photos linked to the same object and task?
3. Is there a written standard that defines each state, with reference images?
4. Which object types and how many sites are represented?
5. Are people, guests or customers visible, and how are they handled?
6. Do you also hold video of the tasks that change these states?
7. How long are photos retained in the app or system?
8. Who owns the app data: you or the software vendor?

Discuss your requirements

List the objects, states and sites you need, and how labels must be defined. Describe the state data to SourceX: it looks for US businesses that hold suitable images, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case. It reviews each request and follows up about potential sources; a request does not guarantee a match. See also the product and object data hub, the humanoid training data guide, the page on licensing images and inspection photos and the robotics and embodied AI use case.

Discuss your humanoid training-data requirements with SourceX

Sources

• The "something something" video database for learning and evaluating visual common sense — template labels for actions that change object states.
• Rescaling Egocentric Vision — kitchen actions from narration in head-mounted video.
• IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting — assembly state detection benchmark.
• SAPIEN: A SimulAted Part-based Interactive ENvironment — articulated objects and part motion.
• Beyond Dents and Scratches: Logical Constraints in Unsupervised Anomaly Detection and Localization — missing and misplaced objects as logical anomalies.
• Inner Monologue: Embodied Reasoning through Planning with Language Models — success detection as planner feedback.