Equipment Controls and Interfaces: Buttons, Switches, Dials and Screens

Why controls are their own recognition problem

A humanoid asked to start a dish machine, check a cooler display or reset a tripped machine has to find the panel, pick the right control among many look-alikes, read its current state and confirm the result after acting. Controls differ from products and parts in four ways. They are small in the frame. Many are told apart only by a legend or icon. Their state is often a few pixels, such as a rocker tilted slightly or a lamp lit versus dark. And their meaning depends on the specific machine and its software version.

This page covers the recognition side. Which control to use, and in what order, comes from equipment operating procedures; how hard and how long to press comes from demonstrations or robot data. Servicing procedures show the link: OSHA's energy-control rule requires employers to develop, document and use procedures, with a narrow documentation exception, that cover specific shutdown, isolation and securing steps when employees service equipment (OSHA, 29 CFR 1910.147), and those steps involve physical controls that someone must find. Whether a robot may perform such steps is a safety decision outside this page; see lockout/tagout procedures. This is not legal advice.

Control types and what each asks of the data

Analog gauges are a recognition task of their own, covered in analog gauge and meter reading. Screens and fault codes are covered in error codes and HMI screens, and printed warnings in workplace signage and safety labels. State labeling in general, including open, closed and on/off, is in object state recognition.

What published work covers, and the gap

This page cites no public benchmark built for physical control panels, because none was verified for it, so treat any supplier's claim of a "panel dataset" as something to test with the checks below. The nearest published work labels parts and affordances on other objects.

• SAPIEN builds a simulator around a large collection of articulated objects for household robotics, with part-detection and motion-recognition benchmarks (Xiang et al., 2020).
• OakInk annotates affordances on 1,800 common household objects and records about 50,000 hand-object interactions, but its authors note that it does not cover dynamic interaction with movable parts of articulated objects (Yang et al., 2022).
• HANDAL provides category-level pose, handle affordance segmentation and reconstructed meshes for 212 objects in 17 categories of hardware and kitchen tools, per its arXiv abstract (Guo et al., 2023).

The label types in these works, such as part masks, affordance regions and articulation states, are the right kind to request; affordance and part labels covers how they are specified. The objects are household items and hand tools, though. Industrial and food-service panels are denser and more varied, and their screens change with firmware, so the images must come from your target equipment. The distinction between recognizing an object and manipulating it is explained in object recognition data vs manipulation training data.

Where the expected state comes from

Images show what a control is set to; they do not say what it should be set to. In food service, for example, a correct reading on a cooler or dish machine display depends on the operator's food-safety rules. FDA's Food Code is model guidance for state and local authorities and retailers, issued in a 2022 edition that jurisdictions adopt voluntarily, so the rules in force can differ by location (FDA, 2022). Confirm whether a newer edition applies. Take "in range" and "alarm" definitions from the site's own standard, and record the standard's version beside the label.

Supplier types

For where commissioned capture fits against licensing what exists, see existing footage vs commissioned recording. If no existing asset fits, SourceX can scope a commissioned collection of new recordings.

What to specify

• Equipment scope: make, model, software version and panel variant, including language and handedness of layout.
• Label set: panel layout, control type, legend text or icon, state from a closed vocabulary that includes unknown, actuation type (press, flip, rotate, slide, touch, pull) and safety class.
• State coverage: every state of every control, including alarm displays and a powered-off panel.
• Views: distance bands from arm's length to across the room, oblique angles, and the heights a robot's camera would have, plus some views with a hand, glove or tool partly blocking the panel.
• Lighting: overhead glare, dim rooms with backlit displays, and exposure settings that keep lamps from saturating.
• Video: short clips for blinking lamps, scrolling displays and the moment before and after a press.
• Splits: hold out whole equipment models and sites, not individual images.

Illustrative annotation record

# Illustrative only — synthetic equipment, identifiers and values
panel_id: panel_site031_unit07_front
equipment: {class: commercial_dish_machine, make: make_A, model: model_900, software: v2.4}
capture: {camera: head_rgb, distance_m: 0.6, angle_deg: 20, lighting: overhead_glare, session: s_0412}
controls:
  - {id: c01, type: push_button_momentary, legend: {text: "START", language: en},
     state: {lamp: off}, actuation: press, bbox: [412, 288, 468, 344], safety_class: normal}
  - {id: c04, type: rocker_switch, legend: {text: "I / O"},
     state: {position: "I"}, actuation: flip, safety_class: normal}
  - {id: c09, type: display, state: {screen_text: "FILL", alarm: false, screen_ref: screens/unit07_fill.png},
     actuation: touch, safety_class: normal}
  - {id: c12, type: estop_latching, state: {latch: released}, actuation: press, safety_class: safety_critical}
transition_clip: clips/c01_press_0412.mp4    # before and after a press, for state-change labels
excluded: actuation force, key travel, audio feedback (not present in images)

Quality and acceptance criteria

1. State coverage matrix. Image counts per control state per equipment model are delivered, and controls below your threshold are flagged.
2. Legend legibility. A sample is reviewed at the resolution a robot camera would deliver, and the share of readable legends is reported.
3. Ambiguous states. Two annotators label rocker and knob positions on a sample; agreement is reported per control type.
4. Screen traceability. Every screen image carries the software version that produced it.
5. Splits. No equipment model appears in both training and test data unless your claim is within-model.
6. Safety class. Safety-critical controls are labeled and spot-checked.
7. Lamps. Sampled lamp images are not saturated, and blinking states are covered by clips.

What this data cannot support

Images do not tell a robot how to operate a control. Press force, travel, detents, hold-to-confirm timing and touchscreen response need demonstration, teleoperation or instrumented data; see when video is not enough and doors, drawers and cabinets for an articulated-part example. They do not choose which control to use; that comes from procedures. A panel recognized on one model does not guarantee the next model, so coverage is counted per model. And recognizing a control is not permission to actuate it: keep safety classes in the labels and keep operating rules in the policy.

Questions to ask suppliers

1. Which makes, models and software versions do the images cover, and is the version recorded per image?
2. Were controls imaged in every state, including alarm states and powered off?
3. Are the images photographs of panels, screenshots from the software or manufacturer renders?
4. At what distances and angles were they taken, and with what cameras?
5. Who holds the rights in panel artwork, software screens and manual diagrams: you, the equipment maker or a software vendor?
6. Do any images include people, badges or screens that display customer or site data?
7. Can the equipment's response to a press, such as a lamp or display change, be shown in clips?
8. Can images be linked to asset IDs and work orders?

Discuss your requirements

If your humanoid has to operate specific machines, describe the equipment classes, controls and states you need recognized to SourceX. SourceX looks for US businesses that hold suitable images, checks the data and the supplier's licensing permissions, and coordinates pricing, allowed uses and delivery in a license. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation and engage specialist capture partners, scoped case by case. SourceX reviews each request and follows up about potential sources; a request does not guarantee a match.

See also the product and object data hub, the humanoid training data guide, the page on licensing images and inspection photos and the robotics and embodied AI use case.

Explore sourcing and licensing options with SourceX

Sources

• SAPIEN: A SimulAted Part-based Interactive ENvironment — articulated household objects with part-detection and motion-recognition benchmarks.
• OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction — affordance annotations on household objects and the stated gap for movable parts.
• HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructions — handle affordance segmentation on hardware and kitchen tools.
• Food Code 2022 — model food-safety guidance adopted voluntarily by jurisdictions.
• 29 CFR 1910.147 The control of hazardous energy (lockout/tagout) — documented procedures with shutdown and isolation steps.