Kitchen Equipment and Utensils Data

Why kitchens are a priority domain and a difficult one

Food service is one candidate domain for general-purpose robots because many tasks repeat and the layout is semi-structured. For perception it is also hard. A commercial kitchen mixes shiny stainless steel, wet surfaces, steam, transparent containers, soft food and hundreds of small tools in drawers and on racks. Objects change state constantly: a pan is empty, then full, then hot.

The data question splits into three: what do the things look like, what are they called and how do they differ, and what happens when someone uses them. This page covers the first two, with notes on the third. For the distinction between recognition and manipulation data, see object recognition data vs manipulation training data.

What published kitchen datasets show, and what they do not

Public kitchen research is mostly video. EPIC-KITCHENS-100 contains 100 hours of head-mounted video, 20 million frames and 90,000 actions across 700 variable-length videos recorded in 45 kitchens; annotators narrated the footage first, then transcription, parsing and temporal segmentation followed (Damen et al., 2020). It is a reference for how kitchen activity data is annotated. It was recorded in home kitchens, not commercial ones, and it is a video-and-action dataset, so it does not replace per-object images, models or attributes.

For objects themselves, Google Scanned Objects offers more than one thousand 3D-scanned household items under a Creative Commons license, preprocessed for the Ignition Gazebo and Bullet simulators (Downs et al., 2022). Household scans cover mugs, bowls and bottles but are unlikely to cover a combi oven, a 20-quart mixer or a steam-table pan. Commercial kitchens use different equipment from homes, which is the gap a buyer usually needs to fill.

What kitchen object data can and cannot support

It can support:

• Recognition and retrieval of utensils, cookware, small appliances, storage containers and serving ware.
• Equipment recognition and state reading, such as which burner is on or where the door handle is. Controls are the subject of equipment controls and interfaces.
• Simulation assets when 3D models and consistent scale are supplied.
• Language grounding through names and attributes ("8-inch chef's knife", "half-size hotel pan").

It cannot support:

• Handling policies. Weight distribution, handle temperature, liquid slosh and the force needed to lift a full stockpot are not in a photograph. See physical property annotations and material attributes for manipulation.
• Food and process state. Doneness, cleanliness and spoilage need their own labels; see food item recognition.
• Safe behavior near heat and blades. Marking a knife as sharp is an attribute; deciding how to hand one over is a safety design problem. See hazard properties of objects.

Supplier types

Operators and training kitchens may also hold video of kitchen work. That belongs under demonstrations, where what human demonstration video can do for robot learning explains the limits.

Image conditions that matter in kitchens

Define capture conditions in the requirement, because defaults do not cover them. Stainless steel and glass reflect their surroundings and overexpose under spot lighting; see transparent and reflective objects. Steam and condensation degrade depth sensors. Objects are stacked and nested: pots inside pots, stacked trays, utensils in a crock. Wet, dirty and food-stained versions look different from catalog images. Pans and utensils from different makers have similar shapes with small differences that matter, such as handle length or lid fit.

Illustrative package

# Illustrative kitchen object package - synthetic values
scope: smallwares, cookware, appliances, containers for a commercial kitchen
imagery:
  catalog_packshots: per-SKU, 4-8 views, plain background
  in_situ: racks, drawers, prep tables, dish pit; 3 lighting conditions
  states: [clean, wet, soiled, hot_steam_visible, stacked, nested]
models: {3d_mesh: optional, units: mm, license_id: per asset}
attributes:
  [item_id, name, category_path, material, dimensions_mm, capacity_ml,
   handle_type, heat_rating_note, sharp: bool, dishwasher_safe_claim]
labels: [bbox or mask, instance_id, container_contents: empty|full|unknown]
privacy: staff faces removed or avoided; no food-brand packaging in frame unless licensed

Acceptance criteria

1. Equipment coverage against a menu. List the equipment your target operation uses and check how many are represented, at what number of views.
2. State coverage. Report counts for clean vs soiled, empty vs full, closed vs open.
3. Scale and units. Dimensions must be consistent and traceable to measurement or specification.
4. Label audit. Check 50 items for correct names; commercial kitchens use names that differ from retail ones.
5. No contamination of hold-out items. Keep some items and some kitchens out of training.

Permissions and provenance

Kitchen footage and photos can include employees, customers, trademarked packaging and proprietary recipes or layouts. Ask about notice and consent for people, site permission for photography, and food-safety rules that may limit image capture. Brand imagery on equipment and on ingredients adds trademark questions; see trademarks in training images. Blurring identifiable faces does not answer who may license the images.

Starting from the task, not the room

A kitchen is not one task. Dish-pit work needs recognition of soiled and stacked tableware; line work needs recognition of pans, tools and ingredients in use; storage work needs recognition of containers and labels on shelves. Each wants a different mix of views and states. Write the requirement per task area, because a single "kitchen" dataset that is mostly clean prep-table images will not transfer to the dish pit.

Seasonal and menu changes also matter. Operators swap smallwares and containers when menus change, so ask how recently images were taken and whether the equipment list is current.

Limitations and extra preparation

Expect to add task data, safety labels and calibration for any manipulation work. Containers need their own treatment (containers, totes and bins). Model transfer from household datasets to commercial kitchens should be tested rather than assumed.

Evaluating a sample

Ask for a sample covering at least three equipment types you actually use and one you do not, then check how the supplier handles the unfamiliar case. Count how many images show an item in a clean state against a used state, and open a handful at full size to look at the handles, lids and knobs, since those are the parts a manipulation team will care about later. Finally, compare names: if the supplier's labels say "pot" where your operators distinguish a sauce pot from a stock pot, plan a relabeling pass before any training.

Questions to ask suppliers

1. Are the kitchens commercial, home or training kitchens, and how many distinct sites?
2. Which equipment categories are covered, and which are missing?
3. Do images include wet, soiled, stacked and steam-covered conditions?
4. Are 3D models scanned or from manufacturer CAD, and what are the licenses?
5. Who appears in the frames, and what notice did they receive?
6. Which food and beverage brands are visible, and do you hold rights to license those images?
7. Are there hygiene or security rules that restricted where images were taken?
8. Can items be imaged again from specified viewpoints?

Discuss your requirements

If you can name the equipment list and the kitchen type, describe the kitchen object requirement to SourceX. SourceX reviews the requirement and follows up about potential sources; whether new recording in a commercial kitchen is feasible is discussed case by case, and a request does not guarantee a match. Start from the humanoid training data guide or the object data hub. The catalog entry for first-person video of skilled manual work covers the video side.

Describe the workplace demonstration data your robotics project needs

Sources

• Rescaling Egocentric Vision — scale and annotation method of the EPIC-KITCHENS-100 kitchen video dataset.
• Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items — scanned household objects under a Creative Commons license.