Short answer
Cluttered-scene data shows many objects touching, overlapping and partly hidden on surfaces such as workbenches, shelves and sinks, with labels for each visible instance and, ideally, for hidden extent and removal order. Bin-picking sets cover one clutter type. Workplace clutter adds mixed object classes, reflective or wet surfaces, and fixtures. Ask for occlusion levels, instance masks and held-out scenes.
The buyer's problem
Real workplaces are messy. A sink holds dishes, a bench holds tools and parts, a shelf holds items pushed together at odd angles. A detector trained on isolated objects loses recall when items overlap, and a grasp planner fed a wrong mask picks the wrong thing. Most public clutter data comes from bin picking or retail shelves, which differ from a workbench or a sink in object variety, surfaces and what counts as clutter.
This page covers sourcing clutter images outside warehouse bins. For bins, totes and shelves in fulfillment, see containers, totes and bins and the warehouse hub. For the evaluation measures, see object recognition evaluation metrics.
What published clutter datasets show
MetaGraspNet targets bin picking. It provides 217,000 synthetic RGB-D images across 82 article types, with annotations for detection, amodal perception, keypoints, manipulation order, and both parallel-jaw and vacuum grasp labels, plus more than 2,300 real annotated RGB-D images in five difficulty levels and an unseen-object set (Gilles et al., 2022). Three features are directly reusable as requirements: amodal labels (the full extent of a partly hidden object), manipulation order (which item to remove first) and graded difficulty levels.
SKU-110K shows the other extreme of clutter, densely packed retail shelves with tightly packed, often identical objects. Its authors developed components for handling overlapping detections in such scenes (Goldman et al., 2019). Identical neighbors create a counting and separation problem rather than a recognition one.
Neither is a sink or a workbench. Treat them as templates for label design and difficulty grading, not as coverage.
Types of clutter to separate
| Type | Example | Main difficulty | Label need |
|---|---|---|---|
| Piles | Parts in a tray, dishes in a sink | Heavy overlap, unknown order | Amodal masks, removal order |
| Tight rows | Items on a shelf | Identical neighbors, boundaries | Instance separation, counts |
| Scatter | Tools on a bench | Mixed classes, shadows | Class and instance labels |
| Containers in containers | Utensils in a drawer | Depth and enclosure | Container relations |
| Liquids and wet surfaces | Sink with water, soap | Reflections, transparency | State and surface tags |
| Background texture | Patterned bench, wire rack | False edges | Background tags |
Sinks and benches often combine transparent items and reflections; see transparent and reflective objects. Fixtures and furniture create occluders and surfaces of their own; see workplace furniture and fixtures.
Illustrative annotation package
# Illustrative: synthetic values only
scene:
scene_id: scn_5120
location_type: lab_workbench # sink | workbench | shelf | drawer
site_id: site_022
clutter_level: 4 # 1 (isolated) .. 5 (heavy overlap), define in guidelines
sensors: [rgb, depth_tof]
objects:
- {inst_id: 1, class: beaker_250ml, visible_fraction: 0.62, amodal_mask: yes, removal_rank: 2}
- {inst_id: 2, class: pipette_rack, visible_fraction: 1.00, amodal_mask: yes, removal_rank: 1}
- {inst_id: 3, class: unknown_object, visible_fraction: 0.20, amodal_mask: no, removal_rank: null}
relations: [{a: 1, b: 2, relation: leaning_on}]
split: heldout_site
The visible_fraction field lets you stratify evaluation by occlusion. The unknown_object class lets annotators mark items outside the vocabulary rather than guessing.
Sourcing
Operating businesses are the supply: laboratories, kitchens, workshops, repair benches, clinics and back-of-house areas hold photographs and camera footage of real surfaces. Most were taken for documentation, inventory or complaint records, so viewpoints vary, labels are absent and people or screens may appear. Annotation providers can add instance and amodal masks, and a commissioned shoot can be designed around your robot's camera height and approach angle. Hybrid plans are common: existing photographs for diversity, a short commissioned set for evaluation. See long-tail object coverage for how rare items appear, and laboratory consumables and instruments for one environment's object set.
Mask quality matters at occlusion boundaries; see segmentation masks for products.
Defining difficulty levels
Published bin-picking work grades scenes into difficulty levels; borrow the idea and define your own levels in the specification. A workable scale combines the number of instances, the share of objects with visible fraction below some threshold, and the presence of reflective or transparent items. Level 1 might be well-separated objects, and a top level might be piles with heavy overlap and mixed materials. Require every scene to carry its level, and report model results per level so that an improvement in easy scenes is not mistaken for progress on hard ones. The thresholds are yours to set, since they depend on your sensors and the task.
Acceptance criteria
- Occlusion strata reported. Counts of instances per visible-fraction band and per clutter level.
- Annotation guidelines shared, including how touching or stacked items and unknown objects are treated.
- Mask audit. Review a sample at boundaries; measure agreement between two annotators.
- Scene diversity. Number of distinct sites, surfaces and lighting setups, not only images; successive frames of the same pile are one scene.
- Held-out splits by site and by object instance for evaluation.
- Hard negatives. Include empty regions and look-alike items.
- Depth quality. If depth is included, check holes on shiny or dark items.
What this data cannot support
Clutter images show the state of a scene, not how it responds to touch. They do not say whether removing the top item will topple a stack, or how a wet dish slides. Manipulation order labels are an annotator's judgment unless validated on a robot. Static images cannot tell you motion or contact. Datasets from one workplace type may not transfer to another, and cleaned or staged scenes understate real mess. Grasp feasibility needs grasp annotations for products or interaction data, and requirements depend on your sensors and architecture.
Permissions and provenance
Workbenches and sinks sit in areas where people work; images may include hands, faces, screens, notes, samples, customer property or regulated materials. Ask what was in frame, what consent applies, and who can license training use. This is not legal advice, and masking part of an image does not by itself settle permission questions.
Questions to ask suppliers
- Which surfaces and workplace types are covered, and how many distinct sites?
- Are instance masks provided, including amodal masks for hidden parts?
- How is clutter level defined, and what is the distribution?
- Are failures retained, such as images with missed items or unusual objects?
- What camera type, height and angle were used?
- Is depth included, and how reliable is it on glossy or transparent items?
- What people, screens or confidential items appear, and how were they handled?
- Who owns the images and annotations, and can both be licensed for training and evaluation?
Discuss your requirements
If you can describe the surfaces, objects and camera positions your robot will meet, describe the requirement to SourceX. SourceX reviews it, follows up about potential US businesses that hold suitable images, and checks the data and licensing permissions. If nothing fits, SourceX can scope a commissioned collection of new recordings. A request does not guarantee a match. Start from the humanoid training data guide or the product and object data hub. The site's entries on images and inspection photos and robotics and embodied AI give more context.
Discuss your humanoid training-data requirements with SourceX
Sources
- MetaGraspNet: A Large-Scale Benchmark Dataset for Scene-Aware Ambidextrous Bin Picking via Physics-based Metaverse Synthesis — MetaGraspNet: 217k synthetic and 2.3k+ real RGB-D bin-picking images, 82 article types, amodal and manipulation-order annotations.
- Precise Detection in Densely Packed Scenes — SKU-110K: densely packed retail shelves with identical adjacent objects.