Short answer
Material recognition data pairs images with labels such as glass, metal, fabric or ceramic, and sometimes with finish and rigidity attributes. It helps a perception model decide what an object is made of before contact. It does not measure stiffness, friction or fragility, so manipulation planning still needs physical-property data and, for control, robot interaction data.
The buyer's problem
A humanoid that picks a ceramic mug, a cardboard sleeve and a steel pan from the same counter needs different grip force, approach and speed for each. Vision is often the only sensor available before touching the object, so teams look for images labeled by material and surface finish. The difficulty is that "material" is used loosely in catalogs. A product listing may say "stainless" when the visible surface is a painted polymer, or "plastic" for six resin families that deform very differently.
This page covers sourcing material-labeled images and attributes. For the property fields that planners need beyond appearance, see physical property annotations for mass, friction, fragility and compliance. For the broader recognition-versus-handling distinction, see object recognition data versus manipulation training data.
What published material datasets show
The Materials in Context Database (MINC) covers 23 material categories and was about ten times larger than earlier material datasets when released; its authors report 85.2% mean class accuracy for patch classification and 73.1% for full-scene per-pixel segmentation (Bell et al., 2014). Two takeaways matter to a buyer. First, material recognition is learnable from in-the-wild photographs, not only from lab captures. Second, the scene-level number is lower than the patch number, which is the number closer to what a robot camera faces.
Amazon Berkeley Objects shows the catalog route. It pairs 147,702 product listings and 398,212 unique catalog images with up to 18 metadata attributes per product, plus 7,953 products with artist-designed 3D meshes carrying physically based materials, and it includes a material estimation benchmark (Collins et al., 2021). Catalog metadata is therefore one source of material text, and rendered material maps are another, but neither is a measurement of the physical object.
The YCB object set sits at the other end. It selects everyday objects that vary in shape, size, texture, weight and rigidity, and distributes scans together with physical properties (Calli et al., 2015). It is a reminder that rigidity and weight are usually recorded by a separate measurement step, not inferred from photographs.
What material labels can and cannot support
| Need | Material-labeled images help | Still required |
|---|---|---|
| Choose a grasp family (suction, pinch, wrap) | Yes, as a prior on surface porosity and finish | Grasp labels or trials on the actual gripper |
| Estimate slip risk | Partly, finish is a cue | Measured friction against your gripper pads |
| Detect fragile items | Partly, glass and thin ceramic are visible | Fragility labels from handling tests or supplier specs |
| Distinguish look-alike materials | Partly, depends on resolution and lighting | Close-range or polarized capture for hard cases |
| Predict deformation under load | No | Compliance measurement and interaction data; see deformable product properties |
Material appearance also depends on illumination. A brushed-metal surface can read as bright, dark or colored depending on what it reflects. Transparent and mirror-like items behave worse in depth sensors; see transparent and reflective objects for that specific failure.
Where supply comes from
Sourcing routes differ by label origin:
- Manufacturers and brands know the actual composition (resin code, alloy, coating) and may hold it in specification sheets, but image rights and spec-sheet access are separate questions.
- Retailers and e-commerce operators hold many images with free-text descriptions. Material fields are often inconsistent and need normalization.
- Operating businesses (kitchens, workshops, laboratories) hold photographs of real in-use items, with wear and residue, and rarely with material labels.
- Annotation providers can add material labels to images you already hold. Ask how they decide ambiguous cases, such as painted metal.
Existing assets are the starting point for this page because the image supply is large. Commissioned collection becomes relevant when you need close-range, polarized or multi-lighting captures of a fixed object list. Note the match to product catalog data for robot perception, which covers the catalog side more generally.
Illustrative material attribute package
# Illustrative: synthetic values only
object_id: obj_00417
images: [img_a_front, img_b_oblique, img_c_closeup]
material_labels:
primary: ceramic_glazed # from a closed vocabulary you define
secondary: [silicone_base_ring]
label_source: expert_visual # expert_visual | spec_sheet | lab_test | catalog_text
label_confidence: high
surface:
finish: gloss # matte | satin | gloss | brushed | textured
coating: unknown
wet_or_oily_in_use: sometimes
rigidity_class: rigid # rigid | semi_rigid | flexible | soft (coarse bins)
notes: "Glaze crazing visible on rim; not a defect label."
Keep the vocabulary closed and versioned. A free-text field cannot be audited, and two suppliers will not use the same words. The object attribute schema page discusses how to align fields across sources, and packaging material properties covers the packaging case.
Three traps in material labels
Appearance versus composition. A label such as "metal" can describe what the surface looks like, what the part is made of, or what the catalog says. A chrome-plated plastic handle satisfies the first and third and fails the second. Decide which you need. For grasp choice, surface appearance and finish often matter most; for recycling or hazard handling, composition matters, and images alone are weak evidence.
Class imbalance hides the hard cases. Public-style material sets tend to be dominated by common classes such as wood, fabric and painted surfaces. Rare but consequential classes, such as thin glass, laminated board or silicone, may have a few hundred examples at most. Ask for per-class counts and for the number of distinct objects behind them.
Context leaks the label. Models can learn that kitchens contain ceramic and workshops contain steel, then fail when a ceramic part appears on a bench. Ask whether the suppliers varied scene context for each material, and test on images where context and material disagree.
Acceptance criteria you can set
- Label provenance recorded per field. A label from a spec sheet and a label from a quick visual check should not share a column unlabeled.
- Held-out split by object, not by image. Multiple photographs of one mug in both train and test sets overstate accuracy.
- Agreement check. Have a second reviewer relabel a random subset and report disagreement by material class. Confusable pairs (painted metal versus plastic, glass versus clear acrylic) deserve separate rows.
- Lighting and view diversity stated. Ask for the count of lighting conditions and capture distances, not just image totals.
- Closed vocabulary with definitions. Reject datasets whose class definitions exist only in annotators' heads.
Limitations and additional preparation
Material is not a single property. Two objects of the same class can differ in thickness, wall construction or coating, and the image cannot show those. Photographs also record the surface of the object, not what is under it. A foil-wrapped item and a bare metal item can look alike, and a woven cover can hide a rigid frame. Treat the label as a statement about appearance unless the supplier documents a measurement.
Expect to normalize vocabularies, deduplicate near-identical catalog images, and check that attribute text does not contradict the picture. Rights are a layer of their own: photographers, brands and retailers may each hold rights in a catalog image, and reuse for model training is a distinct question from reuse for display. See product image rights for the diligence questions. This is not legal advice.
Questions to ask suppliers
- Where does each material label come from: spec sheet, lab test, visual review or catalog text?
- Is the vocabulary closed, and can you share the definitions and any change log?
- How were ambiguous finishes or coated materials labeled?
- Are there close-up images of the surface, or only full-object views?
- How many lighting conditions and backgrounds are represented?
- Can images be grouped by physical object instance so you can split by object?
- Are wear, residue, wetness and damage represented, or only new items?
- Who holds rights in the images and attribute text, and can they license training use?
- Is there a datasheet or data card describing collection and known gaps? Documentation formats such as datasheets for datasets give a reasonable checklist.
Discuss your requirements
If you can name the objects, material classes and finishes your robot must tell apart, describe the requirement to SourceX. SourceX reviews it and follows up about potential US sources, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. A request does not guarantee a match. Begin at the humanoid training data guide or the product and object data hub. The catalog entry for images and inspection photos shows how image assets are described on the site.
Explore sourcing and licensing options with SourceX
Sources
- Material Recognition in the Wild with the Materials in Context Database — MINC: 23 material categories, patch and scene-level material recognition accuracy.
- ABO: Dataset and Benchmarks for Real-World 3D Object Understanding — ABO: catalog images with metadata attributes and physically based materials; material estimation benchmark.
- Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols — YCB set: everyday objects varying in rigidity with physical properties recorded.
- Datasheets for Datasets — Datasheets for datasets as a documentation checklist for suppliers.