Transparent and Reflective Objects: Data for Hard-to-See Items

The buyer's problem

A humanoid in a kitchen, a laboratory or a parts room meets drinking glasses, beakers, bottles, clear packaging and polished fittings. Active depth cameras often return holes, background depth seen through the object, or specular speckle on these surfaces. A grasp planner fed that depth places the gripper in empty space or a wrong location. Teams discover this late, because standard benchmarks and most catalog data feature opaque, matte items.

This page is for perception leads deciding what data to source for these items. If you also need to know which grasps work once geometry is known, see grasp annotations for products. For general depth and scan data on products, see point clouds and depth scans of products.

What published work shows

ClearGrasp addresses transparent objects from a single RGB-D image. It uses neural networks to predict surface normals, transparent-object masks and occlusion boundaries, then uses them to correct the depth, and it reports improved grasping of transparent objects when its output feeds a manipulation pipeline. The authors built a synthetic training set of more than 50,000 RGB-D images and a real-world benchmark of 286 RGB-D images (Sajjan et al., 2019).

Three lessons carry over to purchasing. First, the approach is hybrid: abundant synthetic data for training and a small, carefully measured real set for testing. Second, the supervision is geometric, not just labels: masks, normals and corrected depth. Third, the real benchmark is small, which tells you that real, correctly measured ground truth for transparent items is scarce and worth specifying carefully.

Which items fall in scope

The laboratory consumables and instruments and kitchen equipment and utensils pages describe two environments where these items cluster. Fill level and content are state questions; see object state recognition.

What data to specify

Aligned RGB and depth from the sensor type your robot will use. Structured light, time-of-flight and stereo fail differently on the same glass, so data from one sensor does not guarantee behavior on another.

Ground-truth geometry. Options include CAD or scanned meshes of the object with a known pose, aligned depth from a method that handles the surface (such as a matched opaque replacement or coated copy), and manually verified masks. Ask the supplier which was used and how pose alignment error was checked.

Lighting and background variety. Transparent objects appear differently against dark, light and patterned backgrounds. Reflective items depend on the surrounding scene. Specify the variety you need and require it as metadata.

Multiple views and poses. Include upright, lying and stacked arrangements, and occlusion by other objects.

Object state. Empty, partly filled and full; with and without labels or caps.

Illustrative acceptance package

# Illustrative: synthetic values only
scene_record:
  scene_id: scn_0381
  sensor: tof_rgbd_A            # name the sensor class and firmware if known
  rgb: scn_0381_rgb.png
  depth_raw: scn_0381_depth_raw.png
  depth_ground_truth: scn_0381_depth_gt.png   # source: matched_opaque_copy | cad_render | manual
  gt_alignment_error_mm: 1.8                  # reported method, not assumed
  object_masks: scn_0381_mask.png
  surface_normals_gt: optional
  objects: [{id: glass_017, material: clear_glass, state: half_full}]
  lighting: window_side_daylight
  background: dark_cloth

Sourcing routes

Existing data is thin. Manufacturers and testing laboratories sometimes hold images of reflective parts for inspection, and those are often taken under lighting designed to suppress or exploit specular behavior, which a robot will not have. Food-service sites and laboratories hold everyday photographs of glassware, but depth with ground truth is unlikely to exist. Because of that, this guide assumes a mixed route: use existing images for appearance and a smaller commissioned collection with measured geometry for the evaluation set. SourceX's role is sourcing, rights review, licensing and delivery coordination; capture and annotation are scoped case by case.

A capture protocol to put in the specification

Ask for a fixed protocol rather than ad hoc photography. For each object, capture several poses on at least two backgrounds of different brightness. Repeat under at least two lighting setups, such as diffuse overhead and a single directional source, and add a case with a window or lamp visible behind the object. For reflective items, include rotations around the vertical axis, because the highlight moves and so does the apparent edge. Record sensor distance and exposure settings. For liquids, capture empty, partial and full states with the same pose. Keep each object's frames together under an object ID so you can split by object. This list is a starting point that you adapt to your sensor and task.

Acceptance criteria

• Ground-truth method documented and its alignment error reported on a sample.
• Sensor identity recorded for every depth frame.
• Held-out split by object type and by background. A model that has seen the same glass on every background is not tested.
• Failure cases retained. Frames where raw depth is missing are valuable, not noise; do not accept a set in which those frames were filtered out.
• Synthetic labeled apart from real, with the renderer and material models named, see synthetic renders versus real images.

What this data cannot support

Better perception does not make handling safe. Glass and thin plastics are fragile, slippery when wet and may contain liquid, so grasp force, speed and spill avoidance need physical property annotations and interaction data. Models trained on one sensor may not transfer to another. Datasets of a few object classes will not cover novel shapes. Electronics and small shiny parts add another problem class; see electronic components recognition.

Questions to ask suppliers

1. Which depth sensors captured the data, and are raw frames included?
2. How was ground-truth geometry obtained, and what is its measured error?
3. How many distinct transparent and reflective object types, and how many instances of each?
4. What lighting and backgrounds were used?
5. Are failed or missing-depth frames retained?
6. Is any data synthetic, and with which renderer and material settings?
7. Were liquids and fill levels included?
8. Who holds rights in the images, meshes and any depth outputs?

Discuss your requirements

If you can name the glass, plastic or metal items, the sensor and the settings your robot must handle, describe the requirement to SourceX. SourceX reviews it and follows up about whether US businesses hold suitable material; if nothing fits, SourceX can scope a commissioned collection of new recordings. A request does not guarantee a match. Start from the humanoid training data guide or the product and object data hub. The catalog entry on sensor and IoT data is a related category.

Discuss your humanoid training-data requirements with SourceX

Sources

• ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation — ClearGrasp: depth correction for transparent objects, 50,000+ synthetic RGB-D training images and a 286-image real benchmark.