Fragile Item Handling Demonstrations: Glassware, Ceramics and Produce

Task anatomy

"Careful handling" is a bundle of sub-skills. In a restaurant dish pit, a worker stacks wet plates and pulls glasses from a rack. In a produce packhouse, a worker lays ripe fruit into trays without bruising. In a retail stockroom, someone unpacks ceramic mugs from molded inserts. The objects differ, but the structure is similar.

• Approach. Move toward the item with reduced speed, check for edges, wet surfaces or neighbors.
• Grasp. Choose a hold: rim, base, two-handed cradle, pinch at a stem, palm support under a fruit.
• Lift and carry. Keep the item stable; avoid knocking neighbors; manage slosh in liquid-filled glasses.
• Place. Lower with controlled speed, release when contact is confirmed, sometimes adjust.
• Recover. Catch a slip, re-seat a stack, set aside a chipped piece.
• Judge. Decide what is too damaged to pack or serve.

Failure modes include slipping with wet hands, double-stacking glasses that bind, applying too much pinch to soft produce, and bumping a neighbor during placement. Some of these are rare in routine footage.

What video shows and what it misses

Fragility is a property of the object in relation to force. A video of a worker holding a wine glass looks much like a video of the same worker holding a plastic cup. What differs is what would happen if the grip tightened or the placement was abrupt. Video cannot fully express this, which is why tasks like this lean on other modalities when video is not enough.

Which data helps each part

For perception. Video and images help recognize glassware, ceramics, fruit ripeness and packaging states. Transparent and reflective objects are hard for depth sensors and some segmentation models; see rgbd demonstration capture.

For grasp selection. Hand-object interaction labels from head-mounted video help; see hand-object interaction annotation.

For force control. Contact-rich robot datasets pair vision with force and other channels. RH20T contains more than 110,000 contact-rich real-world manipulation sequences, with visual, force, audio and action data per sequence and a human demonstration video and language description for each robot sequence (Fang et al., 2023). It illustrates the kind of sensing a force-sensitive task can use; it was not collected in food service or packhouse settings, and its findings should not be taken as evidence for any fragile task.

For realistic context. Kitchen video at scale shows how people actually handle cookware. EPIC-KITCHENS-100 provides 100 hours of head-mounted video across 45 kitchen environments with narrated action segments (Damen et al., 2020). It is a daily-life kitchen collection rather than commercial food service, and fragility outcomes are not its focus.

For outcome labels. Whether an item survived, chipped or was rejected must be recorded by the people on site; an annotator watching video often cannot tell.

Supplier types

• Commercial kitchens and restaurants. Glassware, plates and serviceware at pace; see commercial kitchen food prep and restaurant bussing and reset.
• Dish rooms. Stack, rack and sort; see dishwashing and dish racking.
• Packhouses and grocers. Soft produce and graded fruit; see produce packing and grading.
• Retail stockrooms and glass or ceramics sellers. Unpacking, display and packing of breakables.

Existing footage in these settings was typically recorded to monitor service or training. Purpose-built recordings can add close views, force sensors and breakage logging, at the cost of scoping and breakage risk. Compare routes in existing footage vs commissioned recording.

Capture considerations

• Camera. A head-mounted view with the hands and the item in frame, plus a close fixed view of the placement surface.
• Lighting and glare. Glassware causes reflections that hide edges; use diffuse lighting and note conditions.
• Frame rate. Fast slips and catches need frame rates high enough to see them.
• Instrumentation. Pressure-sensing gloves, instrumented trays or force plates under the surface, where the demonstrators and site allow.
• Object log. Record each item type, size, mass and condition, because the same pose with a thick tumbler and a thin flute is different data.
• Staging. Breakage is costly. Decide up front who pays for replacement items and whether deliberately risky behavior is allowed.

Illustrative specification

# Illustrative only - synthetic values
task_family: fragile_handling_v0
objects: [tumbler_A, stemware_B, ceramic_mug_C, plate_D, produce_soft_E]
object_log: [mass_g, rim_thickness_mm, wet_or_dry, condition]
views: [head_mounted, fixed_close_on_placement_surface]
sensors: [optional glove_pressure, optional tray_force]
episodes: pick -> carry -> place -> release, plus recovery and reject decisions
labels:
  grasp_type: [rim, base, cradle, pinch, palm_support]
  outcome: [clean, slip_caught, chipped, dropped, rejected]
  step_segments: start/end per phase
metadata: [site_id, operator_id (pseudonymous), pace_class, surface_type]
rights: "training and evaluation use; breakage cost allocation per agreement"

Limits and extra preparation

Demonstrations by skilled people show good handling, but they rarely include enough failures to teach where the limit is. Human hands have tactile sensing and compliance that robot hands may lack, so strategies do not transfer directly; see the human-to-humanoid embodiment gap. Transparent items challenge depth sensing. Outcome labels need on-site input. Evaluation should use held-out objects and sites, because a model that handles one glass shape may not handle another.

Questions to ask suppliers

1. Which fragile objects appear, and can you provide an object list with mass, material and condition?
2. Is hands-and-item visibility consistent across clips, including under glare or steam?
3. Were force, pressure or tactile sensors used, and how were they calibrated?
4. How are outcomes (clean, chipped, dropped, rejected) recorded, and by whom?
5. Are slips and recoveries included or edited out?
6. At what pace were items handled, and was it normal service pace or slowed for the camera?
7. Who bears the cost of breakage during commissioned recording?
8. How many people, sites and item types are represented?
9. What consent covers the workers and any customers in view?

Discuss your requirements

If your robot needs to handle breakable items, describe the objects, sensing and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources, and can scope a commissioned collection of new recordings if no existing asset fits; a request does not guarantee a match. For wider navigation, see the workplace task demonstration hub and the humanoid training data guide. The catalog entry on egocentric video of manual work is related.

Describe the workplace demonstration data your robotics project needs

Sources

• RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot — contact-rich robot sequences with force, audio, vision and paired human video.
• Rescaling Egocentric Vision — large-scale head-mounted kitchen video with narrated action segments.