Deformable Object Handling Demonstrations: Textiles, Cables, Bags and Film

The buyer's problem

A rigid object has a pose; a towel has a state with no clean description. It can be flat, crumpled, folded in many ways, draped over an edge or tangled. A cable has kinks and a hidden path; a plastic bag has an opening that may or may not be where you expect; film clings to itself. The same nominal task, "put it in the box," produces visibly different configurations each time.

Published learning systems report progress on such tasks. π0 trains a flow-matching action expert on a pretrained vision-language model, using data from single-arm, dual-arm and mobile manipulators plus the Open X-Embodiment collection, and demonstrates dexterous tasks including laundry folding, table cleaning and box assembly (Black et al., 2024). ACT reports 80-90% success on 6 real-world fine-grained tasks from about 10 minutes of demonstrations (Zhao et al., 2023). These are robot-data results. They suggest the tasks are feasible for robots, not that human video of them is sufficient. The single-garment case has its own page: garment folding and textile handling.

Object classes and what each demands

Perception is the first obstacle. ZeroWaste targets deformable, often translucent objects in extreme clutter on a recycling sorting line, and is presented as the first in-the-wild industrial-grade waste detection and segmentation dataset (Bashkirova et al., 2021). It speaks to segmentation of deformable items, not to handling, but it illustrates how much annotation effort the appearance problem alone takes.

Capture specifics

Viewpoints. Include at least one overhead view that shows the full object state and one close view of the hands. Single-view footage hides folds beneath other layers. See multi-view synchronized task recording.

Backgrounds and lighting. Plain or contrasting surfaces help state estimation. White sheets on white tables, black cables on dark floors and clear bags on glass all defeat segmentation. If the production environment is cluttered, record both a clean setup and the real one.

Resolution and pace. Thin cables and film edges need higher resolution and shorter exposures than box handling. Fast production pace can blur them; see working pace in demonstrations.

Material description. Record the material, size and weight of every item, plus condition (wet, new, worn). A cotton towel and a polyester sheet behave differently, and the video will not say which.

Hand tracking. Grasp points and pinch locations matter. Where hands go under the material, tracking fails; ask how often.

Labels that help

• Object mask over time, with a state phase per segment (flat, grasped, lifted, folded, stacked).
• Keypoints for corners, hems, plug ends and bag openings where definable.
• Grasp events: which hand, which feature, pinch or whole-hand.
• Outcome with a measurable criterion: folded edge alignment, bag opened, cable seated.
• Failures: slips, tangles, snags, with recovery steps.

Expect labeling to cost more, and check whether agreement is lower than for rigid objects, because state boundaries are fuzzy. Pilot on a few runs and measure agreement before scaling; see annotation quality for task video.

What the data cannot support

Video does not carry stiffness, friction, weight distribution or tension. Two bags may look identical and behave differently when lifted. A cable's springiness shows only in how it moves. A humanoid needs observations and actions from its own body, and for tasks that depend on touch or tension, force or tactile data. When video is not enough lists those cases. Human hand strategies, such as flicking a sheet to flatten it, may not transfer to a robot gripper at all; consult retargeting human hand demonstrations.

Choosing between existing footage and new recording

Operators who handle textiles, cable or bags may hold video; ask whether a laundry films for loss claims or process review, whether a packing line has overhead cameras, or whether a cable assembler records station work for training. These recordings are good for deciding whether a supplier does the task you want and how much variety exists. They are weak as training data for hand-level handling: the camera is far away, hands are small in the frame, and the material state is hard to see.

Commissioned recording is the usual route for fine handling data. Set up the overhead and hands views, define the materials, and have workers do the real task at their normal pace with a second pass at deliberate pace if you need clearer hand trajectories. For a comparison of the two routes, read existing footage vs commissioned recording.

One more consideration is consistency of the material. Laundry and packing operations process mixed batches, so a "towel" might be six different towels. Decide whether you want a controlled sample, which makes labels cleaner, or a realistic mix, which makes the model more robust, and record the batch identifiers either way.

Supplier types

Laundries, linen services, manufacturers handling cable or wire, packers and fulfillment operations do these tasks daily and have staff with consistent technique. Their existing footage tends to be fixed-camera, wide, and aimed at throughput, so expect to commission close-up capture. Production lines are tightly safety-managed; schedule recording so as not to interrupt work.

Illustrative specification: towel and sheet handling

# Illustrative only - synthetic values
task: "fold and stack hotel towels, two sizes"
materials: [bath_towel_cotton, hand_towel_cotton]  # weights recorded per batch
views:
  overhead: 4K, 30 fps, plain dark table
  hands: head_mount 1080p, 60 fps
labels:
  mask: per-frame, towel only
  phase: [flat, grasp, lift, fold1, fold2, stack]
  keypoints: four corners, visible only
  outcome: edge_offset_mm  # measured on stacked towel after the run
variation: 3 colors, 2 sizes, 2 surfaces, 2 operators
not_included: tension, friction, robot actions

Questions to ask suppliers

1. Which materials and sizes do your workers handle, and can they be recorded and weighed?
2. Can you set up overhead and close views without disrupting operations?
3. What background and lighting choices are possible?
4. How will state phases and keypoints be labeled, and how is agreement measured?
5. Are failures such as tangles and slips kept?
6. How do workers' hand strategies differ, and is that recorded?
7. How do you handle brand labels and tags on garments or bags?
8. What are the safety limits on camera placement near machinery?

Discuss your requirements

If you are specifying deformable-handling data, describe the materials, tasks and label needs to SourceX. SourceX reviews the requirement and follows up about potential sources; new recordings are scoped case by case and a request does not guarantee a match. See the demonstration hub and the humanoid training data guide for the surrounding map.

Describe the workplace demonstration data your robotics project needs

Sources

• π0: A Vision-Language-Action Flow Model for General Robot Control — pi0 dexterous tasks including laundry folding.
• Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware — ACT results on fine-grained tasks from limited demonstrations.
• ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes — ZeroWaste deformable object segmentation in clutter.