Short answer
Opening containers, lids and packaging is a family of constrained, articulated motions: twisting, prying, peeling, cutting, pulling tabs and unfolding flaps. Useful demonstrations show each mechanism from several angles, label the opening type and the state before and after, and record failures. Video shows the motion but not the torque, grip force or latch resistance.
Task anatomy
Every opening task has a mechanism, and the mechanism determines what the hands must do. A robot that opens a screw cap needs a different skill from one that peels a film seal, even though both are "open the container."
| Mechanism | Example objects | Key motion | Typical failure |
|---|---|---|---|
| Screw cap | Bottles, jars | Grip, counter-rotate, unscrew | Slip, cross-thread, over-tight |
| Snap or press lid | Takeout tubs, deli containers | Find edge, pry or press | Lid flips, container deforms |
| Hinged lid | Boxes, bins, laptop-like cases | Lift and rotate about a hinge | Hand blocks hinge |
| Peel seal | Film lids, pouches | Pinch corner, peel at angle | Tears, no corner purchase |
| Tab or pull strip | Cartons, tape-closed boxes | Locate tab, pull along line | Strip breaks |
| Cut | Tape, strapping, shrink wrap | Tool use, controlled depth | Cuts contents |
| Flap and fold | Cardboard cartons | Unfold, hold, fold back | Flaps spring closed |
Unboxing has an additional sequence: place the package, find the opening, open, remove contents, flatten or discard packaging. That sequence is a long-horizon task in its own right. Workplaces see it in receiving rooms, kitchens, labs and supply closets.
What related datasets show
Datasets of hand-object manipulation illustrate what rich capture can look like. ARCTIC provides 2.1M video frames with 3D hand and object meshes and dynamic contact information, showing bimanual articulation of objects such as scissors and laptops, recorded with eight allocentric views plus one egocentric view and ground truth from motion capture (Fan et al., 2023). TACO offers 2.5K bimanual motion sequences across tool-action-object combinations, with 12 third-person views plus one egocentric view, and 3D hand-object meshes, 2D segmentation and behavior triplet annotations (Liu et al., 2024).
At the other end of scale and detail, the "something something" database contains more than 100,000 crowdsourced videos across 174 classes defined as caption templates in its original release, with the object noun replaced by "something" (Goyal et al., 2017). Its classes include actions on objects, and it is useful for action recognition, though it does not give hand meshes or mechanism metadata.
The contrast matters for buying. Mocap-grade rigs produce 3D contact and meshes but in a lab, on a limited set of objects. Crowdsourced clips cover many objects but give only a caption. Workplace recordings fall between: real objects and conditions, with the capture quality you design. See hand-object interaction annotation for the labels that sit between those poles.
What video can support, and what it cannot
Video supports recognition of the opening action, identification of the mechanism, step order, success or failure, and the visual states "closed" and "open." It can support learning where to grasp and where the opening starts, particularly if the grasp point is labeled.
It cannot show the torque needed to break a seal, the grip force that avoids crushing a soft container, the stiffness of a latch or the slipperiness of a wet lid. A humanoid hand also differs from a human hand in finger count, compliance and strength, so a human's two-handed twist may need a different strategy. Contact-rich cases may require force or tactile sensing; see when video is not enough. The pieces of the problem that involve doors, hinges on furniture and drawers are covered in door, drawer and cabinet operation.
Capture and labels
- Views. A close-up view of the opening point is essential; head-mounted video alone often loses it behind the hands. Add a second angle from above or the side.
- Object log. Record container type, material, size, closure type and contents. Note whether it is new, previously opened or deliberately hard to open.
- Labels. Mechanism type; grasp points; start and end of each phase; outcome; failure type; whether tools were used; open/closed state per frame.
- Variation. Different tightness, wet versus dry hands, gloves, reclosing after use, packages from multiple suppliers.
- Gloves. Many workplaces require gloves, which change appearance and tracking. See gloved hands in workplace video.
Illustrative specification
# Illustrative only - synthetic values
task_family: "open and empty received packaging"
mechanisms: [screw_cap, snap_lid, peel_seal, tape_cut, flap_carton]
objects_per_mechanism: 12 distinct items (log brand, size, material)
views:
close_up: 1080p, 60 fps, 30 cm from opening point
head_mount: 1080p, 30 fps
overhead: 1080p, 30 fps
labels:
mechanism, grasp_point, phase, open_state, outcome, failure_type, tool_used
variation: [new vs previously opened, gloved vs bare, wet vs dry]
failure_runs: kept and flagged, share set by buyer
excluded: torque, grip force, robot actions
Quality and acceptance criteria
Examples a buyer can set: the opening point visible in a set share of frames of each opening phase; every mechanism represented by enough distinct objects that a held-out object is possible; failures present and labeled; start and end states recorded; consistent file naming linking object log to clip. Review a random sample against the log. Where the object is destroyed in opening, such as a peel seal, plan enough supply for multiple runs.
Reading the sample for hidden weakness
Opening tasks look easy in a demo and fail in details. When you review a supplier's sample, look at the first two seconds of each opening: does the worker find the opening point by sight, or do they already know where it is because they have opened this product a thousand times? Experienced staff skip the search phase, and a model trained on them will not see how to find an unfamiliar tab. Include some runs with unfamiliar packages, where workers search, test and adapt.
Look also at how the video handles contents. If the container is opaque and the contents are hidden, the "emptying" part may be invisible. If contents are brand-sensitive, the supplier may crop them out, which removes the end state you wanted to label. Settle in advance whether contents are shown and, if not, what substitute labels are used.
Finally, check the repetition design. Ten openings of one jar are ten samples of one object. Prefer a smaller number of repetitions across more objects, and keep a few objects and a few suppliers' packaging aside for evaluation. Demonstration dataset splits explains how to hold these out.
Supplier types
Kitchens open cans, tubs and bags all day; warehouses and receiving docks open cartons and strapping; laboratories open vials, sealed pouches and tubes. Each sees different closure families. Request a mix, and confirm which products' packaging may appear on camera, since product labels and supplier names carry brand and, in some cases, contractual restrictions. See warehouse and fulfillment data for the neighboring supply route.
Questions to ask suppliers
- Which closure types do your staff handle, and how many distinct products per type?
- Can packages be provided in quantity for repeated, recorded openings?
- Will you record the object log: brand, size, material, contents?
- Can the opening point be filmed up close without blocking the work?
- Are failures such as tears and slips retained?
- Do staff wear gloves, and can some runs be bare-handed where permitted?
- How are product labels and customer-specific packaging handled?
- Who approves recording near customer goods?
Discuss your requirements
If you need opening and unpacking demonstrations, describe the mechanisms, objects and label needs to SourceX. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match. Use the demonstration hub and the humanoid training data guide for adjacent decisions.
Describe the workplace demonstration data your robotics project needs
Sources
- ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation — ARCTIC bimanual articulated hand-object capture.
- The "something something" video database for learning and evaluating visual common sense — Something-Something crowdsourced video classes.
- TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding — TACO bimanual tool-action-object sequences.