Tote and Container Handling Demonstrations

Task anatomy

A tote is the basic unit that moves between stations in many fulfillment operations: a plastic container, usually stackable, with handholds, a label and sometimes a lid. Handling a tote sounds trivial and splits into several tasks. A worker picks a tote from a stack, a cart, a rack or a conveyor; checks the label; carries or pushes it; places it on a conveyor, shelf, station or another tote; and stacks or unstacks in columns. Variants include nested empties, lidded totes, full totes with uneven loads and foldable containers.

Humanoid pilots often start here because the object is standard, the paths are short and the tasks repeat. The details still break systems: a tote with a bowed side, a stack that leans, a tote caught on a conveyor guide, a tote that is heavier than its size suggests, a label on the wrong face or a lid that comes off during a lift. Each is a failure case a dataset has to include to be useful.

This page covers sourcing demonstrations of tote tasks. It sits in the warehouse cluster of the humanoid data guide. For emptying cases into totes, see decanting cases into totes; for items inside the tote, bin-picking image datasets.

Task variants and what each needs

Fulfillment centers and third-party logistics operators perform all of these, often with conveyors, goods-to-person stations and put walls; see goods-to-person station tasks and put walls and order consolidation.

What published warehouse data shows

Published warehouse manipulation benchmarks illustrate how much structure annotated data can carry. ARMBench was collected in an Amazon warehouse pick-and-place workcell and provides images, videos and metadata for more than 235K pick-and-place activities on more than 190K unique objects, with tasks including object segmentation in clutter, object identification and defect detection (Mitash et al., 2023). It documents robotic picking of items, not human handling of totes, so it is a reference for annotation scale and schema, not a source of tote demonstrations. Scale figures from other studies are what those studies used; your requirements depend on architecture, task, embodiment and use.

What the data can support

• Perception. Tote detection, pose, fill state and label location, including in stacks and on cart racks.
• Step understanding. Segmenting reach, grasp, lift, carry and place from video, with language descriptions.
• Evaluation. Comparing a system's outcomes (placed, aligned, stable) with human outcomes on held-out stations and shifts.
• Pilot scoping. Cycle times, path lengths and tote weights drawn from records and observation.

If you plan to use human video for policy learning, see what human demonstration video can and cannot do and human demonstrations versus teleoperated robot demonstrations.

What this data cannot support

• Robot actions and states. Human video has no joint positions, end-effector commands, grasp forces or commanded torques. A humanoid requires robot-specific observations and actions to carry a tote.
• Embodiment match. Humans use two hands with fingers, hips and knees, and shift weight in ways a humanoid may not. Grip strategy, lift height and speed differ; see the embodiment gap.
• Load information. Video shows a tote, not its mass or center of gravity. Weights must come from records or scales.
• Occlusion-free views. Hands and handholds are often hidden by the tote itself; head-mounted views help but introduce head motion.
• Contents. Tote contents are usually client goods, and detailed product data cannot be released without client permission.

Illustrative capture specification

# Illustrative only - synthetic values, not an available dataset
capture: tote_handling_v0
site: fc_021
cameras:
  - {id: cam_head, mount: head_mounted, fps: 60, shutter: fast, calibration: intrinsics_recorded}
  - {id: cam_fix_1, mount: ceiling_oblique, fps: 30}
  - {id: cam_fix_2, mount: low_side, fps: 30}
sync: {method: shared_clock_plus_clap_event, max_offset_ms: 20}
scene_variants: [stack_pick, conveyor_place, cart_load, lidded_carry]
tote_types: [std_600x400_a, std_600x400_b, foldable_c]
load_weights_kg: [0, 4, 9, 14]   # buyer sets
labels:
  steps: [reach, grasp, lift, carry, place, release]
  outcome: [ok, readjusted, dropped, jammed]
  tote_id: tote_hash
operators: {count: 8, ids: pseudonymous, glove_use: recorded}
exclusions: [robot_actions, forces]

Values are placeholders chosen by the buyer. A team that needs force information would add instrumented handles or a handheld capture device, since business footage cannot supply it.

Sourcing and permissions

Existing footage is usually from fixed cameras placed for safety or throughput, with wide views and limited hand detail. Commissioned recording can place cameras, record head-mounted views, script variants and failure cases, and measure load weights, but it takes time, access and consent. See existing footage versus commissioned recording and 3PL providers as data suppliers.

Ask who owns tote contents in the frame, whether clients agreed to recording, how worker consent was handled and whether labels or screens show confidential data. Blurring does not on its own settle these questions. This is not legal advice.

Quality and acceptance checks

Buyers set the thresholds. Checks worth writing into an acceptance test include the following.

• Handhold visibility. Sample clips and measure how often the hands and the tote's handholds are both visible during the grasp.
• Variation coverage. Tabulate tote type, load, stack height and surface (conveyor, cart, floor). Gaps here become failures in a pilot.
• Failure retention. Confirm that drops, jams and re-grips remain in the data, since removing them makes the task look easier than it is.
• Synchronization. For multi-camera sets, verify the stated offset using a visible event.
• Weight and label records. Check that weights come from a scale or record, not an estimate, and that tote IDs link to the right clips.
• Held-out design. Split by site, operator or tote type according to the claim you want to make.

A closely related task is parcel sortation, where the object stream is more variable and the label matters more.

Questions to ask suppliers

1. Which tote tasks are recorded: stack, conveyor, cart, station, lid or carry?
2. What camera positions and frame rates were used, and are hands and handholds visible?
3. What tote types, sizes and load weights appear, and are weights recorded?
4. Are jams, drops, stack failures and corrections kept?
5. Are cameras synchronized, and how was the offset measured?
6. How many sites, shifts and workers are represented?
7. Are tote contents visible, and how are client goods and labels handled?
8. What notice and consent did workers receive, and which clients agreed to recording?
9. Can the supplier add head-mounted views or scripted variants if existing footage lacks them?

Discuss your requirements

If your pilot starts with tote handling, describe the tasks, tote types and permitted uses to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. If no existing asset fits, SourceX can scope a commissioned collection of new recordings. A request does not guarantee a match, and nothing is contracted until a supplier agrees. The catalog entry for first-person video of skilled manual work describes a related collection program.

Describe the workplace demonstration data your robotics project needs

Sources

• ARMBench: An Object-centric Benchmark Dataset for Robotic Manipulation — Published warehouse pick-and-place benchmark showing the scale and structure of annotated warehouse manipulation data.