Short answer
Mobile manipulation demonstrations record a person or robot moving between stations and handling objects at each one, so the data must link where the body is to what the hands are doing. Specify a floor-plan reference, body and head pose, continuous recording across room transitions, and labels that mark both navigation segments and manipulation segments.
The buyer's problem
Bench-top demonstrations assume the robot stays put. Real workplaces do not. Setting a table, restocking a supply closet or moving samples between lab stations involves walking, turning, opening doors, carrying and handling objects in different rooms. A model that must do this needs training examples where movement and manipulation are joined, not two separate datasets stitched together.
The two halves change each other. Where the base stops determines what the arm can reach; carrying an object changes how the body can move; arriving at a station sets up the first grasp. Buyers often find that archives contain one half only, with clips that start at the station and end when hands leave the object. For the single-station case, see bimanual task demonstrations.
What published work shows
Mobile ALOHA demonstrates the pattern in teleoperation. Its whole-body interface has the operator tethered to a mobile base, backdriving it as they work, and the authors report 50 demonstrations per task, with co-training on static ALOHA data raising success by up to 90% (Fu et al., 2024). The point for a buyer is that mobile manipulation was learned from demonstrations that included base motion and arm motion together, supported by additional static manipulation data.
DROID, a large teleoperated dataset, shows the other axis: scene diversity. It reports about 76,000 trajectories, 564 scenes and 86 tasks collected by 50 data collectors over 12 months, with a replicated rig at 13 institutions and calibration, depth and language instructions per episode (Khazatsky et al., 2024). It is a tabletop-arm dataset rather than a mobile one, but the packaging, with calibration and language per episode, is a good model for what to request.
For task coverage, BEHAVIOR-1K defines 1,000 everyday activities in 50 scenes in a simulator that handles rigid, deformable and liquid objects, drawing candidate activities from time-use surveys and WikiHow (Li et al., 2024). It is a benchmark and a source of task ideas, not demonstration data from workplaces.
What to capture for human demonstrations
| Stream | Why it matters | Notes |
|---|---|---|
| Egocentric video | Hands and objects as seen by the worker | Head or chest mount; see head-mounted camera capture |
| Body and head pose | Where the body is and how it moves | IMU suit, visual-inertial tracking, or estimation from video |
| Floor-plan position | Links stations to places | Map with named stations; periodic position fixes |
| Fixed overview cameras | Route and room context | Hard to cover large areas; overlap needed |
| Door and fixture states | Moves between rooms | Label open, closed, held |
| Carried load | Changes gait and manipulation | Label object and weight class |
Position tracking is the hard part. GPS fails indoors, and visual odometry from a head camera drifts over long routes. Ask how the provider localizes the worker and how drift is corrected, for example with fiducial markers at stations or anchor points on the floor plan.
Segmentation needs two label families on one timeline: navigation (from station A to B, with obstacle and door events) and manipulation (grasp, carry, place). Mark transitions explicitly, including the moment the worker sets down a load to open a door, which is often where policies fail. Doors, drawers and cabinets and pushing carts and trolleys cover two frequent transitions.
Existing footage or commissioned recording
Existing footage rarely fits. Fixed cameras cover a single room, so a worker disappears at the doorway, and body-worn video, where it exists, was shot to document an incident or a training point, not to follow a route. If a supplier does hold continuous body-worn recordings of multi-room work, ask for a sample and check whether the room transitions are intact, but expect to add floor-plan and station labels afterward.
Commissioned recording lets you design the route: named stations, deliberate obstacles, varied start points and loads, and a repeatable position reference. It also means site access across rooms, which is a larger permission surface than one workstation. Corridors, stairwells and shared areas bring in bystanders, and some rooms may be off limits. A pilot on one floor, with three or four stations, is usually enough to test localization and labeling before widening the route. Use pilot collection scoping to frame it.
What the data cannot support
Human walking is not the locomotion of a wheeled or legged robot. The path a person takes, the clearance they allow and their speed through a doorway all differ from what a humanoid's balance controller or a wheeled base can do. Human video offers the structure of the route and task and the visual appearance of places; it gives no base velocity commands, odometry or footstep targets for a specific robot. Those need robot data or retargeting, discussed in the human-to-humanoid embodiment gap. Stairs and ladders raise further issues; see workplace movement: stairs and ladders.
Whole-building data is also costly to cover. A route that works in one building teaches little about another layout. Plan evaluation with held-out buildings or floors.
Illustrative capture specification
# Illustrative only - synthetic values
site: site_022 (lab building, 2 floors, 6 named stations)
task: "collect samples from stations 1-4, deliver to station 6"
streams:
egocentric: head_mount, 1080p, 30 fps
body_pose: imu_suit, 60 Hz
position: floor-plan fixes at each station (marker scan) + visual odometry between
overview: 3 fixed cameras, corridor and station 6
sync: shared timecode; drift check at start and end of each run
labels:
segments: [navigate, manipulate, wait]
events: [door_open, load_pickup, load_place, obstacle_avoid]
station_ids: from floor plan
metadata: map version, lighting, occupancy level
not_included: robot odometry, base commands, joint states
Quality and acceptance criteria
Examples to set: position error against known station markers under a threshold; no gaps in the timeline across room transitions; navigation and manipulation segments cover the run without overlap; door and carried-load events labeled in at least a set share of runs; sampled clips show hands during every manipulation segment. See demonstration video acceptance criteria for the general rules.
Questions to ask suppliers
- How is the worker localized indoors, and what is the drift over a full route?
- Are recordings continuous across rooms, or split at doors?
- What floor plan and station naming scheme will be delivered?
- How do you track body pose, and what is its stated error?
- Can routes, load weights and start stations be varied?
- How are other people in corridors handled for consent and redaction?
- Are there areas where recording is restricted?
- How are cameras synchronized and checked for drift?
Discuss your requirements
If you need movement and manipulation recorded together, describe the building types, routes and labels to SourceX. SourceX describes a separate physical-world path for new recordings of hands-on work, where each opportunity has its own requirements. It reviews your request and follows up about potential sources; a request does not guarantee a match. See the demonstration hub and the humanoid training data guide.
Describe the workplace demonstration data your robotics project needs
Sources
- Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation — Mobile ALOHA whole-body teleoperation, 50 demonstrations per task.
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — DROID scale, rig, 15 Hz control and per-episode calibration and language.
- BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation — BEHAVIOR-1K activities, scenes and simulator coverage of liquids and deformables.