Short answer
A pilot collection is a small run of the full commissioned pipeline: one or two sites, a few demonstrators, a handful of tasks, every sensor and label type, delivered in the final format. It tests the capture spec, consent process, annotation guidelines and your training pipeline on real files, and it sets go/no-go criteria and a frozen specification before you commit to scale.
What a pilot has to answer
A commissioned demonstration program can fail in ways no document review will reveal: a head camera that loses the hands on one task, a site that can only host two hours a week, labels that annotators cannot agree on, or files your loader cannot parse. A pilot exists to surface those failures while they are cheap. It should answer six questions:
- Capture. Does the rig record what the model needs, at the quality the spec assumes?
- Operations. Can the host run sessions without disrupting work, and how long do setup and teardown take?
- People. Do demonstrators tolerate the devices for a full session, and does the consent process work, including withdrawals?
- Labels. Can annotators apply the guidelines consistently?
- Pipeline. Do the delivered files decode, align and load in your own training and evaluation code?
- Economics. How many usable task minutes come out of each recorded hour, and how much annotation does each minute need?
What a pilot is not
A pilot is not a sample of an existing dataset. Reviewing samples from a supplier's archive is a different exercise, covered in requesting and reviewing dataset samples.
Nor is a pilot a model-performance test. Its volume is far too small to predict what the full dataset will do for your model, or to tell you how much data you need; published figures on that question are collected in how many demonstrations you need. Use the pilot to test the pipeline, not to estimate success rates.
Scope: small in volume, complete in kind
Shrink every dimension except completeness. Every stream, label type, metadata field and deliverable in the full specification should appear in the pilot at least once.
| Dimension | Illustrative pilot scope | Rule of thumb |
|---|---|---|
| Sites | One or two | Include the hardest setting you plan to use, not only the easiest |
| Demonstrators | Three or four with different experience | Never a single person |
| Tasks | Two to four | Include the hardest task in each task family |
| Sensors and streams | Every stream in the full spec | Nothing added later without another pilot |
| Labels | Every label type on a subset | Double-annotate part of it |
| Deliverables | Final file formats, metadata, calibration reports, rights documents | Same packaging as production |
The capture choices being tested come from the head-mounted camera capture specification and from the setting decision in staged workcells vs live operations. A common pattern is to pilot in a staged or off-shift setting and add one live session if live recording is planned.
Why several demonstrators, even in a pilot
Demonstration quality changes what a model learns. In a study of offline learning from human demonstrations, results depended on dataset quality; the multi-human datasets contained 300 demonstrations from 6 operators grouped as worse, okay and better, alongside 200 demonstrations from a single proficient operator (Mandlekar et al., 2021). A single-person pilot hides the spread of quality you will see at scale. With three or four people you can check whether to screen, train or label demonstrators by proficiency before the main program.
Make the pilot rig the template
Standardization is what lets multi-site data be pooled. DROID replicated the same collection rig at 13 institutions and recorded calibration, depth and language instructions for every episode (Khazatsky et al., 2024). Treat the rig that survives the pilot as version 1.0: device models, firmware, mounts, calibration routine, file naming and metadata fields. After that, changes go through written change control, with a note in the metadata of every clip recorded under the new version.
Test every stream end to end
Each additional stream is another chance for a sync, format or privacy problem. Ego-Exo4D, for example, combines video with multichannel audio, eye gaze, 3D point clouds, camera poses and IMU (Grauman et al., 2023). For your pilot, every stream should be shown decoded, time-aligned and reviewed by the person who will use it. If nobody downstream uses a stream in the pilot, question whether it belongs in the full specification.
Pilot metrics and go/no-go criteria
| Metric | How to measure | Example go criterion (buyer sets) |
|---|---|---|
| Usable yield | Usable task minutes ÷ recorded minutes, after QC | Above the share assumed in the budget |
| Hand visibility | Detector plus manual check on a sample | Meets the per-task target |
| Synchronization | Slate or tap test across streams | Within tolerance in every session |
| Label agreement | Double annotation of a subset | Boundary and label agreement above set levels |
| Session logistics | Setup, teardown and interruption logs | Fits the host's available window |
| Consent outcomes | Opt-outs, withdrawals, bystander incidents | No unresolved incidents |
| Pipeline ingestion | Load the full delivery and run a short training or evaluation job | No manual fixes required |
| Storage per hour | Bytes per recorded hour, by stream | Consistent with the budget for scale |
Record the results in a short pilot report that both sides sign. Problems found during recording, rather than after delivery, belong in the program's in-collection quality control checks for the full run.
From pilot to full program
- Freeze the specification. Turn the pilot findings into spec v1.0, listing every change from the draft.
- Re-estimate cost and schedule from measured yield and annotation time, not from assumptions.
- Write the coverage plan for sites, demonstrators, objects and conditions; see coverage planning for demonstrations.
- Decide what happens to pilot data. It can join training if it meets spec v1.0, or be kept as a small held-out check set.
- Re-pilot when you change device, site type or task family.
Illustrative pilot plan
# Illustrative only — synthetic IDs and example values
pilot: shelf_replenishment_pilot_v0
duration: two recording weeks plus one review week
sites:
- site_017 (staged replica, off-shift)
- site_024 (one live session, if host approves)
demonstrators: [operator_A12, operator_B07, operator_C33, operator_D41]
tasks: [case_open_and_face, mixed_shelf_restock, damaged_item_removal]
streams: head camera, two fixed cameras, IMU, session audio slate
labels:
all_clips: task and step segments, outcome
subset: hand-object contact, language instruction; 20% double-annotated
deliverables: final container format, metadata sidecars, calibration reports,
consent register extract (pseudonymous), pilot report
go_no_go: metrics table above, thresholds agreed before recording starts
Limitations
- One or two sites cannot reveal variance across sites; that only appears at scale.
- Demonstrators behave differently in their first sessions with a device; discount early clips when judging yield.
- A pilot of human recording produces no robot actions. If your stack also needs teleoperation or handheld-gripper data, pilot that separately with the relevant specialists.
- Passing the pilot does not show that the full dataset will improve your model; it shows the pipeline works.
Questions to ask collection providers
- Can the pilot use exactly the devices, firmware and packaging planned for the full program?
- Which sites and demonstrators are available, and how representative are they of the full plan?
- How will usable yield be measured and reported for each session?
- Who annotates the pilot, and can a subset be double-annotated?
- How soon after each session can raw files be handed over for pipeline testing?
- What rights apply to pilot data if the full program does not go ahead?
- How would cost and schedule change if measured yield is lower than planned?
- How are consent withdrawals during the pilot reflected in the delivered files?
Discuss your requirements
If you are planning a commissioned program and want to start with a pilot, describe the tasks, sites, streams and labels to SourceX. SourceX reviews the requirement and follows up about potential sources; whether a commissioned collection is feasible depends on the opportunity, and the site's physical-world path describes how new recordings of hands-on work are coordinated with businesses. A request does not guarantee a match. See also the workplace task demonstration hub and the humanoid training data guide.
Describe the workplace demonstration data your robotics project needs
Sources
- What Matters in Learning from Offline Human Demonstrations for Robot Manipulation — demonstration quality and multi-operator datasets.
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — a standardized rig replicated across institutions with per-episode calibration.
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives — example of a multi-stream capture design.