Short answer
Demonstrator skill changes what a model learns. Published offline-learning studies find results depend on demonstration quality, and the mix of operators is a variable to specify, not ignore. Specify proficiency as labeled metadata with a stated rubric, capture several operators per task, keep expert and novice splits traceable, and never assume expert footage is uniformly optimal or novice footage uniformly useless.
The buyer's problem
A supplier says its footage shows "experienced technicians". That can mean ten-year veterans doing precise work, or anyone on the payroll who was recorded on a quiet day. Skill affects the data in at least four ways: speed, smoothness, error rate and the strategy chosen. A model trained on the footage will absorb whichever it sees.
Two opposite mistakes are common. One is to filter to "experts" and lose variety and recoveries. The other is to pool everyone and discover later that the policy hesitates, because it imitates beginners' pauses. The fix is not a rule about which to buy but a specification that makes skill visible so you can choose per use.
What published work shows
Mandlekar et al. studied six offline learning algorithms on five simulated and three real-world tasks, using datasets of varying quality, and report that results depend on demonstration quality (Mandlekar et al., 2021). Their multi-human datasets contain 300 demonstrations from 6 operators, grouped into Worse, Okay and Better operator subsets, and they compare these with a proficient-human set of 200 demonstrations from one operator. The takeaway for buyers is that operator mix is a first-order variable, not a nuisance, and that results from one algorithm and task family do not transfer automatically to yours.
For human video, proficiency labels have been used as a task in their own right. Ego-Exo4D records skilled activities such as sports, music, dance and bike repair with egocentric and multiple exocentric cameras, with 740 participants, 13 cities and 1,286 hours (arXiv v4), includes expert commentary, and offers proficiency estimation among its benchmarks (Grauman et al., 2023). It shows that proficiency can be annotated and benchmarked, but its activities are not warehouse, kitchen or maintenance tasks, so its scale says little about what an industrial buyer should expect.
What skill changes in the data
| Aspect | Typical expert pattern | Typical novice pattern | Training consequence |
|---|---|---|---|
| Pace | Consistent, few pauses | Slower, hesitations | Policy speed and idle-time behavior |
| Path | Direct, economical | Searching, corrective | Smoothness of imitated motion |
| Grasp choice | Matched to object | Trial and error | Grasp diversity and failure modes |
| Errors | Rare, quickly corrected | More frequent, sometimes unrecovered | Coverage of recovery behavior |
| Strategy | Shortcuts, parallel actions | Textbook order | Multimodality of demonstrations |
| Safety habits | Embedded, sometimes tacit | Explicit or absent | Safety-related behavior coverage |
Experts may also use tricks that rely on human strength or dexterity a robot lacks; their "optimal" motion may be infeasible for a humanoid. See the embodiment gap.
What proficiency labels can and cannot support
They support filtering and weighting demonstrations, training skill or quality estimators, analyzing what a policy imitates, and building evaluation slices. They cannot show that a demonstration is good for a particular robot, that expert strategies transfer across embodiment or that proficiency in the supplier's rubric matches yours. A rubric developed for human training or performance review often rewards speed and customer outcomes that do not map to robot learning.
Illustrative specification
# Illustrative only - bands, counts and fields are examples a buyer would set
task_id: pack_order_v0
operators:
per_task_min: 6
bands: [novice, competent, expert] # supplier-defined, rubric attached
rubric: tenure + supervisor rating + measured error rate on a calibration task
per_clip_labels:
operator_id: operator_A12 # pseudonymous, stable across clips
proficiency_band: expert
proficiency_basis: supervisor_rating | tenure | measured | annotator_judgment
time_on_task_s: 74.2
error_count: 0
smoothness_note: optional free text
outcome: success
splits:
by_operator: train / validation / test with no operator in two sets
by_band: reported for each split
exclusions:
- clips where the operator knew they were being scored (flag, do not delete)
The proficiency_basis field is the one buyers most often omit. A label from tenure is a different quality of evidence than a measured error rate.
Specification guidance
Define the rubric before recording. Decide whether skill means speed, accuracy, safety or consistency. Document how raters were selected and trained, and measure their agreement.
Require multiple operators per task. One expert's habits become the dataset's habits. Six operators per task is an illustrative example, not a recommended minimum; the right number depends on architecture and task.
Keep failures and corrections. Novice runs often contain the recoveries experts avoid; see failure and recovery demonstrations and the outcome fields in success and outcome labels.
Watch the observer effect. People record differently when they know skill is rated. Note whether the footage is routine work or a performance for the camera; see working pace in demonstrations.
Split by operator. Random clip splits leak operator style between training and test. See demonstration dataset splits.
Acceptance criteria
- Every clip has an operator ID and proficiency band with a stated basis.
- A sample is re-rated by a second rater, with agreement reported.
- The mix of bands per task is reported, and no band is absent for a task without disclosure.
- Time-on-task and error counts distinguish bands in the direction the rubric predicts; if they do not, the rubric needs review.
- Operators are not shared across train and evaluation splits.
Using proficiency in training and evaluation
Three common designs are worth naming. One trains only on the expert band to maximize consistency, which gives clean behavior but narrow coverage. Another trains on all bands with quality weights or a quality-conditioned input, which keeps variety and lets the model be steered. A third trains on mixed data and evaluates by band, which tests whether the model degrades on novice-style situations. Which suits you depends on architecture and task, and published offline-learning comparisons do not settle it for yours. Whatever you choose, keep the proficiency field in the delivery so you can change your mind, and keep raters' notes so disagreements can be revisited. If expert labels come from supervisors, ask how they were collected, because ratings given for pay or promotion decisions may not describe motion quality.
Questions to ask suppliers
- How was proficiency defined and who rated it?
- Is each band based on measurement or opinion?
- How many distinct operators appear per task, and how many clips each?
- Can the same operator be traced across sites, with a pseudonymous ID?
- Did operators know they were being rated?
- Were novices genuinely new to the task or experienced workers asked to act like beginners?
- Are unedited errors and corrections kept?
- What consent covers use of skill ratings, which can be personnel information? See recruiting demonstrators and worker consent and identifiable people.
Performance ratings about named individuals carry employment and privacy implications. This is not legal advice; the supplier should confirm who may license such labels.
Discuss your requirements
If your project depends on how skilled the demonstrators were, describe the tasks, proficiency bands and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match. Start from the humanoid training data guide or the workplace task demonstration hub. If no existing asset fits, SourceX can scope a commissioned collection of new recordings, and use the demonstration video acceptance criteria to check any delivery.
Describe the workplace demonstration data your robotics project needs
Sources
- What Matters in Learning from Offline Human Demonstrations for Robot Manipulation — dependence on demonstration quality; multi-human operator subsets.
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives — skilled-activity recordings and proficiency estimation benchmark.