Competency Assessments and Skill Sign-Offs as Proficiency Labels for Demonstration Data

Why proficiency labels matter

Demonstration quality is not uniform, and learning methods respond to it. A study of six offline learning algorithms across five simulated and three real-world manipulation tasks found that results depend on the quality of the demonstrations. Its multi-human datasets contained 300 demonstrations from six operators, grouped as Worse, Okay and Better, alongside a 200-demonstration set from one proficient operator (Mandlekar et al., 2021). The finding is a reason to ask where quality labels for your own data will come from.

Businesses already produce such labels as a by-product of training and compliance. A forklift operator's practical sign-off, a line worker's certification on a station, a technician's graded task observation: each is a human judgment about performance, made against a standard, by someone qualified to make it. If a recording shows that person doing the task, the judgment can become a label.

For the recordings themselves, see instruction-paired demonstration datasets; this page covers the grading layer and how to source it.

Record types worth distinguishing

Many of these records come out of training programs; the course content itself is covered in LMS training courses, and spoken coaching in trainer feedback and corrections.

Observed assessments are usually the strongest because they sit closest to a specific performance. A skills matrix rating is a person-level summary and can mislead when mapped to a single clip.

Illustrative mapping from records to clips

The structure is Illustrative; every value is synthetic.

# Illustrative: proficiency label attached to one demonstration clip
clip_id: clip_site_031_0192
operator_ref: operator_A12          # pseudonymous, no name
task_ref: "pallet_wrap_station"
assessment:
  type: observed_assessment
  rubric_id: rubric_station_v3      # supplier rubric, version kept
  criteria_met: 9
  criteria_total: 11                # synthetic values
  assessor_role: "shift_lead"
  assessed_on_this_clip: true       # vs. assessed on a different attempt
level_at_time: "qualified"          # qualified | trainee | senior
level_scale_source: "supplier_defined"
consent_scope: "training_and_evaluation"

The field assessed_on_this_clip matters. If the assessment was of a different day's performance, the label is an attribute of the operator at that time, not of the clip.

How a buyer would use the labels

• Stratified evaluation. Report results by operator level, so you can see whether a policy trained on mixed quality degrades on better or worse demonstrations.
• Filtering and weighting. Down-weight or drop low-proficiency clips before imitation learning, and compare against using everything.
• Failure analysis. Pair failed assessment criteria with the recording to see which sub-steps experts do differently.
• Rubric-to-reward ideas. Criteria can seed progress or quality scoring for the task, in the way checklists as success criteria describes.

Treat these as hypotheses to test on your data. The cited study shows that quality shapes outcomes in its setup; it does not show that any particular business rubric predicts usefulness for a humanoid.

The mapping work is the real cost

Many businesses hold assessments in one system and recordings in another, with nothing joining them. Someone must match an operator, a date and a station to a clip, and decide which clips the assessment actually covers. That work is manual or semi-manual and its error rate is rarely recorded.

Three mapping rules keep the labels honest. First, join on the attempt wherever possible, not on the person. Second, record how the join was made (direct reference, timestamp match, supervisor recollection) so you can drop the weakest links. Third, keep unmapped records out of the labeled set instead of assigning the operator's current level to old footage; levels change, and a trainee's clip should not be tagged with the level they reached a year later.

Quality and acceptance criteria

• A stated share of assessments is linked to a specific attempt, with the join method recorded per link.
• The rubric version at assessment time is kept; if the rubric changed, both are present.
• Level definitions are supplied in words, not only as numbers.
• A sample is re-rated by a second assessor, and the agreement is reported as a figure the supplier measured, not assumed.
• Evaluation splits hold out operators entirely, because the same person's clips share style and can leak across the split.

What this data cannot support

Competency records are not robot data and cannot train control. They hold no joint states, actions or forces; even paired with video they describe a human who passed, not the motion a robot would need. Human proficiency is also not robot proficiency: an expert's technique relies on hand dexterity, strength and vision that a humanoid's body may not match, so "best human" is not automatically "best demonstration for this robot." The embodiment problem is covered in the human-to-humanoid embodiment gap.

Further limits:

• Classroom realism. Assessments in training facilities may use simplified setups that differ from live production.
• Rubric portability. One employer's level 3 is another's level 2; labels are comparable only inside one rubric.
• Selection effects. Sign-offs exist for people who attempted them. Failed attempts may not have been recorded.
• Assessor noise. Without double rating, a score may reflect the assessor as much as the performer.

Permissions and provenance

This is the highest-friction record type in the procedures cluster, because it is employment and learning data about identifiable people. Learners may be trainees, students or minors in some programs, and institutions often have their own policies on use of assessment records. Pseudonymising the operator reference does not by itself settle whether the individual agreed to this use or whether the employer may license it. Ask for the consent basis for each record type and have counsel review; this is not legal advice. See worker consent and identifiable people.

Questions to ask suppliers

• Which records are observed assessments of a specific attempt, and which are periodic ratings?
• Is the rubric available, with its version and the level definitions?
• Can each assessment be linked to a specific recording, or only to a person and date?
• How many assessors graded, and was any sample double-rated?
• Were operators told that assessments could be used for model training or evaluation?
• Are failed or repeat attempts retained, or only passes?
• Who owns the records: the employer, a training provider or a certification body?
• Can scores be provided under pseudonymous operator references without a key that identifies individuals?

For how scores relate to wider quality ratings of procedures themselves, see procedure quality scoring.

Discuss your requirements

If you need graded demonstrations, describe the tasks, the level of grading detail and the consent scope in a buyer request. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match. If existing records cannot be paired with recordings, SourceX can scope a commissioned collection of new recordings. Start from the procedures hub or the humanoid training-data section.

Tell SourceX what your evaluation set needs to cover.

Sources

• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation — demonstration quality affects offline learning; Worse, Okay and Better operator groupings.