Privacy and preparation
Robotics demonstration diversity audit for thousands of similar clips
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
To measure skill diversity in thousands of similar clips, stop counting clips and count distinct cells: each object handled, each action performed and each operating condition, with failures and interventions tagged. A robotics demonstration diversity audit then reports how many cells are covered, how concentrated the clips are in the biggest cell and the effective number of distinct cells, so you can see whether ten thousand clips teach one skill or many.
Video and sensors
Robotics demonstration
Key takeaways
- Clip count is a weak signal; robot learning research reports that adding environments and objects helps generalization more than adding demonstrations of the same setting.
- The audit unit is a cell: one object, one action and one condition, defined before anyone counts footage.
- Three numbers summarize the grid: cell coverage, top-cell share and the effective number of cells.
- Failures, recoveries, interventions and outcomes are tagged per clip, because a library of only clean runs hides the variation a policy most needs.
- Consent, worker visibility and camera metadata are audit fields too; a varied clip without them cannot be offered.
- The audit decides whether to offer the library as it is, narrow it to the cells that are well covered, or record more before offering.
How do you measure skill diversity in thousands of similar video clips?
You measure it by turning the library into a grid of objects, actions and operating conditions and counting how the clips spread across that grid. Ten thousand clips of one worker taping one carton size on one bench fill a single cell, however long the footage runs.
A robotics demonstration diversity audit has four steps. First, fix the audit unit: what counts as one task, one object class and one action verb. Second, tag every clip with those values plus the condition it was recorded in and whether it shows a clean run, a failure, a recovery or a human intervention. Third, score the grid. Fourth, decide whether to offer the library as it is, narrow it, or record more.
The output is not a single diversity score. It is a table a robotics team can read in a minute: which object and action pairs are covered, how deep each one is, which conditions are missing and how much of the footage is near-duplicate repetition.
Why clip count overstates what a robot policy can learn
Volume matters less than spread. In a study of imitation learning for manipulation, Lin and colleagues collected more than 40,000 demonstrations and reported that a policy's ability to handle new objects and new environments followed a roughly power-law relationship with the number of training environments and objects, and that adding environments and objects did far more than adding demonstrations within one setting.
The released policies from that work were trained on 32 environment-object pairs with 50 demonstrations per environment, across four tasks such as pouring water and folding a towel. The design itself is the lesson: the authors budgeted demonstrations across many pairs rather than piling them into one.
Large public robot datasets describe themselves the same way. DROID reports about 76,000 trajectories collected across 564 scenes and more than 80 tasks by 50 collectors over 12 months. BridgeData V2 reports 50,365 demonstrations of 13 skills across 24 environments within 60,096 trajectories. Open X-Embodiment pooled data from 22 robots at 21 institutions covering 527 skills. Each headline leads with scenes, skills and environments, not hours.
For a business holding workplace footage, the consequence is direct. A robotics team reading your inventory will ask how many distinct things the clips show before it asks how many clips there are, and the audit lets you answer with evidence instead of a total.
Setting the audit unit before counting anything
Define the task boundary, the object classes and the action verbs in writing before tagging a single clip, or the counts will drift as people tag. A task boundary says where a demonstration starts and stops: for example, from picking up an empty carton to placing a sealed one on the outbound conveyor.
Object classes should be the level at which handling differs. Small, medium and large cartons are different objects if the grip or the tape path changes; two carton colors are not. Action verbs should name what the hands do: pick, place, fold, tape, insert, scan, wrap, cut. Keep the list short enough that two taggers agree on most clips.
Conditions are the third axis: lighting, station layout, shift, the operator, the tool version and anything else that changes what the camera sees or what the hands must do. Record them as columns, not free text.
If the footage will be packaged in a common robot learning format, the tags map cleanly onto it. The LeRobot dataset format, for instance, keeps natural-language task descriptions mapped to integer IDs in a tasks file and stores per-episode records with lengths and tasks, so a consistent task vocabulary in the audit becomes the task index a buyer loads.
The demonstration diversity audit by object and action
The audit sheet below is the asset. Fill one row of tags per clip, then compute the measures in the third column across the library. The red-flag column says what reading should change the offer.
At a glance
The demonstration diversity audit by object and action
| Audit field | What to record per clip | Library-level measure | Red flag |
|---|---|---|---|
| Task boundary | Task ID from the written task list; start and end timestamps | Share of clips that start and end inside the defined boundary | Many clips cut mid-task or span two tasks |
| Object class | One class from the fixed object list | Clips per object class; classes with fewer than a set minimum | One class holds most of the library |
| Action verb | Primary verb, plus secondary verbs if the task chains them | Distinct object-action pairs observed versus pairs the operation actually performs | Pairs the operation performs daily never appear |
| Condition tags | Lighting, layout, shift, tool version, operator code | Distinct conditions per object-action pair | Every pair recorded under one condition only |
| Outcome | Success, failure, recovered, abandoned | Outcome mix per pair | Only successes, or outcomes not recorded at all |
| Intervention | Whether a second person or a supervisor stepped in, and when | Clips with interventions per pair | Interventions edited out of the footage |
| Near-duplicate cluster | Cluster ID from visual similarity grouping | Clips per cluster; effective number of distinct clips | A few clusters account for most clips |
| Camera metadata | Camera ID, mount, resolution, frame rate, clock source | Pairs covered by more than one viewpoint | Missing frame rate or timestamps on a share of clips |
| Consent and visibility | Consent record ID for each visible person; faces or screens in view | Share of clips with a matching consent record | Any clip with a visible person and no record |
| Rights scope | Site, customer property or third-party items in frame | Clips whose content the business can license | Customer-owned goods or screens shown without permission |
Scoring the grid: coverage, concentration and effective variety
Three numbers summarize the grid, and none of them needs specialist software. Cell coverage is the number of object-action-condition cells with at least a minimum number of clips, divided by the cells your operation actually performs. Top-cell share is the fraction of all clips that sit in the single largest cell. The effective number of cells is the exponential of the Shannon entropy of the clip distribution across cells: it equals the cell count when clips are spread evenly and falls toward one as they pile into a single cell.
Within a cell, check for near-duplicates. Group clips by visual similarity using frame embeddings or a simpler perceptual hash, then report clusters per cell. The Vendi Score, an open-source diversity metric, takes a set of samples and a similarity function and returns a number that can be read as the effective number of unique elements, which is the same idea applied to the clips themselves.
As a rule of thumb, not a published threshold, a top-cell share above roughly a third, or an effective number of cells well under half of the cells covered, means the library is mostly one skill with a few side examples. Agree the minimum clips per cell with the buyer, because the right depth depends on how hard the task is to learn.
- Cell coverage: covered cells divided by cells the operation performs.
- Top-cell share: clips in the largest cell divided by all clips.
- Effective number of cells: exp of the entropy of the clip shares across cells.
- Within-cell duplication: clips per near-duplicate cluster, or an effective-number score on clip embeddings.
- Outcome balance: share of clips per pair that show a failure, recovery or intervention.
Worked example: what an illustrative packing station library reveals
The grid below is illustrative and does not describe any real company. A distribution site holds 10,000 clips from overhead cameras on six packing benches. Tagging shows four object classes and five actions, recorded across day and night shifts.
The readout: 14 of the 20 object-action pairs the site performs are covered with 50 or more clips, but the top cell, medium carton taping on the day shift, holds 58% of all clips. The effective number of cells is about 6, far below the 14 covered pairs, and near-duplicate clustering finds that most taping clips come from two benches with identical layouts. Only 3% of clips show a recovery, and interventions were trimmed by the original video editor.
| Object and action (illustrative) | Clips | Conditions seen | Outcomes tagged | Audit reading |
|---|---|---|---|---|
| Medium carton, tape | 5,800 | Day shift, benches 1 and 2 | Success only | Overrepresented; sample down before offering |
| Small carton, fold and insert | 1,400 | Day and night, four benches | Success and some recoveries | Strong cell; offer as is |
| Poly mailer, insert and seal | 1,100 | Day only | Success only | Add night shift and failures |
| Large carton, wrap fragile item | 350 | Two benches | Success and recoveries | Thin but varied; offer with a depth note |
| Irregular item, bag and label | 40 | One bench | Not tagged | Below minimum; record more or exclude |
Which variation and recovery evidence does the footage leave out?
Most libraries are missing the same things, because footage recorded for training or quality review favors clean runs by experienced staff. The audit should list what is absent, not only what is present, since a buyer plans further collection around the gaps.
At a glance
Which variation and recovery evidence does the footage leave out
Check each object-action pair against the list below and record a yes, a no or a count. A no is not a reason to withhold the library; it is a line in the offer that tells the buyer what the footage cannot teach.
- Operators: more than one person per pair, including newer staff whose motions differ from an expert's.
- Conditions: both lighting regimes, both shifts, a reconfigured station, a worn or replacement tool.
- Object variation: damaged, oversized, unusual or mislabeled items, not only the standard SKU.
- Failures: a dropped item, a torn tape run, a missed scan, kept in the clip with its outcome tagged.
- Recoveries: the motion that fixes the failure, inside the same continuous clip.
- Interventions: when a supervisor or second worker stepped in, with the moment marked.
- Outcome evidence: a downstream record such as a pass at the scale or a reject, linked by time.
Consent, worker visibility and camera metadata the audit has to confirm
A varied library is still unusable if the people in it did not agree to this use or the clips lack the metadata a robotics team needs. Treat both as audit fields, scored per clip like objects and actions.
For people, check that every visible worker has a consent or release record that covers training AI systems and licensing to third parties, and note where faces, badges or screens appear. Large research collections set the expectation: Ego4D, for example, describes collecting from consenting participants with de-identification where relevant. The specific notice and consent rules depend on where the footage was recorded, so counsel should review the records before an offer.
For cameras, record the camera ID, mount position, resolution, frame rate and the clock each camera used. If several cameras cover one bench, note whether their clocks were synchronized, because unsynchronized views cannot be fused into one episode later.
When the audit says narrow the offer or record more
The audit usually leads to one of four decisions, and the numbers make the choice for you rather than a general feeling about the footage.
At a glance
When the audit says narrow the offer or record more
There is a meaningful exception: a library that is narrow on purpose can still be valuable. If the operation performs one rare, difficult task, deep coverage of that single cell with many operators and many failure cases may be exactly what a team cannot get elsewhere. In that case, say so plainly and score depth and outcome balance inside the cell rather than coverage across cells.
The audit also has limits. It measures what was recorded, not whether a model will learn from it, and similarity clustering can merge clips that differ in ways that matter physically, such as grip force. Treat the scores as a description for the buyer, not a quality grade.
- Offer as is: coverage is broad, top-cell share is moderate, outcomes are tagged and consent records match.
- Narrow: offer only the well-covered pairs, sample the dominant cell down and list the rest as out of scope.
- Record more first: key pairs are below the minimum, failures were edited out or consent records are missing for a meaningful share of clips.
- Stop: footage shows customer property, screens or people the business cannot license, and redaction would remove the task itself.
Taking an audited clip library to SourceX
Once the audit supports an offer, a company can find out whether it qualifies with SourceX through a brief assessment, without uploading any footage. Keep the grid and the missing-variation list close by for that conversation.
Buyers usually want a steady history and real volume from one system, but a rare, hard task can qualify on less, and SourceX assesses each case on its own; a deep single-cell library is that kind of case. No video is delivered until the company has approved the scope and terms and signed, and footage measured in terabytes stays on the company's own storage or goes out on encrypted drives instead of being hosted by SourceX.
Frequently asked questions
How many demonstrations per object and action are enough?
There is no universal number; it depends on how hard the motion is and how much the objects vary. Research on manipulation suggests spreading demonstrations across more environments and objects helps generalization more than adding more of the same setting, so set a per-cell minimum with the buyer and spend extra recording on missing cells first.
Is the effective number of cells the same as a diversity score?
It is one diversity score, computed on the distribution of clips across your object, action and condition grid. It falls toward one when clips concentrate in one cell. Pair it with a within-cell near-duplicate measure, because evenly spread cells can still be filled with nearly identical clips.
Should failed attempts be cut from demonstration footage?
No. Keep failures and the recoveries that follow them inside the same clip and tag the outcome. Editing them out removes the variation a policy most needs and makes the library look cleaner than the work really is.
Can a library of one repeated task still be worth offering?
Yes, when the task is rare or difficult and the footage is deep: many operators, many conditions and real failure cases inside that one task. Describe it honestly as a single-skill library and score depth and outcome balance rather than coverage across tasks.
Do I need consent records before running the audit?
You can run the counting part first, but the audit is not complete until each clip with a visible person is matched to a consent or release record covering AI training and licensing. Counsel should review those records for the places the footage was recorded.
Sources
- Lin et al. collected over 40,000 demonstrations and found generalization scales roughly as a power law in the number of environments and objects, with diversity mattering more than demonstrations per setting. Source
- The released policies were trained on 32 environment-object pairs with 50 demonstrations per environment across four tasks. Source
- DROID reports 76k trajectories collected across 564 scenes and more than 80 tasks (86 in v1, 84 in the April 2025 revision) by 50 collectors over 12 months. Source
- BridgeData V2 contains 60,096 trajectories, including 50,365 demonstrations of 13 skills across 24 environments. Source
- Open X-Embodiment pooled data from 22 robots across 21 institutions covering 527 skills. Source
- LeRobot dataset v3 maps natural-language task descriptions to IDs and keeps per-episode records of lengths and tasks. Source
- The Vendi Score takes samples and a similarity function and returns a value read as the effective number of unique elements. Source
- Ego4D describes collection from consenting participants with de-identification where relevant. Source
See if your company qualifies
A short company assessment. No data uploads are needed.