Short answer
Accept a demonstration video delivery only after it passes file checks (every clip decodes, constant frame rate, no dropped or duplicated frames, expected resolution), content checks (task and hands visible, usable exposure and blur), sync and calibration checks, annotation and metadata checks against the plan, and a rights check on every clip. Agree each test, threshold, sample size and remedy in the contract before delivery.
Where this checklist fits
This page adds video-specific tests to the general process in acceptance testing for dataset deliveries, which covers manifests, lots and sign-off. Catching bad recordings while collection is still running is a different job, covered in quality control during collection; detailed label metrics are in measuring annotation quality in task video.
Every pass criterion below is an example. You set the thresholds from your model's needs and write them into the contract before the first file arrives.
How to run acceptance
Run automated checks on every file. Review content manually on a sample stratified by site, demonstrator, device and task, so a single bad station cannot hide in an average. For the manual part, attribute sampling plans of the kind defined in ISO 2859-1, which indexes lot-by-lot sampling schemes by an acceptance quality limit and applies to records as well as products, give a ready framework for sample sizes and accept/reject numbers (ISO, 2026); confirm which edition your contract references. Record every result against the clip ID so failures can be traced and re-delivered.
Checklist
1. File integrity
| Check | Example pass criterion (buyer sets) | Evidence to request |
|---|---|---|
| Completeness | Every file in the manifest is present and its hash matches | Manifest with hashes |
| Full decode | Every clip decodes end to end without errors | Decode log |
| Container and codec | Match the specification | Stream probe report |
| Duration | Matches metadata within tolerance | Probe report vs metadata |
| Duplicates | No identical or near-identical clips | Hash and similarity report |
2. Video technical quality
| Check | Example pass criterion (buyer sets) | Evidence to request |
|---|---|---|
| Frame rate | Constant; no variable-frame-rate segments | Per-frame timestamp report |
| Dropped or duplicated frames | Below a set fraction per clip | Per-clip counts |
| Resolution | As specified; no upscaled footage | Probe report; sample frames |
| Stabilization and cropping | Absent where the spec forbids them | Device settings log |
| Exposure | Clipped-pixel share below threshold on sampled frames | Exposure statistics |
| Blur | Share of blurred frames in task segments below threshold | Sharpness score distribution |
3. Content usability
| Check | Example pass criterion (buyer sets) | Evidence to request |
|---|---|---|
| Correct task | Sampled clips match their task code | Reviewer log |
| Hand visibility | Per-task share of time with both hands visible meets target | Per-task report, not an overall average |
| Task completeness | Clip includes the start and end of the task instance | Reviewer log |
| Off-task content | Below a set share of clip time | Reviewer log |
| Viewpoint | Mount and angle as specified | Device and mount metadata |
4. Synchronization and calibration
| Check | Example pass criterion (buyer sets) | Evidence to request |
|---|---|---|
| Calibration present | Intrinsics for every device unit, extrinsics for every multi-camera session | Calibration files |
| Calibration quality | Reprojection error below threshold | Calibration reports |
| Stream alignment | Slate or tap offset within tolerance in every session | Sync test results |
| Timestamps | Monotonic, with the clock source documented | Timestamp report |
Planar-pattern calibration is standard practice (Zhang, 2000), and toolboxes such as Kalibr write a PDF report, a text summary and YAML that can be delivered as evidence (ETH Zurich ASL). Robot datasets such as DROID ship calibration with every episode (Khazatsky et al., 2024); per-session calibration is a reasonable expectation for commissioned human capture too.
5. Annotations
| Check | Example pass criterion (buyer sets) | Evidence to request |
|---|---|---|
| Coverage | Every required label layer present on every clip | Label inventory |
| Agreement | Double-annotated subset meets boundary and label thresholds | Agreement report with sample size |
| Gold-set audit | Your own audit of a sample meets threshold | Your audit results |
| Versioning | Guideline version recorded on every label | Label metadata |
6. Metadata and coverage
| Check | Example pass criterion (buyer sets) | Evidence to request |
|---|---|---|
| Schema | All records validate against the agreed schema | Validation log |
| Coverage | Counts per factor meet the plan, gaps explained | Coverage report |
| Held-out factors | Reserved sites or operators absent from training lots | Split report |
Field definitions are in the metadata schema for workplace demonstration clips, and quotas in coverage planning for demonstrations.
7. Rights, consent and redaction
| Check | Example pass criterion (buyer sets) | Evidence to request |
|---|---|---|
| Permission reference | Every clip points to a consent or permission record | Register extract |
| Redaction | Spec applied; sampled frames show no missed faces or screens | Redaction report and version |
| Withdrawals | Clips of withdrawn individuals removed | Withdrawal log |
| Permitted uses | Metadata matches the license | License schedule |
Large datasets document consent and de-identification as part of the release; Ego4D describes consenting participants and de-identification procedures (Grauman et al., 2021). Ask for equivalent documentation. Redaction trade-offs are covered in face and bystander redaction.
Red flags
- Variable frame rate across the archive and no per-frame timestamps.
- Clips re-encoded several times, or upscaled to meet a resolution requirement.
- Calibration "available on request" that never arrives.
- Hand visibility reported only as one average across all tasks.
- Agreement figures without a gold set or a sample size.
- Consent described in aggregate ("all staff signed") with no per-clip reference.
- Metadata filled with defaults: identical timestamps, one operator ID everywhere.
- Sample clips visibly better than the bulk delivery.
Remedies to agree in advance
Write down what happens when a lot fails: re-delivery of failed clips, re-annotation, a credit, or rejection of the lot, each with a deadline in the contract. Agree who pays for re-testing, and whether accepted clips can later be rejected if a rights problem surfaces.
Questions to ask suppliers
- Will you run the automated checks before delivery and share the reports?
- How do you detect and report dropped frames and variable frame rates?
- Which calibration files and reports accompany each device unit and session?
- What hand-visibility rate does each task meet, and how was it measured?
- How was annotation agreement measured, and on what sample?
- How does each clip reference its consent or permission record?
- What remedies apply if a lot fails acceptance?
- Were the sample clips produced by the same pipeline as the bulk delivery?
Discuss your requirements
If you are drafting acceptance terms for a demonstration video purchase, share your tests and thresholds with SourceX along with the data you need. SourceX checks the data and the supplier's licensing permissions, agrees allowed uses in a license, and coordinates delivery; it reviews each requirement and follows up about potential sources, though a request does not guarantee a match. For broader diligence, see the training data due diligence checklist, the workplace task demonstration hub and the humanoid training data guide.
Explore sourcing and licensing options with SourceX
Sources
- ISO 2859-1 Sampling procedures for inspection by attributes - Part 1: Sampling schemes indexed by acceptance quality limit (AQL) for lot-by-lot inspection — attribute sampling plans for lot acceptance (edition details only partially verified).
- A Flexible New Technique for Camera Calibration — planar-pattern calibration.
- Kalibr — calibration reports that can serve as delivery evidence.
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — calibration delivered per episode.
- Ego4D: Around the World in 3,000 Hours of Egocentric Video — documented consent and de-identification.