Acceptance Testing for Dataset Deliveries

Why acceptance needs its own plan

A sample that looked good and a bulk delivery that works are different things. Deliveries fail in ordinary ways: files missing from the manifest, a second camera that dropped frames at one site, label guidelines that drifted halfway through, a consent register that covers only some operators, or a held-out site that ended up in the training lot. Acceptance is where those problems are found, and the tests should be written down before the license is signed.

This page covers the general process for any modality. The tests specific to workplace video are in acceptance criteria for demonstration video deliveries. Testing a trained policy on a robot is a different job, covered in real-world policy evaluation protocols. The tests themselves come from the last section of your requirements specification. Judging an offer before purchase belongs to the robotics dataset evaluation checklist and to requesting and reviewing dataset samples; this page starts when files arrive.

The acceptance sequence

1. Quarantine on receipt. Land every delivery in storage that is separate from training and holdout storage, so a failed lot never reaches a training job.
2. Automated checks on every file. Manifest completeness, checksums, readability, schema validation and duplicate detection.
3. Sampled review. People check content, labels and metadata on a sample stratified by site, operator, device and task, so one weak station cannot hide in an average.
4. Rights and provenance review. Every record resolves to a permission record and an item in the license schedule.
5. Decision. Accept, accept with conditions, or reject the lot, recorded against lot and file IDs.
6. Remedy loop. Re-delivery, re-annotation or exclusion of failed items, then re-test only what changed.

Deliver in lots, for example by site or collection week, so a defect holds up one lot rather than the whole purchase. Draw the review sample yourself; a sample the supplier chose tells you what the supplier's best work looks like. If you want a formal sampling scheme, attribute sampling plans of the kind in ISO 2859-1 index single, double and multiple plans by an acceptance quality limit and apply to records as well as products (ISO, 2026). Edition details for that standard were only partly confirmed when this page was written, so name the edition in the contract.

Illustrative test families

Documentation is part of the delivery

Make the datasheet a deliverable. Gebru and colleagues propose that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process, recommended uses and more (Gebru et al., 2018). In acceptance terms that gives you something to test against: the composition the datasheet states should match the counts you measure, and the limitations section should not be empty. Fields and variants are covered in datasheets and data cards for robotics datasets, and the glossary defines the datasheet for datasets.

For a shared vocabulary, ISO/IEC 5259-1:2024 gives an overview, terminology and a framework for assessing and improving data quality across the data life cycle. The 5259 series also has parts on quality measures, management requirements, a process framework and governance (ISO/IEC, 2024). Use it to organize a quality plan if your organization works that way. The thresholds for your task remain yours to set.

Severity classes and remedies

Agree in advance who pays for re-testing, how long the supplier has to remedy a failure, whether accepted items can be rejected later if a rights problem surfaces, and what happens to rejected copies. Deleting a rejected lot is a handling duty, so say how it is done. NIST SP 800-88 Rev. 2, the media sanitization guidelines published in September 2025, is one reference for that wording (NIST, 2025).

Illustrative acceptance schedule

# Illustrative only — synthetic identifiers; every threshold is an example the buyer sets
acceptance_schedule: AS_2026_007
lots: [lot_01_site_A, lot_02_site_B]
quarantine: incoming_quarantine      # separate from training and holdout storage
window: fixed in the contract; silence is not acceptance
automated_checks_on_all_files:
  - manifest_complete_and_checksums_match
  - every_file_opens_with_declared_reader
  - schema_validates: schema_v1.2.json
  - no_duplicate_ids
sampled_review:
  drawn_by: buyer
  stratify_by: [site, operator, device, task]
  sample_size: set from the plan (for example 200 units per lot)
  limits: {critical: 0, major: "2% of sampled units", minor: "5% of sampled units"}
rights_review:
  every_record_links_to: [permission_record_id, license_schedule_item]
holdout_check:
  rule: no holdout site_id or operator_id appears in any training lot
documentation_required: [datasheet, coverage_table, known_issues_list, tool_versions]
outcomes: [accept, accept_with_conditions, reject_lot]
remedies: [redelivery_deadline, re_annotation, partial_acceptance, price_adjustment, termination_right]

Put the same tests in the requirement you send through the buyers page, so suppliers can say before any license is drafted which tests they can meet.

Extra tests for robot-log deliveries

Data from robots, teleoperation rigs, handheld capture devices or instrumented setups needs tests that business archives never raise. Timestamps should increase monotonically, the measured control rate should match the declared one, action values should sit inside the robot's joint limits, and calibration should be present for every session. DROID is a useful reference for the metadata expectation: it recorded camera calibration, depth and a language instruction with each episode (Khazatsky et al., 2024). The field-level version is in the robot trajectory dataset specification, and timing is covered in sensor synchronization requirements.

What acceptance testing cannot tell you

• Whether the data will improve a model. Acceptance shows that a delivery matches the specification, not that the specification was right. A small pilot (scoping a pilot collection) and a separate held-out evaluation set (training vs evaluation datasets) answer that question.
• Whether consent was meaningful. A register that exists is not evidence that notice was adequate for your uses. That is a diligence question for counsel. This is not legal advice.
• Whether the bulk matches the sample. Only an independently drawn sample, taken from the full delivery, tests that.
• Whether redaction preserved training value. Spot checks can pass while hands or parts are obscured; check the content criteria for your modality.

Red flags

• Acceptance defined as "no objection within a period", so silence counts as approval.
• Review samples chosen by the supplier.
• Quality reported as one average across all sites and devices.
• Validation scripts that only the supplier can run.
• A datasheet promised "after delivery".
• Rights evidence stated in aggregate ("all staff signed") rather than per record.

Questions to ask suppliers

1. Will you run the automated checks before delivery and share the reports and tool versions?
2. Can the delivery be split into lots so that acceptance happens per lot?
3. Which identifiers (site, operator, device, task) are in the metadata, so sampling and holdout checks can be stratified?
4. How was label quality measured, on what sample size, and against which guideline version?
5. Which datasheet fields can you complete now, and where are known limitations listed?
6. How does each record link to its permission record, and what is excluded?
7. What remedies and deadlines apply if a lot fails, and how are rejected copies deleted?
8. Can we run our own checks on a sample we draw before the formal acceptance window opens?

Discuss your requirements

SourceX looks for US businesses that hold the data you describe, checks the data and the supplier's licensing permissions, agrees pricing and allowed uses in a license, and coordinates delivery and payment. The acceptance tests and remedies you want belong in those terms, so describe them in your requirement. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match, and nothing is contracted until a supplier agrees. For the rest of the sequence, see the procurement hub and the humanoid training data guide.

Explore sourcing and licensing options with SourceX

Sources

• Datasheets for Datasets — datasheets as documentation that accompanies every dataset.
• ISO/IEC 5259-1:2024 Artificial intelligence - Data quality for analytics and machine learning (ML) - Part 1: Overview, terminology, and examples — data quality framework and the 5259 series.
• ISO 2859-1 Sampling procedures for inspection by attributes - Part 1: Sampling schemes indexed by acceptance quality limit (AQL) for lot-by-lot inspection — attribute sampling plans for lot acceptance (edition details only partially verified).
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — per-episode calibration, depth and language.
• NIST SP 800-88 Rev. 2 Guidelines for Media Sanitization — reference for specifying how rejected copies are deleted.