Acceptance criteria for training data delivery: thresholds to write into the order

The process around them (inspection windows, rejection notices, cure periods) is covered in dataset acceptance testing; the AI training data procurement hub shows where acceptance sits in the buying cycle.

What turns a quality promise into an acceptance criterion

A quality promise becomes an acceptance criterion when two people running the stated test on the same delivery would reach the same verdict. "Complete, high-quality ticket data" fails that test; "resolution_code non-missing in at least [y]% of closed tickets, checked on all records with the attached validator" passes. ISO/IEC 5259-2 sets out data quality measures [1], and one annotation vendor's guidance on dataset service terms likewise says to state how each term is measured and what remedy applies [2].

Write every criterion with seven parts:

1. Unit: file, record, field value, label or document; per-label and per-record accuracy differ.
2. Denominator: the exact population ("closed tickets created 2022-01-01 to 2024-12-31"), never "the dataset".
3. Coverage: every record, or a random sample of stated size and selection method.
4. Threshold: at least, at most or exactly, per stratum or overall.
5. Method: validator, schema version, normalization rules or annotation guideline version.
6. Who and when: buyer, supplier with a buyer witness, or an independent auditor, within a fixed window.
7. Consequence: rejection, credit or replacement records; remedies for a defective data delivery compares them.

The guide to data supplier SLAs lists the surrounding service terms.

Gate checks that run on every file and record

Mechanical properties should be checked on 100% of the delivery, because software can test every record cheaply and sampling would only estimate what you can measure exactly. Treat them as gates: a failure stops further testing until re-delivery.

Duplicate rates depend on the method. Lee et al. found one sentence repeated more than 60,000 times in C4, used both exact-substring and MinHash-based near-duplicate matching, and showed that training on deduplicated data cut verbatim emission of memorized text about tenfold [6]. Write the normalization, similarity measure and threshold into the criterion.

Attach the schema itself to the order.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "support_ticket_v3",
  "type": "object",
  "required": ["ticket_id", "created_at", "channel", "status", "messages"],
  "properties": {
    "ticket_id": {"type": "string", "pattern": "^T-[0-9]{8}$"},
    "created_at": {"type": "string", "format": "date-time"},
    "channel": {"enum": ["email", "chat", "phone_transcript", "web_form"]},
    "status": {"enum": ["open", "pending", "solved", "closed"]},
    "resolution_code": {"type": ["string", "null"]},
    "messages": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "required": ["author_role", "sent_at", "body"],
        "properties": {
          "author_role": {"enum": ["customer", "agent", "system"]},
          "sent_at": {"type": "string", "format": "date-time"},
          "body": {"type": "string", "minLength": 1}
        }
      }
    }
  }
}

Name the validator and its version, and state whether format keywords such as date-time fail a record or are only reported.

Sampled checks: label accuracy, field accuracy and residual personal data

Anything that needs human judgment, such as whether a label, transcription or redaction is correct, is measured on a random sample, so the criterion must state how many items are reviewed, how they are drawn, who adjudicates and how many errors the sample may contain, not only a target rate.

In manufacturing acceptance sampling, a single sampling plan is a pair (n, c): review n items and reject the lot if more than c are defective [7]. The NIST/SEMATECH handbook frames plans around the acceptable quality level (AQL), the producer's baseline that should pass with high probability, and the lot tolerance percent defective (LTPD), poor quality the consumer wants accepted only with very low probability [7][8]. ANSI/ASQ Z1.4 indexes plans by lot size, inspection level and AQL, with switching rules for a continuing stream of lots [9], which suits refresh deliveries. For a one-off purchase the buyer's main concern is the LTPD; acceptance sampling for dataset deliveries covers adapting these plans.

Define the defect before the number: a wrong label, an item with any wrong label, or a record with any wrong field, judged against a named guideline version by a named adjudicator.

Worked example: sizing a label-accuracy sample

Suppose the ML lead wants a delivery with 2% or more defective items accepted at most 5% of the time (LTPD 2%, consumer's risk 5%). These are example inputs, not recommendations. Binomial calculations for a delivery much larger than the sample give:

All three plans protect the buyer equally at 2%. The zero-error plan, however, rejects a 0.5%-defect delivery more often than it accepts it, which invites disputes. Reviewing about three times as many items buys a plan a competent supplier can expect to pass. Record n, c and the LTPD in the schedule so either side can recompute them.

Three practices keep sampled criteria fair:

• Draw the sample yourself or through an independent party, from the full delivery, with a recorded random seed and strata where slices matter. Pre-payment acceptance audits are sold as an independent service [10].
• Count only human-confirmed defects. Northcutt et al. estimated an average label error rate of at least 3.3% across the test sets of 10 widely used datasets, yet human reviewers confirmed only about 51% of the candidates their algorithm flagged [11].
• Do not read agreement as accuracy. Krippendorff's alpha on a double-annotated subset (1 is perfect reliability, 0 is chance-level agreement) [12] shows annotators were consistent; only an adjudicated sample shows they were correct.

Residual personal data works the same way. The Presidio project states that, because it relies on trained models, there is no guarantee it will find all sensitive information [13]. For HIPAA-covered data, OCR guidance recognizes Safe Harbor (removing 18 listed identifiers, with no actual knowledge that the rest could identify someone) and Expert Determination, and sets no numerical threshold for "very small" risk [14]. Require the method record or expert report for this exact delivery plus a sampled review; residual PII audit sampling sets out plans.

Provenance, consent and license evidence as acceptance conditions

A delivery whose records pass every data test should still fail if its records cannot be traced to a source the license covers. These criteria are checked against documents, so name the documents and how they join to the data.

• Source register coverage: every record's source_id resolves to a register entry giving system of origin, collection period and permission basis. Threshold: all records.
• Consent and exclusions: where the license relies on consent or a contractual permission, every record maps to that basis, and a full join against the opt-out and deletion list returns zero matches.
• License evidence, not labels: the Data Provenance Initiative's audit of more than 1,800 text datasets found license omission above 70% and error rates above 50% on popular hosting sites [15]. Require each source's permission documents.
• Documentation: a datasheet covering motivation, composition, collection process and recommended uses [16], or Croissant metadata, a schema.org-based JSON-LD vocabulary describing files and record-level structure [17] that a script can check against the delivery.

One dataset licensor's published process, for example, includes a final QA step covering consent records and metadata accuracy [18].

With SourceX, each dataset goes through rights review and is delivered under a license defining the included records, permitted uses, term and delivery method, and diligence materials on source, rights, preparation and allowed use are prepared per dataset. Personal details are removed or replaced before delivery and a sample is checked after processing, but no de-identification method is perfect, so keep a residual-PII criterion. See how SourceX works with data buyers.

Criteria that tighten for pre-training, fine-tuning, evaluation and RAG

The same records can pass for one use and fail for another, so tie each threshold to the use the license grants. For high-risk systems, EU AI Act Article 10(3) requires training, validation and testing data sets to be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose [19]. As of October 2026, Regulation (EU) 2026/1744 has amended the AI Act; secondary sources report that Annex III high-risk obligations now apply from 2 December 2027 while the regulation also amends Article 10 [20].

Illustrative acceptance schedule for a licensed support-ticket dataset

An acceptance schedule is the contract annex that lists each criterion with its population, method, threshold and consequence. This one fits a license of support-ticket histories for fine-tuning and evaluation; bracketed values are placeholders.

Illustrative example: invented to show structure; it does not describe an available dataset. Not legal advice; adapt with counsel.

Format, transfer channel and security terms belong in the technical delivery specification, which the schedule should reference rather than repeat. If payment is staged, map gate checks and sampled checks to separate milestone payments tied to acceptance.

Drafting mistakes that make acceptance criteria unenforceable

Acceptance disputes often trace back to a target without a denominator, a method or a sample design. Watch for:

• Averages hiding a failed slice. Set per-stratum thresholds wherever a slice matters to the model.
• Criteria proven only on the pre-contract sample. Results carry over only if the full set is drawn and prepared the same way; request the sample on that basis.
• Deemed acceptance by silence. Defects often surface later, during training.
• Unversioned criteria. Schema or guideline changes between refreshes need a matching schedule version; recurring service levels belong in SLAs for recurring data deliveries.

Writing acceptance terms for a data purchase?

If you want acceptance tests settled before anything is delivered, describe the records you need, the uses you want licensed and the thresholds that matter to your model. SourceX looks for US companies that hold that data, checks the data and the supplier's licensing permissions, and manages the license and delivery; nothing is delivered until an agreement is executed and the supplier approves the terms. Describe your data and acceptance requirements.