Skip to content

Rights and contracts

Dataset acceptance dispute test: one shared test beats more examples

By SourceX Editorial · Reviewed by Noah Loul ·

8 sections

Short answer

A dataset acceptance dispute test is one measurement both parties agree before anyone argues: the records it covers, the sample and seed, item checks written as code or anchored rubrics, the reference answers, the pass threshold and who runs it. Trading examples cannot settle a dispute, because examples carry no denominator. Run the shared test, then route failures: data defects get fixed, while rights, terms and task mismatch need their own track.

Checklist

Dataset acceptance dispute

This page covers dataset acceptance dispute test: one shared test beats more examples.Illustrative

Key takeaways

  • Examples are chosen to persuade and carry no denominator, so trading more of them hardens a dispute instead of settling it.
  • A shared test is one both sides can rerun to the same answer: fixed records, sample, item checks, reference answers, threshold and runner.
  • Name the standard behind every disputed quality word, such as an ISO/IEC 25012 characteristic or an ISO/IEC 5259-2 measure, before anyone tests.
  • Route each confirmed failure: data defects to remediation, rights gaps to a rights review, price and volume to renegotiation, task mismatch to re-scoping.
  • Keep one rejection reason log both parties can audit, with the test version, denominator and route on every line.

Why don't more examples settle a dataset dispute?

Because each side picks examples to make its case, and a hand-picked example has no denominator. In an illustrative dispute, a buyer sends 40 incomplete records and the supplier answers with 50 complete ones. Both have proved only that each kind exists. Neither has said how common each kind is, in which records, measured how, or against what threshold.

More examples also shift the argument without anyone noticing. The buyer's examples may test whether a field is filled; the supplier's may test whether filled values are right. Both can be true at once, and each new batch answers a question the other side did not ask.

A dataset acceptance dispute test replaces that exchange with one measurement both parties agree before it runs. This page covers how to build it, which standards give disputed words a fixed meaning, how to route a failure the test confirms and the log both sides keep. The cure and escalation workflow after a confirmed miss is covered in this site's guide to acceptance disputes.

What makes an acceptance test shared instead of one side's opinion?

A test is shared when either party can run it on the same frozen records and get the same result, and when both signed its definition before seeing that result. The parties do not have to agree on the outcome. They have to agree on the measurement.

Commercial law already rewards fixing the method in advance. Under UCC section 2-513, a place or method of inspection the parties fix is presumed to be exclusive, and if the agreed method becomes impossible, inspection falls back to the default rules unless the parties clearly made that method an indispensable condition. Article 2 governs sales of goods and may not reach a data license, so counsel should judge whether it applies; the drafting lesson holds either way.

Reproducibility depends on how each check is written. Anthropic's guide to agent evaluations, published January 9, 2026, calls code-based graders fast, cheap, objective and reproducible, describes model-based graders as non-deterministic and human graders as expensive and slow, and sets a plain bar for any judged task: two domain experts should independently reach the same pass or fail verdict. Data teams already work this way; the Great Expectations project describes its Expectations as extensible unit tests for data that give teams a common language for quality checks and can document every validation run.

The shared test card: seven fields both parties sign

The card is the asset: seven fields that turn a disagreement into a measurement. Each field closes one recurring source of dispute, and the third column names an outside reference both parties can point to instead of their own definitions.

Sign the card before delivery when you can and before any re-test when you cannot. A card written after one side has seen results invites a second fight about whether it was written to fit them. Anthropic's guide makes the same point about tasks: if a check depends on a detail the task never stated, a correct answer can fail through no fault of its own.

The shared test card: seven fields both parties sign
FieldWhat to fix in writingOutside reference to citeDispute it prevents
1. QuestionThe task and intended use the data must serveA datasheet's motivation and recommended usesClean data judged against the wrong task
2. Records in scopeFiles, snapshot date, row counts and content hashesA machine-readable description such as a Croissant fileEach side testing a different version
3. SampleSize, method, strata and random seedThe sampling script stored with the cardEach side testing its own pick
4. Item checksCode assertions for structure, anchored rubrics for judgmentExpectation-style unit tests and the two-expert ruleChecks that depend on who runs them
5. Reference answersWho made them, from which source, and how one is challengedThe source record itself, cited by IDA wrong answer key
6. Decision ruleThreshold, the quality characteristic it measures and how sampling error is handledAn ISO/IEC 25012 characteristic or ISO/IEC 5259-2 measureArguing about what accurate means
7. Runner and replayWho runs it, in what environment, and how the other side reruns itPinned tool and script versionsResults nobody else can reproduce

Which quality words need a named standard before testing?

Every word in the acceptance criteria that a reasonable person could read two ways. Tie each one to a named characteristic and a measure, or the test will measure one reading while the other side argues for the other.

At a glance

Which quality words need a named standard before testing

Two ISO families supply the vocabulary. ISO/IEC 25012:2008 sorts data quality into fifteen characteristics seen from two points of view, inherent to the data and dependent on the system that holds it, and is meant for setting quality requirements, defining measures and planning evaluations. ISO/IEC 5259-2:2024 extends this to analytics and machine learning with a data quality model and measurable characteristics, builds on ISO/IEC 25012 and ISO 8000, and covers implementing measures for a specific analytics or ML task.

Citing a standard settles the meaning, not the threshold. The contract still has to say which measure applies, on which fields, at what level and with what tolerance for sampling error. Here is how six words that often go undefined translate into testable terms:

  • Complete: which fields are required, whether a field the source system never captured counts as missing, and the share of sampled records that must carry every required value.
  • Accurate: accurate against what, the source record or the real-world fact, and how the reference answer is chosen when the two differ.
  • Consistent: within one record, across records about the same entity or across systems, and which system wins a conflict.
  • Current: the snapshot date the records must reflect and how changes made after it are treated.
  • De-identified: the method, the fields covered and who checked it; this is a privacy and rights question as much as a quality one.
  • Representative: the population and strata the sample must match, stated before delivery.

The acceptance decision tree: test problem, data problem or something else?

Run the tree from the top and stop at the first question that ends in a route. Its purpose is to separate disputes about the measurement from disputes about the data, and both from disputes no data fix can solve.

Do not skip question 2. When two runs of the same card disagree, the causes worth checking first are mundane: a different snapshot of the records, a sampling script run without the stated seed, a library or locale change that alters how dates and text encodings are parsed, or a rubric applied by reviewers who were never calibrated against each other. Each is a defect in the test, not in the data, and fixing it costs far less than another round of examples.

Illustrative example: a buyer says a set of field-service work orders is incomplete. Instead of trading examples, the parties sign a card with six required fields, a 400-record random sample with a stated seed and a 95 percent threshold. Both runs return 93 percent, or 28 failing records. Ten trace to an extraction defect, which the supplier fixes. Eighteen lack a field the source system never captured, which is a scope question rather than a defect, so the parties re-scope that field and re-test on a fresh sample. All figures are illustrative.

The seven questions, in order:

  • 1. Is there a signed shared test card? If not, stop arguing about records and write one; if so, continue.
  • 2. Do both parties' runs of the card give the same result? If not, fix the test, its environment or its versions and rerun; if so, continue.
  • 3. Does the delivery miss the decision rule by more than sampling error? If not, accept, or take the second-stage sample the agreement describes; if so, continue.
  • 4. Is the failure a defect in records the supplier agreed to deliver? If so, remediate and re-test on a fresh sample; if not, continue.
  • 5. Does the failure come from rights, provenance or privacy? If so, send it to a rights review on its own track; if not, continue.
  • 6. Is the disagreement really about price, volume or timing? If so, renegotiate the terms; if not, continue.
  • 7. Is the data correct but unfit for the buyer's task? If so, re-scope the request or end the deal; if not, escalate to the independent expert the agreement names.
A decision tree: three questions about the signed test card, matching runs and a miss beyond sampling noise, each with an exit, then a fan-out to four routes for a confirmed failure: fix the data, rights review, renegotiate terms or re-scope the task.
The acceptance decision tree, condensed from the seven questions in this section.

When will a technical fix not end the dispute?

When the confirmed failure is about rights, commercial terms or task fit. Cleaning, relabeling and redelivering change records; they cannot create permission, change a price or make data suit a task it was never collected for.

Rights failures are common enough to deserve their own track. The MIT Media Lab's large-scale audit of more than 1,800 text datasets (Nature Machine Intelligence, 2024) found popular hosting sites dropping the license for more than 70 percent of datasets and misstating it for more than 50 percent, and the correct licenses were often more restrictive than the ones listed. If a buyer's objection is that provenance cannot be shown, no number of extra examples answers it; documents do.

Commercial and task disputes often hide inside quality language. A buyer that wants a lower price may describe adequate data as incomplete, and a buyer whose task changed may describe correct data as wrong. Datasheets for Datasets proposes that every dataset document its motivation, composition, collection process and recommended uses; field 1 of the card does the same job, so the parties can see whether the data failed the stated use or the stated use moved.

The rejection reason log both sides keep

Keep one log, shared and append-only, with a line for every group of failing items the test confirms. Its job is to make each rejection auditable by the other party and to show, across deliveries, which kind of problem keeps coming back.

Assign every line to one route before choosing a remedy. A log where most lines route to task fit says the request was wrong; one dominated by data defects says the supplier's process was. The example entries continue the illustrative work-order case.

Review the log by route at every delivery, not only when a dispute starts. If the same reason code returns after a remedy, the remedy did not reach the cause. If task-fit lines keep growing, the request needs rewriting before the next delivery, and another cure cycle will not help. A short log read regularly is worth more than a long one opened only after relations sour.

The rejection reason log both sides keep
FieldWhat to recordWhy both sides need itIllustrative entry
Reason codeOne code from an agreed listLets patterns show across deliveriesMISSING_REQUIRED_FIELD
Test card and versionThe card ID and version that produced the failureTies the line to a signed measurementCard 3, version 2
Items and denominatorFailing item IDs and the sample sizeTurns examples into a rate18 of 400 sampled records
Measured value and thresholdThe result, the threshold and the uncertaintyShows whether the miss is beyond noise93 percent against 95 percent
Reproduced byWhich party reran the card, and whenConfirms the failure is not one side's runBoth parties, same result
RouteTechnical, rights, commercial or task fitSends the line to the right ownerTask fit: field never captured
Remedy and re-testThe action taken and the fresh-sample resultCloses the line with evidenceField re-scoped; re-test passed

Putting the acceptance test into a SourceX data request

Write the card's first three fields into the request itself, so the test exists before any record is chosen. Each SourceX dataset is located on request from U.S. businesses, not drawn from an inventory, and the request-data form asks buyers for scope, volume, language, format, timing and licensing needs; the intended use and a draft acceptance test belong next to them.

Before any data is delivered, the record owner approves the scope and terms and puts its name to the agreement, so the scope the test checks is fixed in writing first. Nothing in that sequence assures that a delivery will pass a buyer's test or improve a model. The shared test is how buyer, supplier and record owner find out, on terms all three accepted in advance.

For a buyer, the practical step is to submit the request with the task, the intended use and the acceptance test you would run, so the record owner can judge early whether its records can meet it.

Frequently asked questions

What is a dataset acceptance test?

A defined measurement that decides whether a delivered dataset meets the agreed criteria: which records, which sample, which item checks, which reference answers, which threshold and who runs it. It is shared when both parties signed it before seeing results and either can rerun it to the same answer.

Should a buyer send failing examples during a dispute?

Yes, as leads rather than proof. Examples show that a problem exists and where to look, but they carry no denominator. Turn each one into a check on the shared test card and measure how often the problem occurs on the agreed sample.

Does the UCC decide how a licensed dataset must be inspected?

UCC Article 2 governs sales of goods, and whether it reaches a data license depends on the contract and the jurisdiction, so counsel should advise. Section 2-513 is still a useful model: a method of inspection the parties fix is presumed exclusive, which is the effect a signed test card aims for.

Which standards define data quality terms for a contract?

ISO/IEC 25012:2008 sets out fifteen data quality characteristics for structured data, and ISO/IEC 5259-2:2024 adds a data quality model and measures for analytics and machine learning. Citing them fixes what a word means; the contract still sets the measure, fields, threshold and tolerance.

What if the data passes the test but still fails the buyer's task?

Then the test checked the wrong thing or the task changed. Compare the problem with the intended use in field 1 of the card: if the use moved, re-scope the request; if the use was stated and the test missed it, add a check and re-test instead of treating the delivery as defective.

Sources

  • UCC section 2-513 lets a buyer inspect goods before payment or acceptance in any reasonable manner, presumes a place or method of inspection fixed by the parties to be exclusive, and falls back to the default rules if that method becomes impossible unless it was an indispensable condition. Source
  • The MIT Media Lab audit of more than 1,800 text datasets (Nature Machine Intelligence, 2024) found license omission above 70 percent and error rates above 50 percent on popular hosting sites, with correct licenses often more restrictive than those listed. Source
  • ISO/IEC 25012:2008 defines a data quality model of fifteen characteristics considered from inherent and system-dependent points of view, for setting requirements, defining measures and planning evaluations. Source
  • ISO/IEC 5259-2:2024 defines a data quality model and measurable data quality characteristics for analytics and machine learning, building on ISO/IEC 25012 and ISO 8000. Source
  • The Great Expectations project describes Expectations as expressive and extensible unit tests for data that give teams a common language for data quality tests and generate documentation for each set of validation results. Source
  • Anthropic's guide to agent evaluations (January 9, 2026) says a good task is one where two domain experts independently reach the same pass or fail verdict, that ambiguity becomes noise in metrics, and that code-based graders are objective and reproducible while model-based graders are non-deterministic. Source
  • Datasheets for Datasets proposes that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses. Source
  • MLCommons Croissant is a format that combines dataset metadata, resource file descriptions, data structure and default ML semantics in a single file. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies