Skip to content

Rights and contracts

AI dataset acceptance dispute: what happens when criteria are missed

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

An AI dataset acceptance dispute begins when buyer and supplier disagree on whether a delivery met its criteria. Settle it with evidence: freeze the delivery and the test, re-run the agreed measurement together, attribute each failure to a supplier defect, a test error, an ambiguous specification, a source-system limit or sampling noise, then cure, re-scope or escalate to an agreed expert. The pilot agreement should fix each step in advance.

Checklist

AI dataset acceptance

This page covers AI dataset acceptance dispute: what happens when criteria are missed.Illustrative

Key takeaways

  • Most acceptance disputes are about the measurement, so freeze the files, script and gold set before arguing about the data.
  • Attribute every failing item to one cause before choosing a remedy; buyer test errors need no supplier cure.
  • When the uncertainty interval includes the threshold, take the second-stage sample the agreement describes instead of negotiating.
  • Give suppliers a written way to challenge individual gold labels, and re-score every delivery a changed label affects.
  • Name an independent expert and the questions they may decide before the pilot starts.

Why do acceptance results get disputed even when the criteria are written down?

Because a criterion is only as settled as the measurement behind it. Two parties can agree that label accuracy must reach a stated level and still disagree about which sample was drawn, which version of the test script ran, whether the gold labels were right and whether a field was ever present in the source system.

The gold labels are a frequent and underrated source of conflict. The cleanlab label-errors project documents label errors in the test sets of ten widely used benchmarks, including ImageNet, MNIST and IMDB. If curated public test sets contain wrong labels, a buyer's internal gold set built in a few weeks can too, and a supplier who is scored against it has a legitimate grievance.

Agreement statistics add another layer. Artstein and Poesio's survey of inter-coder agreement treats annotated data as reliable when coders agree to an extent set by the purpose of the study, and compares coefficients that handle chance agreement differently. A threshold written without naming the coefficient and the sample invites a dispute about arithmetic rather than about the data.

What should a dataset pilot agreement say about failed acceptance criteria?

It should fix, before any data moves, how a failure is measured, who can challenge the measurement, what evidence a rejection notice must carry, how the cause is attributed, what cure looks like and who decides if the parties still disagree. Most acceptance disputes are arguments about one of those points that nobody wrote down.

At a glance

What should a dataset pilot agreement say about failed acceptance criteria

Commercial law offers useful models, with a caveat. Article 2 of the Uniform Commercial Code governs sales of goods, and whether it applies to a license of data is a question for counsel in the governing jurisdiction. Still, two of its provisions are worth copying by contract: section 2-515 lets the parties agree to a third-party inspection of disputed goods and agree that its findings bind them, and section 2-508 gives a seller who is rejected before the time for performance runs out a chance to cure with a conforming delivery.

  • Measurement protocol: the test script and its version, the sample frame, the sampling method and seed, and the agreement coefficient if labels are scored
  • Measuring party and observer rights: who runs acceptance, and whether the other side may watch or re-run it
  • Gold set terms: who built it, how a supplier may challenge an individual gold label, and how a corrected label is recorded
  • Rejection notice content: the criterion missed, the measured value with its uncertainty, the record IDs sampled and the failing items
  • Attribution categories the parties will use to assign the cause of a miss
  • Cure scope and window, and whether the cure is re-delivery, correction in place or re-scoping
  • Re-test rules: a fresh sample, the same protocol, and how many cure rounds are allowed
  • Partial acceptance of batches or subsets that pass
  • An escalation step to an agreed independent expert, and whether the expert's finding binds
  • Evidence retention: both parties keep the frozen delivery, scripts and outputs until the dispute closes

The evidence-based dispute and cure workflow

The workflow below is the asset: eight steps from rejection to close-out, each producing evidence the next step depends on. Time boxes are left to the agreement; the illustrative values in the last column show the shape a pilot might use, not a recommendation.

The evidence-based dispute and cure workflow
StepWho actsEvidence producedExit conditionIllustrative time box
1. Rejection noticeBuyerCriterion missed, measured value and interval, sampled record IDs, failing itemsSupplier acknowledges receiptWithin the inspection window
2. FreezeBothHashes of the delivered files, test script version, gold set versionBoth hold identical frozen copies2 business days
3. Joint reproductionBoth, or buyer with supplier observingRe-run output on the frozen sampleSame result reproduced, or the discrepancy explained5 business days
4. AttributionBothEach failing item tagged to one cause categoryAgreed tags, or a list of contested items5 business days
5. Remedy choiceSupplier proposes, buyer acceptsWritten cure plan per causePlan accepted3 business days
6. Cure and re-testSupplier cures; buyer re-tests on a fresh sampleNew measurement under the same protocolPass, partial pass or second failurePer cure window
7. Expert determinationAgreed independent expertWritten finding on the contested items or measurementFinding issuedPer the expert's procedure
8. Close-outBothAcceptance certificate or termination record, with the evidence indexSigned record5 business days

Whose error is it? Attributing the cause of a miss

Assign every failing item to one cause before discussing remedies, because each cause points to a different party and a different fix. A supplier asked to re-deliver records that were scored against wrong gold labels will deliver the same records again, and the dispute will repeat.

When a single item seems to fit two causes, record both and let the stronger evidence decide. A label that disagrees with the gold set and also with the source record is a supplier defect; one that disagrees with the gold set but matches the source record points at the test. Keeping the tags at item level also shows patterns, such as one source system or one batch producing most of the defects.

Whose error is it? Attributing the cause of a miss
CauseTelltale evidenceWho carries the fixUsual remedy
Supplier defectDelivered value differs from the source record or the agreed transformationSupplierCorrect or replace the affected records
Buyer test defectGold label wrong, script bug, or wrong sample frameBuyerFix the test and re-measure; no supplier cure
Ambiguous specificationBoth readings of the criterion are defensibleSharedAmend the specification in writing, then re-test
Source-system limitThe field or event never existed in the source systemShared, decided by the agreementRe-scope, or accept with a documented gap
Sampling noiseInterval around the measured value straddles the thresholdNeitherSecond-stage sample under the agreed plan
Rights or privacy defectMissing permission evidence, or personal data left inSupplierSeparate fast track; not a quality cure

How do you tell a real miss from sampling noise?

Compare the threshold with the uncertainty of the measurement, not just the point estimate. If the interval around the measured value includes the threshold, the sample cannot tell the parties whether the delivery passed, and the right response is the second-stage sample the agreement should already describe.

The NIST/SEMATECH e-Handbook of Statistical Methods frames acceptance sampling as a decision to accept or reject a lot, and stresses that its purpose is to decide whether the lot is likely to be acceptable, not to estimate its quality. Its catalog of plans includes double, multiple and sequential plans that call for a further sample before deciding. Writing one of those into the pilot turns a borderline result into a procedure instead of a negotiation.

Small samples are the usual culprit. Card and colleagues showed that many NLP experiments are underpowered, so a result that clears or misses a line by a small margin on a few hundred items deserves suspicion either way. Size the acceptance sample for the difference that would change the decision.

What happens when the dispute is about the gold labels themselves?

Treat challenged gold labels as a separate adjudication with its own record, and re-score the delivery only after it ends. A supplier should be able to flag specific gold items with evidence from the source record; the flagged items then go to a blind re-annotation by someone who did not create the gold set.

Decide the decision rule in advance: how many independent annotators, which agreement measure, and what happens when they split. Artstein and Poesio's survey compares coefficients suited to different annotation tasks and notes that the values of some are hard to interpret, so name the coefficient and the threshold in the agreement rather than relying on a rule of thumb.

  • Log every gold label change with the old value, new value, reason and adjudicator
  • Re-score all deliveries measured against the changed items, not only the disputed one
  • Exclude items the adjudicators cannot resolve from both the numerator and denominator
  • Report how many gold labels changed; a high rate signals a test problem, not a supplier problem

When should an independent expert decide?

Bring in an expert when the parties agree on the facts of the measurement but not on how to read them, or when a few contested items decide whether the threshold was met. Expert determination is built for narrow technical questions of exactly this kind.

WIPO's Arbitration and Mediation Center describes expert determination as consensual, intended for discrete technical, scientific or related business issues, and binding unless the parties agree otherwise; it can be preceded by mediation or followed by arbitration. UCC section 2-515 shows the same idea for goods, allowing parties to agree that a third party's inspection findings bind them. Name the expert's qualifications, the questions they may decide and whether their finding is final.

Worked example (illustrative): a support-ticket pilot that missed label accuracy

All figures here are illustrative. A buyer pilots a supplier's labeled support tickets with a criterion that intent labels must reach 95 percent accuracy on a 400-item random sample scored against the buyer's gold set. The buyer measures 92 percent and sends a rejection notice listing the 32 failing items.

Joint reproduction on the frozen sample gives the same 32 failures. Attribution splits them: 14 are supplier defects where the intent label contradicts the ticket thread, 11 are gold labels the supplier challenges with quotes from the ticket, and 7 fall on an intent category whose definition both readings fit.

Blind re-annotation upholds the supplier on 9 of the 11 challenged gold labels. The parties amend the definition of the ambiguous category, the supplier corrects its 14 defects and the pattern behind them across the full delivery, and a fresh 400-item sample scores 96 percent. The exception: 2 contested items stay unresolved, and the agreement's rule excludes them from scoring rather than handing them to an expert for so little at stake.

What must the pilot prove before the larger purchase?

A pilot earns the larger order when both sides can reproduce the measurement, the attribution categories have been used on real failures, and a cure cycle has actually run within its window. A pilot that passed on the first try proves less about the relationship than one that failed, was attributed and was cured cleanly.

  • The test script, sample frame and gold set produce the same result for both parties
  • Gold label challenges were possible and were handled under the written rule
  • At least the common cause categories were applied to real failing items
  • Cure turnaround and re-test fit the windows the full contract will use
  • Rights and privacy evidence were reviewed on their own track, not folded into quality
  • The supplier can keep producing records of the same kind at the volume the full order needs

Requesting records with acceptance terms agreed up front

Acceptance disputes are cheapest when the records come with a documented source and the measurement is agreed before delivery. SourceX works from buyer requests rather than an inventory, locating a U.S. business with suitable records each time; the scope, volume, language, format, timing and licensing fields of that request are the natural place to write the acceptance protocol down. The records' owner then approves scope and terms and signs, and only after that does a delivery reach the buyer for testing.

Frequently asked questions

What is an AI dataset acceptance dispute?

It is a disagreement between a data buyer and supplier about whether a delivery met its agreed acceptance criteria. It usually turns on the measurement: which sample, which test version, whether the gold labels were right, or whether a field ever existed in the source system.

Can a supplier challenge the buyer's gold labels?

Only if the agreement allows it, which is why a pilot should. A good clause lets the supplier flag specific gold items with evidence, sends them to blind re-annotation, and re-scores every delivery affected by a changed label.

Does the Uniform Commercial Code right to cure apply to datasets?

UCC Article 2 governs sales of goods, and whether it reaches a data license depends on the contract and jurisdiction, so counsel should advise. Parties can write a cure right and a third-party inspection step into the agreement either way.

What if the measured result is just below the threshold?

Check whether the uncertainty interval around the result includes the threshold. If it does, the sample cannot decide the question, and the agreement should call for a second-stage sample under the same protocol.

Should rights or privacy problems go through the same cure process?

No. A missing permission or residual personal data is a different kind of defect with different urgency, and it should have its own fast track, separate from label or field accuracy disputes.

Sources

  • UCC section 2-515 gives parties a right to inspect, test and sample disputed goods and lets them agree to a third-party inspection whose findings bind them. Source
  • UCC section 2-508 lets a seller whose nonconforming tender is rejected cure with a conforming delivery within the contract time, and in some cases a further reasonable time. Source
  • WIPO describes expert determination as a consensual procedure for technical, scientific or related business issues, binding unless the parties agree otherwise. Source
  • The NIST/SEMATECH e-Handbook says acceptance sampling decides whether a lot is likely acceptable rather than estimating its quality. Source
  • The NIST/SEMATECH e-Handbook describes single, double, multiple and sequential lot acceptance sampling plans. Source
  • Artstein and Poesio survey inter-coder agreement coefficients, note that reliability is judged against the purpose of the study, and discuss the difficulty of interpreting coefficient values. Source
  • The label-errors project documents and corrects label errors in the test sets of ten common machine learning benchmarks. Source
  • Card et al. (EMNLP 2020) show many NLP experiments are underpowered and significant differences tend to be inflated. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies