Skip to content

AI uses for records

Data pilot acceptance threshold: three numbers to sign before results

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

A data pilot acceptance threshold is a decision rule signed before anyone sees a result. Fix three numbers: the smallest gain worth paying for, derived from what the data costs; the largest loss you will tolerate on tasks that already work; and the number of tasks needed to tell either from noise. Then assign every outcome, including inconclusive, to a buy, extend, walk-away or stop action in advance.

Checklist

Data pilot acceptance

This page covers data pilot acceptance threshold: three numbers to sign before results.Illustrative

Key takeaways

  • The threshold is not the score you hope for; it is the smallest improvement that pays for the data, written down before the pilot runs.
  • Judge the confidence interval, not the point estimate: accept only when its lower bound clears the smallest worthwhile gain.
  • Add a harm margin on tasks that already succeed, borrowed from non-inferiority trials, where FDA guidance expects the margin to be set in advance.
  • A pilot too small to resolve its own threshold is designed to come back inconclusive; size the task count first.
  • Bind each outcome to a commercial action on the same card, because formal acceptance is hard to reverse once given.

What is a data pilot acceptance threshold, and why set it first?

A data pilot acceptance threshold is the written rule that turns a pilot's result into a decision: the gain a candidate dataset must show, the loss it may not cause, and the evidence needed to tell either from noise. It has to be signed before anyone sees a result, because afterwards every number becomes negotiable.

The failure it prevents is ordinary. A team runs a pilot, sees a 2-point gain and then argues about whether 2 points is good. Without a prior rule, the answer drifts toward whatever the most invested person wanted. The Center for Open Science makes the same point about research: a plan written and registered before the study separates planned tests from unplanned ones, because one set of data cannot both suggest a hypothesis and confirm it.

A pilot threshold is also different from a record-level quality gate. Record gates, such as a defect limit in a random audit, decide whether a delivery conforms to what was ordered. A pilot threshold decides whether the data changes a model enough to be worth buying. The sections below set that number, size the pilot so it can resolve it, and tie each possible outcome to an action both sides agreed in advance.

Should the threshold be a target score or the smallest gain worth paying for?

Use the smallest gain worth paying for. A target score describes what you hope the data achieves; the smallest worthwhile gain describes what it must achieve for the purchase to make sense, and only the second can turn a result into a yes or a no.

Statisticians call this the smallest effect size of interest. Daniel Lakens's 2017 primer on equivalence testing uses it to set the bounds of the two one-sided tests procedure, and warns that researchers often read a non-significant result, wrongly, as proof that an effect is absent. A pilot threshold borrows both ideas: the bound comes from what matters in practice, and a null result is not a rejection on its own.

For a data purchase, what matters is economic. Express the full cost of the data in the unit the pilot measures. Illustrative: a support team handles 40,000 tickets a year, and the license, integration and review effort together cost about as much as the reviewer time for 1,200 tickets. The break-even gain is 1,200 divided by 40,000, or 3 percentage points of auto-resolution rate; below it, even a real improvement loses money. The page on the business case for buying training data shows how to convert gains into money and why pilot gains deserve a haircut before they are projected.

Write the threshold in absolute points on the buyer's own metric, defined so that two qualified reviewers would give the same pass or fail verdict on a task. Anthropic's guidance on agent evaluations defines a task as a test with fixed inputs and success criteria and runs several trials of each, because model outputs vary between runs. Where the output is a document rather than a pass or fail, a blinded comparison works: OpenAI's GDPval has professionals from the same occupation rank AI and human deliverables without knowing which is which and label each AI deliverable better, as good as or worse.

Model builders already write cutoffs this way. NVIDIA's Nemotron training FAQ says stage cutoffs are set experimentally from benchmark evaluations, as rules of the form: when the model reaches X percent on one benchmark and roughly Y percent on another, stop, because going further costs performance elsewhere. A pass mark paired with a limit on collateral loss is the same shape a pilot threshold needs.

How much regression on work that already succeeds will you accept?

Name a harm margin: the largest loss you will tolerate on guarded tasks, the ones the current system already handles, in exchange for the gain. Without one, a dataset that lifts the target metric while quietly breaking existing behavior can pass.

Clinical research has a mature version of this rule. The FDA's 2016 guidance on non-inferiority trials asks a trial to rule out a loss larger than a non-inferiority margin that was specified in advance. Apply the same logic to a pilot: score the guarded set in both arms and require the lower bound of the paired difference to stay above the margin.

Illustrative: the guarded set holds 300 tickets the current model resolves correctly, and the margin is 1 point. If the 95 percent interval for the difference on that set runs from minus 2.4 to plus 0.3 points, the pilot fails on harm, whatever the primary metric did. Add a count rule for errors that are unacceptable at any rate, such as a wrong refund or an exposed customer record, so a few severe cases cannot hide inside an average.

How many tasks does the pilot need to resolve the threshold?

Enough that the confidence interval is narrower than the gap between the smallest worthwhile gain and the gain you expect. If the interval is wider than that gap, the pilot was built to come back inconclusive, and the threshold cannot do its job.

Anthropic's research note on the statistics of model evaluations warns that models need a large underlying difference to register a significant result, so small differences will likely go undetected. It also reports that standard errors can be up to three times larger once questions that share a source are clustered. Both effects push the task count up.

A planning approximation for paired pass or fail outcomes: tasks needed equals (1.96 times the square root of d, divided by h), squared, where d is the share of tasks on which the two arms disagree and h is the half-width the interval must reach. Illustrative: with a 3-point threshold, an expected gain near 8 points and one task in five changing outcome between arms, h is 5 points and the pilot needs about 308 paired tasks. If the expected gain is only 5 points, h falls to 2 points and the count rises to about 1,921.

Estimate d from a dry run, raise the count for clustering, and treat the figures below as a starting point rather than a full power analysis:

How many tasks does the pilot need to resolve the threshold?
Half-width the interval must reachTasks if 1 in 5 tasks differTasks if 1 in 3 tasks differWhere it fits
8 points121201Screening data expected to deliver a large gain
5 points308513Confirmatory pilot with clear headroom above the threshold
3 points8541,423Expected gain only modestly above the threshold
2 points1,9213,202Rarely affordable; lower the threshold through price or scope first

The threshold card: nine entries to sign before the first run

The threshold card is one page holding every number and consequence the pilot decision depends on, signed and dated before any arm is trained or scored. Each row gives how to set the entry, an illustrative value consistent with the examples above, and who signs it.

Keep the signed original. The Center for Open Science encourages researchers who revise a plan before analysis to file a new version that links to the original and lists each change with its reason. A pilot card should work the same way, and any change made once results are visible is logged as a deviation, never as an edit.

The threshold card: nine entries to sign before the first run
EntryHow to set itIllustrative entrySigns
Decision and armsOne purchase question; current setup against current setup plus the candidate data, with the same model snapshot, recipe and budgetFine-tune on licensed resolved tickets, or notDecision owner
Primary metricOne metric in the unit of value, with a pass rule two reviewers would apply the same wayAuto-resolution rate on sealed real ticketsEvaluation lead
Smallest worthwhile gainFull cost of the data divided by annual volume, in the metric's unit3 points (1,200 of 40,000 tickets)Finance and evaluation lead
Harm marginLargest tolerated loss on guarded tasks, judged on the interval's lower boundMinus 1 point on 300 guarded ticketsProduct owner
Severe-error ruleA count limit for errors that are unacceptable at any rateNo increase in wrong refunds or exposed customer recordsRisk or compliance
Resolution planTasks, trials per task, interval method and clustering unit, sized before the run308 paired tasks, 3 trials each, 95 percent interval clustered by accountEvaluation lead
Decision zonesAccept, extend, walk away, harm and void, each defined on interval boundsAccept only if the lower bound is at or above 3 pointsDecision owner
Commercial consequenceWhat each zone does to the purchase, scope, price and sample licenseInconclusive allows one extension of 200 new tasks with no retuningProcurement and supplier
Deviation logDated record of any change after signing, with the reason and both signaturesGrader rubric clarified on day 3; card version 2 filedBoth signers

Which decision does each pilot outcome trigger?

Read the confidence interval against the card, never the point estimate alone. Six outcomes cover every result, and each maps to one reading and one commercial action agreed before the run.

The second row is where unsigned pilots go wrong. A gain that is probably real but may sit below break-even invites a fresh argument about whether the threshold was too strict. The card settles it in advance: one extension with a fixed number of new tasks, or a repricing rule written before the pilot, such as offering a scope whose cost brings the break-even gain down to the interval's lower bound. The technical verdict stays inconclusive at the signed threshold either way.

The figure plots the four primary-metric outcomes as illustrative intervals against zero and the 3-point threshold.

Which decision does each pilot outcome trigger?
OutcomeIllustrative 95 percent intervalReadingAction
AcceptPlus 4.2 to plus 9.1 points, guarded set within marginThe gain is real and worth its costBuy at the signed scope and price
Extend oncePlus 1.1 to plus 6.3 pointsGain likely real, possibly below break-evenRun the pre-agreed extension or apply the repricing rule
InconclusiveMinus 1.8 to plus 4.6 pointsNo detectable effect, which is not proof of noneWalk away at this scope unless the card allowed an extension
Walk awayMinus 0.9 to plus 2.4 pointsAny gain is too small to pay forEnd the evaluation license and return or delete the sample
HarmGuarded set minus 2.4 to plus 0.3 pointsRegression beyond the 1-point marginStop, whatever the primary metric shows
VoidAny result after a protocol breachGrader, model snapshot or test set changed mid-runRerun under a newly signed card
Four illustrative confidence intervals plotted against a zero line and a 3-point smallest worthwhile gain, labeled accept, extend once, inconclusive and walk away.
Reading a pilot interval against the signed threshold. Illustrative intervals, not measured results.

Could a passing result be an artifact of the setup?

Yes, so the card should name the stress tests an accept must survive before money moves. A gain can shrink against a stronger comparison arm, vanish on fresh tasks or fail to repeat across trials, and none of that shows up in a single run.

Anthropic's guidance says an evaluation should be revised when grading is unfair, tasks are ambiguous, valid solutions are penalized or the harness constrains the model. Each of those can push one arm's score up or down for reasons unrelated to the data. GDPval's authors note that its tasks are one-shot, so they do not test the revision cycles real work involves; a pilot inherits whatever limits its own task set has.

Three conditions are worth writing into the accept zone:

  • The lower bound clears the threshold on the sealed set and on a fresh slice of tasks drawn after the card was signed.
  • The comparison arm is the strongest configuration the team could deploy without the data, tuned with the same effort as the data arm.
  • The result holds when trials are rerun with new seeds, using the same interval method and clustering unit.

Why the threshold belongs in the contract before acceptance

Because formal acceptance is hard to reverse. Once a buyer accepts a delivery, its options narrow sharply, so a threshold that lives only in an evaluation notebook gives the buyer little to stand on later.

U.S. federal procurement rules show how final acceptance is meant to be. FAR 46.501 treats acceptance as acknowledgment that supplies or services conform with the contract's quality and quantity requirements, and the fixed-price inspection clause at FAR 52.246-2 makes acceptance conclusive except for latent defects, fraud, gross mistakes amounting to fraud or as the contract otherwise provides. Commercial contracts differ, but the lesson carries over: write the decision rule and the consequence of each zone into the pilot terms before the result exists.

Record-level quality has its own threshold language. NIST's Engineering Statistics Handbook describes acceptance sampling plans by two quality levels, the acceptable quality level and the lot tolerance percent defective, and two risks, the producer's risk of rejecting a good lot and the consumer's risk of accepting a bad one, with typical values between 0.2 and 0.01. A pilot threshold is the model-level counterpart: the harm margin protects the buyer, and the extension rule protects the supplier from a pilot too small to show a real gain.

The supplier should countersign the parts that bind it: the sample scope, the delivery format and what happens in each zone. It should not see the sealed tasks or the guarded set. Terms vary by jurisdiction and deal, and counsel should review the specific wording.

Putting the threshold into a data request

Send the card's decision fields with the request, so the supplier knows what the data must do before anyone prepares a sample. A request that names the metric, the smallest worthwhile gain, the guarded tasks and the consequence of each zone lets a supplier say early whether its records can plausibly clear the bar.

SourceX does not keep a stock of datasets: it sources each one from U.S. businesses after a buyer asks, and the request form is where a buyer sets out scope, volume, language, format, timing and licensing needs. Adding the threshold card turns a broad ask into a testable one, for example outcome-linked support or claims records with the resolution recorded for each case. On the supply side, the pilot sample's scope is fixed in writing before records move: the contributing company has to approve it, along with the terms, and sign the agreement ahead of any delivery.

None of this predicts whether a given dataset will pass. The card's value is that the answer, whichever it turns out to be, is one both sides already agreed to accept.

Frequently asked questions

Can we change the acceptance threshold after seeing pilot results?

Only as a logged deviation, never as a quiet edit. The technical verdict stays tied to the signed threshold; what can change is the commercial response, and only through an extension or repricing rule written into the card in advance. A threshold moved after results are visible no longer tests anything.

What if the pilot gain is not statistically significant?

Then the pilot found no detectable effect, which is not the same as showing the data has no effect. If the interval's upper bound sits below the smallest worthwhile gain, walk away, because any gain is too small to pay for. If the interval still reaches above the threshold, the result is inconclusive, and only a pre-agreed extension should continue the test.

Should a pilot threshold be relative or absolute?

Absolute points on the buyer's own metric are safer. A relative improvement looks large on a weak baseline, such as a move from 2 to 3 percent auto-resolution described as a 50 percent gain, while the economic value depends on the absolute change multiplied by volume.

Should the supplier see the threshold card?

The supplier should see and countersign the decision rule, the metric definition and what each outcome means commercially. It should not see the sealed tasks or the guarded set, because any look at them lets later preparation fit the test.

How is a pilot threshold different from a record-level acceptance criterion?

A record-level criterion checks whether a delivery conforms, for example through a defect limit in a random audit sized with producer's and consumer's risks. A pilot threshold checks whether the data improves a model enough to justify the purchase. Most purchases need both: conformance first, value second.

Sources

  • The Center for Open Science describes preregistration as a research plan written before the study and submitted to a registry, separating planned from unplanned work, since the same data cannot both generate and test a hypothesis. Source
  • The Center for Open Science encourages a new preregistration before analysis when plans change, keeping and linking the original and listing the changes and reasons. Source
  • Lakens's 2017 equivalence testing primer sets equivalence bounds from the smallest effect size of interest and warns that non-significant results are often wrongly read as evidence of no effect. Source
  • Anthropic's agent evaluation guide defines a task by its inputs and success criteria, runs multiple trials because outputs vary, and says to revise evals with unfair grading, ambiguous tasks or a constraining harness. Source
  • OpenAI's GDPval uses graders from the same occupations who blindly classify AI deliverables as better than, as good as or worse than human ones, and its tasks are limited to one-shot evaluation. Source
  • NVIDIA's Nemotron training FAQ says stage cutoffs are set experimentally from benchmark evaluations, as rules of the form X percent on one benchmark and roughly Y percent on another. Source
  • The FDA's November 2016 guidance on non-inferiority trials requires ruling out a loss larger than a prespecified non-inferiority margin. Source
  • Anthropic's statistics note says small capability differences will likely go undetected and that clustered standard errors can be up to three times naive ones. Source
  • FAR 46.501 treats acceptance as acknowledgment that supplies or services conform with applicable contract quality and quantity requirements. Source
  • FAR 52.246-2 makes acceptance conclusive except for latent defects, fraud, gross mistakes amounting to fraud or as otherwise provided in the contract. Source
  • NIST's Engineering Statistics Handbook defines sampling plans by the AQL and LTPD and the producer's and consumer's risks, with typical risk values from 0.2 to 0.01. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies