Skip to content

AI uses for records

Minimum useful dataset pilot: four gates that can reject a bad dataset

By SourceX Editorial · Updated

9 sections

Short answer

A minimum useful dataset pilot is the cheapest sequence of tests that can reject a bad dataset before a full purchase; it is not built to prove a good one. Run four gates in cost order: documents, a random record audit with a preset defect limit, an audit of your own test set, and a small-dose training probe with harm and futility rules. Any gate can stop the pilot; passing earns only a confirmation test.

Agent run

Minimum useful dataset pilot

This page covers minimum useful dataset pilot: four gates that can reject a bad dataset.Illustrative

Key takeaways

  • Rejecting a dataset needs far less evidence than accepting one, so design the first pilot to say no.
  • Order the gates by cost: documents, record audit, test-set audit, then a small training probe.
  • A 52-record audit that rejects at three or more defects passes a 2 percent defect rate about 91 percent of the time and a 10 percent rate only about 10 percent of the time.
  • A broken test set voids the pilot instead of rejecting the dataset.
  • Ruling out a useful gain on task performance can take hundreds of tasks, which is why the cheap gates run first.

What can a minimum useful dataset pilot decide?

It can decide one thing reliably: stop now. A minimum useful dataset pilot is the smallest set of tests whose failure would justify walking away from a dataset before a full purchase, with every threshold written down before the first record is opened.

Acceptance is a different question with a different budget. To accept, you need evidence that the dataset improves held-out tasks against a strong baseline across repeated trials; to reject, one well-defined failure is enough. A pilot sized for acceptance spends weeks on datasets a 52-record audit would have screened out, while a pilot sized for rejection is honest about its limit: passing it earns a confirmation test, not a contract.

This page sets out four rejection gates in cost order, the sample sizes behind them, the result table to file before starting, and the ways a small pilot can wrongly reject a dataset that would have helped.

Why saying no takes fewer records than saying yes

Detecting a high defect rate takes a few dozen records, while proving a low one takes hundreds. If you audit n records and find no defects, the rule of three puts the one-sided 95 percent upper bound on the defect rate at about 3 divided by n, an approximation a 2023 paper indexed in PubMed examines for exactly this zero-event case, so showing a rate below 1 percent needs roughly 300 clean records.

Rejection runs the other way. If a dataset truly has a 10 percent defect rate, the chance that 29 random records contain no defect at all is under 5 percent, so a defect is almost certain to surface. The same arithmetic gives 14 records for a 20 percent rate and 59 for a 5 percent rate.

The catch is false alarms. Reject on the first defect and a 29-record audit also rejects a dataset whose true defect rate is only 2 percent about 44 percent of the time. A usable plan sets both risks at once, which is what industrial acceptance sampling has done for decades: NIST's engineering statistics handbook designs single sampling plans from two points, an acceptable quality level the buyer should usually pass and a lot tolerance percent defective it should usually reject.

The four rejection gates, in cost order

Run the gates from cheapest to most expensive and stop at the first failure, so most bad datasets never reach a GPU. Each gate has a reject rule fixed in advance, a minimum sample and a stated blind spot, because a gate that cannot detect a problem should never be read as clearing it.

Gate 3 is the only one that trains a model, and it runs at a small dose by design. NVIDIA's Nemotron training FAQ says its team validates synthetic data through small-scale ablations, training models on the data to check that they learn something from it; a purchase deserves the same check before the full run. The full protocol:

The four rejection gates, in cost order
GateWhat it checksMinimum sampleReject whenBlind spot
0. DocumentsTask spec and acceptance test, datasheet, field dictionary, rights scope that covers the pilot useThe full document setAny item is missing or contradicts the sampleSays nothing about record quality
1. Record auditRandom records from the delivery frame, checked against written defect codes52 records in the default planThree or more defectsDefects rarer than about 2 percent; usefulness for your tasks
2. Test-set auditYour held-out tasks: two-expert agreement, reference solutions, no overlap with the candidate20 to 50 tasks drawn from real failuresNever rejects the dataset; a failed audit voids the pilotFailure types your tasks do not cover
3. Small-dose probeCandidate slice against a volume-matched filler on the same model, three trials per taskSet by the futility tableHarm: regressions significantly outnumber fixes; futility: upper bound below the minimum useful gainGains that appear only at full volume or on larger models
Confirmation, after the minimum pilotFull dose, strongest baseline, fresh seedsPowered for acceptanceAny confirmation test failsGeneralization beyond the pilot's task families
A staircase of four rejection gates rising in cost, documents, a 52-record audit, a test-set audit of 20 to 50 real failures and a paired small-dose probe, each with an exit to reject or void, leading to a confirmation test.
The four-gate rejection protocol; sample sizes are planning values from this article, not measured results.

Sizing the record audit: two risks, one plan

Pick the defect rate you would accept and the rate you must reject, then choose the smallest sample and defect count that keep both risks near 10 percent. The plans below come from exact binomial calculations, the approach NIST's handbook uses for single sampling plans; they are planning arithmetic, not measured results.

Define a defect before sampling: a missing or unverifiable outcome field, a broken link between records that should join, an exact or near duplicate, a residual direct identifier, or a record outside the agreed scope. Draw the sample at random from the full delivery frame, never from records the supplier chose, and audit without seeing any quality score the supplier attached. Estimating an error rate with a confidence interval, rather than making an accept-or-reject call, is a different calculation covered in the guide to audit sample sizes.

If the delivery pools several source companies or years, split the audit across them in proportion to volume and record defects per stratum, because one company with a broken export can hide inside a passing total. Rejecting a single stratum and re-cutting the delivery without it is often a better result than rejecting the whole dataset.

Sizing the record audit: two risks, one plan
Pass at defect rateReject at defect rateAudit sampleReject when defects reachChance of passing: good / bad
1%10%38 records294% / 10%
2%10%52 records391% / 10%
2%10%, with both risks at 5%89 records597% / 5%
5%20%32 records493% / 9%
0.5%5%77 records294% / 10%
2%5%258 records992% / 10%

Is your test set fit to reject anything?

Audit your own held-out tasks before trusting any training result, and treat a failed audit as a void pilot, not a rejected dataset. A test with wrong reference answers or leaked items can make a good dataset look useless or a bad one look strong.

Anthropic's January 2026 guide to agent evals says 20 to 50 simple tasks drawn from real failures is a good start because early effects are large and small samples suffice, that a good task is one where two domain experts would independently reach the same verdict, and that a 0 percent pass rate across many trials usually signals a broken task rather than an incapable agent. Label errors are common even in famous benchmarks: Northcutt and colleagues estimated in 2021 that the test sets of ten widely used datasets average at least 3.3 percent label errors.

Leakage is the other failure. EleutherAI's evaluation harness documents a decontamination procedure that marks a test item as contaminated when it shares a 13-gram with training data, following the method of the GPT-3 paper. Run that check between the candidate slice and your tasks before training; overlap discovered afterward voids the probe.

The small-dose probe: harm and futility rules

Train the same pinned model twice, once with a slice of the candidate and once with a volume-matched filler of data you already hold, then score both on the held-out tasks with three trials each and compare them task by task. Reject for harm if regressions significantly outnumber fixes, and for futility if the upper end of the interval on net improvement falls below the smallest gain worth paying for.

Task-by-task pairing is what makes a small probe informative. Anthropic's note on eval statistics recommends paired differences because a shared question list removes the variance from question difficulty, and the exact McNemar test in statsmodels uses only the discordant tasks, those that passed in one arm and failed in the other. With small task counts, prefer exact tests: a 2025 position paper on arXiv argues against central-limit-theorem intervals for evaluations with fewer than a few hundred data points.

Illustrative: a probe scores 80 held-out tasks with three trials each. Nine tasks move from fail to pass and four from pass to fail, a net gain of 5 tasks, or 6.3 points. The exact McNemar test on those 13 discordant tasks gives a two-sided p-value of about 0.27, so there is no evidence of harm, and the 95 percent interval on net improvement runs from about -2.5 to +15 points, so a 10-point gain cannot be ruled out either. Neither rule fires and the dataset moves to confirmation, which is the point: a probe this size can stop a dataset, but it cannot vouch for one.

Futility is expensive to prove, which is why it comes last. If the dataset truly adds nothing, the number of tasks needed for the 95 percent upper bound on net improvement to fall below the minimum useful gain, with 80 percent probability, grows with the share of tasks that flip between arms and with the inverse square of the gain you care about:

The small-dose probe: harm and futility rules
Share of tasks that flip between armsMinimum useful gain of 10 pointsMinimum useful gain of 5 points
5%40 tasks157 tasks
10%79 tasks314 tasks
20%157 tasks628 tasks
30%236 tasks942 tasks

Could a small pilot reject a dataset that would have helped?

Yes, and the protocol should name how. A small pilot can wrongly reject a useful dataset when the dose is too small to show a gain that appears at full volume, when the probe model is too small to use the data, when the defect codes punish harmless variation, or when the test set misses the cases the dataset was bought to fix.

Dose is the usual culprit. Generalization error typically falls as a power law in training set size, with exponents that have to be measured problem by problem (Hestness and colleagues, 2017), so a tenth of the data can produce a gain too small to detect even when the full set would clear the bar. Model size comes second: NVIDIA's FAQ says its small-scale ablations generally use models under 10B parameters and that results seem to generalize better above about 3B, which it presents as an observation rather than a firm threshold.

Two rules guard against both. Set the minimum useful gain for the dose you test, not for the full purchase, and log a futility stop as not shown at this dose whenever the probe scored fewer tasks than planned. The opposite error is just as real: a positive probe can vanish under a stronger baseline, a proper holdout or repeated trials, which is why passing gate 3 never ends the evaluation.

The preregistered result table: six outcomes and what follows

File the result table with every rule filled in before the pilot starts; afterward, only the evidence column changes. Exactly one outcome applies, set by the first gate that fails, and nothing in the table is a measured result.

Clinical research has long worked this way. ICH E20 on adaptive designs, published by the FDA, says trials can stop early for efficacy or futility using appropriate pre-specified stopping rules; a dataset pilot borrows the discipline without the regulatory machinery, so a disappointing number cannot be explained away after the fact.

The preregistered result table: six outcomes and what follows
OutcomeTriggered byEvidence to recordNext step
Reject: documentsA gate 0 item is missing or contradictedWhich item, and the supplier's responseStop; reopen only with complete documents
Reject: defectsGate 1 defects reach the preset countDefect codes per record, sample frame and draw methodStop, or request a re-cut that removes the defect source
Void: test setGate 2 audit fails or overlap is foundFailed tasks, disagreement rate, overlapping itemsRepair the test set, then rerun gates 2 and 3
Reject: harm or futilityA gate 3 rule is met at the planned task countPaired pass and fail counts, interval, trials per task, seedsStop; record the dose and model size tested
Not shown at this doseGate 3 stopped before the planned task countTasks scored, interval, reason for stoppingDecide whether a larger probe is worth its cost
Continue to confirmationAll four gates passedEvery gate's evidence, with thresholds unchangedRun the full-dose test against the strongest baseline

Putting the gates into a data request

Send the gates with the request, so a supplier knows before any sampling what will be audited and what ends the pilot. Include the defect codes, the audit plan, how the pilot sample will be drawn from the delivery frame, the permitted pilot use and the acceptance test a later confirmation would apply.

SourceX works from requests rather than an inventory: buyers describe scope, volume, language, format, timing and licensing needs, and records are sourced from U.S. businesses to fit. On the supplier side, the company approves scope and terms and signs before any records are delivered, so even a pilot sample moves under agreed limits. Neither point means a given request can be filled or that a dataset will pass your gates.

Frequently asked questions

How many records should a dataset pilot audit?

Enough to separate the defect rate you would accept from the one you must reject. As planning arithmetic, 52 random records with rejection at three or more defects passes a 2 percent defect rate about 91 percent of the time and passes a 10 percent rate only about 10 percent of the time; tighter limits need more records.

Can a small pilot prove a dataset is good?

No. A small pilot can show that a dataset fails a defined test. Showing that it improves your tasks against a strong baseline across repeated trials needs a larger confirmation test powered for that purpose.

Should the supplier choose the pilot sample?

No. Agree the delivery frame and draw the sample at random from it, yourself or by an agreed method, so the audit reflects what a full delivery would contain rather than the supplier's best records.

What happens if my own test set fails its audit?

The pilot is void, and the dataset is neither accepted nor rejected. Fix the tasks, check them for overlap with the candidate again and rerun the probe; rejecting a dataset on a broken test is one of the most avoidable pilot errors.

Why not train on the full dataset straight away?

Because most failures show up earlier and cheaper. A missing document or a 10 percent defect rate can be found without any training, while a full run costs far more and still needs the same audited held-out tasks to interpret.

Sources

  • Anthropic's guide to agent evals (January 9, 2026) says 20-50 simple tasks drawn from real failures is a great start because large early effects mean small samples suffice, that two domain experts should independently agree on a good task's verdict, and that 0% pass@100 usually signals a broken task. Source
  • Anthropic's note on eval statistics (November 19, 2024) recommends paired-differences tests, which eliminate variance from question difficulty, and power analysis, noting that small evals give wide intervals. Source
  • The NIST/SEMATECH e-Handbook designs single sampling plans from two points, the AQL (accepted with probability 1 minus alpha) and the LTPD (accepted with probability beta), solving for sample size and acceptance number with binomial sums. Source
  • A 2023 paper indexed in PubMed treats the rule of three as an approximation to the one-sided 95% binomial upper bound when zero events are observed. Source
  • Northcutt, Athalye and Mueller (2021) estimated an average of at least 3.3% label errors across the test sets of 10 commonly used datasets. Source
  • EleutherAI's lm-evaluation-harness marks a test document as contaminated when it shares a 13-gram with training data, following Appendix C of the GPT-3 paper. Source
  • statsmodels implements an exact McNemar test that uses the binomial distribution on the discordant cases, those that are one in one sample and zero in the other. Source
  • A 2025 position paper on arXiv argues against using central-limit-theorem intervals in LLM evals with fewer than a few hundred data points. Source
  • NVIDIA's Nemotron training FAQ says synthetic data is validated through small-scale ablations, generally on models under 10B parameters, with results seeming to generalize better above about 3B as an observation, not a firm threshold. Source
  • Hestness et al. (2017) found that generalization error falls as a power law in training set size, with exponents that must be measured empirically. Source
  • ICH E20, Adaptive Designs for Clinical Trials, published by the FDA, says trials can stop early for efficacy or futility using appropriate pre-specified stopping rules. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies