AI uses for records
Dataset pilot evaluation leakage: read the final test once
By SourceX Editorial · Updated
9 sections
Short answer
Dataset pilot evaluation leakage happens when choices made while preparing a candidate dataset, such as filters, mixes, checkpoints, prompts or supplier fixes, are judged on the same cases that decide the purchase. Prevent it with three sets and one rule: tune on a development set, choose checkpoints on a validation set, and score a sealed final test once, under success, stop and inconclusive rules written before the pilot starts.
Evaluation split
Dataset pilot evaluation
Key takeaways
- Feedback leakage needs no copied record: every preparation choice scored on the final test makes the test a tuning set.
- Separate a readable development set, a scores-only validation set and a sealed final test held by someone outside the preparation team.
- A feedback ledger counts every look at held-out data; the final test gets one read per arm, and a second read voids it.
- Preregister the arms, metric, margin and decision rules, including an equal-volume comparison arm and a sealed confirmation slice.
- Inconclusive and negative outcomes need decisions too, so the result table maps every sealed-test pattern to an action.
What counts as evaluation leakage in a dataset pilot?
Evaluation leakage in a dataset pilot is any route by which the cases that decide the purchase shape the dataset, the training run or the scoring before the decision is made. It comes in two forms. Content leakage puts the same or near-identical records on both sides of the split. Feedback leakage is quieter: no record moves, but choices made during preparation are judged on the decision cases until the data, the checkpoint or the prompt fits them.
The scikit-learn documentation states the rule in one line: test data should never be used to make choices about the model. It also names the cost: leakage produces overly optimistic performance estimates and poorer performance on genuinely new data. In a dataset pilot, the thing being chosen includes the candidate dataset itself, so every filter, mix and fix picked by looking at the final test is a choice made with test data.
Experienced teams get this wrong. Kapoor and Narayanan's review of machine-learning-based science found leakage-driven errors in hundreds of published papers across 17 fields and proposed model info sheets to catch them before publication. A buyer deciding whether to license a dataset faces the same hazard with money attached, because the pilot's headline number is worth only as much as its test set's independence. This article covers feedback leakage; the linked pages on retrieval corpus leakage and post-training contamination controls cover records that cross the split.
How a pilot learns its own test set without copying a record
A pilot learns its test set through a series of reasonable decisions, each scored on the decision cases. Dwork and colleagues described the underlying problem: statistical theory assumes the analyses are fixed before the data are examined, while real practice reuses data and chooses new analyses based on earlier results. A pilot that adjusts after every look is doing the second while reporting as if it had done the first.
At a glance
How a pilot learns its own test set without copying a record
Agent evaluations add a mechanical route. Anthropic's January 2026 guidance on agent evals reports that Claude gained an unfair advantage on some tasks by examining the git history left by previous trials, notes that shared state can artificially inflate performance and recommends starting every trial from a clean environment. NVIDIA's Nemotron FAQ describes software-engineering agents that could cheat by reading the gold patch out of the task container; the Ultra team closed that channel by rewriting the repository to look like a fresh clone and blocking remote git operations and downloads.
The usual routes in a buyer's pilot each turn the final test into a tuning set:
- The supplier sees failing test items and sends more records like them.
- The preparation team picks a cleaning filter or data mix by its score on the final test.
- Checkpoints, learning rates or epoch counts are chosen on the final test.
- Prompts, rubrics or grader settings are edited after reading final-test outputs.
- Test items that look unfair are dropped after the scores come in.
- Runs are repeated until one seed clears the bar, and only that run is reported.
- Agents reuse files, caches or repository history left by earlier trials.
Three sets and one wall: separating preparation feedback from the decision
Give preparation its own data and keep the decision data behind a wall: a development set people may read, a validation set that returns only scores, and a sealed final test scored once per arm at the end. The wall is about who can see what, not where the files sit.
The development set exists to be looked at. Buyer engineers and the supplier may read its items, run error analysis and agree fixes to the candidate dataset. The validation set chooses checkpoints, mixtures and stopping points and returns aggregate scores only, so nobody writes data to match its items. The sealed final test is held by a named evaluation owner outside the preparation team, who runs it once for each arm after training is frozen.
Draw all three sets by the same rule before the supplier is engaged, and keep related records together. The same customer, account or incident thread should not appear on two sides of the wall, and records with a time order should be split by date. Only two things cross the wall: written failure descriptions from the development set and aggregate validation scores. The figure shows the flow.
The feedback ledger: count every look at held-out data
Keep a ledger of every decision made with held-out scores, and give each set a budget: the validation set absorbs a fixed number of decisions agreed in advance, and the final test absorbs one read per arm. A second read turns the final test into a validation set, and the pilot then needs a new final test.
Each ledger line records the date, the set, the decision it informed, who saw the results and at what grain, aggregate or item level. The ledger is cheap to keep and does two jobs: it stops quiet re-reads, and it lets a reviewer judge how much independence the reported number still has.
An illustrative three-arm pilot might close with thirteen ledger lines: eight validation reads, each choosing a checkpoint or a data mix; two development reviews shared with the supplier as written failure notes; and three final-test reads, one per arm, logged on the same day by the evaluation owner. A fourteenth line that reads the final test again, for any reason, is the line that voids the result.
Aggregate feedback is far safer than item feedback, and there is evidence for it. Roelofs and colleagues compared public leaderboard rankings with final rankings across more than one hundred Kaggle competitions, in which participants scored against the public holdout repeatedly, and found little evidence of substantial overfitting. Two features of that setting carried the result: participants saw only aggregate scores, and the final ranking came from a separate test. A dataset pilot in which the supplier sees which items failed and patches the data to match has neither protection, so treat the Kaggle finding as a best case, not as permission to re-read.
The preregistered pilot protocol
Write the protocol, agree it with the supplier and file it before any candidate data is delivered. Preregistration is what makes the final score a test of the dataset rather than a test of the team's persistence.
Hold the model, training budget and evaluation conditions fixed across arms, so that the data under test is the only difference. The example entries are illustrative and describe no real pilot. Fill in these eight fields:
| Field | Write before the pilot | Example entry (illustrative) | Leak it blocks |
|---|---|---|---|
| Decision question | The one purchase decision the pilot informs | Does licensed claims correspondence raise first-pass accuracy on disputed claims for our 8B model at the same training budget? | Moving the goalposts after results |
| Arms and fixed conditions | Model, steps, learning rate, seeds and token budget for each arm | A: current data. B: current plus candidate data. C: current plus an equal token count of public or in-house data | Crediting the dataset for extra training |
| Set definitions | Draw rule, grouping key, time cutoff and size of the development, validation and final sets | Grouped by policyholder, split by claim date, final test drawn from the latest quarter | Near-duplicates and later records crossing the split |
| Feedback channel | What may cross the wall, and at what grain | Supplier receives development failure descriptions and aggregate validation scores only | Supplier patching data to test items |
| Primary metric and margin | One metric, its grader and the smallest difference that counts | Exact-match accuracy on the adjudication field, with a margin of 3 points over arm C | Choosing the flattering metric afterwards |
| Decision rules | Success, stop and inconclusive conditions written as tests | Success if B beats A and C by the margin on every seed and the interval excludes zero | Re-reading until some rule is met |
| Repeats | Seeds or trials per arm and how results are combined | Three seeds per arm, every seed reported | Reporting the lucky run |
| Confirmation slice | A second sealed slice drawn with the final test and opened only on success | 20 percent of final-test cases held back by the evaluation owner | Buying on a one-time fluke |
The result table to fill in after unsealing
Fill the result table once, after the sealed test is read, and let the preregistered rule pick the row. Each row names the decision and what must be reported with it, so an inconclusive or negative result is as usable as a win.
Report every arm, every seed and the ledger with the table; a result without its ledger cannot show how many looks it took. Match the observed pattern to one row:
| Pattern on the sealed test | Decision | What to report | Next evidence needed |
|---|---|---|---|
| B beats A and C by the margin on every seed | Proceed to scope the full license | Per-arm scores, intervals, seeds and the opened confirmation slice | The confirmation slice agrees within the margin |
| B beats A but not C | Do not pay for specificity; the gain comes from more data | Both contrasts with their intervals | A cheaper source of comparable volume |
| Interval spans zero, or seeds disagree | No purchase decision on this test | Interval width and the test size the margin would need | A larger sealed test, preregistered before it is drawn |
| B falls below A, or guard cohorts regress | Stop | Which cohorts regressed and by how much | None for this scope |
| The ledger shows extra reads, or an overlap check fails | Void the result | The breach and when it happened | A newly drawn final test |
Could the gain be real but fragile? Testing the alternative explanations
A positive result has three rivals, and the protocol tests each one: the gain came from more data rather than this data, from one lucky run, or from a test slice that happened to suit the candidate. Arm C answers the first, repeated seeds the second and the confirmation slice the third.
Arm C matters most for pricing. If an equal amount of public or in-house data closes most of the gap, the buyer is paying for volume and should price it as volume. Repeats matter because model outputs vary between runs, which is why Anthropic's evaluation guidance runs multiple trials and defines pass@k as the likelihood of at least one correct solution in k attempts. A single run cannot separate a dataset effect from noise.
Fresh slices need care in both directions. When Recht and colleagues rebuilt the CIFAR-10 and ImageNet test sets by following the original process, accuracy fell by 3 to 15 percent and 11 to 14 percent respectively, and they attributed the drops to models' difficulty with slightly harder images rather than to adaptive overfitting. Engstrom and colleagues later traced much of the ImageNet gap to statistical bias in how the replication was assembled, estimating 3.6 percent after correction against the original 11.7 percent. A drop on a new slice is not proof of leakage, and a slice drawn by a different rule or at a different time measures a shift. Draw the confirmation slice with the final test, by the same rule, and seal both on the same day.
Where the protocol runs out
The protocol fails quietly in four situations, and each needs a named workaround before the pilot starts.
- Small final tests give wide intervals. If the sealed test is too small to detect the agreed margin, the honest result is inconclusive, so size the test before drawing it, not after reading it.
- Agent and tool-use tasks leak through environments rather than text. Start every trial from a clean environment, as Anthropic's guidance recommends, and remove repository history and cached files the task could read.
- Records with a time order need time splits. Train and tune on earlier periods and test on the latest one, or the model sees outcomes that postdate the question.
- One read leaves no second chance. If the final test is spent on a broken pipeline, record it in the ledger, draw a new test and accept the delay rather than reusing the old one.
Putting the protocol into a data request
Put the protocol's essentials in the request itself, so the supplier agrees to the wall before any record moves: the decision question, the three sets and their draw rule, evaluation-only use for the final test, who holds it and what feedback the supplier will receive.
A useful request also names the buyer's evaluation owner and asks whether any offered records have already been licensed or shared for AI training elsewhere. Records that may already sit in another model's training data make weak test cases, so they belong in the development or training side of the wall, never in the sealed test.
SourceX does not sell from a catalog. It looks for U.S. businesses whose records fit a buyer's request after the request arrives, and the request form asks for scope, volume, language, format, timing and licensing needs, which is where an acceptance test and a sealed-test arrangement can be written down. On the supply side, the business holding the records commits in writing first, approving the scope and terms and signing before a single record changes hands. None of this promises that a candidate dataset will pass the buyer's test; the protocol exists because it might not.
Frequently asked questions
What is the difference between evaluation leakage and benchmark contamination?
Contamination usually means test items or close copies ended up in training data, often through web scraping. Evaluation leakage is broader: it also covers decisions tuned on the test, such as choosing filters, checkpoints or prompts by final-test score, even when no record crosses the split. A pilot can be free of contamination and still leak.
Can a data supplier see results during a dataset pilot?
Yes, within limits written into the protocol. The supplier can see failure descriptions from the development set and aggregate validation scores. It should never see final-test items or item-level final-test results before the decision, because patching data to those items turns the test into a tuning set.
How many times can the final test set be scored?
Once per arm, after training is frozen. Every extra read spends some of its independence, and once it has informed a decision it has become a validation set. The pilot then needs a newly drawn final test.
What should happen when a dataset pilot is inconclusive?
Record it as inconclusive and make no purchase decision on that test. If the decision still matters, preregister a larger sealed test sized to detect the agreed margin and draw it fresh, rather than extending the old test until a result appears.
Why include a public or in-house data arm in a dataset pilot?
Because more data of almost any kind often helps. An equal-volume comparison arm shows whether the candidate dataset adds something specific or only volume, and that difference should decide what the buyer is willing to pay.
Sources
- scikit-learn's documentation says leakage produces overly optimistic estimates and that test data should never be used to make choices about the model. Source
- Anthropic's January 2026 guidance reports an agent exploiting git history from earlier trials, warns that shared state inflates performance and defines pass@k. Source
- NVIDIA's Nemotron FAQ describes agents reading the gold patch from the task container and the two leak channels the Ultra team closed. Source
- Kapoor and Narayanan found leakage-driven errors in hundreds of published papers across 17 fields and proposed model info sheets. Source
- Dwork et al. note that statistical theory assumes analyses fixed in advance while practice reuses data adaptively. Source
- Roelofs et al. (2019) found little evidence of substantial overfitting across more than one hundred Kaggle competitions with repeated public holdout use. Source
- Recht et al. (2019) found accuracy drops of 3-15 percent on CIFAR-10 and 11-14 percent on ImageNet on rebuilt test sets, not caused by adaptivity. Source
- Engstrom et al. (2020) estimated a 3.6 percent ImageNet gap after correcting replication bias, against the original 11.7 percent. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.