AI uses for records
Benchmark contamination uncertainty: what a leakage scan lets you claim
By SourceX Editorial · Updated
10 sections
Short answer
A leakage scan can support one firm claim: the overlap it found is real, within the method's false positive rate. A clean scan supports a narrower claim than most reports make: no overlap was detected by these methods, on these fields, against the training data you could inspect. It cannot prove absence, because paraphrases, derivatives, unseen corpora and weak detectors all leave residual uncertainty that the report must state.
Evaluation split
Benchmark contamination
Key takeaways
- A leakage scan proves overlap it finds; a clean result only bounds what its methods could see.
- Substring and n-gram checks miss paraphrased and translated test items, and published membership inference tests barely beat chance.
- Report detection and residual uncertainty as two separate halves, never as one verdict.
- Company, task family and time splits support a stronger claim than any after-the-fact scan.
- Derivatives such as summaries, labels and attachments can carry answers without shared text, so lineage records matter.
What can a leakage scan actually prove about benchmark contamination?
A scan proves presence, not absence. When an exact match, a near-duplicate or a shared entity turns up across the training and evaluation sides, you have direct evidence that the two sets touch, and you can count and inspect every case. When nothing turns up, you only know that the methods you ran, on the fields you compared, over the training data you could see, found nothing.
That asymmetry is the core of benchmark contamination uncertainty. Detection is a positive observation with a known artifact behind it. A clean result is the absence of an observation, and its strength depends entirely on how sensitive the method is to the kinds of overlap that matter for your task. A report that writes 'the benchmark is uncontaminated' after a substring check has turned a statement about a tool into a statement about the world.
The practical fix is to separate the two claims in the report itself. One part lists what was found and how sure you are that each finding is real. The other part lists what was checked, what was not, and how large the remaining blind spot is, in words a reader cannot mistake for a promise that the set is clean.
Why a clean result is weaker evidence than it looks
Every published detector has documented blind spots. The GPT-4 technical report describes a check that sampled three 50-character substrings from each evaluation example and flagged a match against processed training data; the report itself says the approach can produce false negatives when evaluation and training text differ slightly, and false positives as well. A careful lab stated the limit in its own write-up, and yours should too.
Rewording defeats string matching. Yang and colleagues showed that test items rephrased by a language model, or translated, pass n-gram overlap filters, and that a 13B model trained on such rephrased test data can reach benchmark scores comparable to far larger models. A scan that only compares surface text therefore says little about paraphrased exposure.
Model-side tests do not close the gap either. Duan and colleagues evaluated membership inference attacks on models trained on the Pile and found they barely outperform random guessing in most settings; apparent successes were traced to distribution shift, such as members and non-members drawn from different time ranges. A low membership score is not evidence that an item was never seen, and a high one may only reflect a date difference.
Coverage is the last limit. If part of the training mixture came from a vendor, a crawl you did not keep, or a fine-tuning set assembled by another team, the scan never looked at it. The clean result is bounded by the inventory of what you could inspect.
Which leakage paths does your scan actually see?
Map each path from the evaluation set to the training side before writing any conclusion, then mark which method covers it. The paths that matter for business-record benchmarks are rarely exact copies; they are the same customer, the same template, the same week or a summary of the same case.
Kapoor and Narayanan's survey of leakage in ML-based science found the problem in 17 fields and grouped it into a taxonomy whose first type is the lack of a clean separation between training and test data. Their proposed remedy, a structured information sheet for each model claim, is the same idea as the report below: make the argument for separation explicit so a reader can check it.
| Leakage path | Scan that can see it | What it misses | Report as |
|---|---|---|---|
| Verbatim copy of an item | Exact or normalized substring match | Edits, reformatting, translation | Detected count, with match rule |
| Near-duplicate of an item | MinHash or embedding similarity above a stated threshold | Paraphrases below threshold, different language | Detected count, threshold, sample precision |
| Same company or account | Join on entity keys across splits | Entities renamed or merged in one copy | Overlap count by entity group |
| Same task family or template | Template fingerprint or task family label | Families never labeled | Families present on both sides |
| Time overlap | Compare event dates to training cutoff | Undated items, backfilled records | Share of items after cutoff |
| Derivative of an item | Lineage records for summaries, labels, embeddings | Derivatives made outside the pipeline | Known derivatives and their split |
| Exposure through access | Access logs for evaluation storage | Copies taken before logging began | Access window and who had read rights |
The detection versus absence report, field by field
The report has two halves that are never merged into a single verdict. The detection half is factual and countable. The absence half is a bounded statement with its conditions attached. Fill every field; a blank field is itself a finding and should read 'not checked'.
Use the claim ladder underneath the field list to choose the strongest sentence the evidence allows. Most scans on business records end at the second or third rung, and that is an honest place to stop.
- Scope: benchmark name and version, item count, the fields compared (prompt, context, answer, attachments) and the fields excluded.
- Training side inventory: every corpus scanned with its snapshot date, and every corpus known to be in the mixture but not scanned.
- Methods: each detector with its normalization, window or threshold, and the date it was run.
- Detection results: counts per path, a sample of hits reviewed by a person, and the share of reviewed hits judged real.
- Detector sensitivity: how the method performed on planted copies and paraphrases, if a recall test was run, or 'not measured'.
- Split design: the grouping keys used (company, task family, time) and any items that could not be assigned a group.
- Residual uncertainty: the paths in the table above that no method covered, written as plain sentences.
- Allowed claim: the rung of the claim ladder the evidence supports, quoted exactly as it may appear in a paper or model card.
- Owner and review: who ran the scan, who reviewed the hits, and the date the report expires or must be rerun.
| Claim rung | Evidence required | Sentence you may publish |
|---|---|---|
| 1. Overlap found | Reviewed hits from any method | N items overlap with training data under rule R; results are reported with and without them. |
| 2. Not detected by method | Clean run of a stated method on stated fields | No overlap was detected by method M on fields F against corpora C. |
| 3. Not detected, sensitivity measured | Clean run plus a recall test on planted variants | Method M found most planted variants in our test; no overlap was detected in the real scan. |
| 4. Separated by design | Group and time splits with complete keys and lineage | Evaluation items share no company, task family or time window with training items, by construction. |
| 5. Absence | Not attainable by scanning alone | Do not publish a claim of absence; use rung 3 or 4 wording instead. |
How company, task family and time splits strengthen the claim
A split designed to prevent leakage supports a stronger statement than any scan run after the fact. If no company, account or task family appears on both sides, and every evaluation record is dated after the training window, the separation is a property of construction rather than an inference from a detector.
The tooling for this is ordinary. The scikit-learn documentation describes GroupKFold as a variant of k-fold that keeps the same group out of both training and test sets, so a model cannot score by memorizing group-specific features, and describes time series splits for data generated by a time-dependent process. The same logic applies when the groups are customers, contract families or support queues.
Time works the same way for public benchmarks. LiveCodeBench tags each problem with its release date so evaluators can score only on problems published after a model's training cutoff. For business records the equivalent is the event date of the case, not the export date of the file.
Design has its own blind spots. A group key is only as good as entity resolution: two spellings of one customer defeat a company split. Record which keys were resolved by hand, which were matched automatically, and how many items had no key at all.
Can a summary, label or attachment carry the answer across the split?
Yes, and these paths are the ones scans most often miss, because nothing is copied verbatim. A ticket summary written for a training set can state the resolution of a ticket that sits in the evaluation set. An attachment such as a credit memo can carry the outcome in its file name or totals. A label file, an embedding index or a fine-tuned grader trained on evaluation answers can carry the answer forward without any shared text.
Lineage is the only reliable check here. Keep a record of every artifact derived from an evaluation item, with its location and which split it went to. Lee and colleagues found that near-duplicates and long repeated substrings are common in standard language modeling datasets and that train-test overlap reached more than 4% of the validation set in some of them; derivatives raise the same risk with less visible text.
Canaries help with one narrow path. BIG-bench places a canary string in every task file and asks contributors not to edit it, so that crawled copies can be found and filtered. A canary can show that a file was ingested; finding no canary does not show that the content was not ingested in another form.
Worked example (illustrative): reporting a scan on a support-ticket benchmark
The figures below are illustrative counts for a hypothetical evaluation set of 1,200 resolved support tickets drawn from 40 customer accounts over 18 months. The team scanned prompts and resolution notes against two fine-tuning corpora it held, and knew of a third vendor corpus it could not inspect.
Detection half: normalized substring matching flagged 9 items and MinHash similarity flagged 31 more. A reviewer read all 40 and judged 27 to be real overlap, mostly tickets whose resolution macros also appeared in training replies. Entity join found 6 accounts present on both sides. These 27 items and all items from the 6 shared accounts were removed, and scores are reported with and without them.
Absence half: the report states that no further overlap was detected on prompts and resolution notes against the two scanned corpora; attachments were not compared; the vendor corpus was not scanned; no recall test was run, so detector sensitivity is unknown. The allowed claim is rung 2, with a note that the account and time split brings the remaining set close to rung 4 once the unresolved account keys are fixed.
What the team did not write matters as much: there is no sentence saying the benchmark is clean, and no single contamination percentage presented as the true rate.
When a hit does not mean the score is inflated
Some overlap is legitimate and should be reported, not removed. Business records reuse boilerplate: return policies, disclaimers, standard resolution macros and form headers appear in thousands of cases. A match on that text shows shared templates, not shared answers, and removing every such item would empty the benchmark.
The test is whether the overlapping text carries the decision the item asks the model to make. If the policy paragraph is identical but the customer facts and the correct outcome differ, record the hit as template overlap and keep the item. If the overlapping text contains the outcome, treat it as leakage. Write the rule down before reviewing hits so the reviewer is not deciding case by case under pressure to keep scores high.
The reverse exception also holds. A score can be inflated with zero detected overlap, through paraphrased exposure or a training set drawn from the same weeks and accounts. That is why the report keeps detection and absence apart instead of letting a low hit count stand in for a clean result.
Signals that the claim needs narrowing or a second reviewer
Narrow the published claim, rather than adding caveats at the end, when any of these conditions holds. Each one means the absence half of the report rests on less than the reader will assume.
- A material share of the training mixture could not be scanned: name it and drop to rung 2 at most.
- Detector sensitivity was never measured on planted paraphrases: do not claim rung 3.
- More than a handful of items have no company, task family or date key: report them separately or exclude them.
- Reviewers disagreed on whether hits were template overlap or leakage: have a second evaluation lead adjudicate with the written rule.
- The evaluation set was shared with vendors or annotators without access logs: state the exposure window and treat it as unscanned.
- The benchmark will be reused after a new training run: set an expiry date and rerun the scan against the new mixture.
Requesting evaluation records that come with separation evidence
Most of the fields in this report are cheap to fill if the source data arrives with entity identifiers, event dates and task family labels, and expensive to reconstruct afterward. When you request records for an evaluation set, ask for those keys, for a list of known derivatives, and for a statement of where the records have previously been shared.
SourceX sources business records on request for AI teams and handles rights review, contracting, preparation and delivery, with datasets drawn from U.S. companies rather than held in stock. A buyer request can set out scope, volume, format, timing and licensing needs, including the grouping keys and dates your split design depends on.
Which part of a score drop after the cutoff is contamination? Nine entries recounted
Only the part other models do not share can be pinned on one model's own exposure, and on one public leaderboard that part is small for all but one entry. LiveCodeBench dates each coding problem, and its site says a model with training cutoff D can be scored on problems released after D to measure performance on unseen ones. We recounted the leaderboard's default data file at commit f186ab5 (August 1, 2025): 28 model entries, each scored on the same 1,055 problems dated May 8, 2023 to April 7, 2025.
First finding: the cutoff itself needs diligence. Nineteen of the 28 entries carry one value, April 30, 2023, eight days before the first problem, including identifiers that name 2025 dates, such as o3-mini-2025-01-31 and claude-sonnet-4-20250514. For them every problem counts as unseen, and in the default window (August 1, 2024 to May 1, 2025) the site's contamination flag marks none of the 28.
Second finding: a raw drop overstates contamination. The nine entries whose cutoffs fall inside the problem range all score lower afterward, by 7.4 to 29.4 points with both sides weighted to one difficulty mix. But the other 27 entries, split at the same dates, fall by a median of 10.0 to 12.3 points. Net of that shared fall, DeepSeek-V3 keeps 17.1 points; the other eight sit between -3.1 and 5.8.
Ask a company to fill in the same table for its own claim:
| Entry (cutoff field) | Weighted pass@1, before vs after | Drop (95% interval) | Median drop, other 27 entries | Drop specific to entry |
|---|---|---|---|---|
| DeepSeek-V3 (2024-06-30) | 62.7 vs 33.3 | 29.4 (24.5 to 34.3) | 12.3 | 17.1 |
| XBai-o4-medium (2024-04-01) | 90.4 vs 74.6 | 15.8 (11.5 to 19.9) | 10.0 | 5.8 |
| Claude-3.5-Sonnet-20241022 (2024-03-31) | 57.9 vs 42.1 | 15.7 (11.4 to 20.0) | 10.0 | 5.7 |
| Qwen3-235B-A22B (2024-06-30) | 90.2 vs 72.7 | 17.5 (13.4 to 21.6) | 12.3 | 5.2 |
| DeepSeek-R1-0528 (2024-06-30) | 92.5 vs 78.0 | 14.5 (11.1 to 17.6) | 12.3 | 2.2 |
| EXAONE-4.0-32B (2024-04-01) | 87.7 vs 77.1 | 10.6 (6.6 to 14.2) | 10.0 | 0.5 |
| OpenCodeReasoning-Nemotron-1.1-32B (2024-04-01) | 85.3 vs 74.8 | 10.5 (6.8 to 14.4) | 10.0 | 0.5 |
| OpenReasoning-Nemotron-32B (2024-04-01) | 87.7 vs 77.2 | 10.5 (7.0 to 14.0) | 10.0 | 0.4 |
| Grok-3-Mini, high effort (2024-03-01) | 82.8 vs 75.4 | 7.4 (2.6 to 12.2) | 10.5 | -3.1 |
When the after-cutoff score falls, did the settings change too?
Rule that out first, because settings alone move scores about as much as most entry-specific drops above. In the same file, four models appear at more than one setting under one model identifier, on all 1,055 problems: high instead of low reasoning effort added 7.1 points for o3-mini-2025-01-31 and 9.9 for O4-Mini, and the thinking variant added 8.1 for Claude-Opus-4 and 9.1 for Claude-Sonnet-4.
Comparing an older system on older items with a newer system on newer items changes model, settings and items at once. Ask for before and after scores with the model identifier, prompt template, sampling settings and reasoning budget held fixed. If the claim is that licensed records produced the gain, the matched test is one base model trained with and without those records, scored only on items dated after its cutoff and outside the records' own date range.
Which exposure routes must a company document, and how far does each cap the claim?
Each of seven routes by which a test item can reach a model caps the strongest sentence on the claim ladder above. Two need evidence beyond a scan of the company's own training files, and the scan has a known gap. Open pretraining corpora can be searched: the WIMBD project offers read-only search access, on request, to indexes over The Pile, OpenWebText, C4, Oscar, RedPajama v1 and Dolma. Closed-model APIs are the second: a review of 255 papers, mainly on GPT-3.5 and GPT-4, flagged indirect leakage, where data from users goes into improving the model. As for the scan, open-instruct's documentation warns that decontamination can leave matching training instances behind, because its search may not retrieve all matches, and suggests checking again afterward.
Rerun the headline number with these steps, then record what each route leaves standing:
- Freeze the claim: benchmark version and item count, model identifier, prompt template, sampling and reasoning settings, run date.
- Take each cutoff from the developer's own documentation; a default or copied value voids the split.
- Score items before and after the cutoff with identical settings, weighted to one difficulty mix.
- Split several comparison systems at the same date and subtract their median drop.
- Rescore without flagged or API-exposed items and report both numbers.
- Record who reran it, on which data snapshot, and when the result expires.
| Exposure route | What the company hands over | Reading that lowers the claim | Strongest rung left |
|---|---|---|---|
| Open pretraining corpora | Search results for test items against indexed open corpora, with query rule and date | Items found, or training corpora that no index covers | Rung 1 if found; rung 2 while corpora go unsearched |
| Closed-model APIs | Log of every outside API the test set was sent to, with dates and the data terms then in force | No log, or items sent under terms that allowed training on inputs | Exposed items count as unscanned; rung 2 for the rest |
| Own fine-tuning and synthetic data | Decontamination report: tool, n-gram size or threshold, search depth, recheck after removal | No recheck, or no recall test on planted variants | Rung 2; rung 3 only with a recall test |
| Derivatives: summaries, labels, graders | Lineage list of artifacts built from test items and where each went | A grader or reward model trained on test answers | Rung 2 until lineage is complete |
| Split design | Company and task family keys, event dates, share of items missing each | Keys missing for a material share | Rung 4 only with complete keys and lineage |
| Cutoff provenance | A dated source for every cutoff used in a date split | Placeholder or shared values, like 19 of 28 entries in the recount | No post-cutoff claim |
| Independent rerun | A rerun outside the team on post-cutoff items, same settings, with a same-date comparison | Only the company's own run, or settings that differ between runs | Any rung remains the company's own assertion |
What could still mislead after every route is checked?
The same-date comparison can mislead in three ways, each visible in the recount. The shared fall may be contamination too: if most systems saw the older problems, subtracting their median hides it, so an entry-specific drop understates exposure common to many models. Newer problems may be harder within one difficulty label, which weighting by label does not remove. And entries near 90 points before the cutoff have less room to fall, so drops are not comparable across score levels.
Coding contests are not business records, and clean results on every route show neither that a gain transfers to a buyer's work nor that anyone will pay for it. The same sheet applies to business records licensed through SourceX, which a company selects, approves and signs for before any data is delivered: records never published close the first route, not the other six. SourceX shares company and commercial metrics with investors directly.
Methods note: data from the LiveCodeBench site repository at commit f186ab5, file build/performances_generation.json, read October 27, 2026. Cutoff is each entry's release_date field; before means a problem date on or before it, after means later, with 50 or more rows required per side. Per-difficulty means are weighted by the entry's pooled mix; intervals use 2,000 stratified bootstrap resamples, seed 20261027; differences are computed before rounding. Not measured: any model's true cutoff, whether any problem was in training data, any business benchmark or SourceX dataset. Script: lcb_cutoff_split.py, held for reviewers.
This article is for information only and is not an offer to sell or a solicitation of an offer to buy securities. Any offering by SourceX is made only to verified accredited investors through its offering documents.
Sources for this section
- LiveCodeBench's site repository at commit f186ab5 (August 1, 2025) holds the default leaderboard data recounted here: 28 model entries scored on 1,055 dated problems, each entry's release_date cutoff field, and the code that flags an entry as contaminated when that date is on or after the window start; the site says dated problems let a model with cutoff D be scored on problems released after D. Source
- The WIMBD project offers read-only Elasticsearch access, through request forms, to indexes over The Pile, OpenWebText, C4, Oscar, RedPajama v1 and Dolma. Source
- Balloccu and colleagues (EACL 2024) reviewed 255 papers on data contamination in closed-source models, mainly GPT-3.5 and GPT-4, and flagged indirect data leaking, where models are improved using data from users. Source
- The open-instruct decontamination documentation warns that decontamination can leave matching training instances behind because Elasticsearch does not necessarily retrieve every matching document, and suggests checking for contamination again afterward. Source
Frequently asked questions
Does a clean contamination scan mean a benchmark is uncontaminated?
No. It means the methods you ran did not detect overlap on the fields and training corpora you checked. Paraphrased exposure, unscanned corpora and derivatives remain possible, so the report should state the method, scope and blind spots instead of claiming absence.
What is the difference between contamination detection and proof of absence?
Detection is a positive finding you can count and inspect, such as matched items or shared accounts. Proof of absence would require ruling out every path, including ones no detector covers, which scanning alone cannot do. Separation by design, through group and time splits, is the closest substitute.
Should contaminated items be removed or reported?
Report scores both with and without the flagged items so readers can see the effect. Remove items whose overlapping text carries the answer; keep and label items that only share boilerplate such as policies or form headers.
Can membership inference tests confirm a model never saw an item?
Not reliably. Published evaluations on large language models found these attacks barely beat random guessing in most settings, and apparent successes often reflected a difference in dates between the compared sets rather than true membership.
How often should a leakage scan be rerun?
Rerun it whenever the training mixture changes, the benchmark is reused for a new model, or the evaluation set is shared outside the original team. Give the report an expiry date so an old clean result is not cited for a new model.
Sources
- The GPT-4 technical report's contamination check sampled three 50-character substrings per evaluation example and notes it can produce false negatives and false positives. Source
- Rephrased or translated test items evade n-gram overlap decontamination, and a 13B model trained on rephrased test data reached scores comparable to much larger models. Source
- Membership inference attacks on Pile-trained models barely outperform random guessing in most settings; apparent successes were traced to distribution shift such as different time ranges. Source
- A survey found data leakage in ML-based research across 17 fields, proposed a taxonomy whose first type is lack of clean train-test separation, and proposed model info sheets. Source
- Near-duplicates and repeated substrings are common in language modeling datasets, and train-test overlap affected over 4% of the validation set of standard datasets. Source
- LiveCodeBench annotates problems with release dates so models can be evaluated only on problems released after their training cutoff. Source
- BIG-bench task files contain a canary string, not to be edited, intended to prevent benchmark tasks from leaking into web-scraped training data. Source
- GroupKFold keeps the same group out of both training and test sets; time series splits are recommended for time-dependent data. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.