AI uses for records
How to check a private dataset for public benchmark overlap
By SourceX Editorial · Updated
10 sections
Short answer
Check a private dataset against public benchmarks at four levels, not one: the text itself, the source documents and entities it came from, the time window, and the task family. Pin the benchmark versions and your dataset hash, run exact, n-gram and near-duplicate scans, join both sides on a source registry, adjudicate every flag by hand and publish a report someone else can rerun.
Evaluation split
Benchmark public private
Key takeaways
- Benchmark overlap has four levels: text, source documents and entities, time, and task family.
- Pin benchmark versions, dataset hashes and every threshold before the first run, or the result cannot be audited.
- A source registry built from benchmark metadata finds shared repositories and filings that text scans miss.
- Rephrased items evade n-gram checks, so near-duplicate candidates need human adjudication.
- Split your own data by company, task family and time so the remaining test set measures skill.
How do you check a private dataset for overlap with public benchmarks?
Screen it in layers and write down every setting. A text scan finds copied items; a lineage join finds records built from the same filings, repositories or tickets a benchmark used; a time check finds records written after the benchmark was published; and a task-family check finds records that rehearse the benchmark's exact question shapes. Any one of these can make a model look better on that benchmark than it is.
Reproducibility is what turns a scan into evidence. If the benchmark version, the normalization rules, the n-gram length and the similarity threshold are not pinned, two teams running the same check will get different flag counts and neither result can be audited later.
This guide covers the screening workflow for a dataset you are about to license, train on or use as an evaluation set. It does not repeat how to remove contaminated rows from a pretraining corpus or how to keep a finished eval set from leaking; the linked pages handle those decisions.
Copied test items are the obvious case, but they are rarely the only overlap that makes a score misleading. The table lists the overlap levels the workflow checks, what each one inflates and the detector that catches it.
| Overlap level | What it looks like | What it inflates | Detector |
|---|---|---|---|
| Verbatim item | A benchmark question and answer pasted into a ticket, wiki page or training note | Memorized answers on that exact item | Hash match after normalization |
| Long shared span | A paragraph of a benchmark passage quoted inside a longer document | Recall of passages the benchmark asks about | 13-gram or suffix-array substring scan |
| Rephrased or translated item | The same question reworded, reordered or translated | Answers that survive surface changes | Near-duplicate and embedding candidates, then human review |
| Shared source document | Your record and the benchmark both quote the same annual report, repository issue or manual | Familiarity with the benchmark's source context | Join on a benchmark source registry |
| Shared entity | Records about the same company, product or codebase the benchmark draws on | Entity-specific facts the model can reuse | Entity key join with group splits |
| Post-publication record | A record written after the benchmark was released that discusses or solves its items | Exposure to published solutions | Created-at date against benchmark release date |
| Task-family rehearsal | Many records that follow the benchmark's exact template, even with new content | Format fit rather than skill | Task-family labels and template matching |
| Marker present | A benchmark canary string or item ID inside the dataset | Direct evidence that benchmark files were ingested | String search for published canaries and IDs |
Which public benchmarks belong on the screening list?
List every benchmark whose score will be reported or used for a decision about a model trained or evaluated with this dataset, then add the benchmarks built from the same kinds of source material. Fix the list before the first run and keep it in the manifest.
The second group is the one teams skip. A dataset of support tickets from a software company should be screened against coding benchmarks built from public issue trackers, even if nobody plans to report those scores, because the same repositories may appear on both sides. A dataset of financial statements should be screened against question-answering benchmarks built from public company reports.
Record the exact version of each benchmark. Popular benchmarks ship subsets, verified editions and new splits over time; the SWE-bench datasets guide, for instance, lists a full set, a Lite subset, a Verified subset, a Multimodal set and a Multilingual set. A scan against one edition says nothing about the others.
- Reported benchmarks: every benchmark named in a model card, launch review or vendor comparison that this dataset feeds.
- Source-sibling benchmarks: benchmarks built from the same repositories, filings, manuals, forums or websites your records cite.
- Format-sibling benchmarks: benchmarks whose task template your records imitate, such as question, table and numeric answer.
- Excluded benchmarks: anything left off the list, with the reason, so a reader can see the boundary of the claim.
The reproducible overlap screening workflow, step by step
Run the steps in order and save the output of each one under the run ID. A later reviewer should be able to take the manifest, the same inputs and the same code and get the same flags.
Steps
The reproducible overlap screening workflow, step by step
Pre-declare thresholds before the first run. Changing the n-gram length or the Jaccard cutoff after seeing the counts is the most common way a screening result stops being trustworthy.
Scan in both directions. Benchmark text inside your dataset matters for training; your dataset's text inside a public benchmark matters if you planned to use the dataset as a private evaluation set.
Keep the false positives. A record of what was flagged and cleared is what lets the next run reuse decisions instead of relitigating them. The signed report should record the items listed below.
Checklist · 7 items
- Run ID, date, and the people who ran and adjudicated the screen.
- Dataset version and file hashes; benchmark names, versions, commits or download dates and splits.
- Normalization config, n-gram length, Jaccard threshold, embedding model and neighbor count, all declared before the run.
- Counts per step: marker hits, exact matches, n-gram flags, near-duplicate candidates, confirmed contaminating items, shared-source and shared-entity records, post-release records.
- Decision log: each confirmed item, the action taken (drop, move to training only, keep with reason) and the reviewer.
- Split description: which group keys were enforced and which were unavailable.
- Known gaps: benchmarks not scanned, fields not inspected, hidden test sets and any access-history concerns.
0 of 7 ticked. Tick each item as you go.
| Step | Input | Method | Saved output |
|---|---|---|---|
| 1. Freeze inputs | Dataset export, benchmark list | Hash the dataset files; record each benchmark's version, commit or download date and split names | Run manifest with hashes and versions |
| 2. Normalize | All text fields on both sides | Lowercase, collapse whitespace, strip markup, apply the same tokenizer to both sides | Normalization config file |
| 3. Marker scan | Normalized dataset | Search for published canary strings, benchmark item IDs and benchmark names | Marker hit list with record IDs |
| 4. Exact match | Normalized items and records | Hash whole items, questions and answers separately and compare | Exact match table |
| 5. N-gram scan | Tokenized text | Flag any shared 13-gram; log the n used and why | Per-item overlap with matched spans |
| 6. Near-duplicate scan | Shingled text | MinHash LSH at a pre-declared Jaccard threshold; embedding nearest neighbors for rephrasings | Candidate pairs with scores |
| 7. Lineage and entity join | Benchmark source registry, dataset metadata | Join on repository, document ID, company and product keys | Shared-source and shared-entity table |
| 8. Time and task-family check | Created-at dates, task labels | Compare record dates with benchmark release dates; map records to benchmark task families | Temporal and family exposure counts |
| 9. Adjudicate and report | All flags | Two reviewers label each flag as contaminating, related or false positive | Signed overlap report and decision log |
Why the 13-gram rule is a starting point and not a verdict
The EleutherAI evaluation harness documents its decontamination method as a 13-gram check: a test document counts as contaminated if any 13-gram overlaps a training document. Its documentation notes that OpenAI's GPT-3 work used N values between 8 and 13 depending on the dataset, and that the harness chose 13 for simplicity. That gives you a defensible default, plus a reason to record whatever n you pick.
Short benchmark items need a smaller n or an exact-match pass, because a 20-token question may share no 13-gram with a lightly edited copy. Long passages need a substring method that scales; the suffix-array ExactSubstr tool released with Google Research's deduplication paper finds repeated substrings across very large text files.
N-grams also miss rewording. The paper Rethinking Benchmark and Contamination for Language Models with Rephrased Samples reports that paraphrased or translated benchmark items can pass n-gram decontamination. That is why step 6 adds MinHash candidates, which the datasketch library implements for estimating Jaccard similarity with LSH indexes, and embedding neighbors, then sends every candidate to a person.
Building a benchmark source registry: repositories, documents and dates
Text matching cannot see that your record and a benchmark item were built from the same source. A source registry can. For every benchmark on your list, extract the fields that identify where each item came from, then join your dataset's metadata against them.
Many benchmarks publish these fields. The SWE-bench datasets guide lists, for each instance, the repository, issue number, issue URL, pull request URL, base commit and a creation date. A private code dataset containing commits or tickets from those same repositories shares sources with the benchmark even when no line of text matches. FinQA's documentation says each example ID is composed of the original report name plus an index, which gives you a document key to join on for financial records.
Record the benchmark's own release date and each item's creation date in the registry. Records about a benchmark item created after its public release are the ones most likely to contain a discussion or a worked solution.
- Registry columns: benchmark, version, item ID, source system (repository, filing, website, manual), source document ID, entity keys (company, product, codebase), item creation date, benchmark release date, task family.
- Dataset columns to join: record ID, source system, document or ticket ID, entity keys, created-at date, task-family label.
- Matching rules: exact key match on document IDs and repositories; normalized name match on companies and products, reviewed by a person; date comparison on created-at against release dates.
Where answers hide: attachments, merged fixes and derived notes
A training example, attachment or derivative can reveal the evaluation answer, and these leaks are easy to miss because the record that carries the answer rarely looks like the benchmark item. Check the places where answers end up after someone has solved a problem, not just where questions are stored.
At a glance
Where answers hide: attachments, merged fixes and derived notes
The FinQA maintainers' own changelog shows how subtle this gets: they fixed a table-to-text function bug that leaked label information and produced retriever results higher than the correct ones. A formatting choice, not a copied item, moved the score.
- Attachments: spreadsheets, PDFs or screenshots attached to tickets where the benchmark's figure or table was worked through.
- Merged fixes: the commit or pull request that resolved a benchmark issue, which is the gold patch in a code benchmark.
- Derived summaries: AI-written notes, meeting summaries or wiki pages that restate a solved item in new words.
- Computed fields: a column that stores the answer a benchmark asks the model to calculate, such as a precomputed ratio from a filing.
- Discussion threads: chat or forum posts quoting the benchmark question and the accepted answer.
Splitting by company, task family and time so the score means something
Even when no item overlaps, a random split of your own dataset can let records about the same company, the same task template or the same week fall on both sides. The score then measures recall of that company or template rather than the skill you care about.
Group splits are the standard fix. The scikit-learn documentation describes GroupKFold as ensuring the same group is not represented in both the testing and training sets, and its time-series section warns that classical shuffled cross-validation on correlated observations gives poor estimates of generalization error; TimeSeriesSplit evaluates on later observations instead.
Use three group keys and record which ones each split honors: a company or customer key, a task-family label, and a time boundary with a gap so the last training week and the first test week do not describe the same events. When a key is unavailable, say so in the report rather than implying the split is clean.
Worked example (illustrative): screening a licensed support and code dataset
Suppose an evaluation team plans to license 40,000 support tickets and linked code changes from a software company and to use 2,000 of them as a private test set. All figures here are illustrative counts, not results from a real scan.
Steps 1 to 6 find 3 marker hits (a benchmark name pasted into internal notes), 11 exact matches and 46 near-duplicate candidates, of which reviewers confirm 19 as contaminating. The lineage join in step 7 finds that about 6% of tickets reference two open-source libraries that also appear as source repositories in a public code benchmark, although no text matches.
The decision: drop the 19 confirmed items and the 3 marker records, move every ticket touching the two shared repositories into training only, and rebuild the test set with a company-and-month group split. The exception is a ticket that cites a shared library only as a dependency version; reviewers keep it, because the record contains no code or discussion from the benchmark's issues, and they log the reason.
Limits of an overlap scan, and when a second reviewer should decide
A clean scan shows the absence of detectable overlap against the benchmarks you listed, in the versions you pinned. It does not show that a model under test never saw the benchmark elsewhere, and it cannot inspect hidden test sets: FinQA, for example, keeps a private test file without gold references for leaderboard scoring.
Narrow the claim or stop when any of these hold: the benchmark list was chosen after seeing results; the dataset has no reliable created-at dates; access logs show the benchmark was downloaded into the same workspace where the records were prepared; or the overlap sits in a field such as a computed answer that the team cannot inspect. In those cases an evaluation lead who did not run the scan should review the report before the dataset is used for a launch decision.
Overlap is a technical finding, not a rights finding. Whether you may use, publish or redistribute either side depends on the benchmark's license and your data agreement, which a separate review covers.
Sourcing held-out records with lineage you can check
The workflow only works if the records arrive with the metadata it joins on: source system, document IDs, entity keys and creation dates. When those fields are missing, the lineage and time checks cannot run and the screening falls back to text matching alone.
Because SourceX sources each dataset on request from U.S. businesses rather than from stock, an evaluation team can write those lineage fields into the request next to scope, volume, format, timing and licensing needs. No supplier delivers anything until it has approved the scope and terms and signed.
Frequently asked questions
What n-gram length should I use to check benchmark overlap?
The EleutherAI evaluation harness uses 13-grams and notes that GPT-3's work used values from 8 to 13 depending on the dataset. Use a smaller n or exact matching for short items such as single questions. Whatever you choose, declare it before the run and record it in the report.
Is a clean n-gram scan enough to call a dataset uncontaminated?
No. Rephrased and translated benchmark items can pass n-gram checks, and records can share source documents or entities with a benchmark without sharing text. Add near-duplicate candidates, a source registry join and a time check, and have people review the candidates.
What is a benchmark canary string and should I search for it?
A canary is a unique string a benchmark asks people to include wherever its data is published, so the data can be filtered from training corpora. BIG-bench publishes one in its task files. Searching for known canaries is cheap, but a missing canary proves little because copies are often edited.
How do group splits relate to benchmark overlap?
They handle the overlap inside your own data. If records about the same company, task template or week sit in both training and test sets, scores rise for reasons unrelated to skill. Splitting by company, task family and time keeps those groups on one side.
Who should review an overlap screening report?
An evaluation lead or dataset engineer who did not run the scan. They should be able to rerun it from the manifest and reach the same counts. Questions about whether either side may be used or published go to the people who handle the data agreement.
Sources
- The EleutherAI harness flags a test document as contaminated on any 13-gram overlap with training data, noting GPT-3 used N between 8 and 13 depending on the dataset. Source
- Each SWE-bench instance records its repository, issue number, issue and pull request URLs, base commit and creation date. Source
- FinQA example IDs combine the original report name and an index, the maintainers fixed a bug that leaked label information into retriever results, and a private test file without references is used for leaderboard scoring. Source
- BIG-bench publishes a canary GUID and asks that it be included wherever task data is published so the data can be kept out of training corpora. Source
- The datasketch library provides MinHash for estimating Jaccard similarity and MinHash LSH indexes for threshold queries. Source
- Google Research released a suffix-array ExactSubstr tool for finding repeated substrings across large text datasets. Source
- scikit-learn's GroupKFold keeps the same group out of both training and test sets, and TimeSeriesSplit evaluates on later observations because shuffled splits misestimate generalization on correlated data. Source
- Rephrased or translated benchmark items can pass n-gram decontamination checks. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.