Skip to content

AI uses for records

How to check a private dataset for public benchmark overlap

By SourceX Editorial · Updated

10 sections

Short answer

Check a private dataset against public benchmarks at four levels, not one: the text itself, the source documents and entities it came from, the time window, and the task family. Pin the benchmark versions and your dataset hash, run exact, n-gram and near-duplicate scans, join both sides on a source registry, adjudicate every flag by hand and publish a report someone else can rerun.

Evaluation split

Benchmark public private

This page covers how to check a private dataset for public benchmark overlap.Illustrative

Key takeaways

  • Benchmark overlap has four levels: text, source documents and entities, time, and task family.
  • Pin benchmark versions, dataset hashes and every threshold before the first run, or the result cannot be audited.
  • A source registry built from benchmark metadata finds shared repositories and filings that text scans miss.
  • Rephrased items evade n-gram checks, so near-duplicate candidates need human adjudication.
  • Split your own data by company, task family and time so the remaining test set measures skill.

How do you check a private dataset for overlap with public benchmarks?

Screen it in layers and write down every setting. A text scan finds copied items; a lineage join finds records built from the same filings, repositories or tickets a benchmark used; a time check finds records written after the benchmark was published; and a task-family check finds records that rehearse the benchmark's exact question shapes. Any one of these can make a model look better on that benchmark than it is.

Reproducibility is what turns a scan into evidence. If the benchmark version, the normalization rules, the n-gram length and the similarity threshold are not pinned, two teams running the same check will get different flag counts and neither result can be audited later.

This guide covers the screening workflow for a dataset you are about to license, train on or use as an evaluation set. It does not repeat how to remove contaminated rows from a pretraining corpus or how to keep a finished eval set from leaking; the linked pages handle those decisions.

Copied test items are the obvious case, but they are rarely the only overlap that makes a score misleading. The table lists the overlap levels the workflow checks, what each one inflates and the detector that catches it.

How do you check a private dataset for overlap with public benchmarks?
Overlap levelWhat it looks likeWhat it inflatesDetector
Verbatim itemA benchmark question and answer pasted into a ticket, wiki page or training noteMemorized answers on that exact itemHash match after normalization
Long shared spanA paragraph of a benchmark passage quoted inside a longer documentRecall of passages the benchmark asks about13-gram or suffix-array substring scan
Rephrased or translated itemThe same question reworded, reordered or translatedAnswers that survive surface changesNear-duplicate and embedding candidates, then human review
Shared source documentYour record and the benchmark both quote the same annual report, repository issue or manualFamiliarity with the benchmark's source contextJoin on a benchmark source registry
Shared entityRecords about the same company, product or codebase the benchmark draws onEntity-specific facts the model can reuseEntity key join with group splits
Post-publication recordA record written after the benchmark was released that discusses or solves its itemsExposure to published solutionsCreated-at date against benchmark release date
Task-family rehearsalMany records that follow the benchmark's exact template, even with new contentFormat fit rather than skillTask-family labels and template matching
Marker presentA benchmark canary string or item ID inside the datasetDirect evidence that benchmark files were ingestedString search for published canaries and IDs

Which public benchmarks belong on the screening list?

List every benchmark whose score will be reported or used for a decision about a model trained or evaluated with this dataset, then add the benchmarks built from the same kinds of source material. Fix the list before the first run and keep it in the manifest.

The second group is the one teams skip. A dataset of support tickets from a software company should be screened against coding benchmarks built from public issue trackers, even if nobody plans to report those scores, because the same repositories may appear on both sides. A dataset of financial statements should be screened against question-answering benchmarks built from public company reports.

Record the exact version of each benchmark. Popular benchmarks ship subsets, verified editions and new splits over time; the SWE-bench datasets guide, for instance, lists a full set, a Lite subset, a Verified subset, a Multimodal set and a Multilingual set. A scan against one edition says nothing about the others.

  • Reported benchmarks: every benchmark named in a model card, launch review or vendor comparison that this dataset feeds.
  • Source-sibling benchmarks: benchmarks built from the same repositories, filings, manuals, forums or websites your records cite.
  • Format-sibling benchmarks: benchmarks whose task template your records imitate, such as question, table and numeric answer.
  • Excluded benchmarks: anything left off the list, with the reason, so a reader can see the boundary of the claim.

The reproducible overlap screening workflow, step by step

Run the steps in order and save the output of each one under the run ID. A later reviewer should be able to take the manifest, the same inputs and the same code and get the same flags.

Steps

The reproducible overlap screening workflow, step by step

Pre-declare thresholds before the first run. Changing the n-gram length or the Jaccard cutoff after seeing the counts is the most common way a screening result stops being trustworthy.

Scan in both directions. Benchmark text inside your dataset matters for training; your dataset's text inside a public benchmark matters if you planned to use the dataset as a private evaluation set.

Keep the false positives. A record of what was flagged and cleared is what lets the next run reuse decisions instead of relitigating them. The signed report should record the items listed below.

Checklist · 7 items

  1. Run ID, date, and the people who ran and adjudicated the screen.
  2. Dataset version and file hashes; benchmark names, versions, commits or download dates and splits.
  3. Normalization config, n-gram length, Jaccard threshold, embedding model and neighbor count, all declared before the run.
  4. Counts per step: marker hits, exact matches, n-gram flags, near-duplicate candidates, confirmed contaminating items, shared-source and shared-entity records, post-release records.
  5. Decision log: each confirmed item, the action taken (drop, move to training only, keep with reason) and the reviewer.
  6. Split description: which group keys were enforced and which were unavailable.
  7. Known gaps: benchmarks not scanned, fields not inspected, hidden test sets and any access-history concerns.

0 of 7 ticked. Tick each item as you go.

The reproducible overlap screening workflow, step by step
StepInputMethodSaved output
1. Freeze inputsDataset export, benchmark listHash the dataset files; record each benchmark's version, commit or download date and split namesRun manifest with hashes and versions
2. NormalizeAll text fields on both sidesLowercase, collapse whitespace, strip markup, apply the same tokenizer to both sidesNormalization config file
3. Marker scanNormalized datasetSearch for published canary strings, benchmark item IDs and benchmark namesMarker hit list with record IDs
4. Exact matchNormalized items and recordsHash whole items, questions and answers separately and compareExact match table
5. N-gram scanTokenized textFlag any shared 13-gram; log the n used and whyPer-item overlap with matched spans
6. Near-duplicate scanShingled textMinHash LSH at a pre-declared Jaccard threshold; embedding nearest neighbors for rephrasingsCandidate pairs with scores
7. Lineage and entity joinBenchmark source registry, dataset metadataJoin on repository, document ID, company and product keysShared-source and shared-entity table
8. Time and task-family checkCreated-at dates, task labelsCompare record dates with benchmark release dates; map records to benchmark task familiesTemporal and family exposure counts
9. Adjudicate and reportAll flagsTwo reviewers label each flag as contaminating, related or false positiveSigned overlap report and decision log

Why the 13-gram rule is a starting point and not a verdict

The EleutherAI evaluation harness documents its decontamination method as a 13-gram check: a test document counts as contaminated if any 13-gram overlaps a training document. Its documentation notes that OpenAI's GPT-3 work used N values between 8 and 13 depending on the dataset, and that the harness chose 13 for simplicity. That gives you a defensible default, plus a reason to record whatever n you pick.

Short benchmark items need a smaller n or an exact-match pass, because a 20-token question may share no 13-gram with a lightly edited copy. Long passages need a substring method that scales; the suffix-array ExactSubstr tool released with Google Research's deduplication paper finds repeated substrings across very large text files.

N-grams also miss rewording. The paper Rethinking Benchmark and Contamination for Language Models with Rephrased Samples reports that paraphrased or translated benchmark items can pass n-gram decontamination. That is why step 6 adds MinHash candidates, which the datasketch library implements for estimating Jaccard similarity with LSH indexes, and embedding neighbors, then sends every candidate to a person.

Building a benchmark source registry: repositories, documents and dates

Text matching cannot see that your record and a benchmark item were built from the same source. A source registry can. For every benchmark on your list, extract the fields that identify where each item came from, then join your dataset's metadata against them.

Many benchmarks publish these fields. The SWE-bench datasets guide lists, for each instance, the repository, issue number, issue URL, pull request URL, base commit and a creation date. A private code dataset containing commits or tickets from those same repositories shares sources with the benchmark even when no line of text matches. FinQA's documentation says each example ID is composed of the original report name plus an index, which gives you a document key to join on for financial records.

Record the benchmark's own release date and each item's creation date in the registry. Records about a benchmark item created after its public release are the ones most likely to contain a discussion or a worked solution.

  • Registry columns: benchmark, version, item ID, source system (repository, filing, website, manual), source document ID, entity keys (company, product, codebase), item creation date, benchmark release date, task family.
  • Dataset columns to join: record ID, source system, document or ticket ID, entity keys, created-at date, task-family label.
  • Matching rules: exact key match on document IDs and repositories; normalized name match on companies and products, reviewed by a person; date comparison on created-at against release dates.

Where answers hide: attachments, merged fixes and derived notes

A training example, attachment or derivative can reveal the evaluation answer, and these leaks are easy to miss because the record that carries the answer rarely looks like the benchmark item. Check the places where answers end up after someone has solved a problem, not just where questions are stored.

At a glance

Where answers hide: attachments, merged fixes and derived notes

The FinQA maintainers' own changelog shows how subtle this gets: they fixed a table-to-text function bug that leaked label information and produced retriever results higher than the correct ones. A formatting choice, not a copied item, moved the score.

  • Attachments: spreadsheets, PDFs or screenshots attached to tickets where the benchmark's figure or table was worked through.
  • Merged fixes: the commit or pull request that resolved a benchmark issue, which is the gold patch in a code benchmark.
  • Derived summaries: AI-written notes, meeting summaries or wiki pages that restate a solved item in new words.
  • Computed fields: a column that stores the answer a benchmark asks the model to calculate, such as a precomputed ratio from a filing.
  • Discussion threads: chat or forum posts quoting the benchmark question and the accepted answer.

Splitting by company, task family and time so the score means something

Even when no item overlaps, a random split of your own dataset can let records about the same company, the same task template or the same week fall on both sides. The score then measures recall of that company or template rather than the skill you care about.

Group splits are the standard fix. The scikit-learn documentation describes GroupKFold as ensuring the same group is not represented in both the testing and training sets, and its time-series section warns that classical shuffled cross-validation on correlated observations gives poor estimates of generalization error; TimeSeriesSplit evaluates on later observations instead.

Use three group keys and record which ones each split honors: a company or customer key, a task-family label, and a time boundary with a gap so the last training week and the first test week do not describe the same events. When a key is unavailable, say so in the report rather than implying the split is clean.

Worked example (illustrative): screening a licensed support and code dataset

Suppose an evaluation team plans to license 40,000 support tickets and linked code changes from a software company and to use 2,000 of them as a private test set. All figures here are illustrative counts, not results from a real scan.

Steps 1 to 6 find 3 marker hits (a benchmark name pasted into internal notes), 11 exact matches and 46 near-duplicate candidates, of which reviewers confirm 19 as contaminating. The lineage join in step 7 finds that about 6% of tickets reference two open-source libraries that also appear as source repositories in a public code benchmark, although no text matches.

The decision: drop the 19 confirmed items and the 3 marker records, move every ticket touching the two shared repositories into training only, and rebuild the test set with a company-and-month group split. The exception is a ticket that cites a shared library only as a dependency version; reviewers keep it, because the record contains no code or discussion from the benchmark's issues, and they log the reason.

Limits of an overlap scan, and when a second reviewer should decide

A clean scan shows the absence of detectable overlap against the benchmarks you listed, in the versions you pinned. It does not show that a model under test never saw the benchmark elsewhere, and it cannot inspect hidden test sets: FinQA, for example, keeps a private test file without gold references for leaderboard scoring.

Narrow the claim or stop when any of these hold: the benchmark list was chosen after seeing results; the dataset has no reliable created-at dates; access logs show the benchmark was downloaded into the same workspace where the records were prepared; or the overlap sits in a field such as a computed answer that the team cannot inspect. In those cases an evaluation lead who did not run the scan should review the report before the dataset is used for a launch decision.

Overlap is a technical finding, not a rights finding. Whether you may use, publish or redistribute either side depends on the benchmark's license and your data agreement, which a separate review covers.

Sourcing held-out records with lineage you can check

The workflow only works if the records arrive with the metadata it joins on: source system, document IDs, entity keys and creation dates. When those fields are missing, the lineage and time checks cannot run and the screening falls back to text matching alone.

Because SourceX sources each dataset on request from U.S. businesses rather than from stock, an evaluation team can write those lineage fields into the request next to scope, volume, format, timing and licensing needs. No supplier delivers anything until it has approved the scope and terms and signed.

Frequently asked questions

What n-gram length should I use to check benchmark overlap?

The EleutherAI evaluation harness uses 13-grams and notes that GPT-3's work used values from 8 to 13 depending on the dataset. Use a smaller n or exact matching for short items such as single questions. Whatever you choose, declare it before the run and record it in the report.

Is a clean n-gram scan enough to call a dataset uncontaminated?

No. Rephrased and translated benchmark items can pass n-gram checks, and records can share source documents or entities with a benchmark without sharing text. Add near-duplicate candidates, a source registry join and a time check, and have people review the candidates.

What is a benchmark canary string and should I search for it?

A canary is a unique string a benchmark asks people to include wherever its data is published, so the data can be filtered from training corpora. BIG-bench publishes one in its task files. Searching for known canaries is cheap, but a missing canary proves little because copies are often edited.

How do group splits relate to benchmark overlap?

They handle the overlap inside your own data. If records about the same company, task template or week sit in both training and test sets, scores rise for reasons unrelated to skill. Splitting by company, task family and time keeps those groups on one side.

Who should review an overlap screening report?

An evaluation lead or dataset engineer who did not run the scan. They should be able to rerun it from the manifest and reach the same counts. Questions about whether either side may be used or published go to the people who handle the data agreement.

Sources

  • The EleutherAI harness flags a test document as contaminated on any 13-gram overlap with training data, noting GPT-3 used N between 8 and 13 depending on the dataset. Source
  • Each SWE-bench instance records its repository, issue number, issue and pull request URLs, base commit and creation date. Source
  • FinQA example IDs combine the original report name and an index, the maintainers fixed a bug that leaked label information into retriever results, and a private test file without references is used for leaderboard scoring. Source
  • BIG-bench publishes a canary GUID and asks that it be included wherever task data is published so the data can be kept out of training corpora. Source
  • The datasketch library provides MinHash for estimating Jaccard similarity and MinHash LSH indexes for threshold queries. Source
  • Google Research released a suffix-array ExactSubstr tool for finding repeated substrings across large text datasets. Source
  • scikit-learn's GroupKFold keeps the same group out of both training and test sets, and TimeSeriesSplit evaluates on later observations because shuffled splits misestimate generalization on correlated data. Source
  • Rephrased or translated benchmark items can pass n-gram decontamination checks. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies