Skip to content

Privacy and preparation

Benchmark decontamination tradeoffs: did dedup remove the hard cases?

By SourceX Editorial · Updated

9 sections

Short answer

Run a before and after coverage audit. Tag every evaluation and training record with its source, entity group, task family, time period and a difficulty signal before any deduplication or decontamination pass, then compare counts per stratum afterward. A stratum that loses a much larger share than the whole set, or removed items that are harder than the ones kept, means the filter is reshaping the benchmark rather than cleaning it.

Evaluation split

Benchmark decontamination

This page covers benchmark decontamination tradeoffs: did dedup remove the hard cases.Illustrative

Key takeaways

  • Deduplication and decontamination remove records unevenly, so report losses per stratum, not one overall rate.
  • Tag task family, entity, time window and lineage before the pass; afterward you cannot see what left.
  • Compare baseline error on removed and kept items to see whether the filter took the hard cases.
  • Splitting by customer, entity and time prevents most leakage without deleting anything.
  • Hand-review matched pairs: template twins with different outcomes belong in evaluation, not in the bin.
  • Publish the filter configuration and scan limits with every filtered score.

How do you detect difficult cases lost through deduplication?

You detect them by measuring what each filter removed, stratum by stratum, instead of reporting one overall removal rate. The audit compares the evaluation set and the training set before and after the pass across the dimensions that matter for the claim you want to make: which company or customer a record came from, which task family it belongs to, which time window it covers and how hard it was for a baseline model.

Two different jobs remove records here, and both can skew a benchmark. Training deduplication drops repeated or near-repeated records so a model does not see the same thing many times. Benchmark decontamination drops or flags evaluation items that overlap with training data so a score is not inflated by memorized answers. Lee and colleagues found that train-test overlap affected over 4% of the validation sets of standard language modeling datasets, which is why both steps are now routine.

The tradeoff is that neither filter is neutral. Matching methods are approximate: the datasketch documentation for MinHash LSH says plainly that queries return false positives and false negatives around the chosen threshold. Whatever a matcher catches is decided by how records are written, and in business data the records written most alike are often the ones where a small difference carries the whole answer.

Why removed records are rarely a random sample

Similarity filters remove records that share text, and shared text clusters in predictable places. If those places overlap with the cases your benchmark exists to test, the cleaned set gets easier and the score goes up for the wrong reason.

At a glance

Why removed records are rarely a random sample

Averages hide this. The SemDeDup authors report that semantic deduplication could remove 50% of a LAION subset with minimal loss in overall performance, a useful result for training efficiency. The FairDeDup authors then examined subgroups and found that deduplication reinforced existing biases on the FACET benchmark, and they modified the method to favor keeping samples from underrepresented concepts. The same pattern applies to task families and customers in operational data: a pass can look harmless in aggregate and still hollow out one slice.

  • Template twins: Zendesk macro replies, NetSuite invoice layouts and Jira issue templates make unrelated cases look alike, so the matcher removes the case whose only difference is the field that decided the outcome.
  • Same input, different outcome: two tickets quoting the same error message can end in a refund or a firmware fix. These pairs are hard by design, and a text matcher reads them as duplicates.
  • Entity concentration: one large customer or one site can contribute many similar records, so removal falls on that entity and quietly changes who the benchmark represents.
  • Event waves: an outage or a recall produces a burst of near-identical records in one week, so a filter can delete most of a time window.
  • Short and common text: a long n-gram rule cannot fire on a record shorter than the n-gram, and very common phrases may be excluded on purpose. The GPT-3 authors flagged overlap with 13-grams and ignored 13-grams that matched more than 10 training documents as common phrases, a sensible choice that also means heavy template text is never flagged.

Which splits prevent leakage before any filter runs?

Splitting by the unit that carries shared information prevents most leakage without deleting anything. If every record from one customer, one task family or one period sits entirely on one side of the train and test boundary, near-duplicates within that group cannot inflate the score, and the hard cases stay in the benchmark.

The scikit-learn cross-validation guide makes the general point: when the process that generated the data has a group structure, all samples in a validation fold should come from groups that are not represented in the paired training fold, and time-dependent data should be evaluated on future observations. Allamanis showed what happens when this is ignored in code models, where duplication across training and test corpora inflated reported performance by up to 100% in some cases. Kapoor and Narayanan found leakage errors in 329 papers across 17 fields that use machine learning and sorted them into eight types, so this is a common failure, not an edge case.

Which splits prevent leakage before any filter runs?
Split keyWhat leaks if ignoredBusiness record exampleHow to assign
Company or customerWriting style, product names and recurring issues of one accountAll Salesforce cases and emails from one clientHash the account ID; the whole account goes to one side
Entity such as asset, site or case IDRepeated facts about the same objectEvery ServiceTitan job on one HVAC unitGroup by asset or location ID before splitting
Task familyAnswer patterns shared by one workflowBilling disputes versus shipping exceptionsHold out whole families when testing transfer to new tasks
Time windowLater records that mention the resolution of earlier onesTickets before and after a policy changeCut at a date; evaluation comes after training
Source lineageDerived text that restates the originalA knowledge base article written from a resolved ticketKeep a record and its derivatives on the same side

What should the before and after coverage audit include?

The audit is one table per filter run, with one row per stratum and enough columns to show size, loss, difficulty and a decision. Build it from metadata you attach before the pass, because after removal you can no longer see what was there.

Record the filter configuration with the table: method, threshold, n-gram length or embedding model, the fields compared and the version of each dataset. A table without its configuration cannot be reproduced or compared with the next run.

What should the before and after coverage audit include?
ColumnWhat it recordsHow to compute itWhy it matters
StratumTask family, entity group, time window or sourceFrom tags attached before the passLosses are only visible per slice
Before and after countsRecords in the stratum before and after the passCount by stratum in both versionsShows whether a slice became too small to score
Removal shareFraction of the stratum removedRemoved divided by before countCompared with the overall removal share
Removal ratioHow much harder the filter hit this sliceStratum removal share divided by overall shareA ratio well above 1 flags a skewed slice
Difficulty of removed versus keptBaseline error on removed items and on kept itemsScore a fixed baseline model on both groupsShows whether the hard cases left
Pair review resultWhat the flagged pairs really areHand labels on a sample of matched pairsSeparates true duplicates from template twins
DecisionKeep, move, re-split or report with a caveatReviewer sign-off with a reasonTurns the audit into an action

The removal-ratio and difficulty tests, step by step

Run two tests on every flagged stratum: does the filter remove this slice out of proportion, and are the removed items harder than the ones kept. Either signal alone is a reason to look at pairs by hand; both together mean the benchmark has changed meaning.

Set the flag thresholds before you see the results and write them down. A rule such as removal ratio above 2 or a slice falling below the smallest size you are willing to score is a reasonable starting point, but the right values depend on how many items each stratum has and how precise the reported score needs to be.

Checklist · 8 items

  1. Freeze the before versions of the training and evaluation sets and record their versions or checksums.
  2. Attach stratum tags to every record: task family, entity group, time window, source lineage.
  3. Score a fixed baseline model on the full evaluation set and keep per-item results.
  4. Run the filter and log every removed or flagged item with the record it matched.
  5. Fill the audit table and compute the removal ratio for each stratum.
  6. Compare baseline error on removed items with baseline error on kept items, overall and per flagged stratum.
  7. Hand-label a sample of matched pairs in each flagged stratum: true duplicate, template twin, same input with a different outcome, same entity on a different task, or false match.
  8. Decide per stratum and record the reason next to the score you publish.

0 of 8 ticked. Tick each item as you go.

Can a quoted reply, attachment or help article leak a test answer into training?

Yes, and a text matcher run on the main body of each record will often miss it. The answer to an evaluation item can travel through quoted email replies, attached PDFs, screenshots, summaries, knowledge base articles written after the case closed and later records that mention the outcome.

At a glance

Can a quoted reply, attachment or help article leak a test answer into training

Treat the leak check as an evidence checklist rather than a single similarity score. Each item below should have a named owner and a written result before the evaluation set is frozen.

  • Source lineage: which system and export produced each record, and whether any record was derived from another.
  • Entity groups: customer, account, asset or site IDs used to keep related records on one side.
  • Task families: the workflow label for each item, so whole families can be held out.
  • Temporal boundaries: the cut date, and a search of training data after that date for case IDs from the evaluation set.
  • Near-duplicate checks: method, threshold and fields compared, including attachments and quoted text.
  • Access history: who saw the evaluation items and whether any were pasted into prompts, tickets or documents that later entered training data.
  • Scan limitations: what the checks could not see, such as images without OCR or records too short for the n-gram length.

Worked example: a support-ticket benchmark after a near-duplicate pass

This example is illustrative; the counts and percentages are invented to show the method, not drawn from a real dataset. A team builds a 2,000-ticket evaluation set across six task families from a help desk history, then runs a near-duplicate filter against the training pool. Overall, 9% of evaluation tickets are flagged.

The audit shows the loss is concentrated. Billing dispute escalations lose 31% of their tickets, a removal ratio of about 3.4, while password resets lose 4%. A fixed baseline model got 41% of the removed billing tickets wrong against 22% of the kept ones, so the filter removed the harder half of the slice. Hand review of a sample of the matched pairs finds that most are template twins: agents used the same macro, but the disputed line item and the final credit decision differ.

The fix is structural. The team re-splits by customer account and by month, which moves the template twins to one side instead of deleting them, and reruns the filter. Billing removal drops to near the overall rate. The exception is password resets, where most flagged pairs are true duplicates generated by one automated workflow; there the removal is correct, and the family is reported with its smaller count and a note that it measures little beyond template recognition.

Keep, move or report: deciding what to do with a flagged stratum

Pick the action from what the pair review found, not from the removal share alone. The same removal ratio can mean the filter worked or that it gutted a slice, and only the pairs tell you which.

Checklist · 3 items

  1. Bring in a dataset engineer when matches cluster in attachments, OCR text or code blocks your matcher was not built for.
  2. Bring in the evaluation lead when a single stratum drives the headline score change.
  3. Pause the benchmark release when access history shows evaluation items were pasted into systems that feed training data.

0 of 3 ticked. Tick each item as you go.

Keep, move or report: deciding what to do with a flagged stratum
What the review findsActionWhat to report
True duplicates of training recordsRemove from evaluation or from trainingCount removed and the method used
Template twins with different outcomesKeep in evaluation; move twins out of training by re-splittingThe split key and the before and after counts
Same entity, different taskRe-split by entity rather than deleteEntity-level split and how many groups each side holds
Stratum below the size you can scoreMerge with a related family or drop it from headline resultsThe excluded slice and why
Removed items much harder than keptScore both the full and filtered sets and publish bothThe gap between the two scores and the configuration
Attachments or derivatives not scannedNarrow the claim to what was checked or extend the scanThe scan limitation in plain words

What to ask for when you license records for evaluation

Most of this audit depends on metadata that only the record owner has: account IDs, task labels, timestamps and lineage between a ticket and the article written from it. Asking for those fields at the request stage costs far less than reconstructing them after delivery, and it lets you split by group before any filter runs.

Split keys and lineage fields can be written into the scope from day one when an evaluation team requests records through SourceX, which sources them from U.S. businesses on demand and holds no inventory. The request form takes scope, volume, format, timing and licensing needs, and SourceX handles the rights review, contracting, preparation and delivery between the business and the AI developer.

Frequently asked questions

What is benchmark decontamination?

It is the process of finding evaluation items that also appear, exactly or nearly, in a model's training data and then removing, flagging or separately scoring them. The goal is a score that reflects generalization rather than recall. It usually relies on n-gram overlap, hashing or embedding similarity, so it inherits the blind spots of those methods.

Is deduplication the same as decontamination?

No. Deduplication removes repeated records inside a training set to save compute and reduce memorization. Decontamination compares the evaluation set with the training set so the score is not inflated by overlap. They use similar matching tools, and both should be audited for what they remove.

What similarity threshold should I use?

There is no universal value; it depends on how your records are written and how similar legitimate distinct cases are. Approximate matchers such as MinHash LSH also return false positives and misses near any threshold. Choose a threshold, then review matched pairs near it by hand and record the result with the configuration.

How many matched pairs should be reviewed by hand?

Review every pair in small flagged strata and a random sample in large ones, and record the sample size next to the result. The aim is to see each pair type often enough to choose an action, not to estimate a precise error rate. Add pairs near the threshold, where misclassification is most likely.

Does a clean-subset score prove there is no contamination?

No. A clean subset, such as the low-contamination subset the Llama 2 authors defined by token n-gram matches, shows how a model scores on items one matcher did not flag. It says nothing about paraphrases, translated items or answers that leaked through attachments. Report the matcher, its threshold and its scan limits with any clean-subset score.

Sources

  • Lee et al. found that train-test overlap affected over 4% of the validation sets of standard language modeling datasets and released exact-substring and near-duplicate deduplication methods. Source
  • The datasketch MinHash LSH documentation states that LSH queries return false positives and false negatives relative to the Jaccard threshold. Source
  • The scikit-learn guide recommends group-wise cross-validation when data has a group structure and evaluating time series on future observations. Source
  • Allamanis reports that code duplication across training and test corpora inflated reported performance of code models by up to 100% in some cases. Source
  • Kapoor and Narayanan found leakage errors in 17 fields affecting 329 papers and built a taxonomy of 8 leakage types. Source
  • SemDeDup reports removing 50% of a LAION subset with minimal performance loss using embedding-based semantic deduplication. Source
  • FairDeDup reports that deduplication reinforced existing biases on FACET and modifies SemDeDup to favor keeping underrepresented concepts. Source
  • The Llama 2 paper defines contamination by token n-gram matches and reports results on clean and dirty evaluation subsets. Source
  • The GPT-3 paper used 13-gram overlap to flag contaminated evaluation examples and ignored 13-grams that matched more than 10 training documents as common phrases. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies