Privacy and preparation
Benchmark decontamination tradeoffs: did dedup remove the hard cases?
By SourceX Editorial · Updated
9 sections
Short answer
Run a before and after coverage audit. Tag every evaluation and training record with its source, entity group, task family, time period and a difficulty signal before any deduplication or decontamination pass, then compare counts per stratum afterward. A stratum that loses a much larger share than the whole set, or removed items that are harder than the ones kept, means the filter is reshaping the benchmark rather than cleaning it.
Evaluation split
Benchmark decontamination
Key takeaways
- Deduplication and decontamination remove records unevenly, so report losses per stratum, not one overall rate.
- Tag task family, entity, time window and lineage before the pass; afterward you cannot see what left.
- Compare baseline error on removed and kept items to see whether the filter took the hard cases.
- Splitting by customer, entity and time prevents most leakage without deleting anything.
- Hand-review matched pairs: template twins with different outcomes belong in evaluation, not in the bin.
- Publish the filter configuration and scan limits with every filtered score.
How do you detect difficult cases lost through deduplication?
You detect them by measuring what each filter removed, stratum by stratum, instead of reporting one overall removal rate. The audit compares the evaluation set and the training set before and after the pass across the dimensions that matter for the claim you want to make: which company or customer a record came from, which task family it belongs to, which time window it covers and how hard it was for a baseline model.
Two different jobs remove records here, and both can skew a benchmark. Training deduplication drops repeated or near-repeated records so a model does not see the same thing many times. Benchmark decontamination drops or flags evaluation items that overlap with training data so a score is not inflated by memorized answers. Lee and colleagues found that train-test overlap affected over 4% of the validation sets of standard language modeling datasets, which is why both steps are now routine.
The tradeoff is that neither filter is neutral. Matching methods are approximate: the datasketch documentation for MinHash LSH says plainly that queries return false positives and false negatives around the chosen threshold. Whatever a matcher catches is decided by how records are written, and in business data the records written most alike are often the ones where a small difference carries the whole answer.
Why removed records are rarely a random sample
Similarity filters remove records that share text, and shared text clusters in predictable places. If those places overlap with the cases your benchmark exists to test, the cleaned set gets easier and the score goes up for the wrong reason.
At a glance
Why removed records are rarely a random sample
Averages hide this. The SemDeDup authors report that semantic deduplication could remove 50% of a LAION subset with minimal loss in overall performance, a useful result for training efficiency. The FairDeDup authors then examined subgroups and found that deduplication reinforced existing biases on the FACET benchmark, and they modified the method to favor keeping samples from underrepresented concepts. The same pattern applies to task families and customers in operational data: a pass can look harmless in aggregate and still hollow out one slice.
- Template twins: Zendesk macro replies, NetSuite invoice layouts and Jira issue templates make unrelated cases look alike, so the matcher removes the case whose only difference is the field that decided the outcome.
- Same input, different outcome: two tickets quoting the same error message can end in a refund or a firmware fix. These pairs are hard by design, and a text matcher reads them as duplicates.
- Entity concentration: one large customer or one site can contribute many similar records, so removal falls on that entity and quietly changes who the benchmark represents.
- Event waves: an outage or a recall produces a burst of near-identical records in one week, so a filter can delete most of a time window.
- Short and common text: a long n-gram rule cannot fire on a record shorter than the n-gram, and very common phrases may be excluded on purpose. The GPT-3 authors flagged overlap with 13-grams and ignored 13-grams that matched more than 10 training documents as common phrases, a sensible choice that also means heavy template text is never flagged.
Which splits prevent leakage before any filter runs?
Splitting by the unit that carries shared information prevents most leakage without deleting anything. If every record from one customer, one task family or one period sits entirely on one side of the train and test boundary, near-duplicates within that group cannot inflate the score, and the hard cases stay in the benchmark.
The scikit-learn cross-validation guide makes the general point: when the process that generated the data has a group structure, all samples in a validation fold should come from groups that are not represented in the paired training fold, and time-dependent data should be evaluated on future observations. Allamanis showed what happens when this is ignored in code models, where duplication across training and test corpora inflated reported performance by up to 100% in some cases. Kapoor and Narayanan found leakage errors in 329 papers across 17 fields that use machine learning and sorted them into eight types, so this is a common failure, not an edge case.
| Split key | What leaks if ignored | Business record example | How to assign |
|---|---|---|---|
| Company or customer | Writing style, product names and recurring issues of one account | All Salesforce cases and emails from one client | Hash the account ID; the whole account goes to one side |
| Entity such as asset, site or case ID | Repeated facts about the same object | Every ServiceTitan job on one HVAC unit | Group by asset or location ID before splitting |
| Task family | Answer patterns shared by one workflow | Billing disputes versus shipping exceptions | Hold out whole families when testing transfer to new tasks |
| Time window | Later records that mention the resolution of earlier ones | Tickets before and after a policy change | Cut at a date; evaluation comes after training |
| Source lineage | Derived text that restates the original | A knowledge base article written from a resolved ticket | Keep a record and its derivatives on the same side |
What should the before and after coverage audit include?
The audit is one table per filter run, with one row per stratum and enough columns to show size, loss, difficulty and a decision. Build it from metadata you attach before the pass, because after removal you can no longer see what was there.
Record the filter configuration with the table: method, threshold, n-gram length or embedding model, the fields compared and the version of each dataset. A table without its configuration cannot be reproduced or compared with the next run.
| Column | What it records | How to compute it | Why it matters |
|---|---|---|---|
| Stratum | Task family, entity group, time window or source | From tags attached before the pass | Losses are only visible per slice |
| Before and after counts | Records in the stratum before and after the pass | Count by stratum in both versions | Shows whether a slice became too small to score |
| Removal share | Fraction of the stratum removed | Removed divided by before count | Compared with the overall removal share |
| Removal ratio | How much harder the filter hit this slice | Stratum removal share divided by overall share | A ratio well above 1 flags a skewed slice |
| Difficulty of removed versus kept | Baseline error on removed items and on kept items | Score a fixed baseline model on both groups | Shows whether the hard cases left |
| Pair review result | What the flagged pairs really are | Hand labels on a sample of matched pairs | Separates true duplicates from template twins |
| Decision | Keep, move, re-split or report with a caveat | Reviewer sign-off with a reason | Turns the audit into an action |
The removal-ratio and difficulty tests, step by step
Run two tests on every flagged stratum: does the filter remove this slice out of proportion, and are the removed items harder than the ones kept. Either signal alone is a reason to look at pairs by hand; both together mean the benchmark has changed meaning.
Set the flag thresholds before you see the results and write them down. A rule such as removal ratio above 2 or a slice falling below the smallest size you are willing to score is a reasonable starting point, but the right values depend on how many items each stratum has and how precise the reported score needs to be.
Checklist · 8 items
- Freeze the before versions of the training and evaluation sets and record their versions or checksums.
- Attach stratum tags to every record: task family, entity group, time window, source lineage.
- Score a fixed baseline model on the full evaluation set and keep per-item results.
- Run the filter and log every removed or flagged item with the record it matched.
- Fill the audit table and compute the removal ratio for each stratum.
- Compare baseline error on removed items with baseline error on kept items, overall and per flagged stratum.
- Hand-label a sample of matched pairs in each flagged stratum: true duplicate, template twin, same input with a different outcome, same entity on a different task, or false match.
- Decide per stratum and record the reason next to the score you publish.
0 of 8 ticked. Tick each item as you go.
Can a quoted reply, attachment or help article leak a test answer into training?
Yes, and a text matcher run on the main body of each record will often miss it. The answer to an evaluation item can travel through quoted email replies, attached PDFs, screenshots, summaries, knowledge base articles written after the case closed and later records that mention the outcome.
At a glance
Can a quoted reply, attachment or help article leak a test answer into training
Treat the leak check as an evidence checklist rather than a single similarity score. Each item below should have a named owner and a written result before the evaluation set is frozen.
- Source lineage: which system and export produced each record, and whether any record was derived from another.
- Entity groups: customer, account, asset or site IDs used to keep related records on one side.
- Task families: the workflow label for each item, so whole families can be held out.
- Temporal boundaries: the cut date, and a search of training data after that date for case IDs from the evaluation set.
- Near-duplicate checks: method, threshold and fields compared, including attachments and quoted text.
- Access history: who saw the evaluation items and whether any were pasted into prompts, tickets or documents that later entered training data.
- Scan limitations: what the checks could not see, such as images without OCR or records too short for the n-gram length.
Worked example: a support-ticket benchmark after a near-duplicate pass
This example is illustrative; the counts and percentages are invented to show the method, not drawn from a real dataset. A team builds a 2,000-ticket evaluation set across six task families from a help desk history, then runs a near-duplicate filter against the training pool. Overall, 9% of evaluation tickets are flagged.
The audit shows the loss is concentrated. Billing dispute escalations lose 31% of their tickets, a removal ratio of about 3.4, while password resets lose 4%. A fixed baseline model got 41% of the removed billing tickets wrong against 22% of the kept ones, so the filter removed the harder half of the slice. Hand review of a sample of the matched pairs finds that most are template twins: agents used the same macro, but the disputed line item and the final credit decision differ.
The fix is structural. The team re-splits by customer account and by month, which moves the template twins to one side instead of deleting them, and reruns the filter. Billing removal drops to near the overall rate. The exception is password resets, where most flagged pairs are true duplicates generated by one automated workflow; there the removal is correct, and the family is reported with its smaller count and a note that it measures little beyond template recognition.
Keep, move or report: deciding what to do with a flagged stratum
Pick the action from what the pair review found, not from the removal share alone. The same removal ratio can mean the filter worked or that it gutted a slice, and only the pairs tell you which.
Checklist · 3 items
- Bring in a dataset engineer when matches cluster in attachments, OCR text or code blocks your matcher was not built for.
- Bring in the evaluation lead when a single stratum drives the headline score change.
- Pause the benchmark release when access history shows evaluation items were pasted into systems that feed training data.
0 of 3 ticked. Tick each item as you go.
| What the review finds | Action | What to report |
|---|---|---|
| True duplicates of training records | Remove from evaluation or from training | Count removed and the method used |
| Template twins with different outcomes | Keep in evaluation; move twins out of training by re-splitting | The split key and the before and after counts |
| Same entity, different task | Re-split by entity rather than delete | Entity-level split and how many groups each side holds |
| Stratum below the size you can score | Merge with a related family or drop it from headline results | The excluded slice and why |
| Removed items much harder than kept | Score both the full and filtered sets and publish both | The gap between the two scores and the configuration |
| Attachments or derivatives not scanned | Narrow the claim to what was checked or extend the scan | The scan limitation in plain words |
What to ask for when you license records for evaluation
Most of this audit depends on metadata that only the record owner has: account IDs, task labels, timestamps and lineage between a ticket and the article written from it. Asking for those fields at the request stage costs far less than reconstructing them after delivery, and it lets you split by group before any filter runs.
Split keys and lineage fields can be written into the scope from day one when an evaluation team requests records through SourceX, which sources them from U.S. businesses on demand and holds no inventory. The request form takes scope, volume, format, timing and licensing needs, and SourceX handles the rights review, contracting, preparation and delivery between the business and the AI developer.
Frequently asked questions
What is benchmark decontamination?
It is the process of finding evaluation items that also appear, exactly or nearly, in a model's training data and then removing, flagging or separately scoring them. The goal is a score that reflects generalization rather than recall. It usually relies on n-gram overlap, hashing or embedding similarity, so it inherits the blind spots of those methods.
Is deduplication the same as decontamination?
No. Deduplication removes repeated records inside a training set to save compute and reduce memorization. Decontamination compares the evaluation set with the training set so the score is not inflated by overlap. They use similar matching tools, and both should be audited for what they remove.
What similarity threshold should I use?
There is no universal value; it depends on how your records are written and how similar legitimate distinct cases are. Approximate matchers such as MinHash LSH also return false positives and misses near any threshold. Choose a threshold, then review matched pairs near it by hand and record the result with the configuration.
How many matched pairs should be reviewed by hand?
Review every pair in small flagged strata and a random sample in large ones, and record the sample size next to the result. The aim is to see each pair type often enough to choose an action, not to estimate a precise error rate. Add pairs near the threshold, where misclassification is most likely.
Does a clean-subset score prove there is no contamination?
No. A clean subset, such as the low-contamination subset the Llama 2 authors defined by token n-gram matches, shows how a model scores on items one matcher did not flag. It says nothing about paraphrases, translated items or answers that leaked through attachments. Report the matcher, its threshold and its scan limits with any clean-subset score.
Sources
- Lee et al. found that train-test overlap affected over 4% of the validation sets of standard language modeling datasets and released exact-substring and near-duplicate deduplication methods. Source
- The datasketch MinHash LSH documentation states that LSH queries return false positives and false negatives relative to the Jaccard threshold. Source
- The scikit-learn guide recommends group-wise cross-validation when data has a group structure and evaluating time series on future observations. Source
- Allamanis reports that code duplication across training and test corpora inflated reported performance of code models by up to 100% in some cases. Source
- Kapoor and Narayanan found leakage errors in 17 fields affecting 329 papers and built a taxonomy of 8 leakage types. Source
- SemDeDup reports removing 50% of a LAION subset with minimal performance loss using embedding-based semantic deduplication. Source
- FairDeDup reports that deduplication reinforced existing biases on FACET and modifies SemDeDup to favor keeping underrepresented concepts. Source
- The Llama 2 paper defines contamination by token n-gram matches and reports results on clean and dirty evaluation subsets. Source
- The GPT-3 paper used 13-gram overlap to flag contaminated evaluation examples and ignored 13-grams that matched more than 10 training documents as common phrases. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.