Skip to content

AI uses for records

Duplicate task detection for benchmarks: same task, different names

By SourceX Editorial · Updated

10 sections

Short answer

Normalize every task so names, IDs, dates and amounts become placeholders, then generate candidate pairs three ways: exact hashes, MinHash similarity on the normalized text and embedding similarity on meaning. Have two reviewers label each pair as the same task, a sibling or distinct, group the confirmed duplicates, and keep each group, entity and time window on one side of every split.

Evaluation split

Duplicate task detection

This page covers duplicate task detection for benchmarks: same task, different names.Illustrative

Key takeaways

  • Two tasks are duplicates when solving one hands a model the answer to the other, whatever the names or wording.
  • Replace names, IDs, dates and amounts with placeholders before comparing, so renamed copies collapse on an exact hash.
  • Use MinHash for lexical near-duplicates and embeddings for paraphrases, and tune both on planted pairs.
  • Keep siblings whose differing facts change the answer; they are coverage, not leakage.
  • Split by duplicate group, customer and time window, and search training and few-shot pools for the answers themselves.

How do you find duplicate tasks in a benchmark when the names differ?

Strip the surface away before comparing, then look for sameness of meaning and answer, not of wording. Two tasks are duplicates for evaluation purposes when solving one gives a model the answer, or the exact reasoning path, to the other: same rule, same kind of input, same expected result, whether one says Acme order 4417 and the other says Northwind order 9921.

String matching alone misses these. Yang and colleagues showed that decontamination based on n-gram overlap can be bypassed by simple variations such as paraphrasing or translation, and that a 13B model trained on rephrased test samples could overfit a benchmark and reach performance on par with GPT-4. If rephrasing can fool a decontamination filter, renaming customers in a business task set certainly can.

The protocol below is built for task sets drawn from business records, where the same piece of work recurs with different companies, people and numbers. It combines normalization, three candidate generators, human adjudication with a fixed label set, and split rules that keep duplicate groups, entities and time windows apart.

Exact, renamed, paraphrased or sibling: which kind of duplicate are you facing?

Name the duplicate type first, because each type needs a different detector and a different action. A dataset engineer who treats every high-similarity pair the same will either delete legitimate coverage or keep leaks.

Exact, renamed, paraphrased or sibling: which kind of duplicate are you facing?
TypeWhat it looks likeDetector most likely to catch itDefault action
ExactSame text after whitespace and case normalizationHash of normalized textKeep one, record group size
RenamedSame text with different company, person, ID, date or amountHash or MinHash after placeholder normalizationKeep one, record group size
ParaphrasedSame request and answer, reworded or translatedEmbedding similarity plus human reviewKeep one, record group size
SiblingSame rule and template, materially different facts that change the answerTemplate or rule ID; embedding clustersKeep as coverage; same split side as its family
Entity leakDifferent tasks about the same customer or account across splitsEntity group keyMove the whole entity group to one split
Derivative leakA later email, attachment or summary that states the answerAnswer-string search and linked-record traceRemove the derivative from any training or few-shot pool

The semantic duplicate review protocol

Run these seven steps in order and keep the output of each, so a reviewer can audit how a pair was found and why it was kept or removed. Version the protocol and the thresholds with the evaluation set.

At a glance

The semantic duplicate review protocol

  • 1. Define the unit and the answer: one task is one prompt plus its starting records plus its expected outcome. Store an answer key or answer hash for every task so answer-equivalence can be checked mechanically.
  • 2. Normalize: replace company names, person names, record IDs, dates, amounts, addresses and order numbers with typed placeholders, such as COMPANY, ORDER_ID or DATE. Keep the original alongside.
  • 3. Exact pass: hash the normalized text and group identical hashes. Renamed duplicates collapse here.
  • 4. Lexical near-duplicate pass: shingle the normalized text and use MinHash with locality-sensitive hashing to retrieve pairs above a Jaccard threshold. Start conservatively and tune on planted pairs.
  • 5. Semantic pass: embed each task and retrieve its nearest neighbors by cosine similarity. Send every pair above a review threshold, plus a random sample just below it, to human review.
  • 6. Adjudicate: two reviewers label each candidate pair as same task, sibling, related or distinct using the definitions in this page. A third person settles disagreements, and the agreement rate is reported.
  • 7. Act and split: merge same-task pairs into duplicate groups, attach entity and time keys, and assign whole groups to one split. Record what was removed, from where and why.

Choosing candidate generators and thresholds you can defend

Use more than one generator, because each one misses a different kind of duplicate, and tune each threshold against pairs you planted rather than a number copied from another project.

MinHash with locality-sensitive hashing scales to large sets but is approximate. The datasketch documentation says plainly that a query will return false positives, sets below your threshold, and miss some qualifying sets as false negatives, while sets with higher Jaccard similarity always have a higher probability of being returned. That is acceptable for candidate generation as long as a later step checks the pairs.

Lexical methods cannot see meaning. Embedding-based methods can: the SemDeDup work used embeddings from pre-trained models to find semantic duplicates, pairs that are similar in meaning but not identical in wording. For task sets of tens of thousands of items, the sentence-transformers paraphrase_mining function compares texts against each other in chunks and returns the pairs with the highest cosine similarity, retrieving up to 100 neighbors per text by default.

Set thresholds by planting known duplicates. Take 50 tasks, write a renamed version and a paraphrased version of each, mix them into the set and measure how many each pass recovers. Report that recall next to the duplicate rate, because a low duplicate rate from a detector that misses half the planted pairs says little.

Worked example: a support-operations task set with renamed duplicates

In this illustrative case, a team builds 1,200 evaluation tasks from resolved support tickets across 300 customer companies. Every count is illustrative and only shows how the protocol's outputs fit together.

Two tasks read: refund order 4417 for Acme, the item arrived damaged and was reported within the return window; and Northwind wants its money back on order 9921 because the product came broken, reported eight days after delivery. After normalization they differ only in wording; the embedding pass pairs them, and both reviewers label them the same task because the same return-policy rule produces the same outcome.

  • The exception in this example: 150 sibling pairs stayed, because applying the return rule to different facts, such as a report outside the window or a missing photo, is exactly the coverage the evaluation needs.
  • The limit: the team cannot see the training data of a third-party model, so the scan proves the set is internally clean and separated from the team's own fine-tuning data, not that no vendor model has seen similar tickets.
Worked example: a support-operations task set with renamed duplicates
StepOutput (illustrative)What the reviewer checks
Exact pass on normalized text38 groups covering 92 tasksPlaceholder rules did not merge tasks whose amounts change the answer
MinHash pass140 new candidate pairsPrecision on a 30-pair sample; recall on planted renamed pairs
Embedding pass260 new candidate pairsRecall on planted paraphrases; a sample below the threshold
Human adjudication110 same task, 150 sibling, 140 related or distinctAgreement between the two reviewers and how ties were settled
Group and split150 tasks removed (54 exact, 96 adjudicated); 1,050 unique tasks split by customer company and monthNo group, company or month on both sides

When a near-duplicate is legitimate coverage, not leakage

Keep siblings that share a rule but differ in a fact that changes the correct answer, and keep recurring work if your score is meant to mirror the workload. Deleting all similarity removes the variation that shows whether a model applies a rule or memorizes a pattern.

Real workloads repeat. If a large share of real tickets are password resets, an evaluation that keeps one password-reset task understates how often the model will face them. Collapse the duplicates into one representative per group for scoring, and give that representative a weight equal to its group size when you report a workload-weighted result. Report the unweighted result too, so a reader can see both the unique-task and the volume view.

The test for a sibling is counterfactual: change the fact that differs and ask whether the expected answer changes. If it does, the pair tests rule application and both tasks stay. If it does not, the pair is a duplicate and one copy goes.

Why company, task-family and time splits matter as much as text matching

Split on the groups that carry hidden answers, not only on the text, because two textually distinct tasks about the same customer or the same month can leak just as badly. Kapoor and Narayanan's survey found leakage errors in papers across 17 fields that adopted machine learning and proposed a taxonomy of eight leakage types, the first of which is a lack of clean separation between training and test data.

Group splits are standard tooling. The scikit-learn guide describes GroupKFold as a variant of k-fold that ensures the same group is not represented in both testing and training sets, and TimeSeriesSplit as returning earlier folds for training and a later fold for testing. Use the customer company as the group key for entity leakage, the duplicate-group ID for paraphrases and the record month for temporal leakage, and check every key on every split.

Leakage between training and test is not rare. Lee and colleagues found that train-test overlap affects over 4% of the validation sets of standard language-modeling datasets, and that removing it allows more accurate evaluation. A business task set built from a few systems of record, with repeat customers and templated replies, has every reason to carry more overlap, not less.

Could an attachment, summary or training example reveal the evaluation answer?

Yes, and text-similarity scans will not find it, because the leaking item does not look like the task. Trace each task's linked records and search for the answer itself.

In support and back-office data, the answer often appears later in the same thread: the agent's closing note, a refund confirmation email, a credit memo PDF or a monthly summary that lists the outcome. If any of those sit in the training set, the retrieval index or the few-shot examples used during evaluation, the model can copy the answer without solving the task. Search those pools for the answer key, the final record values and the ticket IDs of every evaluation task, and remove or quarantine what you find.

Record what you scanned and what you could not. A scan limited to text fields will miss answers inside images, spreadsheets or attachments that were not extracted, and a scan of your own pools says nothing about data a third-party model saw during pretraining.

What a duplicate review report should state

Publish the review with the evaluation set so a buyer or auditor can judge how clean it is without rerunning it. Keep the report to facts a second team could reproduce.

At a glance

What a duplicate review report should state

  • Source lineage: which systems, date ranges and companies the tasks came from.
  • Normalization rules, generator settings, thresholds and the embedding model version.
  • Planted-pair recall for each pass and reviewer agreement on adjudicated pairs.
  • Counts by duplicate type, groups removed and siblings kept with the reason.
  • Split keys used, with a check that no group, entity or time window crosses a split.
  • Pools scanned for derivative leaks, and the pools and formats not scanned.
  • When to stop and escalate: if planted recall stays low after tuning, or reviewers disagree on what counts as the same task, narrow the set to task families where the definition holds and bring in the evaluation lead before scoring.

Sourcing fresh tasks when deduplication shrinks the set

A strict review often leaves fewer unique tasks than planned, especially in families where work repeats. Topping up with tasks from different companies and time windows restores coverage without reintroducing the same groups. For that top-up, SourceX gathers task records from additional U.S. businesses whenever a team files a request naming scope, volume, language, format, timing and licensing, and the duplicate review above should still run on every batch that arrives.

Frequently asked questions

What is a near-duplicate task in an evaluation set?

It is a task that a model could solve by reusing the answer or reasoning from another task, even though the wording, names or numbers differ. Renamed copies and paraphrases count; siblings whose differing facts change the correct answer do not.

Is n-gram overlap enough to decontaminate a benchmark?

No. Research on rephrased samples showed that paraphrasing or translation can bypass n-gram decontamination, so add placeholder normalization, embedding-based candidate search and human review.

What similarity threshold should I use for embedding-based duplicate detection?

There is no universal value, because scores depend on the embedding model and the text. Plant known renamed and paraphrased pairs, measure recall at several thresholds and pick the one that keeps review workload manageable at the recall you need.

Should duplicates be deleted or down-weighted?

Delete extra copies for unique-task scoring, but keep the group size. If you report a workload-weighted result, give the kept representative a weight equal to its group size so frequent work is not understated.

How do I stop the same customer leaking across train and test splits?

Assign a group key such as the customer company and split by group, so all of that customer's tasks land on one side. Group-aware splitters like GroupKFold do this, and a time-ordered split handles leakage from later records.

Sources

  • Yang and colleagues report that string-matching decontamination such as n-gram overlap can be bypassed by paraphrasing or translation, and that a 13B model trained on rephrased test samples can reach benchmark performance on par with GPT-4. Source
  • Lee and colleagues report that language-modeling datasets contain many near-duplicates and that train-test overlap affects over 4% of the validation sets of standard datasets; deduplication allows more accurate evaluation. Source
  • SemDeDup uses embeddings from pre-trained models to identify and remove semantic duplicates: data pairs that are semantically similar but not exactly identical. Source
  • Kapoor and Narayanan found leakage errors in papers across 17 fields using machine learning and proposed a taxonomy of eight types of leakage. Source
  • The datasketch MinHash LSH documentation states that queries return false positives and false negatives, while sets with higher Jaccard similarity always have higher probability of being returned. Source
  • The sentence-transformers paraphrase_mining function compares texts against each other and returns the pairs with the highest cosine similarity, retrieving up to top_k (default 100) neighbors per text. Source
  • The scikit-learn cross-validation guide describes GroupKFold as ensuring the same group is not in both training and testing sets, and TimeSeriesSplit as training on earlier folds and testing on a later one. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies