AI uses for records
Synthetic variation train test leakage: why a paraphrased task is not a new test
By SourceX Editorial · Updated
10 sections
Short answer
A paraphrase changes the wording of a task but not what the model must know to solve it: the same source record, entities, reasoning path and answer. If one version trains the model and another tests it, the test measures recall, not generalization. Assign every variation to a task family before generating anything, split by family, entity and time, and treat similarity scans as an audit, not the split.
Evaluation split
Synthetic variation train
Key takeaways
- A paraphrase keeps the facts, reasoning and answer of a task, so it does not make a new test.
- Assign family IDs before generating variations, then split whole families.
- Add entity, solution-artifact and time keys when the evaluation claim needs them.
- Derivatives such as KB articles, summaries and attachments can leak answers across a clean task split.
- Similarity scans audit the split; they cannot replace it and they miss shared rules and renamed entities.
Why can paraphrased training tasks remain contaminated evaluation cases?
Because the thing a test is meant to probe survives the rewording. A task's identity is its inputs, the facts it depends on, the steps that solve it and the answer that ends it. Rephrasing the request, swapping a name or reformatting a table leaves all four in place. When the model trains on one wording and is scored on another, it can succeed by recognizing a problem it has already solved.
The fix is a split rule, not a better filter. Give every source task a family identifier before any synthetic variation is produced, generate variations inside the family, and send whole families to either training or evaluation. Then add entity and time boundaries where the use case needs them. Similarity scans still run afterward, but they check the split; they cannot create it.
This page gives a variation map that classifies common changes as same family or new family, a table of split keys, a list of derivatives that leak answers, an illustrative worked example and the one case where paraphrases belong in the test set on purpose.
What does the model keep when only the wording changes?
Think of a support task: a customer on an annual plan asks why a mid-cycle seat change produced a prorated charge, and the resolution cites a billing rule and a credit memo. A generator can produce dozens of versions: formal and casual, in Spanish, with the customer renamed, with the seat count changed from five to seven, as an email or a chat. Every one still teaches the same rule, the same reasoning chain and the same shape of answer.
If version 12 is in training and version 31 is in the test set, a correct answer on version 31 tells you that the model saw the rule, not that it can apply billing rules it has never seen. That is the question most buyers of an evaluation actually want answered. Item-level random splits make this leak the default whenever variations are generated before the split, because sibling variations land on both sides by chance.
The same reasoning applies to entities and time. A model that saw several cases about one customer can answer a held-out case about that customer from remembered context, and a model trained on records from after an event can answer a test case set before it with knowledge no real user would have had.
What do published contamination studies show about rephrased test items?
Yang and colleagues showed that rephrasing or translating benchmark items lets them pass n-gram overlap checks, the most common decontamination method, and that a model trained on rephrased test items can reach very high scores on that benchmark. Their LLM Decontaminator tool quantifies rephrased samples relative to a benchmark; its README lists, for example, 21 of the 164 HumanEval problems as having rephrased counterparts in the CodeAlpaca training set.
Exact and near duplicates were already common before generation became cheap. Lee and colleagues found duplicated text in all four common language datasets they studied and reported that train-test overlap affected over 4 percent of the validation sets of standard datasets. Zhang and colleagues took the opposite route with GSM1k: they wrote fresh problems matched in style and difficulty to GSM8K, found that some model families scored lower on the fresh set than on the original, and chose not to release the new set publicly to keep it out of future training data.
Together these results point at one conclusion for teams that generate variations of their own tasks: a scan that compares strings will miss paraphrases, a scan that compares meaning will miss some too, and the only reliable protection is to keep related items from crossing the split in the first place.
Which split unit should you fix before generating any variation?
Decide what counts as related before you create the relationships. The scikit-learn documentation describes the general problem: when the process that produced the data yields groups of dependent samples, the independence assumption behind random splits breaks, and you need group-aware splits to learn whether a model generalizes to unseen groups. For time-correlated data it recommends splits that train on the past and test on the future. Synthetic variation creates dependent groups on purpose, so it needs the same treatment.
Use the strictest key your evaluation claim requires. A claim about new problems needs a family split; a claim about new customers needs an entity split on top; a claim about future performance needs a time cut as well.
In practice, the family ID should be written by the generator, not inferred later. Every generation call takes a parent ID and writes it into the output record, along with the variation type and the generator or prompt version. Source tasks get their IDs from the system they came from, such as a ticket number or invoice ID, so families can be traced back to records. When one record yields several tasks, those tasks share a record key even if they are different families, and the stricter split decides where they go.
Mixed keys need a rule for conflicts. If a family belongs to an entity that also appears in held-out families, either move the whole entity to one side or drop the family; do not split the entity silently. Log every such decision, because the counts of moved and dropped families are part of the evaluation report.
| Split key | What it groups | When it is required | Common mistake |
|---|---|---|---|
| Task family | All variations derived from one source task and its solution | Always, once any variation is generated | Assigning the family after generation from similarity scores |
| Source record | Tasks built from the same ticket, invoice, contract or thread | When one record yields several tasks | Treating two questions about one contract as independent |
| Entity | Tasks that share a customer, vendor, employee or account | When the claim is generalization to new entities | Renaming the entity and assuming the link is broken |
| Solution artifact | Tasks resolved by the same runbook, KB article, macro or policy clause | When the answer is mostly the artifact | Splitting by request while the artifact sits in training |
| Time window | Tasks created or resolved within a period | When the claim is performance on future work | Random split across a policy or price change |
Which variations create a new test, and which do not?
Use the map below to tag every generated variation with a family decision. Same family means the variation shares the source task's identity and must stay on its side of the split. New family means the change alters what has to be known or done, so the result can be a separate item, provided it is also checked against the other split keys.
| Variation | What changes | What stays the same | Family decision |
|---|---|---|---|
| Surface paraphrase or tone change | Wording, register, length | Facts, steps, answer | Same family |
| Translation | Language | Facts, steps, answer | Same family |
| Entity rename or swap | Names, IDs, addresses | Rule, steps, answer shape | Same family; check entity key too |
| Number perturbation inside one rule | Amounts, counts, dates | Rule and reasoning path | Same family |
| Format or channel change | Email to ticket, prose to table | Content and answer | Same family |
| Distractor added or removed | Irrelevant context | Required facts and answer | Same family |
| Different governing rule or policy clause | The rule that decides the outcome | Domain and style only | New family if the rule is absent from training |
| Different failure cause with same symptom | Root cause and fix | Surface complaint | New family |
| Same task on a later policy version | Applicable rule text and date | Request type | New family only with a time cut |
Could a derivative or attachment reveal the evaluation answer?
Often, yes, and this leak survives even a perfect family split of the tasks themselves. Business records spawn derivatives that restate the answer in another form. If any of them enter training, the held-out task is partly answered in advance. List the derivatives of every source record and assign them to the same side as the record.
- Knowledge base articles, macros and FAQ entries written from resolved tickets.
- Summaries, call notes and handoff notes that restate the resolution.
- Attachments such as the corrected invoice, the signed amendment or the screenshot of the fix.
- Model-generated reasoning traces or reference solutions produced while building the training set.
- Later records that quote the earlier one, such as a reopened ticket or a follow-up email thread.
- Embeddings or retrieval indexes built from the full corpus before the split was made.
How does a family split change a support resolution test? (illustrative)
This example is illustrative; the counts are invented to show the method. A team has 2,000 resolved billing tickets and generates five paraphrases of each, giving 12,000 items. A random 90/10 item split puts about 1,200 items in the test set. Because each family has six members, nearly every test item has at least one sibling in training, so the test mostly measures recall.
With the family rule, the team first assigns family IDs to the 2,000 source tickets, then holds out 200 whole families, including their paraphrases, so the test has about 1,200 items with no siblings in training. Next it checks the solution-artifact key: 40 of the held-out families resolve through two KB articles that also resolve training families. It either moves those families to training or removes those KB articles from the training corpus and records the choice.
Finally it applies a time cut, holding out families from the most recent quarter, because a pricing change took effect then. Scores on the strict split will usually be lower than on the random one. That gap is the measurement the random split was hiding, not a regression.
When do paraphrases belong in the test set on purpose?
The exception is an invariance test. If the question is whether a model gives the same correct answer however a user phrases the request, paraphrases are the point. Build them from held-out families only, keep them together as a paired group, and score consistency across the group as its own metric next to accuracy.
The rule still holds in a weaker form: no member of a paired test group may have a sibling in training. Report invariance and generalization separately, because a model can be perfectly consistent across paraphrases of a problem it memorized. Translation tests follow the same pattern; they measure language robustness only when the family is unseen in every language.
What do similarity scans after the split catch, and what do they miss?
Run scans as an audit of the split, in increasing cost: exact and normalized string matching, n-gram overlap, embedding similarity, then an LLM judge on the closest pairs, which is the approach LLM Decontaminator uses. Record each method, its threshold and what it flagged. A flagged pair across the split is a bug in the family assignment, not a judgment call to accept.
Scans cannot see shared rules, shared entities that were renamed, or derivatives that restate an answer in different words, and their thresholds trade missed paraphrases against false alarms. They also say nothing about what a pretrained model saw before your project. Write those limits into the evaluation report so that readers know which leaks were ruled out by design and which were only searched for.
Which records keep the family boundary checkable?
A family split is only as trustworthy as the lineage behind it. Source records with stable IDs, links to the KB articles and attachments they produced, entity keys and creation and resolution dates let a team assign families before generation and prove later that no sibling crossed the line. Records stripped of those links force teams back to similarity scans.
SourceX can find operational records with that lineage intact at U.S. businesses when a team asks, using the request-data form's fields for scope, volume, language, format, timing and licensing. Ask for record IDs, link tables and dates in the delivery specification, because they are cheaper to keep at export than to rebuild.
Frequently asked questions
Does paraphrasing a training example create a new test case?
No. A paraphrase keeps the facts, reasoning steps and answer of the original, so a model that trained on one version can answer the other from memory. Treat all paraphrases of a task as one family and keep the family on one side of the split.
What is a task family split?
It is a split in which every item derived from the same source task, including paraphrases, translations, renamed and reformatted versions, carries one family ID and the whole family goes to training or to evaluation. It is the group split idea from classical machine learning applied to synthetic variation.
Can n-gram decontamination catch paraphrased test items?
Not reliably. Research on rephrased samples showed that rewording or translating benchmark items lets them pass n-gram overlap checks. Embedding and LLM-based checks catch more, but none of them replace assigning families before generation.
Is a random split ever acceptable with synthetic variations?
Only when variations are generated after the split, from training items alone, and no source record, entity or solution artifact is shared across it. If variations already exist, group them first and split by group.
How should we report results from a family split?
Report the split keys used, the number of families and items on each side, the scan methods and thresholds used as an audit, and any families moved or removed. If you also ran a paraphrase invariance test, report consistency separately from accuracy.
Sources
- Yang and colleagues show that rephrased or translated test items evade n-gram decontamination and that training on rephrased test data can produce very high benchmark scores. Source
- The LLM Decontaminator README describes a tool that quantifies rephrased samples relative to a benchmark and lists 21 of 164 HumanEval problems with rephrased counterparts in CodeAlpaca. Source
- The scikit-learn user guide explains that grouped, dependent samples break the independence assumption of random splits and recommends group-aware and time-ordered splits. Source
- Lee and colleagues found duplicates in all four datasets studied and train-test overlap affecting over 4 percent of standard validation sets. Source
- The GSM1k authors built fresh problems matched to GSM8K, observed lower scores for some model families and withheld the set to prevent contamination. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.