Privacy and preparation
Split integrity after a dataset refresh: checks before you retrain
By SourceX Editorial · Updated
9 sections
Short answer
To keep a dataset refresh from crossing the train-test boundary, assign splits by a deterministic rule on a stable group key, freeze the test group manifest and time cutoff, and rerun a regression checklist on every new version. Check updated and late-arriving records, reissued IDs, near-duplicates and derivatives such as summaries that restate test answers before anyone retrains or compares scores.
Evaluation split
Split integrity
Key takeaways
- A seeded shuffle reproduces a split only for identical input, so a refresh needs a rule on a stable group key plus a frozen test manifest.
- Refreshes leak through re-exported updates, late records dated before the cutoff, reissued IDs and derivatives that restate test answers.
- Choose the split unit that production must generalize across, often company plus time, and test how each one can break.
- Run the ten-check regression checklist on every version and publish its counts in the version notes.
- Near-duplicate scans are evidence under a stated method and threshold, not proof that no leakage exists.
How do you keep a dataset refresh from crossing the train-test boundary?
Make split membership a property of a stable key, not of row order, and test it again on every refresh. Assign each company, customer, case or other group to a split by a rule that gives the same answer whatever else is in the file, freeze the list of test groups and the time cutoff, then run a regression checklist on each new version before anyone retrains or re-scores against the old test set.
The checklist matters because a refresh changes more than the row count. New records arrive for entities already in the test set, old records are updated and re-exported, late records appear with dates before the cutoff, identifiers can be reissued, and new derivatives such as summaries or knowledge articles can restate test answers. Any one of these can move evaluation material into training without anyone touching the split code.
This page is about that regression step for refreshed data. How to choose split ratios and why to split by group in the first place is covered in the train-test split guide linked below; when to replace an evaluation set altogether is covered by the eval refresh cadence guide.
Why does a seeded random split stop being reproducible once new rows arrive?
Because a seed reproduces a shuffle of a particular input, and a refresh is a different input. Add a thousand rows or change their order and the same seed sends many existing rows to different splits, so records that were in test last quarter can land in training this quarter.
Dataset tooling makes the same point about versions. The TensorFlow Datasets versioning guide says that when the major version changes, existing data has changed and the slicing API does not necessarily return the same set of records for a given slice; only patch and minor changes keep slices stable. The Hugging Face datasets library lets you pin a dataset to a tag, branch or commit through its revision parameter. Pinning keeps old experiments reproducible, but it does not make a new version's split agree with the old one.
What does survive a refresh is a deterministic rule on a stable key, for example assigning a group to test when a keyed hash of its group ID falls in a chosen range, combined with a stored manifest of the test group IDs from the first release. The rule places new groups consistently; the manifest makes sure no existing test group can move. Record the hash key, the range and the manifest's own checksum with each dataset version.
Which unit should the split hold constant: entity, time or task family?
Hold constant whichever unit the model must generalize across in production, and often two of them at once. Splitting rows when the rows are not independent leaks shared context; scikit-learn's documentation on grouped data says that to measure generalization to unseen groups, all samples in the validation fold must come from groups not represented at all in the paired training fold. Its TimeSeriesSplit trains on earlier folds and tests on the next, which is the time-ordered version of the same idea.
Each unit protects against a different shortcut, and each has a refresh-specific way of failing.
| Split unit | What it protects | How a refresh breaks it | Check to add |
|---|---|---|---|
| Company or customer | Generalization to businesses the model has not seen | New records for a test company arrive and are assigned by row, not by group | Every new record's group ID is looked up in the frozen test manifest first |
| Entity within a company | Generalization to unseen accounts, assets or patients of a known business | Entity IDs are reissued or merged in the source system between exports | Crosswalk old to new IDs before assignment; count unmapped IDs |
| Time | Forecast-style evaluation on periods after training | Backdated or late-arriving records land before the cutoff after the test period was cut | Compare event time with export time; quarantine records whose event predates the last cutoff |
| Task family | Generalization to task types held out on purpose | New templates or categories map an old task into a family that sits in training | Re-run the family classifier on old test items and diff the labels |
| Source system or site | Robustness to a new system, region or workflow | A supplier adds a second system that copies records from the first | Check for shared record IDs or near-identical text across systems |
What should the refresh split regression checklist include?
Run the checks below on every new version, in this order, before training or evaluation uses it. Each row says what to test, how, the pass rule and what to do when it fails. The pass rules are deliberately strict; loosen one only with a written reason that ships with the dataset version.
| Check | How to run it | Pass rule | If it fails |
|---|---|---|---|
| 1. Version pinned | Record the dataset version, revision or delivery ID and the previous one | Both versions identified and retrievable | Stop; no comparison is possible without the prior version |
| 2. Test manifest intact | Compare the frozen test group IDs with the groups assigned to test now | Every manifest group still in test; none in train or validation | Restore the assignment from the manifest, then rerun all checks |
| 3. New groups assigned by rule | Apply the stored hash rule to groups first seen in this version | Assignment matches the rule for every new group | Fix the pipeline; never hand-place a group |
| 4. Identifier continuity | Map old IDs to new IDs with the supplier's crosswalk or stable keys | No test group split across an old and a new ID | Merge under the old ID and treat both as test |
| 5. Updated records | Find records whose ID appeared in the previous version with a later updated time | Updated test records stay in test; the old copy is replaced, not kept | Deduplicate on ID plus update time and reassign |
| 6. Temporal cutoff | List records with event time before the last cutoff that were not in the previous version | None placed in a test period that is already frozen | Quarantine and log them as late arrivals |
| 7. Exact and near duplicates | Hash normalized text and run a near-duplicate search across splits | No test item has an exact or near match in training | Drop the training copy or move its group to test |
| 8. Derivatives and attachments | Search training for summaries, articles, replies or files built from test cases | No derivative of a test case in training | Remove the derivative or exclude the case from scoring |
| 9. Score comparability | Re-score a fixed reference model on the old and new test sets | Change explained by documented additions or removals | Report scores on old and new sets separately |
| 10. Change record | Write the counts from checks 2 to 9 into the version notes | Notes published with the version | Release is blocked until notes exist |
Which derived records can carry a test answer into training?
Summaries, knowledge articles, follow-up cases, quoted replies and attachments built from a test case can all carry its answer, and in business records this is a leak that row-level checks easily miss. A support ticket in test may be resolved by a knowledge base article written afterward from that very ticket; a closed case may spawn a follow-up that quotes its resolution; an email reply quotes the whole earlier thread; an invoice PDF restates the amount a model is asked to predict. If the derivative sits in training, the model can learn the answer without seeing the test row.
Duplicate text is the measurable part of this problem. Lee and colleagues report that deduplication reduces train-test overlap that affects over 4% of the validation sets of standard datasets, which they say allows more accurate evaluation. Their numbers come from public text corpora, not business exports, so treat them as a sign that overlap is common rather than an estimate for your data.
Kapoor and Narayanan's survey of leakage in machine-learning-based science builds a taxonomy of eight leakage types, and duplicates across train and test and temporal leakage are among the ones a refresh can introduce. For derivatives, the practical test is lineage: every derived item should carry the IDs of the records it was made from, so the checklist can look up whether any of those IDs is a test case. Where lineage is missing, fall back to near-duplicate search on key phrases and identifiers, and accept that it will miss paraphrases.
Worked example: a quarterly support-ticket refresh (illustrative)
All figures are invented for illustration. A team evaluates a ticket-routing model on a test set of 300 companies, frozen at the first delivery, with a cutoff at the end of the second quarter. The supplier's second delivery adds a quarter of new tickets and re-exports everything updated since the first.
Check 2 passes. Check 5 finds that roughly 9% of test tickets were updated after the first export, mostly status and tag changes, and the pipeline had loaded the new copies as fresh rows assigned by the hash of the ticket ID rather than the company ID; about half of those copies went to training. Check 6 finds 140 tickets created before the cutoff that were missing from the first export because they had been merged into other tickets at the time. Check 8 finds 37 knowledge articles in training whose source-ticket field points to test tickets.
The fix is to deduplicate on ticket ID plus update time, assign by company, quarantine the late tickets and exclude the 37 linked tickets from scoring until the articles are removed from training. Zendesk's incremental export documentation describes the mechanism behind the first finding: subsequent responses may contain duplicate items, and it recommends filtering on ticket ID plus updated time. The team reports both the old and the cleaned test score, with the counts, in the version notes.
When should the test set itself change after a refresh?
Change it only when the old set no longer measures the decision you care about, and then version it as a new set rather than editing it in place. Fixing a leak is a cleanup, not a new test set; adding a new product line, region or task family is a new test set.
The exception to freezing is a correction. If the refresh shows that test labels were wrong, for example tickets routed to a team that the supplier later merged, fix the labels and say so, because an evaluation that preserves known errors is consistent but misleading. Report results on both the corrected and the original version for at least one cycle.
Datasheets for Datasets asks maintainers whether older versions will continue to be supported and, if not, how their obsolescence will be communicated. Answer that question in the version notes for the evaluation set too: which version is current, which earlier scores are still comparable, and which checks a reader should rerun before comparing across versions.
Where the checklist stops working, and what to escalate
The checklist depends on stable keys and lineage, so it weakens when either is missing. If a supplier rotates pseudonymization keys between deliveries, the same customer gets a new token and check 4 cannot connect the versions; ask for a stable keyed token or a crosswalk before the next delivery rather than trying to rematch on de-identified text.
Near-duplicate search is a scan, not proof. It finds copied and lightly edited text, misses paraphrase and translation, and its threshold trades false alarms against misses. Record the method and threshold in the version notes and describe the result as no matches found under that method, not as no leakage.
Escalate to the data owner when check 2 or 4 fails twice in a row, because that points to a pipeline or supplier process problem rather than a one-off. Escalate to whoever owns benchmark claims when check 9 shows an unexplained score change, because published comparisons may need correction.
What to settle with a data supplier before the next delivery
Most of the checklist is cheap when the supplier provides four things with each delivery: stable record and group IDs across versions, an updated time on every record, a list of deletions and merges, and source-record IDs on anything derived. Ask for them in the first request, not after the first leak.
Because SourceX gathers each dataset from U.S. businesses only after a buyer's request, the request form, with its scope, volume, language, format, timing and licensing fields, is the place to require stable keys and lineage fields. De-identification is done by SourceX through approved processing routes, so a buyer who needs pseudonyms to stay constant across refreshes should say so in the request.
Frequently asked questions
What is split integrity after a dataset refresh?
It means every record still sits in the split its group was assigned to before the refresh, and nothing in training reveals a test answer. A refresh can break it through new records for test entities, re-exported updates, late records, reissued IDs and derivatives. A regression checklist run on each version confirms it holds.
Can I fix leakage by reshuffling the whole dataset with a new seed?
Not without losing comparability. A full reshuffle moves groups that were in test into training, so a model trained on the old split has now seen part of the new test set and old and new scores no longer mean the same thing. Keep the frozen test manifest and assign only new groups by rule.
How often should the refresh checks run?
On every new version before it is used for training or scoring, including small incremental deliveries. Routine updates and re-exports can leak as easily as large releases. Automating the checks in the ingestion pipeline makes this cheap.
Are near-duplicate checks enough to rule out contamination?
No. They catch copied and lightly edited text but miss paraphrases, translations and facts restated in structured fields. Combine them with lineage checks on derived items and with group and time rules, and report the method and threshold you used.
What if the supplier cannot provide stable IDs across deliveries?
Then the safest option is to treat each delivery's overlap with the test set as unknown until you can match it. Ask the supplier for a crosswalk or a stable keyed token, and in the meantime quarantine records that could belong to test groups rather than adding them to training.
Sources
- scikit-learn's guide to grouped cross-validation requires validation samples to come from groups absent from the paired training fold, and TimeSeriesSplit tests on the fold after its training folds. Source
- TensorFlow Datasets says a major version change means the slicing API does not necessarily return the same records for a given slice. Source
- The Hugging Face datasets library loads a specific dataset version through a revision parameter that takes a tag, branch or commit hash. Source
- Deduplication reduces train-test overlap that affects over 4% of the validation sets of standard datasets, allowing more accurate evaluation. Source
- A survey of leakage in ML-based science builds a taxonomy of eight leakage types, including overlap between training and test sets and temporal leakage. Source
- Zendesk's incremental exports can return the same item more than once, and Zendesk recommends deduplicating tickets on ID plus updated time. Source
- Datasheets for Datasets asks whether older versions will continue to be supported and how their obsolescence will be communicated. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.