Skip to content

AI uses for records

Temporal dataset split leakage: audit the high-water mark, not the cutoff

By SourceX Editorial · Updated

9 sections

Short answer

Temporal dataset split leakage happens when a split passes its cutoff check but examples still contain information written later: revised documents, records linked afterward, labels that settled after the cutoff, statistics computed over the whole period, or a model trained past the test window. Audit each example's high-water mark, the latest write time of anything it contains, then purge, roll back or re-snapshot whatever crosses the boundary.

Evaluation split

Temporal dataset split

This page covers temporal dataset split leakage: audit the high-water mark, not the cutoff.Illustrative

Key takeaways

  • A cutoff filters one timestamp per record; an example contains many, and the latest of them decides whether it is clean.
  • Time-based splits change model rankings: TabReD found that eight industrial datasets rank methods differently under time-based and random splits.
  • Labels settle late: Stripe notes cardholders can dispute a charge up to 120 days after payment, so recent dispute rates keep moving.
  • An as-of join set to search forward or nearest, or a merged pull request that closes an older issue, writes later facts into earlier examples.
  • Ask for field write times, link creation times and label settlement times in the data request, or the split cannot be audited at all.

How can a temporal split leak the future when every date passes?

It leaks when the cutoff is applied to one timestamp per record while the example carries many, some written after the cutoff. A ticket opened in March passes a March cutoff even if its status, linked records, attachments and label were all written in May.

The decision this page supports is narrow: can you trust the score from a temporal holdout you have already built, and if not, what do you repair? Answering it takes three kinds of evidence: the write time of every value an example contains, the time each outcome label became final, and the training cutoff of every model in the evaluation. Without the first, a split cannot be audited; it can only be believed.

Designing the split in the first place, choosing a decision timestamp, a gap and boundaries at policy changes, is covered in the guide to temporal splits for business AI data linked below. This page starts after that work, with a split that looks right and may not be.

What a leaky temporal split does to the score

It pulls the score back toward what a random split would report, hiding the drop a temporal split exists to measure. Published benchmarks show that drop can be large and that it changes which methods look best.

Wild-Time, a benchmark of distribution shift over time presented at NeurIPS 2022, reports accuracy of 97.99 on held-out data from the training years against 79.50 on later years for its Yearbook task, and 73.00 against 61.33 for MIMIC readmission prediction, and its authors conclude that existing methods do not close the gap. TabReD, a set of eight industrial tabular datasets that all ship with time-based train, validation and test splits, finds that evaluating on time-based splits ranks methods differently from the random splits common in benchmarks.

A split that leaks lands somewhere between those two numbers, and nobody can say where without an audit. The risk is not only a flattering score but the wrong model choice, because a leak helps some methods more than others.

Seven channels that carry later facts across the cutoff

Check seven channels separately, because each hides behind a different timestamp. For each one, the list gives how the leak happens and, where a system documents it, the mechanism that writes later facts into earlier records.

  • Revised documents. A knowledge base article, contract or policy exported today is its latest revision. Wikis usually keep history: MediaWiki's revisions API lists a page's revisions between two timestamps, newest first by default, so the version live on the decision date can be fetched in place of the current one.
  • Linked records. Links are often created after the record they point to. GitHub's documentation says merging a linked pull request into a repository's default branch closes the linked issue automatically, so an issue filed before the cutoff can carry a closed state and a fix written after it.
  • Late labels. Outcomes settle on their own clock. Stripe notes that cardholders can dispute a charge up to 120 days after payment, and sometimes later, so dispute rates for recent dates can still change; a fraud label read 30 days after payment is not yet final.
  • Join direction. The pandas merge_asof function picks the last earlier row by default, but its forward and nearest searches can select a row whose key is later than the record's own time, and rows with the same timestamp match unless exact matches are switched off.
  • Whole-period statistics. Normalization constants, vocabularies, deduplication clusters and customer-level counts computed over the full dataset carry test-period information into every training row.
  • Entity carryover. The same customer, asset or account appearing in both periods brings its later history along whenever a feature summarizes the entity rather than the moment.
  • The model itself. A pretrained model may have seen public outcomes from the test window. Roberts and colleagues used GPT training cutoffs as a natural experiment and found statistically significant trends linking pass rates on Codeforces and Project Euler problems to the problems' release dates, which they read as evidence of contamination.

The high-water mark audit: one timestamp per example that settles it

Give every example a high-water mark, the latest write time of any value it contains, and compare that mark with the boundary the split promises. An example whose mark crosses the boundary leaks, whatever its nominal date says.

Compute the mark from the write times of field values, attachments and linked records, the time the label became final, and the snapshot date of any reference corpus or statistic the example draws on. Three rules follow. A training example's mark, label included, must fall before the test window opens, less any embargo set by the label's settling period. A test example's inputs must carry a mark at or before that case's own decision time. A test label may be written after the decision, but no later than the evaluation snapshot.

Many systems expose only a record-level last-modified time. Treat it as an upper bound: a record last modified before the boundary is clean for its own fields, while one modified after it is a possible breach that needs field-level history to clear, not an automatic leak. Record which examples were cleared by field history and which only by the upper bound.

Report the audit with denominators: examples checked, examples breaching each rule, breaches by channel, and examples whose mark cannot be computed because a write time is missing. Unknown is its own category, never a pass. The split leakage card lists what to record so a second team can repeat the audit.

The high-water mark audit: one timestamp per example that settles it
Card fieldWhat to recordWhy it matters
Nominal cutoff and clockThe date and the timestamp field the split filtered onShows which of the example's other clocks were never checked
High-water mark distributionLatest write time per example, summarized separately for train and testMeasures directly how far past the boundary examples reach
Breaches by channelCounts for revisions, links, labels, joins, statistics, entities and modelsPoints each repair at the channel that caused it
Unknown marksExamples with any missing write time, and the fields involvedUnknown cannot be counted as clean
Embargo and label settlingGap between the periods and the label's settling ruleKeeps unsettled outcomes out of training
Reference snapshotsDates of knowledge bases, corpora and statistics the examples useRetrieval and normalization can leak even when rows do not
Model training cutoffsStated cutoff of each evaluated model against the test windowSeparates split leakage from pretraining contamination

Checking revisions and linked records past the nominal cutoff

Follow every link one hop out and fetch every referenced document at its decision-date revision; an audit that checks only a record's own fields misses both channels. They are hard because the leak sits not in the row being filtered but in something the row points to.

For revisions, ask whether the source keeps history and for how long. A wiki or document store with a revisions API lets you swap the current text for the version that was live on the decision date. A store that overwrites in place cannot, and every example citing it gets an unknown mark for that field.

For links, record when the link was created as well as when the linked record was. A credit memo issued in May against a March invoice, a follow-up case opened after closure, or a pull request merged after an issue was filed all point backward in time while being written forward. Cut each link at the decision time: keep links that existed then, and drop the later ones or move them to the label side.

Settle link depth before the audit starts. One hop covers direct references. A second hop matters when a linked record itself summarizes later events, such as a parent case whose notes were updated after the child case closed.

A worked audit of six examples near one cutoff (illustrative)

Of six illustrative examples near a June 30, 2025 cutoff, four breach and two pass, and each breach needs a different repair. The dates are invented to show how the high-water mark changes a verdict; they come from no real dataset.

Read each row across: the nominal date is what the split filtered on, the high-water mark is the latest write time found in the example, and the verdict follows the three rules above. The figure plots the same six examples as spans from nominal date to high-water mark.

A worked audit of six examples near one cutoff (illustrative)
ExampleNominal date and splitHigh-water mark and channelVerdict
A: support ticketMay 12, 2025, trainMay 17, 2025, own fields onlyKeep
B: refund requestMay 20, 2025, trainAug 2, 2025, policy article revisedRoll back to the May revision
C: invoice disputeJun 3, 2025, trainJul 21, 2025, credit memo linkedCut the link; the memo moves to the label side
D: card paymentJun 18, 2025, trainSep 30, 2025, dispute label settledDrop from training; dispute labels need an embargo
E: renewal requestJul 8, 2025, testJul 8, 2025, inputs at decision timeKeep
F: account escalationJul 15, 2025, testJun 2026, customer lifetime count recomputedRecompute the feature as of Jul 15, 2025
Six illustrative examples drawn as spans from nominal date to high-water mark around a June 30, 2025 cutoff; four spans cross their boundary and are marked as breaches, two stay clean.
Illustrative: the six examples from the worked audit. A span that crosses its boundary leaks, whatever its nominal date.

Will newer records muddy the comparison instead?

Sometimes, and the audit should say so rather than hide it: newer records can add noise, change the task or make old and new scores incomparable. That is the strongest alternative explanation for a score that falls after a leak is fixed, and it needs its own evidence.

Version three things for both periods: the source scope (which systems, teams and record types were included), the policy rules in force, and the label definitions. If any of them changed between training and test, a lower test score may reflect a different task rather than a repaired leak. Compare the test window's mix of task types, outcomes and channels with the distribution the model is meant to serve, and report any divergence next to the score.

When a historical comparison matters, keep a frozen split whose examples, snapshots and labels never change, and add a refreshed split for current performance. Report both, and never join their scores in one trend line. A buyer comparing suppliers or model versions needs the frozen split; a buyer deciding whether to deploy needs the refreshed one.

Repair, rebuild or report: what to do with a leaky split

Repair when breaches sit in channels with recoverable history, rebuild when unknown marks dominate, and report what remains either way. The choice follows from the leakage card, not from how far the score moved.

Checklist · 5 items

  1. Late labels only: add an embargo before the test window at least as long as the label's settling period, and drop training examples whose labels settled inside it.
  2. Revisions or links with history: roll documents back to their decision-date revisions, cut links created after the decision, then run the audit again.
  3. Whole-period statistics or entity features: recompute them within each period, as of each example's decision time.
  4. Unknown marks above a threshold fixed in advance: rebuild from dated snapshots or replace the source, because repairs to those examples cannot be verified.
  5. Model contamination: move the test window past every evaluated model's stated training cutoff, or label the result as possibly contaminated.

0 of 5 ticked. Tick each item as you go.

Putting auditability into a data request

Ask for the timestamps the audit needs before the data arrives, because adding them later is often impossible. A request that states the intended use and an acceptance test, such as a ceiling on the share of unknown high-water marks, lets a supplier say up front whether its sources can meet it.

Fields to ask for: a write time for every field value, or the change history that implies one; revision history for documents and knowledge articles, with its retention limits; creation times for links between records; the time each outcome label became final and the rule that decides when it counts as settled; and snapshot dates for any reference data delivered with the records.

SourceX assembles buyer datasets from U.S. businesses when a request arrives rather than selling from stock, so a requirement for write times, revision history and label settlement dates becomes part of qualifying each source. Whether a given company's systems kept that history is checked source by source; nothing here implies that a particular dataset exists or would pass.

The audit has limits. It finds leaks only through channels that can be timestamped, so it cannot prove a split is clean when a source overwrote its history. It does not measure how much a leak inflated a score until the repaired split is run again. And it does not cover leakage through annotation instructions or duplicated test items, which need their own checks.

Frequently asked questions

What is temporal leakage in a train and test split?

It is information from after the split boundary reaching an example that is supposed to predate it. A record can pass the cutoff check on its creation date while its status, linked records, revised documents or final label were written later, so the model sees part of the future it is meant to predict.

How long should the gap between training and test periods be?

At least as long as the outcome label takes to settle. For card disputes that can mean 120 days or more, since Stripe notes cardholders can dispute a charge up to 120 days after payment and sometimes later. Set the gap from the label's settling rule, not from a round number.

Does a temporal split protect against pretraining contamination?

Not by itself. A split controls what your training data contains, not what a pretrained model saw. Roberts and colleagues found pass rates on coding problems tracked the problems' release dates relative to GPT training cutoffs. Place the test window after the stated cutoff of every model you evaluate, or report the result as possibly contaminated.

Can you audit a split if the source system keeps no change history?

Only partly. Without write times, an example's high-water mark is unknown, and unknown should not count as clean. You can still check labels, joins and derived statistics, but fields that were overwritten in place need dated snapshots or a different source.

Should a leaky split be repaired or replaced?

Repair it when breaches sit in channels with recoverable history, such as document revisions, link creation times or late labels. Replace it when a large share of examples have unknown high-water marks, because a repair you cannot verify leaves the score as uncertain as before.

Sources

  • TabReD is a collection of eight industrial tabular datasets that all come with time-based train, validation and test splits; evaluating on time-based splits leads to a different ranking of methods than random splits. Source
  • Wild-Time (NeurIPS 2022 Datasets and Benchmarks) reports in-distribution versus out-of-distribution accuracy of 97.99 versus 79.50 on Yearbook and 73.00 versus 61.33 on MIMIC-Readmission, and concludes existing methods do not close the gap. Source
  • Roberts et al. use GPT training cutoffs as a natural experiment and report statistically significant trends between pass rates, GitHub popularity and release dates on Codeforces and Project Euler, evidence of contamination. Source
  • pandas merge_asof performs a backward search by default (last right row with key less than or equal to the left key); forward and nearest searches can select later rows, and allow_exact_matches defaults to True. Source
  • GitHub's documentation says that when a linked pull request is merged into a repository's default branch, its linked issue is automatically closed. Source
  • MediaWiki's API:Revisions lists a page's revisions between rvstart and rvend timestamps, with rvdir defaulting to older (newest first). Source
  • Stripe notes that because cardholders can dispute a charge up to 120 days after payment, and sometimes later, the dispute rate can change for dates more recent than 120 days. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies