Skip to content

AI uses for records

Temporal splits for business AI data with policy change boundaries

By SourceX Editorial · Updated

10 sections

Short answer

To split business AI data by time, choose the moment the task is decided, put everything resolved before a cutoff into training, leave a gap for late outcomes, and test on work that began after it. Then cut again wherever a policy, system or form changed, and hold out whole customers and task families so leaks cannot cross the boundary through repeat entities or later edits.

Evaluation split

Temporal splits business AI

This page covers temporal splits for business AI data with policy change boundaries.Illustrative

Key takeaways

  • Split on the time the predicted outcome was decided, never on export or last-modified dates.
  • Leave an embargo gap so outcomes near the cutoff can settle before testing.
  • Put a boundary at every policy, system or form change, or label each record with its regime.
  • Hold out whole customers, vendors or case families in addition to the time boundary.
  • Restore records to their state at the cutoff and drop post-decision attachments.
  • Use several rolling origins to see whether performance decays over time.

How do you separate training and evaluation data using business time periods?

Separate them with a cutoff on the time a task was decided, not on the time a record was exported or last edited. Training gets work whose outcome was settled before the cutoff, a gap absorbs outcomes still in flight, and evaluation gets work that started after the gap. That mirrors deployment: a model trained on last year's records is used on next quarter's work.

A date alone is not enough for business records. Policies change mid-year, systems are migrated, and the same customers return month after month, so a clean temporal split also needs policy boundaries, entity holdouts and a scan for records that were edited after the cutoff. The plan in this article puts all of those decisions on one page so a buyer and a supplier can agree them before export.

Start from the prediction moment, the point in the workflow where the model will act. Everything the model sees must have existed at that moment, and the outcome it is scored against must have been settled afterwards. Writing that sentence down for the specific task usually settles half of the later arguments about which fields to keep.

Why does a random split overstate models trained on business records?

A random split overstates performance because neighboring records share information the model should not have at prediction time. The scikit-learn user guide makes the point for time series: classical cross-validation such as KFold and ShuffleSplit assumes independent, identically distributed samples, which produces unreasonable correlation between training and test instances on autocorrelated data, so evaluation should use future observations.

Business records are autocorrelated in ways that are easy to miss. Tickets from one outage arrive in a burst and share wording. Invoices from one vendor repeat month after month. A model that has seen ten of a burst in training will look brilliant on the eleventh in a random test set and much weaker on next month's different outage.

The research literature names the error. Kapoor and Narayanan's survey of leakage in machine-learning-based science (arXiv 2207.07048) lists temporal leakage as its own category in a taxonomy of eight leakage types. For language models, Roberts and colleagues (arXiv 2310.10628) used model training cutoffs as a natural experiment and found statistically significant trends between pass rates and problem release dates, the pattern contamination would produce. Time is the axis along which leaks show up.

Which timestamp should define the boundary?

Use the timestamp of the decision or event the model is meant to predict, and check that every input was available before it. Most business records carry several dates, and picking the wrong one quietly moves future information into training.

Before choosing, audit the dates on a sample of records. Compare created, resolved and last-modified times on 100 or so records and look for impossible orders, such as a resolution before creation, or clusters of identical timestamps that betray a bulk migration. Those patterns tell you which field can be trusted as the boundary and which only looks precise.

Which timestamp should define the boundary?
TimestampWhat it marksRisk if used as the boundaryRecommended use
Created or openedWhen the work arrivedIncludes cases whose outcomes were decided long after the cutoffDefines the start of the test window
Decided or resolvedWhen the outcome the model predicts was setLate-closing cases may be missing from the latest periodPrimary boundary for training eligibility
Last modifiedAny later edit, including corrections and reopeningsPulls post-cutoff knowledge into older recordsUse only to find records edited after the cutoff
Posted or effectiveAccounting or legal effective dateCan be backdated, so it may precede the real decisionUse for policy regime assignment where rules key on it
ExportedWhen the dataset was pulledSays nothing about when anything happenedRecord for lineage only
Attachment or note timeWhen supporting files and comments were addedPost-decision notes can state the answerExclude attachments added after the decision time

Where should a boundary go when the rules themselves moved?

Place a boundary wherever the rule that produced the correct answer changed, because records on either side of it teach different answers to the same question. A model trained across the change without knowing it learns an average of two policies and is wrong under both.

At a glance

Where should a boundary go when the rules themselves moved

Public examples are easy to find. In June 2022 the IRS announced an increase in the optional business standard mileage rate effective July 1, 2022, for the rest of that year, citing higher fuel prices. Any expense-review dataset that spans that date contains correct reimbursement decisions under two different rates; a test set drawn from the second half of the year measures something different from a training set drawn from the first.

Inside a company, the same thing happens through approval thresholds, refund windows, pricing rules, coding standards, new form versions and system migrations. Ask the supplier for a change log alongside the records. Where none exists, look for structural signs in the data: a field that starts or stops being filled, a new status value, a step change in approval rates.

  • Cut at the change: train only on the regime the model will be deployed under, if that regime has enough history.
  • Label the regime: keep both sides, add a policy version field to every record, and report test results per regime.
  • Test across it on purpose: train on the old regime and test on the new one to measure how badly a policy change hurts, which is useful when rules change often.

Which ten decisions go in a temporal split plan with policy boundaries?

The plan below is the asset: ten decisions, each written down with an illustrative entry and the check that proves it was applied. Fill it in before the export, because several checks need fields that are lost if the supplier exports only the current state of each record.

Which ten decisions go in a temporal split plan with policy boundaries?
Plan elementDecision to recordIllustrative entryCheck
Prediction momentThe point in the workflow where the model actsWhen a support ticket is first assignedEvery input field existed at that moment
Boundary timestampWhich date field decides train or testTicket resolved timeNo training record resolved after the cutoff
Training windowStart and end of eligible historyIllustrative: 24 months ending at the cutoffCounts per month show no gaps from migrations
Embargo gapPeriod excluded between train and testIllustrative: 30 days, the time most reopenings occurRecords opened in the gap appear in neither split
Test windowThe period evaluatedIllustrative: the 3 months after the gapCovers at least one full business cycle that matters
Policy regimesEach rule change and its effective dateRefund window changed on an illustrative dateEvery record carries a regime label
Entity holdoutCustomers, vendors or sites held out entirelyIllustrative: 10 percent of customer accountsNo held-out entity id appears in training
Task family holdoutTemplates or case types kept for test onlyOne product line's ticketsFamily ids disjoint across splits
Near-duplicate scanMethod and threshold for similar textShingled text similarity over ticket bodies and repliesFlagged pairs across splits reviewed and removed
Access and freezeWho may see test records and when the split was frozenTest set hashed and access-logged at freezeHash unchanged at evaluation time

Could a training record, attachment or later edit reveal the test answer?

Yes, and in business data that is the usual way temporal splits fail. A date boundary stops a test case from appearing in training, but not its echoes: a knowledge-base article written after an incident, a reopened ticket whose correction note names the fix, a recurring invoice that repeats a held-out pattern, or a summary generated from records on both sides.

At a glance

Could a training record, attachment or later edit reveal the test answer

Hold out whole entities as well as periods. scikit-learn's GroupKFold exists for this reason: it keeps all samples from one group on the same side of a split, so the score shows how well the model generalizes to unseen groups. In business data the group is usually the customer, vendor, site or case family, and the same logic applies even when the split is by time.

  • Records edited after the cutoff: restore their state as of the cutoff, or drop them from training.
  • Attachments and notes added after the decision time: exclude them from the inputs of any record.
  • Derived documents such as FAQs, playbooks and templates written after the cutoff: keep them out of training corpora used with this split.
  • Repeat entities: hold out a share of customers or vendors entirely and report results for seen and unseen entities separately.
  • Shared templates and boilerplate: scan for near-duplicates across splits and review the matches by hand.
  • Access history: record who could see test records, so later leaks can be traced.

Worked example: a support-ticket split around a refund policy change

Illustrative only; every figure is invented. A buyer is training a model to recommend dispositions on customer refund tickets. The supplier offers 30 months of tickets, about 4,000 per month. Partway through month 22, the company shortened its refund window, which changes the correct disposition for a recognizable slice of tickets.

The plan sets the prediction moment at first assignment and the boundary at resolved time. Training covers months 1 to 21 and the part of month 22 before the change; the rest of month 22 and months 23 and 24 form the embargo, because most reopenings in this business happen within six weeks; months 25 to 30 are the test window. Every record carries a regime label. Ten percent of customer accounts are held out from training altogether.

Results are then reported in four cells: old regime and new regime, seen and unseen customers. An overall score hides the finding that matters, which is how much accuracy drops on new-regime tickets for customers the model has never seen.

The exception: if the new regime has only a few weeks of history, there may be too little to test on. Then the honest choice is to report results for the old regime, state that the new one is untested, and refresh the evaluation when more months arrive.

When is one cutoff not enough, and what does a rolling origin add?

One cutoff gives one estimate; a rolling origin gives several and shows whether performance is stable over time. scikit-learn's TimeSeriesSplit implements the idea: in the k-th split it trains on the first k folds and tests on the next, so successive training sets are supersets of earlier ones, and its gap parameter excludes samples from the end of each training set before the test set. The library notes that folds should cover equal durations for metrics to be comparable.

Benchmarks for language models use the same logic. LiveCodeBench collects problems continuously, publishes release versions covering successive date ranges, and lets users score models only on problems released inside a chosen window; its README notes that, to counter contamination in the DeepSeek models, the paper reports results only on problems released after August 2023.

For business data, three or four origins a quarter apart are usually enough to see drift. If scores fall steadily across origins, the model is learning patterns that age, and the buyer should plan for regular refreshes rather than a one-time purchase.

Report each origin's score with its window dates and the number of test records, and show the spread across origins rather than only the mean. A model whose score varies widely between quarters is less predictable in deployment than one with a slightly lower but steady score, and the spread is the only place that shows up.

When a temporal split is the wrong test

A temporal split is the wrong test when the deployment is not forward in time, such as a model that will process a backlog of old records, or when the test window is too short to contain the cases that matter, such as a seasonal process measured over one quiet month.

Narrow the scope instead of forcing it. If the most recent period is dominated by a one-off event, such as a product recall, test on it separately rather than letting it define the whole score. If a policy changed so recently that the new regime has almost no history, say so in the evaluation report.

Ask for specialist review when timestamps cannot be trusted. Systems that backfill dates on migration, or that overwrite the resolved time on every reopening, need a dataset engineer to rebuild the event history before any split is meaningful. Until that rebuild is done, treat any reported score as provisional and say so wherever it is quoted.

What should a buyer ask a supplier for about dates and policy history?

A temporal split plan changes what a buyer asks a supplier for: continuous history across the cutoff rather than a convenience sample, the original timestamps rather than export dates, record state as of the cutoff where later edits exist, and a dated list of policy and system changes. Agreeing these terms before export is far cheaper than discovering a leak after training.

SourceX does not keep a catalog of ready-made sets; it sources records on request from U.S. businesses. As a general guide, buyers look for consistent history, typically two years or more, and tens of thousands of records or more in one system, though rarer work can qualify with less and each deal is assessed on its own; two years is also roughly what a training window plus a meaningful test window needs. Buyers can send the split plan with scope, volume, format, timing and licensing needs through the request form.

  • Continuous history covering the training window, the embargo gap and the test window, with monthly record counts
  • Original created, decided, resolved and modified timestamps, with time zones
  • Record state as of the cutoff for any record edited later, or a change history that allows it to be rebuilt
  • Attachment and note timestamps, so post-decision material can be excluded
  • A dated list of policy, form and system changes in the period, with effective dates
  • Stable pseudonymous ids for customers, vendors and sites, so entity holdouts survive de-identification

Frequently asked questions

What is a temporal split in machine learning?

It is a train and test division by time: the model trains on records from before a cutoff and is evaluated on records from after it. It mirrors deployment, where a model trained on past work is used on future work, and avoids training on information that would not have existed at prediction time.

How long should the gap between training and test data be?

Long enough for outcomes near the cutoff to settle. Base it on how long reopenings, corrections or late outcomes usually take in the process, which can be days for some workflows and months for others, and exclude records that start inside the gap from both splits.

Is a temporal split enough to prevent data leakage?

No. Repeat customers, shared templates, records edited after the cutoff and documents written later can all carry test answers into training. Combine the time boundary with entity holdouts, a near-duplicate scan and a check on post-cutoff edits.

What should happen when a business policy changes in the middle of the data?

Mark the change as a boundary. Either train only on the regime the model will be deployed under, or label every record with its policy version and report results per regime, so an average across two sets of rules does not hide the drop.

How many months of history does a temporal split need?

Enough for a training window that covers the business cycles that matter plus a test window long enough to contain the cases you care about. For many operational datasets that means two years or more of continuous records, but the right figure depends on seasonality and case volume.

Sources

  • The scikit-learn user guide says KFold and ShuffleSplit assume i.i.d. samples and that time series should be evaluated on future observations; GroupKFold keeps each group on one side of a split. Source
  • TimeSeriesSplit trains on the first k folds and tests on the next, with successive training sets as supersets, a gap parameter, and equal-duration folds for comparable metrics. Source
  • Kapoor and Narayanan's taxonomy of eight leakage types includes temporal leakage. Source
  • Roberts et al. used GPT training cutoffs as a natural experiment and reported pass-rate trends against release dates that indicate contamination. Source
  • LiveCodeBench collects problems continuously, publishes dated release versions and scores models within chosen date windows to counter contamination. Source
  • The IRS raised the optional business standard mileage rate effective July 1, 2022, for the rest of 2022, citing fuel prices. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies