AI uses for records
Temporal splits for business AI data with policy change boundaries
By SourceX Editorial · Updated
10 sections
Short answer
To split business AI data by time, choose the moment the task is decided, put everything resolved before a cutoff into training, leave a gap for late outcomes, and test on work that began after it. Then cut again wherever a policy, system or form changed, and hold out whole customers and task families so leaks cannot cross the boundary through repeat entities or later edits.
Evaluation split
Temporal splits business AI
Key takeaways
- Split on the time the predicted outcome was decided, never on export or last-modified dates.
- Leave an embargo gap so outcomes near the cutoff can settle before testing.
- Put a boundary at every policy, system or form change, or label each record with its regime.
- Hold out whole customers, vendors or case families in addition to the time boundary.
- Restore records to their state at the cutoff and drop post-decision attachments.
- Use several rolling origins to see whether performance decays over time.
How do you separate training and evaluation data using business time periods?
Separate them with a cutoff on the time a task was decided, not on the time a record was exported or last edited. Training gets work whose outcome was settled before the cutoff, a gap absorbs outcomes still in flight, and evaluation gets work that started after the gap. That mirrors deployment: a model trained on last year's records is used on next quarter's work.
A date alone is not enough for business records. Policies change mid-year, systems are migrated, and the same customers return month after month, so a clean temporal split also needs policy boundaries, entity holdouts and a scan for records that were edited after the cutoff. The plan in this article puts all of those decisions on one page so a buyer and a supplier can agree them before export.
Start from the prediction moment, the point in the workflow where the model will act. Everything the model sees must have existed at that moment, and the outcome it is scored against must have been settled afterwards. Writing that sentence down for the specific task usually settles half of the later arguments about which fields to keep.
Why does a random split overstate models trained on business records?
A random split overstates performance because neighboring records share information the model should not have at prediction time. The scikit-learn user guide makes the point for time series: classical cross-validation such as KFold and ShuffleSplit assumes independent, identically distributed samples, which produces unreasonable correlation between training and test instances on autocorrelated data, so evaluation should use future observations.
Business records are autocorrelated in ways that are easy to miss. Tickets from one outage arrive in a burst and share wording. Invoices from one vendor repeat month after month. A model that has seen ten of a burst in training will look brilliant on the eleventh in a random test set and much weaker on next month's different outage.
The research literature names the error. Kapoor and Narayanan's survey of leakage in machine-learning-based science (arXiv 2207.07048) lists temporal leakage as its own category in a taxonomy of eight leakage types. For language models, Roberts and colleagues (arXiv 2310.10628) used model training cutoffs as a natural experiment and found statistically significant trends between pass rates and problem release dates, the pattern contamination would produce. Time is the axis along which leaks show up.
Which timestamp should define the boundary?
Use the timestamp of the decision or event the model is meant to predict, and check that every input was available before it. Most business records carry several dates, and picking the wrong one quietly moves future information into training.
Before choosing, audit the dates on a sample of records. Compare created, resolved and last-modified times on 100 or so records and look for impossible orders, such as a resolution before creation, or clusters of identical timestamps that betray a bulk migration. Those patterns tell you which field can be trusted as the boundary and which only looks precise.
| Timestamp | What it marks | Risk if used as the boundary | Recommended use |
|---|---|---|---|
| Created or opened | When the work arrived | Includes cases whose outcomes were decided long after the cutoff | Defines the start of the test window |
| Decided or resolved | When the outcome the model predicts was set | Late-closing cases may be missing from the latest period | Primary boundary for training eligibility |
| Last modified | Any later edit, including corrections and reopenings | Pulls post-cutoff knowledge into older records | Use only to find records edited after the cutoff |
| Posted or effective | Accounting or legal effective date | Can be backdated, so it may precede the real decision | Use for policy regime assignment where rules key on it |
| Exported | When the dataset was pulled | Says nothing about when anything happened | Record for lineage only |
| Attachment or note time | When supporting files and comments were added | Post-decision notes can state the answer | Exclude attachments added after the decision time |
Where should a boundary go when the rules themselves moved?
Place a boundary wherever the rule that produced the correct answer changed, because records on either side of it teach different answers to the same question. A model trained across the change without knowing it learns an average of two policies and is wrong under both.
At a glance
Where should a boundary go when the rules themselves moved
Public examples are easy to find. In June 2022 the IRS announced an increase in the optional business standard mileage rate effective July 1, 2022, for the rest of that year, citing higher fuel prices. Any expense-review dataset that spans that date contains correct reimbursement decisions under two different rates; a test set drawn from the second half of the year measures something different from a training set drawn from the first.
Inside a company, the same thing happens through approval thresholds, refund windows, pricing rules, coding standards, new form versions and system migrations. Ask the supplier for a change log alongside the records. Where none exists, look for structural signs in the data: a field that starts or stops being filled, a new status value, a step change in approval rates.
- Cut at the change: train only on the regime the model will be deployed under, if that regime has enough history.
- Label the regime: keep both sides, add a policy version field to every record, and report test results per regime.
- Test across it on purpose: train on the old regime and test on the new one to measure how badly a policy change hurts, which is useful when rules change often.
Which ten decisions go in a temporal split plan with policy boundaries?
The plan below is the asset: ten decisions, each written down with an illustrative entry and the check that proves it was applied. Fill it in before the export, because several checks need fields that are lost if the supplier exports only the current state of each record.
| Plan element | Decision to record | Illustrative entry | Check |
|---|---|---|---|
| Prediction moment | The point in the workflow where the model acts | When a support ticket is first assigned | Every input field existed at that moment |
| Boundary timestamp | Which date field decides train or test | Ticket resolved time | No training record resolved after the cutoff |
| Training window | Start and end of eligible history | Illustrative: 24 months ending at the cutoff | Counts per month show no gaps from migrations |
| Embargo gap | Period excluded between train and test | Illustrative: 30 days, the time most reopenings occur | Records opened in the gap appear in neither split |
| Test window | The period evaluated | Illustrative: the 3 months after the gap | Covers at least one full business cycle that matters |
| Policy regimes | Each rule change and its effective date | Refund window changed on an illustrative date | Every record carries a regime label |
| Entity holdout | Customers, vendors or sites held out entirely | Illustrative: 10 percent of customer accounts | No held-out entity id appears in training |
| Task family holdout | Templates or case types kept for test only | One product line's tickets | Family ids disjoint across splits |
| Near-duplicate scan | Method and threshold for similar text | Shingled text similarity over ticket bodies and replies | Flagged pairs across splits reviewed and removed |
| Access and freeze | Who may see test records and when the split was frozen | Test set hashed and access-logged at freeze | Hash unchanged at evaluation time |
Could a training record, attachment or later edit reveal the test answer?
Yes, and in business data that is the usual way temporal splits fail. A date boundary stops a test case from appearing in training, but not its echoes: a knowledge-base article written after an incident, a reopened ticket whose correction note names the fix, a recurring invoice that repeats a held-out pattern, or a summary generated from records on both sides.
At a glance
Could a training record, attachment or later edit reveal the test answer
Hold out whole entities as well as periods. scikit-learn's GroupKFold exists for this reason: it keeps all samples from one group on the same side of a split, so the score shows how well the model generalizes to unseen groups. In business data the group is usually the customer, vendor, site or case family, and the same logic applies even when the split is by time.
- Records edited after the cutoff: restore their state as of the cutoff, or drop them from training.
- Attachments and notes added after the decision time: exclude them from the inputs of any record.
- Derived documents such as FAQs, playbooks and templates written after the cutoff: keep them out of training corpora used with this split.
- Repeat entities: hold out a share of customers or vendors entirely and report results for seen and unseen entities separately.
- Shared templates and boilerplate: scan for near-duplicates across splits and review the matches by hand.
- Access history: record who could see test records, so later leaks can be traced.
Worked example: a support-ticket split around a refund policy change
Illustrative only; every figure is invented. A buyer is training a model to recommend dispositions on customer refund tickets. The supplier offers 30 months of tickets, about 4,000 per month. Partway through month 22, the company shortened its refund window, which changes the correct disposition for a recognizable slice of tickets.
The plan sets the prediction moment at first assignment and the boundary at resolved time. Training covers months 1 to 21 and the part of month 22 before the change; the rest of month 22 and months 23 and 24 form the embargo, because most reopenings in this business happen within six weeks; months 25 to 30 are the test window. Every record carries a regime label. Ten percent of customer accounts are held out from training altogether.
Results are then reported in four cells: old regime and new regime, seen and unseen customers. An overall score hides the finding that matters, which is how much accuracy drops on new-regime tickets for customers the model has never seen.
The exception: if the new regime has only a few weeks of history, there may be too little to test on. Then the honest choice is to report results for the old regime, state that the new one is untested, and refresh the evaluation when more months arrive.
When is one cutoff not enough, and what does a rolling origin add?
One cutoff gives one estimate; a rolling origin gives several and shows whether performance is stable over time. scikit-learn's TimeSeriesSplit implements the idea: in the k-th split it trains on the first k folds and tests on the next, so successive training sets are supersets of earlier ones, and its gap parameter excludes samples from the end of each training set before the test set. The library notes that folds should cover equal durations for metrics to be comparable.
Benchmarks for language models use the same logic. LiveCodeBench collects problems continuously, publishes release versions covering successive date ranges, and lets users score models only on problems released inside a chosen window; its README notes that, to counter contamination in the DeepSeek models, the paper reports results only on problems released after August 2023.
For business data, three or four origins a quarter apart are usually enough to see drift. If scores fall steadily across origins, the model is learning patterns that age, and the buyer should plan for regular refreshes rather than a one-time purchase.
Report each origin's score with its window dates and the number of test records, and show the spread across origins rather than only the mean. A model whose score varies widely between quarters is less predictable in deployment than one with a slightly lower but steady score, and the spread is the only place that shows up.
When a temporal split is the wrong test
A temporal split is the wrong test when the deployment is not forward in time, such as a model that will process a backlog of old records, or when the test window is too short to contain the cases that matter, such as a seasonal process measured over one quiet month.
Narrow the scope instead of forcing it. If the most recent period is dominated by a one-off event, such as a product recall, test on it separately rather than letting it define the whole score. If a policy changed so recently that the new regime has almost no history, say so in the evaluation report.
Ask for specialist review when timestamps cannot be trusted. Systems that backfill dates on migration, or that overwrite the resolved time on every reopening, need a dataset engineer to rebuild the event history before any split is meaningful. Until that rebuild is done, treat any reported score as provisional and say so wherever it is quoted.
What should a buyer ask a supplier for about dates and policy history?
A temporal split plan changes what a buyer asks a supplier for: continuous history across the cutoff rather than a convenience sample, the original timestamps rather than export dates, record state as of the cutoff where later edits exist, and a dated list of policy and system changes. Agreeing these terms before export is far cheaper than discovering a leak after training.
SourceX does not keep a catalog of ready-made sets; it sources records on request from U.S. businesses. As a general guide, buyers look for consistent history, typically two years or more, and tens of thousands of records or more in one system, though rarer work can qualify with less and each deal is assessed on its own; two years is also roughly what a training window plus a meaningful test window needs. Buyers can send the split plan with scope, volume, format, timing and licensing needs through the request form.
- Continuous history covering the training window, the embargo gap and the test window, with monthly record counts
- Original created, decided, resolved and modified timestamps, with time zones
- Record state as of the cutoff for any record edited later, or a change history that allows it to be rebuilt
- Attachment and note timestamps, so post-decision material can be excluded
- A dated list of policy, form and system changes in the period, with effective dates
- Stable pseudonymous ids for customers, vendors and sites, so entity holdouts survive de-identification
Frequently asked questions
What is a temporal split in machine learning?
It is a train and test division by time: the model trains on records from before a cutoff and is evaluated on records from after it. It mirrors deployment, where a model trained on past work is used on future work, and avoids training on information that would not have existed at prediction time.
How long should the gap between training and test data be?
Long enough for outcomes near the cutoff to settle. Base it on how long reopenings, corrections or late outcomes usually take in the process, which can be days for some workflows and months for others, and exclude records that start inside the gap from both splits.
Is a temporal split enough to prevent data leakage?
No. Repeat customers, shared templates, records edited after the cutoff and documents written later can all carry test answers into training. Combine the time boundary with entity holdouts, a near-duplicate scan and a check on post-cutoff edits.
What should happen when a business policy changes in the middle of the data?
Mark the change as a boundary. Either train only on the regime the model will be deployed under, or label every record with its policy version and report results per regime, so an average across two sets of rules does not hide the drop.
How many months of history does a temporal split need?
Enough for a training window that covers the business cycles that matter plus a test window long enough to contain the cases you care about. For many operational datasets that means two years or more of continuous records, but the right figure depends on seasonality and case volume.
Sources
- The scikit-learn user guide says KFold and ShuffleSplit assume i.i.d. samples and that time series should be evaluated on future observations; GroupKFold keeps each group on one side of a split. Source
- TimeSeriesSplit trains on the first k folds and tests on the next, with successive training sets as supersets, a gap parameter, and equal-duration folds for comparable metrics. Source
- Kapoor and Narayanan's taxonomy of eight leakage types includes temporal leakage. Source
- Roberts et al. used GPT training cutoffs as a natural experiment and reported pass-rate trends against release dates that indicate contamination. Source
- LiveCodeBench collects problems continuously, publishes dated release versions and scores models within chosen date windows to counter contamination. Source
- The IRS raised the optional business standard mileage rate effective July 1, 2022, for the rest of 2022, citing fuel prices. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.