Skip to content

AI uses for records

Temporal holdout for a business dataset: a pilot with preset buy rules

By SourceX Editorial · Updated

9 sections

Short answer

A temporal holdout tests whether historical business records help on work that happened after them, which is the only kind of work a purchased dataset will meet. Train every arm on records resolved before a cutoff, leave a gap, and score on two or more later periods. Fix the model, budget, grader and decision rules before the first run, then read how lift changes with distance from the cutoff.

Evaluation split

Temporal holdout business

This page covers temporal holdout for a business dataset: a pilot with preset buy rules.Illustrative

Key takeaways

  • A time holdout answers the purchase question a random split cannot: whether value learned from past records carries forward to later work.
  • Score at least two successive later periods, so the result shows whether the benefit holds or fades rather than giving a single snapshot.
  • Add an arm that uses only recent records to learn whether older history earns its price.
  • Freeze the model version, training budget, grader and decision rules in a dated plan before anyone sees holdout data.
  • Map each result pattern to a contract term: buy recent years, buy full history, buy with refresh deliveries, stop, or extend once.

What can a temporal holdout tell you before you buy historical records?

It tells you whether records from the past improve a model on work that came after them, how quickly that improvement fades, and whether the oldest years add anything. Those are the three facts a buyer needs before paying for history, because every task a purchased dataset will meet lies in its future.

A temporal holdout business dataset pilot trains and tunes only on records resolved before a cutoff date, leaves a gap, and evaluates on one or more later periods that nobody building the model has seen. It is harder to pass than a random split, and that is the point: it reproduces the conditions the records will face after the purchase.

This page is a protocol for that pilot decision: a preregistered design with three arms, the conditions to hold fixed, controls that keep later information out of training, and a result-reading table that maps each pattern to a decision and a contract term. It reports no results. The mechanics of choosing cutoffs, timestamps and policy boundaries are covered in SourceX's guide to temporal splits for business AI data.

Why a random split cannot answer a forward-looking purchase question

A random split puts earlier and later records on both sides, so the test measures how well a model recalls its own era rather than how it handles the next one. For a purchase decision, that inflates the case for buying.

The evidence for the inflation is consistent across fields. The scikit-learn documentation for its TimeSeriesSplit cross-validator says other cross-validation methods are inappropriate for time-ordered data because they would train on future data and evaluate on past data. Lazaridou and colleagues argued in their NeurIPS 2021 paper on temporal generalization that time-agnostic random splits do not test whether a language model generalizes to data from after its training period. The Wild-Time benchmark, built from six tasks across five datasets, reported an average performance drop of about 20 percent from in-distribution to out-of-distribution data under temporal shift, and found no tested method consistently more robust than ordinary empirical risk minimization.

Leakage compounds the problem. Kapoor and Narayanan's survey of machine-learning-based science found, in its arXiv version, leakage errors in 17 fields affecting 329 papers, and set out eight types of leakage. Business records are especially prone to it, because fields are edited, backfilled and closed long after the event they describe.

Carry-forward, decay and depth: three answers only a time holdout gives

Carry-forward asks whether the licensed records help on the first later period, decay asks whether that help shrinks on periods further from the cutoff, and depth asks whether older years add value beyond recent ones. Only a design that scores the future, in more than one window, can answer all three.

None of them can be read from a single holdout period. One later window gives one estimate; two or three successive windows show direction, and a second arm restricted to recent records shows what the older years contribute. Each answer maps to a contract decision:

  • Carry-forward decides whether to buy at all. If the records do not help on the next period, their fit to the past is irrelevant to the purchase.
  • Decay decides the shape of the deal. Help that fades over successive periods argues for a shorter term or refresh deliveries rather than a one-time purchase of history.
  • Depth decides how much history to pay for. If a recent-years-only arm matches the full history, the older years add volume but not value for this task.

The preregistered protocol for a time-holdout pilot

Write every element in the table into a dated, read-only plan before anyone builds a model or opens holdout data. The Center for Open Science describes a preregistration as a time-stamped, read-only copy of a research plan completed before the research begins, and treats later changes as acceptable only when the final report states them and the reasons.

The protocol has three arms. Arm A is the buyer's current setup without the licensed records. Arm B adds the full licensed history up to the cutoff. Arm C adds only the most recent part of that history. Everything else is identical across arms. The entries in the third column are illustrative and should be replaced with the buyer's own task, dates and thresholds.

The preregistered protocol for a time-holdout pilot
ElementWhat to fix before the runIllustrative entryWhy it matters
Decision questionThe commercial decision the pilot informsLicense 36 months of dispatch history for a scheduling model, or notStops the pilot drifting into open-ended research
Cutoff and gapTraining cutoff and a gap long enough for outcomes to settleCutoff at month 36; two-month gap for late cancellationsKeeps unresolved outcomes out of both sides
Holdout periodsTwo or more successive later windows covering comparable seasonsMonths 39 to 41, then months 42 to 44Shows decay; one window gives one estimate
ArmsA current setup; B plus full history; C plus recent history onlyC uses the last 12 months before the cutoffSeparates carry-forward from depth
Fixed conditionsModel version, training or tuning budget, prompts and retrieval settingsOne pinned model version and an equal token budget per armRules out gains from anything but the records
GraderScoring method, frozen and blind to armCode check on the final schedule state plus a blinded expert samplePrevents scoring drift between arms
Primary metric and minimum liftOne metric and the smallest lift worth paying forShare of tasks solved correctly; minimum lift of 3 pointsDefines success in business terms before results exist
Success ruleA lift that must hold on every holdout periodB beats A by the minimum on both windows, interval above zeroPrevents buying on one lucky window
Stop ruleThe result that ends the pilot earlyB fails to beat A on the first window by any marginSaves the cost of later windows
Inconclusive ruleWhat happens when intervals span zero and the minimum liftScore one more window once, then decideStops thresholds moving after results arrive
A timeline of a time-holdout pilot: a training window, a gap and two later holdout periods, with three arms (current setup, plus full history, plus recent history only) and the decision rules fixed before the run.
The time-holdout pilot design. Illustrative: window lengths are examples, not recommendations for every task.

Holding the model, budget and grader fixed across arms

Any difference between arms other than the records will be credited to the records, so pin the model version, give each arm the same training or tuning budget, freeze prompts and retrieval settings, and score every output with one grader that cannot see which arm produced it. A pilot that upgrades the model halfway through has measured the upgrade.

Budget parity needs a unit. Use tokens processed, optimizer steps or a fixed number of retrieved documents per query, whichever the method consumes, and record it for each arm. If arm B's records are larger than arm C's, either subsample to the same budget or report the comparison as volume plus content rather than content alone.

Graders need the same discipline. Anthropic's engineering guidance on agent evaluations describes three kinds of grader, code-based, model-based and human, and recommends multiple trials because model outputs vary from run to run. In a time-holdout pilot, run each arm several times with different seeds and report the spread, not the best run.

Compute uncertainty on the paired difference between arms over the same tasks. Paired bootstrap resampling, set out for machine translation evaluation by Philipp Koehn in 2004, repeatedly draws test sets with replacement and scores both systems on each draw, which yields an interval for the difference. Use the task as the resampling unit, or the customer or account when tasks cluster by entity.

Keeping later information out of the training arms

Every arm must see the business only as it looked before the cutoff. A filter on each record's creation date is not enough, because business records keep changing after the event they describe, and the changes often encode the outcome the model is supposed to predict.

At a glance

Keeping later information out of the training arms

Six controls catch most of the ways later information slips in:

  • Reconstruct records as of the cutoff, using field history to restore values as they stood then rather than as exported today.
  • Drop fields completed after the event, such as final status, resolution codes or backfilled categories, from the inputs of cases that were still open.
  • Exclude cases still open at the cutoff, or assign them to the gap, so their outcomes cannot leak forward.
  • Report results separately for customers or assets that appear on both sides of the cutoff, so repeat entities do not pass for generalization.
  • Escrow the holdout: have the supplier or a neutral party withhold the later periods until every arm is frozen.
  • Tune prompts, thresholds and retrieval only on a validation window inside the training period, never on the holdout periods.

Reading the result table: six patterns and the decision each supports

Read the results as a pattern across arms and periods, not as one number. The table maps six common patterns to their most likely meaning, the decision the preregistered rules support and the contract consequence. It is a reading guide, not a record of any pilot run.

Two patterns deserve particular suspicion. A lift that appears on a random split but not on the time holdout usually means the random split leaked; a lift confined to the first holdout window means the business is changing faster than the records, which is a pricing fact, not a failure.

Report the full grid, not the winning cell. The pilot report should attach the dated plan, list every arm, period and seed, give each paired interval, and state any deviation from the plan with its reason. A reader who sees only a pooled lift cannot tell a durable effect from one strong window.

Reading the result table: six patterns and the decision each supports
Pattern across holdout periodsMost likely meaningDecisionContract consequence
B beats A on every period, and C matches BRecent records carry the valueBuyLicense recent years; do not pay for depth
B beats A and C on every periodOlder history adds signalBuy full historyDepth justifies its price for this task
B beats A on the first period only, and the gap shrinks laterValue decays as the business changesBuy with refreshShorter term or periodic refresh deliveries
B beats A on a random split but not on the time holdoutLeakage or memorized entitiesDo not buy on this evidenceAudit the export before any new test
Intervals span zero and the minimum liftToo little data to tellInconclusive: apply the preset extension onceHold terms open until the extra window is scored
B trails A on later periodsOutdated rules or practices in the recordsStop, or test a filtered periodExclude superseded periods from scope

Could the lift vanish under a stronger baseline or a second cutoff?

Yes, and the pilot should try to make it vanish before anyone pays. A positive result can disappear when the baseline is today's strongest relevant setup rather than an old one, when the holdout moves to a different cutoff, or when the run is repeated with different seeds.

Three checks address it. First, set arm A at the strongest setup the buyer could deploy without the records, including public data and retrieval it already has, so the lift measures what only the licensed records add. Second, repeat the comparison at an earlier cutoff, in the spirit of what forecasters call evaluation on a rolling forecasting origin, which pairs each test window with training data from before it and averages accuracy across windows. Third, check seasonality: each holdout window should cover the same part of the business cycle as the period it is compared with, or the lift may be a calendar effect.

If the lift survives all three, the evidence is strong for that task and those periods. It still cannot show how the records will perform after the next system migration or policy change, which is why the contract consequences in the result table matter as much as the verdict.

What the pilot needs from a supplier, and how to request it

The pilot needs continuous, dated history that runs past the cutoff, plus the right to withhold later periods until the run. As a rough guide rather than a rule, buyers want years of steady history, usually at least two, and a record count in the tens of thousands or higher from one system, though rare work can let a smaller set qualify; a time-holdout pilot needs enough span for a training window, a gap and at least two later periods.

Ask for decision timestamps rather than export dates, field change history so records can be rebuilt as of the cutoff, and a note of policy or system changes inside the span. Agree pilot terms that let the buyer test before a full license, and record in writing that the holdout periods stay escrowed until the arms are frozen.

SourceX keeps no stock of datasets and goes to U.S. businesses only once a buyer has asked, with the request form capturing what the buyer needs on scope, volume, language, format, timing and licensing. For a time-holdout pilot, include the intended use, the cutoff and holdout design, and the acceptance test the records must pass, so a supplier can say early whether its history spans the windows. If the history is too short for a gap and two later periods, say so in the plan and narrow the claim: a single-window pilot can reject a dataset, but it cannot measure decay. Nothing here implies that a given dataset is available or will pass the test.

Frequently asked questions

What is a temporal holdout in dataset evaluation?

It is a later period of records kept out of all training and tuning and used to score a model once. Because it comes after the training period, it tests whether what a model learned from earlier records carries forward, which is how a purchased dataset will actually be used.

How many holdout periods does a dataset pilot need?

At least two successive periods, fixed before the run. One period gives one estimate; two or more show whether a dataset's benefit holds, grows or fades as the holdout moves further from the cutoff.

Why add an arm that uses only recent records?

It shows whether older history earns its price. If recent records alone match the full history on later periods, the older years add volume but not value for that task, which changes how much history is worth licensing.

What should happen if a time-holdout pilot is inconclusive?

Apply the inconclusive rule written before the run, typically scoring one more holdout period, then decide. Moving the success threshold after seeing results turns a test into a negotiation and makes the evidence useless to procurement.

Can a temporal holdout show how long a dataset stays useful?

It shows the trend across the periods tested, not how the records will perform after the next system migration or policy change. Read decay across periods as a guide to contract term and refresh needs rather than as a lifetime estimate.

Sources

  • scikit-learn's TimeSeriesSplit docstring says other cross-validation methods are inappropriate for time-ordered data because they would train on future data and evaluate on past data, and documents a gap parameter. Source
  • Forecasting: Principles and Practice describes evaluation on a rolling forecasting origin: a series of test sets, each with a training set of earlier observations, with accuracy averaged across test sets. Source
  • Lazaridou and colleagues (NeurIPS 2021) argue that time-agnostic random splits do not test temporal generalization, the ability to generalize to data from beyond the training period. Source
  • The Wild-Time benchmark (six tasks, five datasets) reports an average performance drop of about 20 percent from in-distribution to out-of-distribution data under temporal shift. Source
  • Kapoor and Narayanan's arXiv survey found leakage errors in 17 fields affecting 329 papers and proposed a taxonomy of eight types of leakage. Source
  • Anthropic's engineering post on agent evaluations describes code-based, model-based and human graders and recommends multiple trials because outputs vary between runs. Source
  • The Center for Open Science describes a preregistration as a time-stamped, read-only research plan completed before the research begins, with deviations reported with reasons. Source
  • Koehn (2004) describes paired bootstrap resampling: repeatedly drawing test sets with replacement and scoring both systems on each to estimate the reliability of a difference. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies