Deal economics
How to measure what each extra year of business history adds to a dataset
By SourceX Editorial · Updated
10 sections
Short answer
To measure what another year of business history adds, train the same model on nested windows that reach further back, score each on one sealed holdout from the most recent period, and read the paired change in score per added year with its interval. Volume-matched and leave-one-year-out arms separate more records from older records. The page sets out the experimental method only and reports no measured findings.
Record timeline
Dataset valuation history
Key takeaways
- Measure the marginal utility of a year as the paired change in score when the training window extends back one year, on a sealed recent holdout.
- A fixed-volume ladder and a recent-volume learning curve separate the effect of more records from the effect of older records.
- Leave-one-year-out and regime-break arms show which specific years help and which pull the model toward outdated conditions.
- The holdout definition is the assumption that moves results most; a random split usually overstates the value of old years.
- This is a protocol only: no results are reported, and a score change is not a price.
How do you measure what another year of business history adds?
You measure it by training the same model on nested windows of history that reach progressively further back, scoring every version on one fixed holdout drawn from the most recent period, and reading the change in score as each older year is added. The difference between adjacent windows is the marginal utility of that year for that task, with an uncertainty interval around it.
This page is a protocol for a dataset valuation history depth experiment. It publishes the method only. No results are reported here, because a result belongs to one company's records, one task and one model, and has to be collected before anyone quotes it.
The design matters because two effects are tangled together. An older year adds more records, and it adds records from a different time. The first usually helps, the second can help or hurt. A useful experiment separates them, so a company can say whether an extra year is worth exporting, cleaning and offering, and a buyer can say whether it is worth asking for.
What question does a history depth experiment answer?
The experiment answers one narrow question: for a stated task, model and recent evaluation period, how much does performance change when the training window is extended back by one more year, holding everything else fixed? Write that sentence down with the real task filled in before any data is touched.
Years in business are not history depth. Count only the years that can be exported from one system with links between records intact, such as tickets with their resolution notes in Zendesk, jobs with invoices and callbacks in ServiceTitan, or opportunities with activities in Salesforce. A company founded decades ago may have three usable years after a migration; the experiment runs on the usable years only.
The task should look like something a buyer would train or evaluate on. Examples include predicting ticket category or escalation from the first message, estimating job duration from the work order, or flagging an invoice exception from its line items. Pick one primary task and at most two secondary tasks; every extra task multiplies the number of comparisons and widens the uncertainty you must report.
Population, sampling and the holdout: which records go in
The population is every eligible record in one system of record over the usable history, after the rights and privacy review has cleared the records for this internal test. De-identify before the experiment, not after, so the test runs on the same kind of records a buyer would receive.
The holdout is a fixed slice of the most recent period, sealed before any training run and never used for tuning. Using the latest period matters: a buyer deploys a model on future records, so the test should ask how well older history helps predict recent work. A random split mixes old and new records in both training and test and hides the effect you are trying to measure.
Leave a gap between the end of the training windows and the start of the holdout when records take time to close. A ticket opened in the last week of training may be resolved in the holdout period, and if its outcome is visible during training the score is inflated. The scikit-learn documentation describes TimeSeriesSplit as a variation of k-fold in which successive training sets are supersets of those before them, which is the same nested shape this protocol uses.
Sample at the level the buyer will use. If the unit is the customer or the asset, not the individual ticket, keep all of a customer's records on one side of the split so near-duplicates do not leak.
The history depth ablation experiment: arms and what each isolates
The core of the design is a backward-nested ladder: window one is the most recent year before the holdout, window two adds the year before it, and so on to the oldest usable year. Every other arm exists to explain what the ladder shows. The arms below are illustrative design choices, not prescribed settings; adapt the window length to how fast the business changes. The run order comes first, and its arm letters refer to the table that follows it.
- Run arm F first as a sanity check: if a random split scores far above the temporal holdout, look for leakage before going further.
- Run arms A and C next; together they answer whether depth adds more than volume.
- Run B, D and E only for the windows where A and C disagree, which keeps compute proportional to the question.
- Run G last on the arms that matter, and only if the holdout contains enough rare cases to score.
| Arm | Training data | What it isolates | Read it as |
|---|---|---|---|
| A. Natural ladder | Nested windows of 1, 2, 3 ... k years, all records in each | Total effect of adding a year: more records plus older records | The headline marginal utility per year |
| B. Fixed-volume ladder | Same windows, each subsampled to the record count of window one | Effect of time span and diversity with volume held constant | If B is flat while A rises, volume is doing the work |
| C. Recent-volume control | Most recent window only, subsampled to several sizes | Learning curve from volume alone, with no older data | The benchmark A must beat to show older years add more than size |
| D. Leave-one-year-out | Full history minus one year at a time | Contribution of each specific year, including harmful ones | A year whose removal improves the score is a candidate to drop |
| E. Regime-break split | Full history minus everything before a known change | Whether records before a system migration, policy or pricing change still help | Decides whether pre-change years belong in the offer |
| F. Random-split control | Same data as the deepest window, random rather than temporal holdout | How much a random split overstates performance | A large gap between F and A means leakage or drift, not value |
| G. Rare-case slice | Arms A and B scored only on rare or high-stakes holdout cases | Whether old years add coverage of uncommon situations | Often where depth matters most to buyers |
Variables to record and disclose
Every run should be reproducible from a variables record. Fill this table in before the first run and publish it next to any result, internally or externally.
| Variable | Type | How to set or measure it | Disclose |
|---|---|---|---|
| Window start and end dates | Independent | Calendar boundaries of each nested window | Exact dates and the record count per window |
| Holdout period and gap | Fixed | Most recent period, sealed; gap sized to typical time-to-close | Dates, count and why the gap length was chosen |
| Model and training recipe | Fixed | One model family and hyperparameter search applied identically to every arm | Model, version, search space and budget |
| Random seeds | Nuisance | Several seeds per arm covering initialization, sampling and order | Number of seeds and the spread across them |
| Primary metric | Dependent | One task metric chosen before training, such as macro F1 or mean absolute error | Metric definition and direction of improvement |
| Schema and label changes by year | Covariate | Field availability, code lists and label definitions per year | A per-year table of fields present and label changes |
| Coverage by year | Covariate | Count of distinct categories, products, regions and rare cases per year | Coverage counts per window |
| Preparation effort by year | Cost | Hours to export, clean, link and de-identify each year | Hours per year, kept separate from model results |
Which metrics turn model scores into marginal utility per year?
Report the marginal utility of year k as the score of window k minus the score of window k minus one, on the same holdout, averaged over seeds. That single difference is the number a reader wants, and it should never appear without its interval.
Fit a learning curve to arm C as well. Hestness and colleagues reported that generalization error in deep learning tends to fall as a power law in training set size across several domains, with the exponent set empirically per problem. If arm A sits on the same curve as arm C, older years help only by adding volume. If A sits above the curve, older years add something volume from recent records does not, such as coverage of seasonal or rare cases. If A sits below it, older records are pulling the model toward conditions that no longer hold.
The scikit-learn guide describes a learning curve as the validation and training score of an estimator for varying numbers of training samples, a way to see how much the model benefits from more data. Plot arms A, B and C on one chart with training records on the horizontal axis.
Record-level valuation methods can complement the year-level view. Ghorbani and Zou proposed data Shapley as a metric to quantify the value of each training datum to predictor performance. Averaging such values by year is more expensive than the ladder but can show whether a year's contribution comes from a few unusual records or from the year as a whole.
Set the utility of each year against its preparation effort, kept in a separate column. Older years often cost more to export and clean because fields were added later, codes changed or attachments sit in an archive. A year with a small positive gain and a large cleaning cost may still be worth offering if it fills a coverage gap, but the decision should be visible rather than assumed.
Translate scores into buyer terms only with explicit assumptions. A point of accuracy is not a price. State what the metric change means for the task, for example fewer misrouted tickets per thousand in the holdout, and leave commercial value to the people negotiating with the full context.
How wide is the interval around one added year, and how is it computed?
Report each marginal utility as a paired difference with a confidence interval, computed over the same holdout records and repeated over seeds. Paired comparison removes the variation that comes from which records happen to be in the holdout, which is usually larger than the effect of one year.
Miller's statistical guide to language model evaluations recommends paired differences when two models answer the same questions and clustered standard errors when questions come in related groups. Apply the same idea here: cluster by customer, site or account if holdout records are grouped.
Randomize more than the seed. Bouthillier and colleagues found that data sampling, initialization and hyperparameter choices all add variance large enough to change benchmark conclusions, and recommended randomizing as many sources of variation as possible and judging improvements with variance in view rather than by averages alone.
Correct for the number of comparisons. A ladder with many years produces many adjacent differences; a few will look significant by chance. Pre-register which comparison is primary, usually the full depth against the most recent window, and treat the rest as descriptive.
Which assumption moves the history depth result most?
The holdout definition has the largest effect. Moving the holdout from a recent temporal slice to a random sample can turn a year that hurts into a year that helps, because a random holdout contains old records that older training years naturally match. Lock the holdout first and report the random-split control so readers can see the size of that gap.
Drift is the second large assumption. The Wild-Time benchmark was built because distribution shift caused by the passage of time is common in real applications, and its authors report that the existing methods they tested did not close the performance gap temporal shift creates. Business records drift through new products, price lists, staff, regulations and systems. Arm E tests the known breaks; the per-year schema table catches the quiet ones.
The third is the task. Depth that adds little to a routing classifier can add a great deal to a forecasting or rare-failure task, where each year contributes a handful of the cases that matter. Run arm G before concluding that old years have no value.
Pre-registration checklist and limits of the protocol
Pre-registration means posting a time-stamped, read-only version of the plan before data collection or analysis, according to the Center for Open Science. For an internal study, a dated document in a shared drive that nobody edits after the first run does the same job. Changes are fine if the final write-up lists them with reasons.
At a glance
Pre-registration checklist and limits of the protocol
Checklist · 10 items
- Task, model and primary metric stated in one sentence, with the direction of improvement.
- System of record named, usable history dates given, and the rights and privacy clearance for internal testing recorded.
- Holdout period, gap length and grouping unit fixed and sealed before training.
- Arms chosen from the design table, with window length and subsample sizes listed.
- Seeds, hyperparameter search and compute budget identical across arms.
- Primary comparison named; secondary comparisons labeled descriptive.
- Interval method stated: paired differences, clustering unit, and multiple-comparison handling.
- Per-year schema, label and coverage table completed before results are seen.
- Stopping rule for dropping a year decided in advance, such as a year whose removal does not reduce the score.
- Limits written down: one task, one model, one company, one holdout period; results do not transfer to other tasks or set a price.
0 of 10 ticked. Tick each item as you go.
What the experiment settles before a licensing conversation
For a company considering licensing, the experiment answers a practical question before anyone negotiates: which years are worth the effort of exporting and preparing, and which add cost without adding much. It also produces an honest description of history depth that a buyer can check.
On depth itself, SourceX's general guide is that buyers tend to want consistent history, usually two years or more, and tens of thousands of records or more in a single system, with exceptions for rare work and every case judged deal by deal. The experiment tells a company which of its years carry that weight. Once a company decides which years to offer, SourceX carries those records from qualification and rights review through contracting, preparation, delivery and payment.
None of this experiment is a precondition for qualifying, since SourceX's first step is a short look at the company that involves no uploaded data.
Frequently asked questions
Is more history always more valuable to an AI buyer?
No. Older years add records, but they also add records from conditions that may no longer hold, such as retired products or old pricing. The ablation design separates the volume effect from the age effect so you can see whether a given year helps the task, does nothing or hurts it.
Why use a recent holdout instead of a random test set?
A buyer deploys a model on future records, so the useful test is how well history predicts the latest period. A random test set mixes old and new records and usually overstates the value of old years. Report the random split only as a control.
How many years of history do I need to run this experiment?
You need at least two usable years plus a holdout period to compare anything, and more years give a longer ladder. If the system was migrated, count only the years that came across with links intact. The protocol works with whatever depth exists; it does not set a minimum.
Does this experiment tell me what my data is worth?
No. It measures how a model's score changes with depth for one task, which is one input to a valuation. Price also depends on rights, exclusivity, buyer need, preparation cost and timing, which this protocol does not measure.
Can a buyer run this on a sample before licensing?
Yes, if an evaluation agreement allows training test models on the sample. The same arms apply, but a small sample widens the intervals, so the buyer should plan the sample size from the uncertainty they can accept.
Sources
- Hestness and colleagues reported that deep learning generalization error tends to follow a power law in training set size, with exponents set empirically per domain. Source
- The Wild-Time benchmark addresses distribution shift over time and reports that tested methods did not close the temporal shift gap. Source
- Ghorbani and Zou proposed data Shapley to quantify the value of each training datum to predictor performance. Source
- Bouthillier and colleagues recommend randomizing data sampling, initialization and hyperparameter search and judging improvements with variance in view. Source
- Miller recommends paired differences on shared questions and clustered standard errors for grouped questions when comparing models. Source
- The Center for Open Science defines preregistration as posting a time-stamped, read-only study plan before data collection or analysis. Source
- The scikit-learn guide defines a learning curve as validation and training score for varying numbers of training samples. Source
- The scikit-learn guide describes TimeSeriesSplit, in which successive training sets are supersets of those that come before them. Source
See if your company qualifies
A short company assessment. No data uploads are needed.