Skip to content

AI uses for records

Dataset refresh incremental value study: set the cost bar first

By SourceX Editorial · Updated

12 sections

Short answer

A dataset refresh incremental value study asks whether newer records improve a model's results by more than the refresh costs to license, review, prepare, retrain and evaluate. Turn the full cost into the smallest gain worth paying for, fix it in advance, then compare newer records with an equal count of older ones in each task category. SourceX has not run this study; this page publishes its method and reporting rules.

Value drivers

Dataset refresh incremental

This page covers dataset refresh incremental value study: set the cost bar first.Illustrative

Key takeaways

  • SourceX has not run the refresh value study; this page fixes its method, cost bar and reporting rules before any data exists.
  • A refresh pays only if its gain over an equal count of older records clears a bar set by its total cost, not by the license fee alone.
  • Each task category is scored separately on tasks dated after the current release, because rules, entities, language and procedures age for different reasons.
  • Results are read by where the whole interval sits against the bar: above, below, crossing it, or blocked by a regression on older work.
  • Public studies show models degrading as their data ages, but none measures business records or prices a refresh.

What would a dataset refresh incremental value study decide?

It would decide, for each kind of task a model serves, whether a delivery of newer records changes results by enough to pay for everything the refresh costs: licensing or collecting the records, reviewing their rights, preparing them, retraining and evaluating. A refresh that raises a score by less than that bill is a measured loss, and the study is built to say so plainly.

SourceX has not conducted this study and has published no results from it. What follows is the commission in full: the definitions, the cost ledger, the comparisons, the task categories, the decision rule and the exhibits a finished report would carry. Fixing the design before any data exists means the bar a refresh must clear is set while nobody knows which side of it the results will fall.

The design builds on the dataset refresh marginal utility protocol already on this site, which asks whether a refresh added information. This study asks whether that information was worth its price, per task category. A useful answer is a per-category interval for the gain from newer records, set against a bar derived from cost, with negative and inconclusive categories reported beside positive ones.

What does public research already show about newer data, and what does it leave open?

Each study below finds that models lose accuracy as the gap widens between their training data and the data they meet, but they measure public text and benchmarks, and none puts a price on a refresh. They justify the study; they cannot replace its results.

Lazaridou and colleagues (NeurIPS 2021) found that Transformer-XL language models predicted text written after their training period less well, and that the shortfall widened with time; larger models did not escape it, while continually updating the models with new data reduced it. Longpre and colleagues (2023) pretrained 28 models of 1.5 billion parameters on corpora of different ages and found that a temporal gap between pretraining and evaluation data degraded performance in a way finetuning did not overcome.

Two further results shape how a refresh should be tested. Luu and colleagues (NAACL 2022) built eight tasks from social media, science papers, news and reviews spanning at least five years, and reported stronger misalignment effects than earlier work; continued pretraining on newer text helped only modestly next to finetuning on task data from the target period. Vela and colleagues (Scientific Reports, 2022), testing 32 datasets with four standard model types, found that model quality could erode gradually or drop abruptly after a long run of good performance. The table sets out what each finding suggests for a refresh decision and what it leaves unmeasured:

What does public research already show about newer data, and what does it leave open?
StudyWhat it measuredWhat it suggests for a refreshWhat it cannot tell a buyer
Lazaridou et al., NeurIPS 2021Language model accuracy on text dated after trainingThe value of newer data can grow with the time gapWhether business task outcomes follow the same curve
Longpre et al., 202328 pretrained models built on data of different agesStaleness baked in at pretraining may survive finetuningWhether a finetuning-stage refresh repays its cost
Luu et al., NAACL 2022Eight tasks in four domains over five or more yearsTask data from the target period helps more than extra raw textWhich business task categories are most time-sensitive
Vela et al., Scientific Reports 2022Model quality over time on 32 datasets, four model typesDecay can be sudden, so a calendar schedule can miss itWhat a refresh delivered before a drop is worth

Which definitions have to be fixed before any refresh is scored?

The dates every record carries, and three quantities: incremental value, total refresh cost and the value bar. Most arguments about whether new data helped turn out to be arguments about which date a record carries or what counts as a gain, and both can be settled before anyone looks at a score.

Business records run on several clocks at once. A support ticket has the day the customer wrote, the day it was resolved, the day it was exported and the day it reached the buyer, and the refund rule that governed it has its own effective date. The study dates every record and every evaluation task by the event that decided its outcome, because only that date shows which rules were in force when the work happened.

Which definitions have to be fixed before any refresh is scored?
TermDefinition in this studyRecorded fromMistake it prevents
RefreshOne delivery of records dated after the newest record in the release the model already usesThe delivery manifest and record event datesCounting re-exports of old records as new
Event dateThe date of the action that decided a record's outcome, such as resolution or approvalThe source system's status historyUsing export or delivery dates, which make everything look recent
Freshness gapDays between the newest training record and the event date of the task being scoredTraining manifest and task metadataScoring only old tasks and concluding newer data adds nothing
Newer-record setRefresh records left after duplicates and unchanged revisions are removed using the protocol's novelty classesThe novelty auditCrediting a refresh for records the model has already seen
Incremental valueGain on the primary metric from adding newer records, minus the gain from adding the same count of older recordsPaired scores on the same tasksPaying for volume and calling it recency
Total refresh costEvery cost the refresh causes, from acquisition to release review, in one ledgerInvoices, time logs and compute recordsComparing a gain with the license fee alone
Value barThe smallest incremental value that would repay the total refresh cost over the refresh's useful lifeThe cost ledger and a stated value per metric pointTreating any positive gain as a success

How is the total cost of a refresh counted?

From a ledger, line by line, recorded as the work happens rather than reconstructed afterward. The license fee or collection effort is the line teams remember, and it is rarely the only one that matters.

Rights review recurs with every delivery rather than ending at setup. Each refresh may draw on new sources or arrive under amended terms, and license information does not reliably travel with data: the Data Provenance Initiative's 2023 audit of more than 1,800 text datasets found license omission of over 70 percent, and error rates above 50 percent, on widely used dataset hosting sites. Evaluation grows quietly too. Anthropic's guide to agent evaluations runs several trials per task because model outputs vary from run to run, and every extra trial, grading pass and rebuilt holdout belongs to the refresh's cost.

The ledger then becomes a bar. Divide the total refresh cost by the value the team assigns to one point of the primary metric over the period the refresh will serve before it is replaced. In an illustrative case, a refresh costing the equivalent of 400 staff hours, for a deployment where each point of task accuracy saves 50 staff hours over that period, has to add at least 8 points beyond the older-record control to pay for itself. The value per point comes from the team's own business case and is stated in the plan; the study tests the gain, not the business case. The ledger has six lines:

How is the total cost of a refresh counted?
Cost lineWhat to countUnit to recordOften left out
AcquisitionLicense fee for the refresh, or internal collection and export effortInvoice amount or hoursNegotiation and contract amendment time
Rights and provenance reviewChecking permitted use, sources and consents for the new recordsReviewer hoursRe-review when sources or terms changed since the last release
PreparationDe-identification, normalization and deduplication against the prior releaseHours and processing costRepairing schema changes between releases
EvaluationBuilding dated holdouts, grading and repeated trialsHours and computeRebuilding holdouts the refresh contaminated
RetrainingTraining runs for every arm at the agreed budgetAccelerator hoursFailed and repeated runs
Release reviewRegression checks, sign-off and rollback readinessHoursMonitoring after release

Which comparisons separate newer records from simply more records?

Three training sets, run on the same budget and scored on the same dated tasks: the current release alone, the current release plus the newer-record set, and the current release plus an equal number of older records drawn by the same sampling rule. Incremental value is the gap between the second and the third; the gap between the second and the first mixes recency with volume.

The older-record control carries more weight than it appears to. Muennighoff and colleagues (2023) found that, at a fixed compute budget, training for up to four epochs on repeated data changed language model loss negligibly compared with unique data, with returns from further repetition eventually falling to zero. When reusing what a team already owns comes that close to new text, a refresh has to beat reuse and older volume, not just the previous model.

The marginal utility protocol specifies those arms, budgets, seeds and the holdout freeze. This study adds three requirements:

  • Every arm runs per task category, so a category where newer records help cannot mask one where they hurt.
  • The older-record pool comes from the same systems and segments as the refresh; when no such pool exists, the report says so and substitutes a learning curve on the current release.
  • No raw gain is reported alone: each category's incremental value appears next to its value bar and the cost ledger behind it.

How can an AI team run a historical dataset freshness experiment across task categories?

By sorting evaluation tasks into categories that differ in why time should matter, writing a prediction for each before any training, and scoring newer against older records separately in every category on tasks dated after the current release. The experiment locates where recency adds value, and a category where it adds none is as useful a result as one where it adds a lot.

Four categories, separated by the mechanism that would make old records wrong, cover most business work: rules that change, entities that turn over, language that drifts and procedures that stay put. The examples and predictions in the table are the study's starting hypotheses, written so that they can be refuted, and none of them is a finding.

Each category gets its own holdout of tasks dated after the newest record in the current release, cut at known policy or system changes so one change cannot dominate a category's score. Testing several categories at once needs a correction: SciPy's false_discovery_control function implements the Benjamini-Hochberg procedure and the more conservative Benjamini-Yekutieli variant for tests that are not independent, which category tests on one model rarely are. The experiment runs in six steps, and the category table after them holds the starting hypotheses:

  • Tag every evaluation task with its category, event date and source system, using rules fixed before the refresh arrives.
  • Write one prediction per category and the result that would refute it, then date the plan.
  • Build each category's holdout from tasks dated after the current release, cut at known policy and system changes, and check it for overlap with every training set.
  • Run the three training sets per category on equal budgets, with repeated trials.
  • Report each category's paired gain with its interval, its novelty share and its corrected conclusion, beside its value bar.
  • Publish every category, including those where newer records changed nothing or made results worse.
How can an AI team run a historical dataset freshness experiment across task categories?
Task categoryBusiness exampleWhy newer records might matterPrediction written in advanceResult that would refute it
Rule-boundRefund decisions in Zendesk tickets after a policy changeOlder records teach the superseded ruleLargest gain from newer recordsNo gain beyond the older-record control on post-change tasks
Entity-boundMatching invoices to the current NetSuite vendor listNew vendors, products and prices appear only in recent recordsGain concentrated on tasks naming new entitiesEqual gain on tasks naming only long-standing entities
Language-boundDrafting customer replies in HubSpotTerminology and tone drift slowlySmall gain that grows with the freshness gapA gain that does not change with the gap
Stable procedureRouting Jira issues to the owning componentThe steps rarely changeNo gain beyond volumeA clear gain from newer records over older ones

How will each category's result be read against the value bar?

By where the whole interval sits, not where the point estimate lands. A category's refresh pays for itself only when the interval for its incremental value lies entirely above the bar; it does not pay when the interval lies entirely below; an interval that crosses the bar is inconclusive and is reported that way.

This rule is what makes negative results publishable. Lakens (Social Psychological and Personality Science, 2017) argues that a nonsignificant result does not by itself show an effect is absent, and recommends equivalence tests whose bounds sit at the smallest effect size of interest. Here the value bar plays that role: an interval wholly below it is positive evidence that the refresh does not repay its cost in that category, a finding a buyer can act on rather than a failure to find anything.

One outcome overrides the others. A refresh that lifts recent tasks while breaking older ones has not earned its place, however good its headline. Anthropic's evaluation guide describes moving capability evals with high pass rates into a regression suite that runs continuously to catch backsliding; the counterpart here is a prior-period holdout that every training set must hold. The four outcomes and what each triggers:

How will each category's result be read against the value bar?
OutcomeWhere the interval sitsDecision for that categoryHow the report shows it
Pays for itselfEntirely above the value barTake the refresh for this categoryInterval, bar and cost ledger side by side
Does not payEntirely below the value bar, including below zeroSkip or narrow the refresh for this categorySame format and position as a positive result
InconclusiveCrosses the value barNo decision; add tasks or stop, as the plan specifiesThe interval width and the task count needed to resolve it
BlockedAnywhere, with a regression on the prior-period holdoutHold the release until the regression is fixedRegression size and the tasks affected
A six-line refresh cost ledger converts into a dashed value bar, and four illustrative task-category intervals are read against it: one entirely above (pays), one crossing (inconclusive), one entirely below (does not pay) and one blocked by a regression.
How the study will read each category against its cost-derived bar. Illustrative positions; no results have been measured.

What will the published report contain, and what has to be true before it appears?

Six exhibits, each with a denominator and a result that would count against the refresh, released only once five conditions hold. A report missing any of them is a draft, whoever publishes it.

Rights and disclosure head the conditions because the study runs on records that belong to the companies that supplied them. Aggregate, anonymized reporting needs approval from those companies and from any buyer whose tasks are scored, and no category may be reported in a way that identifies a supplier, a buyer or an acceptance criterion. The release conditions are below, followed by the exhibits:

  • The plan, categories, predictions, value bars and analysis are dated before the refresh is delivered, and every later change is listed with its reason.
  • Data owners and buyers have approved aggregate, anonymized disclosure.
  • Costs come from recorded invoices, time logs and compute records, never from estimates made after scoring.
  • Every category is reported in the same format, inconclusive and negative ones included.
  • Category conclusions carry the false-discovery correction, and categories below the minimum task count set in the plan are marked underpowered instead of dropped.
What will the published report contain, and what has to be true before it appears?
ExhibitWhat it showsDenominatorResult that would count against the refresh
1. Refresh inventoryRecords delivered, novelty classes and event-date range per categoryAll records in the refreshA small newer-record share in categories predicted to need recency
2. Cost ledgerEach cost line in recorded units, and the totalThe whole refreshLines missing, or estimated after the fact
3. Value bar derivationTotal cost, value per point and useful life behind each barOne bar per categoryA bar changed after scoring
4. Category resultsIncremental value with its interval beside each barScored tasks per category after overlap removalIntervals entirely below the bar
5. Regression panelPrior-period holdout scores for every training setPrior-period tasksAny set with newer records scoring below the current release
6. Gap profileIncremental value by freshness-gap bandScored tasks per bandNo link between gap and gain where time sensitivity was predicted

Which alternative explanations could fake a refresh's value, or hide it?

Volume and compute are the strongest alternative explanations for an apparent gain, and the equal-count older control on equal budgets exists to remove them. A refresh can also look worthless for reasons unrelated to its records. Until the study runs, every cell of its results is unknown, and nothing here forecasts them. The ways value can be faked or hidden:

At a glance

Which alternative explanations could fake a refresh's value, or hide it

  • Contaminated tasks: when evaluation items or their answers leak into training, older training sets score well for the wrong reason. GSM1k's newly written problems, built to mirror GSM8k, exposed accuracy drops of up to 8% in the paper's current version, which is why holdouts use tasks dated after the current release and are checked for overlap.
  • An understated ledger: leaving out rights review, evaluation or failed runs lowers the bar and flatters the refresh, so exhibit 2 shows every line in recorded units.
  • An inflated value per point: a higher claimed value lowers the bar, so the plan states it before scoring and the report shows whether the decision survives halving it.
  • Pooled categories: an average across categories can show a modest gain while one category regresses, so no pooled figure appears without the per-category results beside it.
  • Old tasks only: scoring on prior-period work hides recency value by construction.
  • A single snapshot: because quality can hold steady and then fall abruptly, as Vela and colleagues observed, one study date can miss a shift that arrives later; the report states its date and treats its result as dated.
  • Records without outcomes: Luu and colleagues found task data from the target period worth more than extra raw text, so a refresh lacking resolved outcomes may underperform one that has them, and the report states which kind was tested.

Where would a well-run refresh study still mislead?

In transfer, in cost and in coverage. The design measures one model, one set of categories and one cost structure at one date, and each of those can shift after the report appears.

  • One model: a large model trained on the current release may gain less from newer records than a smaller one, so results do not carry across models without a new run.
  • Local costs: the bar reflects one team's costs and value per point, and a team with cheaper review or a higher-value use could reach the opposite decision from identical gains.
  • Thin categories: categories with few dated tasks produce wide intervals and many inconclusive results, which signal a need for more tasks rather than the absence of an effect.
  • Public evidence gap: the published decay studies measure public text and benchmarks, and business records with explicit rules and outcomes may behave differently, which is the reason to run the study at all.
  • Permission: the experiment measures performance; whether a refresh may be used for a purpose depends on the agreement covering it.

How can a buyer request refresh records that this study could measure?

Ask for records that carry what the study needs: event dates from the source system, stable case identifiers linking a refresh to earlier releases, a task category or segment per record, and notes of policy or system changes. Without them, neither the newer-record set nor the dated holdouts can be built.

SourceX keeps no datasets on the shelf and sources each one from U.S. businesses when a buyer requests it. The request-data form takes the buyer's scope, volume, language, format, timing and licensing needs, and a refresh study should add its intended use and an acceptance test there too. The older-record control needs depth as well as recency, and as a general guide buyers look for years of consistent history, typically two or more, in one system. A refresh also has to respect the deal sequence: no records are delivered until the supplying company has given its approval to the scope and terms and its signature to the agreement, so newer records move only inside a scope already approved and signed. SourceX has not run this study and reports no refresh results here.

Methods note: what this design rests on and what no one has measured yet

The definitions, cost ledger, value-bar rule, task categories, decision table, exhibits and release conditions are editorial frameworks written for this page on 2026-10-26. Public evidence comes from the studies of temporal decay, data repetition and benchmark contamination named above, the Data Provenance Initiative audit, Anthropic's evaluation guide, SciPy's documentation and Lakens (2017), each checked on its publisher's domain.

The 400-hour, 50-hour and 8-point figures form an illustrative calculation, not a measurement, and the figure's interval positions are illustrative too. Not measured: any refresh's incremental value, cost, value bar or category result at SourceX or anywhere else; no prediction in the category table has been tested. Each needs refresh deliveries, recorded costs and disclosure approvals that do not yet exist, and the report will not appear until they do.

Frequently asked questions

Has SourceX published results from a dataset refresh value study?

No. SourceX has not conducted the study and has published no results from it. This page publishes the design first: definitions, a cost ledger, a cost-derived value bar, per-category comparisons, a decision table and the exhibits a finished report would include.

Is fresher training data always worth buying?

Not necessarily. Published studies show models losing accuracy as the gap between training and test data grows, but a refresh also costs money and staff time to license, review, prepare and evaluate. It is worth buying only where its gain over older records of the same volume clears the bar its total cost sets.

How do you turn the cost of a refresh into a threshold?

Add every cost the refresh causes, from acquisition and rights review to evaluation and retraining, then divide by the value of one point of the primary metric over the refresh's useful life. The result is the smallest gain, in metric points, that would repay the refresh. Write it into the plan before anything is scored.

Can a refresh be worth it for one task category and not another?

Yes, and the experiment is designed to find out where. Tasks governed by changing rules or new entities may respond differently from stable procedures, so each category is scored on its own dated holdout and judged against its own bar, with a correction for testing several categories at once.

What counts as a negative result in a refresh study?

An interval for incremental value that lies entirely below the value bar. That is evidence the refresh does not repay its cost in that category, not a failure to detect an effect, and the report presents it in the same format and position as a positive result.

Sources

  • Lazaridou and colleagues (NeurIPS 2021) found Transformer-XL language models performed worse on text from beyond their training period, increasingly so over time; larger models did not avoid the decline, and continually updating with new data reduced it. Source
  • Longpre and colleagues (2023) pretrained 28 models of 1.5 billion parameters and found that a temporal shift between pretraining and evaluation data degrades performance in a way finetuning does not overcome. Source
  • Luu and colleagues (NAACL 2022) built eight tasks across social media, science papers, news and reviews spanning five or more years, found stronger temporal misalignment effects than previously reported, and found continued pretraining helped modestly compared with finetuning on data from the target period. Source
  • Vela and colleagues (Scientific Reports, 2022) tested 32 datasets with four standard AI models and found model quality can degrade gradually or escalate abruptly after a long period of good performance. Source
  • Muennighoff and colleagues (2023) found that at a fixed compute budget, training with up to four epochs of repeated data yields negligible changes to loss compared with unique data, and that returns from more repetition eventually diminish to zero. Source
  • The current version of the GSM1k paper (Zhang and colleagues) reports accuracy drops of up to 8% on newly written problems that mirror GSM8k, with several model families showing evidence of overfitting. Source
  • The Data Provenance Initiative's 2023 audit of more than 1,800 text datasets found license omission of over 70% and error rates above 50% on widely used dataset hosting sites. Source
  • Anthropic's engineering guide to agent evaluations runs multiple trials per task because model outputs vary between runs, and describes moving high-pass-rate capability evals into a continuously run regression suite to catch backsliding. Source
  • SciPy's false_discovery_control adjusts p-values to control the false discovery rate using the Benjamini-Hochberg procedure, or the more conservative Benjamini-Yekutieli procedure that remains valid when tests are not independent. Source
  • Lakens (Social Psychological and Personality Science, 2017) argues a nonsignificant result alone does not justify concluding an effect is absent and recommends equivalence tests with bounds set at the smallest effect size of interest. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies