AI uses for records
Dataset refresh incremental value study: set the cost bar first
By SourceX Editorial · Updated
12 sections
Short answer
A dataset refresh incremental value study asks whether newer records improve a model's results by more than the refresh costs to license, review, prepare, retrain and evaluate. Turn the full cost into the smallest gain worth paying for, fix it in advance, then compare newer records with an equal count of older ones in each task category. SourceX has not run this study; this page publishes its method and reporting rules.
Value drivers
Dataset refresh incremental
Key takeaways
- SourceX has not run the refresh value study; this page fixes its method, cost bar and reporting rules before any data exists.
- A refresh pays only if its gain over an equal count of older records clears a bar set by its total cost, not by the license fee alone.
- Each task category is scored separately on tasks dated after the current release, because rules, entities, language and procedures age for different reasons.
- Results are read by where the whole interval sits against the bar: above, below, crossing it, or blocked by a regression on older work.
- Public studies show models degrading as their data ages, but none measures business records or prices a refresh.
What would a dataset refresh incremental value study decide?
It would decide, for each kind of task a model serves, whether a delivery of newer records changes results by enough to pay for everything the refresh costs: licensing or collecting the records, reviewing their rights, preparing them, retraining and evaluating. A refresh that raises a score by less than that bill is a measured loss, and the study is built to say so plainly.
SourceX has not conducted this study and has published no results from it. What follows is the commission in full: the definitions, the cost ledger, the comparisons, the task categories, the decision rule and the exhibits a finished report would carry. Fixing the design before any data exists means the bar a refresh must clear is set while nobody knows which side of it the results will fall.
The design builds on the dataset refresh marginal utility protocol already on this site, which asks whether a refresh added information. This study asks whether that information was worth its price, per task category. A useful answer is a per-category interval for the gain from newer records, set against a bar derived from cost, with negative and inconclusive categories reported beside positive ones.
What does public research already show about newer data, and what does it leave open?
Each study below finds that models lose accuracy as the gap widens between their training data and the data they meet, but they measure public text and benchmarks, and none puts a price on a refresh. They justify the study; they cannot replace its results.
Lazaridou and colleagues (NeurIPS 2021) found that Transformer-XL language models predicted text written after their training period less well, and that the shortfall widened with time; larger models did not escape it, while continually updating the models with new data reduced it. Longpre and colleagues (2023) pretrained 28 models of 1.5 billion parameters on corpora of different ages and found that a temporal gap between pretraining and evaluation data degraded performance in a way finetuning did not overcome.
Two further results shape how a refresh should be tested. Luu and colleagues (NAACL 2022) built eight tasks from social media, science papers, news and reviews spanning at least five years, and reported stronger misalignment effects than earlier work; continued pretraining on newer text helped only modestly next to finetuning on task data from the target period. Vela and colleagues (Scientific Reports, 2022), testing 32 datasets with four standard model types, found that model quality could erode gradually or drop abruptly after a long run of good performance. The table sets out what each finding suggests for a refresh decision and what it leaves unmeasured:
| Study | What it measured | What it suggests for a refresh | What it cannot tell a buyer |
|---|---|---|---|
| Lazaridou et al., NeurIPS 2021 | Language model accuracy on text dated after training | The value of newer data can grow with the time gap | Whether business task outcomes follow the same curve |
| Longpre et al., 2023 | 28 pretrained models built on data of different ages | Staleness baked in at pretraining may survive finetuning | Whether a finetuning-stage refresh repays its cost |
| Luu et al., NAACL 2022 | Eight tasks in four domains over five or more years | Task data from the target period helps more than extra raw text | Which business task categories are most time-sensitive |
| Vela et al., Scientific Reports 2022 | Model quality over time on 32 datasets, four model types | Decay can be sudden, so a calendar schedule can miss it | What a refresh delivered before a drop is worth |
Which definitions have to be fixed before any refresh is scored?
The dates every record carries, and three quantities: incremental value, total refresh cost and the value bar. Most arguments about whether new data helped turn out to be arguments about which date a record carries or what counts as a gain, and both can be settled before anyone looks at a score.
Business records run on several clocks at once. A support ticket has the day the customer wrote, the day it was resolved, the day it was exported and the day it reached the buyer, and the refund rule that governed it has its own effective date. The study dates every record and every evaluation task by the event that decided its outcome, because only that date shows which rules were in force when the work happened.
| Term | Definition in this study | Recorded from | Mistake it prevents |
|---|---|---|---|
| Refresh | One delivery of records dated after the newest record in the release the model already uses | The delivery manifest and record event dates | Counting re-exports of old records as new |
| Event date | The date of the action that decided a record's outcome, such as resolution or approval | The source system's status history | Using export or delivery dates, which make everything look recent |
| Freshness gap | Days between the newest training record and the event date of the task being scored | Training manifest and task metadata | Scoring only old tasks and concluding newer data adds nothing |
| Newer-record set | Refresh records left after duplicates and unchanged revisions are removed using the protocol's novelty classes | The novelty audit | Crediting a refresh for records the model has already seen |
| Incremental value | Gain on the primary metric from adding newer records, minus the gain from adding the same count of older records | Paired scores on the same tasks | Paying for volume and calling it recency |
| Total refresh cost | Every cost the refresh causes, from acquisition to release review, in one ledger | Invoices, time logs and compute records | Comparing a gain with the license fee alone |
| Value bar | The smallest incremental value that would repay the total refresh cost over the refresh's useful life | The cost ledger and a stated value per metric point | Treating any positive gain as a success |
How is the total cost of a refresh counted?
From a ledger, line by line, recorded as the work happens rather than reconstructed afterward. The license fee or collection effort is the line teams remember, and it is rarely the only one that matters.
Rights review recurs with every delivery rather than ending at setup. Each refresh may draw on new sources or arrive under amended terms, and license information does not reliably travel with data: the Data Provenance Initiative's 2023 audit of more than 1,800 text datasets found license omission of over 70 percent, and error rates above 50 percent, on widely used dataset hosting sites. Evaluation grows quietly too. Anthropic's guide to agent evaluations runs several trials per task because model outputs vary from run to run, and every extra trial, grading pass and rebuilt holdout belongs to the refresh's cost.
The ledger then becomes a bar. Divide the total refresh cost by the value the team assigns to one point of the primary metric over the period the refresh will serve before it is replaced. In an illustrative case, a refresh costing the equivalent of 400 staff hours, for a deployment where each point of task accuracy saves 50 staff hours over that period, has to add at least 8 points beyond the older-record control to pay for itself. The value per point comes from the team's own business case and is stated in the plan; the study tests the gain, not the business case. The ledger has six lines:
| Cost line | What to count | Unit to record | Often left out |
|---|---|---|---|
| Acquisition | License fee for the refresh, or internal collection and export effort | Invoice amount or hours | Negotiation and contract amendment time |
| Rights and provenance review | Checking permitted use, sources and consents for the new records | Reviewer hours | Re-review when sources or terms changed since the last release |
| Preparation | De-identification, normalization and deduplication against the prior release | Hours and processing cost | Repairing schema changes between releases |
| Evaluation | Building dated holdouts, grading and repeated trials | Hours and compute | Rebuilding holdouts the refresh contaminated |
| Retraining | Training runs for every arm at the agreed budget | Accelerator hours | Failed and repeated runs |
| Release review | Regression checks, sign-off and rollback readiness | Hours | Monitoring after release |
Which comparisons separate newer records from simply more records?
Three training sets, run on the same budget and scored on the same dated tasks: the current release alone, the current release plus the newer-record set, and the current release plus an equal number of older records drawn by the same sampling rule. Incremental value is the gap between the second and the third; the gap between the second and the first mixes recency with volume.
The older-record control carries more weight than it appears to. Muennighoff and colleagues (2023) found that, at a fixed compute budget, training for up to four epochs on repeated data changed language model loss negligibly compared with unique data, with returns from further repetition eventually falling to zero. When reusing what a team already owns comes that close to new text, a refresh has to beat reuse and older volume, not just the previous model.
The marginal utility protocol specifies those arms, budgets, seeds and the holdout freeze. This study adds three requirements:
- Every arm runs per task category, so a category where newer records help cannot mask one where they hurt.
- The older-record pool comes from the same systems and segments as the refresh; when no such pool exists, the report says so and substitutes a learning curve on the current release.
- No raw gain is reported alone: each category's incremental value appears next to its value bar and the cost ledger behind it.
How can an AI team run a historical dataset freshness experiment across task categories?
By sorting evaluation tasks into categories that differ in why time should matter, writing a prediction for each before any training, and scoring newer against older records separately in every category on tasks dated after the current release. The experiment locates where recency adds value, and a category where it adds none is as useful a result as one where it adds a lot.
Four categories, separated by the mechanism that would make old records wrong, cover most business work: rules that change, entities that turn over, language that drifts and procedures that stay put. The examples and predictions in the table are the study's starting hypotheses, written so that they can be refuted, and none of them is a finding.
Each category gets its own holdout of tasks dated after the newest record in the current release, cut at known policy or system changes so one change cannot dominate a category's score. Testing several categories at once needs a correction: SciPy's false_discovery_control function implements the Benjamini-Hochberg procedure and the more conservative Benjamini-Yekutieli variant for tests that are not independent, which category tests on one model rarely are. The experiment runs in six steps, and the category table after them holds the starting hypotheses:
- Tag every evaluation task with its category, event date and source system, using rules fixed before the refresh arrives.
- Write one prediction per category and the result that would refute it, then date the plan.
- Build each category's holdout from tasks dated after the current release, cut at known policy and system changes, and check it for overlap with every training set.
- Run the three training sets per category on equal budgets, with repeated trials.
- Report each category's paired gain with its interval, its novelty share and its corrected conclusion, beside its value bar.
- Publish every category, including those where newer records changed nothing or made results worse.
| Task category | Business example | Why newer records might matter | Prediction written in advance | Result that would refute it |
|---|---|---|---|---|
| Rule-bound | Refund decisions in Zendesk tickets after a policy change | Older records teach the superseded rule | Largest gain from newer records | No gain beyond the older-record control on post-change tasks |
| Entity-bound | Matching invoices to the current NetSuite vendor list | New vendors, products and prices appear only in recent records | Gain concentrated on tasks naming new entities | Equal gain on tasks naming only long-standing entities |
| Language-bound | Drafting customer replies in HubSpot | Terminology and tone drift slowly | Small gain that grows with the freshness gap | A gain that does not change with the gap |
| Stable procedure | Routing Jira issues to the owning component | The steps rarely change | No gain beyond volume | A clear gain from newer records over older ones |
How will each category's result be read against the value bar?
By where the whole interval sits, not where the point estimate lands. A category's refresh pays for itself only when the interval for its incremental value lies entirely above the bar; it does not pay when the interval lies entirely below; an interval that crosses the bar is inconclusive and is reported that way.
This rule is what makes negative results publishable. Lakens (Social Psychological and Personality Science, 2017) argues that a nonsignificant result does not by itself show an effect is absent, and recommends equivalence tests whose bounds sit at the smallest effect size of interest. Here the value bar plays that role: an interval wholly below it is positive evidence that the refresh does not repay its cost in that category, a finding a buyer can act on rather than a failure to find anything.
One outcome overrides the others. A refresh that lifts recent tasks while breaking older ones has not earned its place, however good its headline. Anthropic's evaluation guide describes moving capability evals with high pass rates into a regression suite that runs continuously to catch backsliding; the counterpart here is a prior-period holdout that every training set must hold. The four outcomes and what each triggers:
| Outcome | Where the interval sits | Decision for that category | How the report shows it |
|---|---|---|---|
| Pays for itself | Entirely above the value bar | Take the refresh for this category | Interval, bar and cost ledger side by side |
| Does not pay | Entirely below the value bar, including below zero | Skip or narrow the refresh for this category | Same format and position as a positive result |
| Inconclusive | Crosses the value bar | No decision; add tasks or stop, as the plan specifies | The interval width and the task count needed to resolve it |
| Blocked | Anywhere, with a regression on the prior-period holdout | Hold the release until the regression is fixed | Regression size and the tasks affected |
What will the published report contain, and what has to be true before it appears?
Six exhibits, each with a denominator and a result that would count against the refresh, released only once five conditions hold. A report missing any of them is a draft, whoever publishes it.
Rights and disclosure head the conditions because the study runs on records that belong to the companies that supplied them. Aggregate, anonymized reporting needs approval from those companies and from any buyer whose tasks are scored, and no category may be reported in a way that identifies a supplier, a buyer or an acceptance criterion. The release conditions are below, followed by the exhibits:
- The plan, categories, predictions, value bars and analysis are dated before the refresh is delivered, and every later change is listed with its reason.
- Data owners and buyers have approved aggregate, anonymized disclosure.
- Costs come from recorded invoices, time logs and compute records, never from estimates made after scoring.
- Every category is reported in the same format, inconclusive and negative ones included.
- Category conclusions carry the false-discovery correction, and categories below the minimum task count set in the plan are marked underpowered instead of dropped.
| Exhibit | What it shows | Denominator | Result that would count against the refresh |
|---|---|---|---|
| 1. Refresh inventory | Records delivered, novelty classes and event-date range per category | All records in the refresh | A small newer-record share in categories predicted to need recency |
| 2. Cost ledger | Each cost line in recorded units, and the total | The whole refresh | Lines missing, or estimated after the fact |
| 3. Value bar derivation | Total cost, value per point and useful life behind each bar | One bar per category | A bar changed after scoring |
| 4. Category results | Incremental value with its interval beside each bar | Scored tasks per category after overlap removal | Intervals entirely below the bar |
| 5. Regression panel | Prior-period holdout scores for every training set | Prior-period tasks | Any set with newer records scoring below the current release |
| 6. Gap profile | Incremental value by freshness-gap band | Scored tasks per band | No link between gap and gain where time sensitivity was predicted |
Which alternative explanations could fake a refresh's value, or hide it?
Volume and compute are the strongest alternative explanations for an apparent gain, and the equal-count older control on equal budgets exists to remove them. A refresh can also look worthless for reasons unrelated to its records. Until the study runs, every cell of its results is unknown, and nothing here forecasts them. The ways value can be faked or hidden:
At a glance
Which alternative explanations could fake a refresh's value, or hide it
- Contaminated tasks: when evaluation items or their answers leak into training, older training sets score well for the wrong reason. GSM1k's newly written problems, built to mirror GSM8k, exposed accuracy drops of up to 8% in the paper's current version, which is why holdouts use tasks dated after the current release and are checked for overlap.
- An understated ledger: leaving out rights review, evaluation or failed runs lowers the bar and flatters the refresh, so exhibit 2 shows every line in recorded units.
- An inflated value per point: a higher claimed value lowers the bar, so the plan states it before scoring and the report shows whether the decision survives halving it.
- Pooled categories: an average across categories can show a modest gain while one category regresses, so no pooled figure appears without the per-category results beside it.
- Old tasks only: scoring on prior-period work hides recency value by construction.
- A single snapshot: because quality can hold steady and then fall abruptly, as Vela and colleagues observed, one study date can miss a shift that arrives later; the report states its date and treats its result as dated.
- Records without outcomes: Luu and colleagues found task data from the target period worth more than extra raw text, so a refresh lacking resolved outcomes may underperform one that has them, and the report states which kind was tested.
Where would a well-run refresh study still mislead?
In transfer, in cost and in coverage. The design measures one model, one set of categories and one cost structure at one date, and each of those can shift after the report appears.
- One model: a large model trained on the current release may gain less from newer records than a smaller one, so results do not carry across models without a new run.
- Local costs: the bar reflects one team's costs and value per point, and a team with cheaper review or a higher-value use could reach the opposite decision from identical gains.
- Thin categories: categories with few dated tasks produce wide intervals and many inconclusive results, which signal a need for more tasks rather than the absence of an effect.
- Public evidence gap: the published decay studies measure public text and benchmarks, and business records with explicit rules and outcomes may behave differently, which is the reason to run the study at all.
- Permission: the experiment measures performance; whether a refresh may be used for a purpose depends on the agreement covering it.
How can a buyer request refresh records that this study could measure?
Ask for records that carry what the study needs: event dates from the source system, stable case identifiers linking a refresh to earlier releases, a task category or segment per record, and notes of policy or system changes. Without them, neither the newer-record set nor the dated holdouts can be built.
SourceX keeps no datasets on the shelf and sources each one from U.S. businesses when a buyer requests it. The request-data form takes the buyer's scope, volume, language, format, timing and licensing needs, and a refresh study should add its intended use and an acceptance test there too. The older-record control needs depth as well as recency, and as a general guide buyers look for years of consistent history, typically two or more, in one system. A refresh also has to respect the deal sequence: no records are delivered until the supplying company has given its approval to the scope and terms and its signature to the agreement, so newer records move only inside a scope already approved and signed. SourceX has not run this study and reports no refresh results here.
Methods note: what this design rests on and what no one has measured yet
The definitions, cost ledger, value-bar rule, task categories, decision table, exhibits and release conditions are editorial frameworks written for this page on 2026-10-26. Public evidence comes from the studies of temporal decay, data repetition and benchmark contamination named above, the Data Provenance Initiative audit, Anthropic's evaluation guide, SciPy's documentation and Lakens (2017), each checked on its publisher's domain.
The 400-hour, 50-hour and 8-point figures form an illustrative calculation, not a measurement, and the figure's interval positions are illustrative too. Not measured: any refresh's incremental value, cost, value bar or category result at SourceX or anywhere else; no prediction in the category table has been tested. Each needs refresh deliveries, recorded costs and disclosure approvals that do not yet exist, and the report will not appear until they do.
Frequently asked questions
Has SourceX published results from a dataset refresh value study?
No. SourceX has not conducted the study and has published no results from it. This page publishes the design first: definitions, a cost ledger, a cost-derived value bar, per-category comparisons, a decision table and the exhibits a finished report would include.
Is fresher training data always worth buying?
Not necessarily. Published studies show models losing accuracy as the gap between training and test data grows, but a refresh also costs money and staff time to license, review, prepare and evaluate. It is worth buying only where its gain over older records of the same volume clears the bar its total cost sets.
How do you turn the cost of a refresh into a threshold?
Add every cost the refresh causes, from acquisition and rights review to evaluation and retraining, then divide by the value of one point of the primary metric over the refresh's useful life. The result is the smallest gain, in metric points, that would repay the refresh. Write it into the plan before anything is scored.
Can a refresh be worth it for one task category and not another?
Yes, and the experiment is designed to find out where. Tasks governed by changing rules or new entities may respond differently from stable procedures, so each category is scored on its own dated holdout and judged against its own bar, with a correction for testing several categories at once.
What counts as a negative result in a refresh study?
An interval for incremental value that lies entirely below the value bar. That is evidence the refresh does not repay its cost in that category, not a failure to detect an effect, and the report presents it in the same format and position as a positive result.
Sources
- Lazaridou and colleagues (NeurIPS 2021) found Transformer-XL language models performed worse on text from beyond their training period, increasingly so over time; larger models did not avoid the decline, and continually updating with new data reduced it. Source
- Longpre and colleagues (2023) pretrained 28 models of 1.5 billion parameters and found that a temporal shift between pretraining and evaluation data degrades performance in a way finetuning does not overcome. Source
- Luu and colleagues (NAACL 2022) built eight tasks across social media, science papers, news and reviews spanning five or more years, found stronger temporal misalignment effects than previously reported, and found continued pretraining helped modestly compared with finetuning on data from the target period. Source
- Vela and colleagues (Scientific Reports, 2022) tested 32 datasets with four standard AI models and found model quality can degrade gradually or escalate abruptly after a long period of good performance. Source
- Muennighoff and colleagues (2023) found that at a fixed compute budget, training with up to four epochs of repeated data yields negligible changes to loss compared with unique data, and that returns from more repetition eventually diminish to zero. Source
- The current version of the GSM1k paper (Zhang and colleagues) reports accuracy drops of up to 8% on newly written problems that mirror GSM8k, with several model families showing evidence of overfitting. Source
- The Data Provenance Initiative's 2023 audit of more than 1,800 text datasets found license omission of over 70% and error rates above 50% on widely used dataset hosting sites. Source
- Anthropic's engineering guide to agent evaluations runs multiple trials per task because model outputs vary between runs, and describes moving high-pass-rate capability evals into a continuously run regression suite to catch backsliding. Source
- SciPy's false_discovery_control adjusts p-values to control the false discovery rate using the Benjamini-Hochberg procedure, or the more conservative Benjamini-Yekutieli procedure that remains valid when tests are not independent. Source
- Lakens (Social Psychological and Personality Science, 2017) argues a nonsignificant result alone does not justify concluding an effect is absent and recommends equivalence tests with bounds set at the smallest effect size of interest. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.