Skip to content

AI uses for records

Dataset value after a model upgrade: a matched retest of the gain

By SourceX Editorial · Updated

9 sections

Short answer

A dataset value model upgrade experiment reruns the original pilot on the new base model under matched conditions: the same frozen tasks, grader, trials and tuning budget, with and without the dataset. Compare the new gain with the old one, then check which of the dataset's original fixes the new model already makes alone. Keep the data if the gain holds, narrow it if it shrinks, retire its training use if it vanishes.

Value drivers

Dataset value

This page covers dataset value after a model upgrade: a matched retest of the gain.Illustrative

Key takeaways

  • A pilot gain belongs to the model it was measured on; a base model upgrade reopens the question.
  • Four arms answer it: the old model with and without the data and the new model with and without it, all on one frozen task set.
  • The retention ratio compares the new gain with the old one; the absorbed share shows how many of the dataset's fixes the new model now makes unaided.
  • Deprecation notices set the deadline: Anthropic gives at least 60 days before retiring a public model, and old-model arms cannot be rerun after that.
  • A shrinking gain is not proof of bad data; the residual fixes often point to the narrower slice still worth licensing or keeping as a test set.

What should a dataset value model upgrade experiment decide?

Whether a dataset that helped the previous base model still earns its place once that model is replaced. The experiment reruns the original comparison on the successor under matched conditions and ends in one of four decisions: carry the data forward, narrow it to the cases the new model still misses, retire it from training while keeping it for evaluation, or declare the run void.

The question matters because pilot gains are conditional. A fine-tuning or post-training gain was measured against one base model, at one point in time, on one task set. When the base changes, the gain can hold, grow, shrink or reverse, and a license renewal, a second purchase or a training budget may depend on which.

Useful evidence is narrow: the same frozen tasks scored four ways, task-level transitions that show which of the dataset's fixes the new model already makes alone, repeated trials, and a recipe re-tuned for the new base with the same budget the old one had. Anything less compares a new model with an old memory.

Why can a measured gain shrink on a stronger base model?

Because how much a fixed set of examples adds depends on what the model already knows. Hernandez and colleagues (2021) found that pre-training effectively multiplies the size of a fine-tuning dataset, so a stronger starting point can reach the same result with less help, or may already be there.

Gudibande and colleagues (2023) saw the same pattern from the other side: fine-tuning on imitation data closed little to none of the gap to a stronger model on tasks the data did not heavily cover, and the authors concluded that better base models were the higher-leverage path. A model upgrade is exactly that kind of change, delivered on a schedule the buyer does not control.

Training teams already treat data value as relative to the current checkpoint. NVIDIA's Nemotron documentation (repository checked October 26, 2026) describes dropping reinforcement-learning prompts that the current checkpoint solves every time, and re-profiling tasks with the best checkpoint when progress plateaus. The same documentation says customization has the least headroom in heavily optimized domains such as math and coding, and that the only way to know for a given domain is to evaluate on your own task set rather than public leaderboards.

None of this says gains must shrink. A stronger base can also make better use of specialized records, especially for work that public data rarely covers. The direction is an empirical question, and the retest is how a buyer answers it.

When does an upgrade force the retest, and how long is the window?

When the model you measured on is deprecated or replaced, and the window closes at retirement. Anthropic's model deprecations page (checked October 26, 2026) says requests to retired models fail, commits to at least 60 days' notice before retiring publicly released models, and advises testing applications on newer models well before the retirement date.

The dates are concrete. On September 30, 2026, Anthropic notified developers that Claude Sonnet 4.5 would retire on November 30, 2026. A team whose dataset gain was measured on that model can still rerun its old-model arms today; after retirement it can only compare against archived outputs, which is acceptable only if the harness, grader and task set are unchanged.

Baselines also move without any deprecation. OpenAI reported on March 5, 2026 that GPT-5.4 matched or beat industry professionals in 83.0% of GDPval comparisons, against 70.9% for GPT-5.2, on knowledge work across 44 occupations. A dataset that once closed a gap on that kind of work may now face a successor that has closed much of it alone.

Treat three events as retest triggers: a deprecation notice for the measured model, a decision to switch base models for cost or capability, and any renewal or expansion of the data license.

Four arms and a gain retention ledger

Run the old and new base models, each with and without the dataset, on one frozen task set, then read four contrasts and three task-level counts. The four-arm structure separates what the upgrade did, what the data still does, and whether the two now overlap.

Keep everything identical across arms except the base model and the presence of the data. If the original pilot used a volume-matched filler arm to separate the dataset's content from extra training, keep that arm on both models too. The ledger below is the core of the preregistered protocol:

Four arms and a gain retention ledger
QuantityHow it is computedWhat it answersDecision use
Original gainOld model with data minus old model withoutWhat the dataset added when it was boughtThe yardstick for retention
Retained gainNew model with data minus new model withoutWhat the dataset adds nowPrimary outcome, reported with a paired interval
Retention ratioRetained gain divided by original gainHow much of the value survivedCarry forward at 0.5 or more (editable example)
Upgrade-alone gapNew model without data minus old model with dataWhether the upgrade alone beats current productionSets the urgency of switching, data or not
Absorbed shareOriginal fixes the new model solves unaided, divided by all original fixesWhich of the data's wins became redundantA high share points to narrowing or retiring
Residual fixesTasks the new model fails alone but solves with the dataWhere the data still earns its placeDefines the narrower scope worth keeping
InterferenceTasks the new model solves alone but fails with the dataWhether the old recipe now hurtsAny cluster triggers a recipe review

What has to stay fixed between the old run and the new one?

Everything that is not the base model or the data: task set version, grader and rubric, harness and tools, trial count, inference settings and tuning budget. A retest that changes two things at once cannot say which one moved the score.

At a glance

What has to stay fixed between the old run and the new one

Vendor launch tables show how easily settings drift. In the same March 5, 2026 announcement, OpenAI notes that the GDPval comparison set GPT-5.4's reasoning effort to xhigh and GPT-5.2's to heavy, so the two models did not run at the same setting. That is a reasonable launch comparison, but in a retest each model should run at a setting chosen by a rule written in advance, with the setting recorded on the ledger.

Use this checklist to freeze the retest before any training run:

  • Task set: the same version and case list, kept private and, where possible, created after the old model's training cutoff so neither model has seen it.
  • Grader: the same rubric, grader model version and adjudication rule; if the grader must change, rescore the archived old-model outputs with the new grader.
  • Trials: the same number of repeated trials per task in every arm, reported as a pass rate and as pass^k.
  • Recipe: the original training recipe re-applied to the new base, plus a re-tuning search with the same budget the original pilot spent.
  • Data: the same dataset version and example count; records added since the pilot are tested as a separate arm.
  • Compute and context: the same training compute cap, context limits and tool access in every arm.

Preregistered rules: carry forward, narrow, retire or void

Fix the verdict rules before the first retest run, with the retained gain as the primary outcome and the absorbed share as the guide to scope. The thresholds below are editable examples, not industry standards:

  • Carry forward: the retained gain's paired interval is above zero and the retention ratio is 0.5 or higher. Keep the dataset in the new recipe at its current scope.
  • Narrow: the retained gain's interval is above zero but the retention ratio is under 0.5. Keep the slice that produces most residual fixes, test that subset in the next round, and revisit the license scope.
  • Retire from training: the retained gain's interval includes or falls below zero and the absorbed share is above one half. Stop training on the data, review the recipe if interference clusters, and consider keeping the records as a private evaluation set, where their value is headroom rather than lift.
  • Void: a frozen element changed between arms, the old-model arms ran on a different task set, or the new model scores so near the ceiling of the task set that no gain could show.

Illustrative retest: a 15-point gain that keeps 5

In this illustrative case, the dataset's gain falls from 15 points on the old model to 5 on the new one, and nearly three quarters of its original fixes are now made by the new model alone. The numbers are constructed to show the arithmetic; they are not measured results.

The frozen set holds 300 private tasks. On the old model, the dataset fixed 60 tasks and broke 15, a net gain of 45 tasks. The new model alone solves 222 tasks, already 9 points ahead of the old production system, and solves 44 of the 60 tasks the dataset used to fix. With the data added, the new model fixes 24 more tasks and breaks 9.

The rules return narrow, not retire. The retained gain is positive, assuming its paired interval clears zero over repeated trials, but the retention ratio of 0.33 is under the 0.5 bar. Seventeen of the 24 residual fixes sit in one stratum, multi-record exception cases, so the next round tests that slice of the dataset on its own.

Illustrative retest: a 15-point gain that keeps 5
Arm or contrastTasks solved of 300RateReading
Old model, no data15050.0%Original baseline
Old model with data19565.0%Original gain of 15.0 points
New model, no data22274.0%Upgrade alone beats current production by 9.0 points
New model with data23779.0%Retained gain of 5.0 points; retention ratio 0.33
Original fixes the new model makes alone44 of 6073.3%Absorbed share above one half
Residual fixes and interference24 fixed, 9 brokenNet +15 tasks17 of 24 fixes in multi-record exception cases
A dot chart on a 0 to 100 percent axis showing the old model at 50 and 65 percent without and with the data, and the new model at 74 and 79 percent, with brackets marking a 15-point original gain and a 5-point retained gain, beside a bar splitting 60 original fixes into 44 absorbed and 16 still needing the data.
Illustrative: invented counts for a 300-task retest; the gain falls from 15 points to 5 after the upgrade.

Could the shrinking gain be noise, a stale recipe or a saturated test?

Yes, each can fake a shrinking gain, so rule all three out before narrowing or retiring a dataset.

Noise first. Run repeated trials in every arm and compare tasks in pairs. Anthropic's January 2026 evaluation guide recommends multiple trials because model outputs vary between runs, and distinguishes pass@k, at least one success in k attempts, from pass^k, the probability that all k attempts succeed. Report both when the workflow must succeed every time.

A stale recipe second. Hyperparameters, data mixtures and example formats tuned for the old base can underuse a new one. That is why the protocol re-tunes on the new base with the same search budget: if the re-tuned arm recovers the gain, the data was fine and the recipe was the problem.

Saturation last. If the new model alone nears the top of the task set, there is no room left to show a gain. The same Anthropic guide notes that an evaluation at 100% tracks regressions but gives no signal for improvement. Add harder private tasks from the stratum where residual fixes cluster, and rerun before concluding the data has nothing left to add. Private tasks written after the newer model's training cutoff also reduce the chance that it has seen public versions of them.

Writing the retest into a data request or renewal

Put the retest trigger and the four-arm design into the original request, so a renewal rests on evidence about the current model rather than last year's pilot.

Licensing terms decide whether a retest is even possible: whether the data may be used to train successor models, whether evaluation use survives the end of training rights, and whether a narrowed scope can be priced on its own. Those are contract questions for counsel, and the clause options are set out in the article on whether a data license covers future model versions.

Datasets from SourceX are sourced from U.S. businesses when a buyer asks, not kept in stock, and the request form collects scope, volume, language, format, timing and licensing needs. Stating the intended use, the acceptance test and the retest trigger makes the requirement testable. SourceX has not published retest results, and no supplier can promise that a dataset will improve a given model.

Frequently asked questions

Do you need to retest a dataset every time the base model changes?

Retest when the measured model is deprecated, when you switch base models, or before renewing or expanding a data license. Minor updates that keep the same model version usually do not need a full four-arm rerun, but a regression check on the frozen task set is cheap insurance.

Can archived results stand in for the old model after it is retired?

Only if the task set, grader, harness and trial count are unchanged, because the old arms can no longer be rerun. If the grader changes, rescore the archived outputs with the new grader so both models are judged the same way. Running the old-model arms inside the deprecation window avoids the problem.

What retention ratio means a dataset is still worth keeping?

There is no universal number; the 0.5 threshold in this protocol is an editable example. Set the bar from the decision at stake, such as the cost of renewal against the value of the residual fixes, and write it down before the retest runs.

Does a smaller retained gain mean the data was low quality?

Not necessarily. A stronger base model can absorb the cases the data used to fix, which says more about the model than about the records. The residual fixes show where the data still helps, and records that no longer add training lift can still serve as a private evaluation set.

Does SourceX promise that a dataset will improve a model?

No. SourceX has not published retest results and describes candidate data characteristics without promising model improvement. A buyer who needs records for a specific task can describe the intended use and the acceptance test, including the retest trigger, on the request-data form.

Sources

  • Anthropic's January 2026 evaluation guide recommends multiple trials because outputs vary between runs, defines pass@k and pass^k, and notes that an evaluation at 100% tracks regressions but gives no signal for improvement. Source
  • Anthropic's model deprecations page says requests to retired models fail, promises at least 60 days' notice before retiring publicly released models, advises testing newer models well before retirement, and lists Claude Sonnet 4.5 as deprecated on September 30, 2026 with retirement on November 30, 2026. Source
  • OpenAI's March 5, 2026 GPT-5.4 announcement reports GDPval wins or ties of 83.0% for GPT-5.4 against 70.9% for GPT-5.2, with reasoning effort set to xhigh for GPT-5.4 and heavy for GPT-5.2. Source
  • Hernandez and colleagues (2021) found that pre-training effectively multiplies the fine-tuning dataset size. Source
  • Gudibande and colleagues (2023) found imitation fine-tuning closed little to none of the gap to ChatGPT on tasks not heavily supported in the imitation data, and called better base models the highest-leverage action. Source
  • NVIDIA's Nemotron training FAQ describes dropping RL prompts the current checkpoint already solves, re-profiling tasks with the best checkpoint when progress plateaus, least customization headroom in math and coding, and evaluating on your own task set. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies