AI data market
Dataset value after a model breakthrough: what decays, what compounds
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
A model breakthrough does not devalue a dataset evenly. Examples of skills the new model already performs lose most of their training value, because the model can generate replacements; Meta's Llama 3.1 fine-tuning data already included over 25 million synthetic examples. Hard cases the model still fails, regression tests, fresh held-out tests and recorded business outcomes hold or gain value. Underwrite that split, not the record count.
Value drivers
Dataset value
Key takeaways
- Breakthroughs reprice datasets by component: solved-skill demonstrations decay, while hard cases, tests and recorded outcomes hold or gain.
- Capability jumps have been fast: a year after their 2023 introduction, scores rose 18.8, 48.9 and 67.3 points on MMMU, GPQA and SWE-bench (AI Index 2025).
- Tests outlive training examples: Anthropic advises graduating saturated capability evals into regression suites that run continuously.
- Fresh tests keep finding gaps: rebuilt ImageNet test sets cut accuracy by 11 to 14 percent across many models (Recht and colleagues, 2019).
- Watch five dated signposts, from GDPval win rates to license renewals, rather than one forecast.
Does a capability breakthrough make a dataset worth less?
Parts of it, yes; the whole of it, rarely. A breakthrough here means a model release that moves a task from usually failing to usually passing. When that happens, examples that teach the now-solved skill lose most of their training value, while tests that check the skill and records of what actually happened keep their value or gain.
Breakthroughs of that kind have been frequent. Stanford HAI's 2025 AI Index reports that a year after MMMU, GPQA and SWE-bench were introduced in 2023, scores rose by 18.8, 48.9 and 67.3 percentage points, with AI systems solving 71.7 percent of SWE-bench problems in 2024, up from 4.4 percent in 2023. On business deliverables, OpenAI's GDPval paper (2025) found that 47.6 percent of Claude Opus 4.1's deliverables on its 220-task gold subset were graded as good as or better than the industry professionals' work, and OpenAI's launch page named Claude Opus 4.1 the best performing model in that test. OpenAI's March 5, 2026 GPT-5.4 announcement reports GPT-5.4 matching or exceeding professionals in 83.0 percent of comparisons, under different settings, so the two figures are not like-for-like.
For an investor the question is therefore not whether a dataset decays but which of its parts do, how fast, and what evidence would show it. The five-part split separates the parts, the scenario matrix shows what different breakthroughs do to each, and the scorecard lists dated signposts.
Which parts of a dataset decay, and which compound?
Five parts behave differently, and the test for each is whether a stronger model can replace it. Solved-skill demonstrations decay because the model can now produce them. Hard cases at the new frontier, regression tests, fresh held-out tests and recorded outcomes cannot be produced by the model whose limits they measure.
Most catalogs mix all five, often inside one delivery. A support archive, for example, holds routine resolved tickets a current model can imitate, rare escalations it still mishandles, tickets that could serve as permanent regression cases, recent tickets no model has seen, and outcome fields recording what the customer actually did next. Valuing that archive by record count treats the five as one.
The replaceability test can be run on a sample without a full training pilot. Give today's strongest model, at its best setting, the inputs of a random sample of records; have a reviewer who cannot see which is which compare its outputs with the recorded ones against the buyer's acceptance test; and count the share judged equivalent. That share is the replaceable fraction. Rerun the same sample after each major release, and the change in the share is the decay rate, measured rather than assumed.
The split, with the question that sorts each record:
| Part of the dataset | Replaceability test | After a breakthrough on its task | Evidence to ask a seller for |
|---|---|---|---|
| Solved-skill demonstrations | Can the new model produce an equivalent example that passes the buyer's acceptance test? | Decays toward the cost of generating it | Share of records a current model already handles |
| Hard cases at the new frontier | Does the new model still fail it over repeated trials? | Holds or gains; becomes the scarce part | Current-model failure rates on a sample |
| Regression tests | Must every future model keep passing it? | Holds; reused at every release | How test items are kept out of training deliveries |
| Fresh held-out tests | Was it created after the model's training cutoff and kept unseen? | Gains as public tests saturate or leak | Creation dates and access controls |
| Recorded outcomes | Does it record what actually happened, which no model can infer? | Holds; anchors both training and tests | Outcome coverage by period |
Why can a stronger model replace its own training examples?
Because a model that performs a skill reliably can generate examples of it, and developers already train on such examples at scale. Meta's Llama 3.1 model card (released July 23, 2024) says its fine-tuning data included publicly available instruction datasets and over 25 million synthetically generated examples, and that the model collection supports using its outputs to improve other models, including synthetic data generation and distillation.
NVIDIA's Nemotron training FAQ describes the same pattern across the pipeline: synthetic data in post-training, synthetic and rephrased sources in pre-training, and validation of synthetic data through small-scale ablations that check models learn something from it. Once a skill is solved, buying human demonstrations of it competes with that pipeline on cost.
Replaceability has limits, and those limits mark what keeps value. A generated example is useful only if something can check that it is right, and that check usually comes from a real test or a recorded outcome. The same FAQ notes that customization has the least headroom in domains that are already heavily optimized, such as math and coding, which suggests that more examples of already well-covered skills add the least.
In business workflows the line usually runs inside a task rather than between tasks. A model that drafts routine invoice-dispute replies well may still misapply a contract's exception clause or miss that a customer escalated twice; the routine replies are replaceable, while the exception cases and the recorded result of each dispute are not. That is why the split is applied record by record, not dataset by dataset.
Why do tests outlive the training examples they came from?
Because a test's job continues after the skill is learned: it checks that the next model still has the skill, and that the skill holds on cases nobody has seen. Anthropic's January 2026 guide to agent evals says regression evals should have a nearly 100 percent pass rate, and that capability evals with high pass rates can graduate into a regression suite run continuously, so tasks that once asked whether something could be done at all then ask whether it can still be done reliably.
Specialization makes those suites more necessary, not less. NVIDIA's Nemotron FAQ says supervised fine-tuning on agentic trajectories often degrades performance outside the target domain, and describes the tactics its teams use to limit regression on unrelated domains. Every new model and every domain fine-tune is another reason to rerun regression tests.
Fresh tests keep finding what public scores hide. When Recht and colleagues (2019) rebuilt test sets for CIFAR-10 and ImageNet following the original procedures, accuracy fell by about 3 to 15 percent on CIFAR-10 and 11 to 14 percent on ImageNet across a broad range of models, even though the best models on the old sets stayed the best on the new ones. Public test material also leaks into training data: Dodge and colleagues (2021) found test examples from other NLP datasets inside C4, a corpus built from a Common Crawl snapshot. Private, dated and unseen tests are the part a breakthrough cannot devalue by memorizing it.
Four breakthrough scenarios and what each does to the five parts
Different breakthroughs reprice different parts, so the scenarios are defined by what the breakthrough does, not by when it arrives. None is a prediction; each names the signpost that would show it is underway, and more than one can run at once in different tasks.
Scenarios also follow one another within a task. A breakthrough that solves the common cases usually leaves a long tail of exceptions, so the skill-solved scenario tends to hand over to the long-tail one; a deployment wave then raises the stakes of every regression. A seller's exposure depends on which scenario its main tasks are in now, not on the market average.
The scenario matrix:
| Scenario | What the breakthrough does | Loses value | Holds or gains value | Signpost that it is underway |
|---|---|---|---|---|
| Skill solved | A release passes the task reliably on public and buyer tests | Demonstrations of that skill | Regression tests; recorded outcomes | Task benchmarks near saturation, as SWE-bench Verified was above 80 percent by January 2026 |
| Frontier moves to the long tail | Common cases pass; rare, multi-step or exception cases still fail | Routine examples | Hard cases; fresh held-out tests | Gaps between public scores and results on buyers' own holdouts |
| Generator with a checker | Models generate and verify their own examples for the task | Most training examples, including some hard cases | Checker calibration sets; recorded outcomes | Model cards reporting larger synthetic shares of fine-tuning data |
| Deployment wave | Capability pulls broad deployment into the dataset's domain | Little in the short run | Regression and fresh tests; outcome records for monitoring | Rising professional-task win rates, such as GDPval's |
Public forecast scorecard: five signposts to check by 2027
Each row is a conditional forecast with a signpost a reader can check, a date to check it by, and the evidence that would weaken it. Status is as read on 26 October 2026. Scoring rule: a forecast is supported if its signpost moves as stated by the check date, and weakened if the disconfirming evidence appears first.
Read the early status with care. The GDPval figures come from different settings, Anthropic's guidance describes one developer's practice rather than a market measure, and a model card reports what one developer used, not what the market buys. No row has reached its check date, so nothing here is scored yet; the table exists so a reader can score it later against the same rule.
The scorecard:
| Forecast | Signpost and status on 26 Oct 2026 | Check by | What would weaken it |
|---|---|---|---|
| Human demonstrations of professional deliverables lose training value | GDPval wins or ties: 47.6 percent for the best model in 2025; 83.0 percent reported for GPT-5.4 in March 2026, settings differ | 31 Dec 2027 | Labs keep buying human demonstrations for occupations where win-or-tie rates exceed half |
| Saturated capability tests become standing regression suites | Anthropic's January 2026 guide recommends graduating saturated capability evals into regression suites | 31 Dec 2027 | Labs report regression suites built only from their own usage logs, with no outside test data |
| Fresh held-out tests gain value as public ones saturate | SWE-bench Verified near saturation above 80 percent, per Anthropic's January 2026 guide | 30 Jun 2027 | New public benchmarks stay unsaturated and uncontaminated for more than two years |
| Synthetic examples displace more human demonstrations | Llama 3.1 model card, July 2024: over 25 million synthetic fine-tuning examples | 31 Dec 2027 | Model cards report human-written data rising as a share of fine-tuning data |
| Licensed data keeps value across model generations | Reddit's January 2024 data licenses: $203.0 million aggregate transaction price, durations over one year, continuous API access plus quarterly transfers | Reddit's annual report for 2026 | Renewals disclosed at lower value, or not renewed |
What could make the five-part split wrong?
Four developments could weaken this thesis, either by flattening the split so that tests and outcomes lose value alongside demonstrations, or by limiting who pays for the durable parts. Weigh them as heavily as the split itself; each comes with a source an investor can track.
- Checkers could reach the hard cases. If generator-and-checker pipelines like those Nemotron's teams validate through ablations extend to rare and multi-step cases, frontier hard cases become replaceable too, leaving recorded outcomes as the main durable part.
- Buyers could build their own tests. Anthropic's guide recommends starting evals from real failures in a team's own product; if labs and deployers build regression and fresh tests from their own usage, outside test data loses buyers.
- Demand could stay concentrated. Reddit's amended S-1 (2024) said substantially all of the contract value associated with its licensing revenue came from one partner at the time, and a seller facing few buyers carries renewal risk however well its catalog splits.
- Private tests can leak. Test items sent to hosted models or shared across buyers can end up in training data, as benchmark items did in C4; a seller that cannot show how it keeps tests unseen holds a decaying test catalog, not a durable one.
Questions to ask a data company before the next model release, SourceX included
Ask how the catalog splits across the five parts and how the company proves it, using definitions you set. Put the same questions to SourceX.
SourceX helps companies turn selected business records into licensed AI datasets and runs sourcing and qualification, rights review, contracting, preparation, delivery and payment. Delivery of any record waits on the holder's approval of the scope and terms and its signature on the agreement. SourceX has not published how its datasets split across these parts, or any commercial metrics; investors get company and commercial metrics from it directly. It is raising capital only from accredited investors, under Rule 506(c), and investors can request materials through its investor page and bring any of these questions:
Checklist · 6 items
- What share of your delivered records would a current frontier model already handle, and how did you measure it?
- Which records are kept as tests and never delivered for training, and what controls keep them unseen?
- How are records dated against model training cutoffs, and how much of the catalog postdates the latest releases?
- What share of records carries a recorded outcome, period by period?
- How have buyers' requests shifted between training examples and evaluation sets across recent model releases?
- Which scenario in the matrix would hurt your catalog most, and what would you do about it?
0 of 6 ticked. Tick each item as you go.
Methods note: how the split, scenarios and scorecard were built
The five-part split, the scenario matrix and the scorecard are editorial frameworks written for this article on 2026-10-26. Evidence comes from Stanford HAI's 2025 AI Index, OpenAI's GDPval paper and launch page (2025) and GPT-5.4 announcement (March 2026), Anthropic's guide to agent evals (January 2026), Meta's Llama 3.1 model card, NVIDIA's Nemotron training FAQ, Recht and colleagues (2019), Dodge and colleagues (2021) and Reddit's 2024 S-1 filings. Check dates are editorial choices, and the scoring rule is stated with the scorecard.
Not measured: the share of any real catalog in each part, how fast each part loses value in money terms, buyers' willingness to pay for tests versus training examples, and SourceX's own catalog. None can be measured from public sources without company records that have not been published.
Frequently asked questions
Do capability breakthroughs make old training data worthless?
Not all of it. Examples of skills a new model already performs lose most of their training value because the model can generate replacements, but hard cases it still fails, regression tests, fresh held-out tests and recorded outcomes keep or gain value. The answer depends on the mix inside the dataset.
Why do regression test sets keep value after a model breakthrough?
Every new model and every fine-tune has to show it kept what earlier models could do. Anthropic's evals guide recommends graduating saturated capability tests into regression suites that run continuously, so a test written for an old capability is reused at each release.
After a breakthrough, can synthetic examples stand in for real business records?
For solved skills, largely, which is why demonstrations decay; Llama 3.1's fine-tuning data already included over 25 million synthetic examples. Synthetic examples still need checks against real tests and recorded outcomes, which a model cannot generate about events it never observed.
How can an investor test a data company's exposure to the next model release?
Ask for the catalog split across the five parts, how the company measured which records a current model already handles, how it keeps test items unseen, and outcome coverage by period. Then check the scorecard's signposts on their dates rather than relying on one forecast.
Does SourceX publish how its datasets split across these parts?
No. SourceX has not published a breakdown of its datasets or any commercial metrics, and shares company and commercial metrics with investors directly. Investors can request materials at sourcex.si/investors; any offering is made only to verified accredited investors.
Sources
- Stanford HAI's 2025 AI Index reports that scores rose by 18.8, 48.9 and 67.3 percentage points on MMMU, GPQA and SWE-bench a year after their 2023 introduction, and that systems solved 71.7 percent of SWE-bench problems in 2024, up from 4.4 percent. Source
- OpenAI's GDPval paper (2025) reports that 47.6 percent of Claude Opus 4.1's deliverables on the 220-task gold subset were graded as better than or as good as the professionals' deliverables. Source
- OpenAI's GDPval launch page says Claude Opus 4.1 was the best performing model in the evaluation. Source
- OpenAI's GPT-5.4 announcement (March 5, 2026) reports GPT-5.4 matching or exceeding industry professionals in 83.0 percent of GDPval comparisons, versus 70.9 percent for GPT-5.2, with different reasoning-effort settings. Source
- Anthropic's guide to agent evals (January 2026) says regression evals should have a nearly 100 percent pass rate, that saturated capability evals can graduate into regression suites, and that SWE-bench Verified scores rose from 30 percent to above 80 percent. Source
- Meta's Llama 3.1 model card says fine-tuning data included publicly available instruction datasets and over 25 million synthetically generated examples, and that outputs may be used for synthetic data generation and distillation. Source
- NVIDIA's Nemotron training FAQ describes synthetic data in pre- and post-training validated by small-scale ablations, notes that agentic SFT often degrades out-of-domain performance, and says customization has least headroom in heavily optimized domains. Source
- Recht and colleagues (2019) found accuracy drops of about 3 to 15 percent on new CIFAR-10 test sets and 11 to 14 percent on new ImageNet test sets, with the best original models remaining best. Source
- Dodge and colleagues (2021) found that C4, a corpus built from a Common Crawl snapshot, contains test examples from other NLP datasets. Source
- Reddit's S-1 says it entered data licensing arrangements in January 2024 with original durations over one year and an aggregate transaction price of $203.0 million, delivered through continuous API access and quarterly data transfers. Source
- Reddit's amended S-1 (2024) said substantially all of the contract value associated with its licensing revenue was derived from one partner at that time. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.