Skip to content

AI data market

Evaluation versus training data: a flywheel test of the market's future

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Evaluation could outgrow training data licensing only if its purchase triggers keep recurring and buyers pay outsiders to meet them: model releases, contamination, regulation, new deployments, reliability demands and independent assurance. No public data measures either market, so the evaluation versus training market future is a conditional bet. Track paid repurchases tied to dated triggers, and watch whether free tooling, synthetic tests or buyer concentration break the loop.

Evaluation split

Evaluation

This page covers evaluation versus training data: a flywheel test of the market's future.Illustrative

Key takeaways

  • Evaluation can outgrow training data licensing only through repeat purchases, because test sets lose value with exposure while training data is absorbed once.
  • Keep training licenses, evaluation data, expert labor and tooling as separate budgets; adding them together inflates every estimate.
  • The flywheel test asks two things of each of six triggers: does it recur, and does the buyer pay an outside supplier to meet it?
  • Customer-specific test programs built from a company's own records have the strongest recurrence logic; private static benchmarks decay fastest.
  • Free tooling such as Inspect, synthetic tests, concentrated buyers and curator conflicts of interest are the risks that could stop the flywheel.

Could evaluation spending outgrow training data licensing?

It could, but only if evaluation keeps being bought again for reasons training data does not share. A training dataset is largely used up once its signal is absorbed into a model. An evaluation set loses value as it is exposed, and every new release, deployment or rule creates a fresh reason to buy. If those reasons recur faster than training data is refreshed, evaluation spending could pass training data licensing even though neither market is measured today.

That is a conditional thesis, and this article treats it as one. Training data licensing occasionally shows up in public filings; evaluation work mostly hides inside research budgets, vendor services and staff time. So no market sizes or shares appear here. Instead the article separates the budgets, builds a test of the evaluation flywheel from dated public signals, sets out four scenarios with signposts, and names the evidence that would prove the idea wrong.

The question matters because the two markets reward different businesses. Training data licensing rewards rights and scale. Evaluation rewards freshness, independence and the ability to keep test material unseen, which favors suppliers of records that have never been on the public web.

Three budgets investors should not add together

Training data licensing, evaluation data and the expert labor behind both are separate budgets with separate purchase triggers, and adding them together is how most market estimates inflate. The table defines each one by what is bought and how its value is used up.

Keep annotation and tooling outside both definitions. Expert hours can produce a training set or a test set, and counting them twice inflates both. Tooling is software, not data: it lowers the cost of running evaluations but supplies none of the unseen tasks a release decision needs. The licensing difference itself, training rights against evaluation-only rights, is covered on this site's comparison of the two license types.

Three budgets investors should not add together
BudgetWhat is boughtHow its value is used upTypical purchase triggerWhere it shows up publicly
Training data licensingRights to use records or content to change model weightsAbsorbed into the model; repeated use adds littleA new model generation or a capability gapOccasionally in filings, such as Reddit's 2024 S-1
Evaluation dataHeld-out tasks, reference answers and graders that measure a model or agentDecays as items leak into training data or stop separating modelsA release decision, deployment, audit or ruleRarely; usually inside research or vendor spending
Expert annotation and reviewHuman time to write, label or judge itemsSpent per task; can serve either budgetAny project that needs human judgmentBundled into services revenue
Evaluation toolingSoftware to run, score and log evaluationsReusable; little marginal cost once builtTeam size and evaluation volumeOpen-source projects and platform pricing

The Evaluation Demand Flywheel Test: six triggers and their breaking points

The flywheel turns only if each purchase trigger both recurs and is paid for outside the buyer's own team. The test lists six triggers, a dated public signal for each, and the evidence that would show the trigger is not producing repeat purchases. A trigger with no paid repurchase counts against the thesis even if the activity behind it is growing.

The triggers also feed each other, which is what makes this a flywheel rather than a list. Releases expose test sets and speed up contamination; contamination forces new sets; new deployments bring rules and audits; reliability demands multiply trials on every set. The loop breaks wherever buyers can meet a trigger with free, synthetic or self-built tests.

Two signals deserve weight because they come from the people who build models. In a post explaining why it no longer evaluates SWE-bench Verified (read October 26, 2026), OpenAI says the benchmark is increasingly contaminated and mismeasures frontier coding progress, and that an audit found at least 59.4 percent of audited problems had flawed test cases that reject functionally correct submissions. And OpenAI's GDPval (2025) measures models on graded tasks from 44 occupations, evidence that model builders now invest in evaluation shaped like real work.

Score the test by counting triggers, not by weighing anecdotes. For each row, mark paid repurchase observed, activity without paid repurchase, or no evidence, and date the mark. Fewer than three rows with observed paid repurchase means the flywheel is not turning as a market, however much evaluation activity there is; five or six means the thesis is passing its own test. The rule is deliberately coarse, because a finer score would imply a precision the public record cannot support.

The figure draws the loop and the three forces that could break it. The table is the checklist an investor can rerun each quarter: for every row, ask whether paid repurchase was observed, and keep the answer with its date.

The Evaluation Demand Flywheel Test: six triggers and their breaking points
Purchase triggerRecurs whenDated public signalDisconfirming evidence
Model releasesEach release needs a go or no-go decision on tasks the model has not seenOpenAI's GDPval (2025) measures models on graded tasks from 44 occupationsReleases judged on public benchmarks alone, with no private test purchases
Contamination and saturationExposed or solved test sets stop separating modelsOpenAI, read October 2026: SWE-bench Verified increasingly contaminated; 59.4 percent of audited problems had flawed testsStatic sets keep separating models for years without refresh
Regulation and auditRules require documented evaluation before or after releaseEU guidelines for systemic-risk general-purpose models: evaluation with state-of-the-art tools, including adversarial testingObligations stay with a few frontier providers who build tests in-house
New enterprise deploymentsEach workflow needs tests drawn from its own recordsGDPval notes it is limited to one-shot tasks, without context building or multiple draftsEnterprises accept vendor scores instead of testing on their own tasks
Reliability requirementsAgents must succeed repeatedly, multiplying trials per itemtau-bench repository: one setup fell from 0.460 at pass^1 to 0.225 at pass^4Reliability checked in production monitoring, not with bought test sets
Independent assuranceBuyers want tests the model builder did not writeScale says its SEAL leaderboards use private datasets that cannot be gamedConflict-of-interest concerns push labs back to self-evaluation
A loop of six evaluation purchase triggers, releases, contamination, regulation, deployments, reliability and independent assurance, each feeding the next, with three amber breaking points: free tooling, synthetic tests and few buyers.
The Evaluation Demand Flywheel Test: six triggers that must recur as paid purchases, and three forces that could break the loop.

Static benchmarks, dynamic environments or customer test programs: which one recurs?

Customer-specific test programs have the strongest recurrence logic, because every new workflow, policy change or model swap creates a new test; static benchmarks have the weakest, because they decay with use, and public ones are never bought at all. Dynamic benchmarks sit between them: they refresh by design but need a steady supply of new source material.

Contamination is real but uneven, which matters for how fast static sets decay. A 2024 study comparing models on GSM8k, a popular grade-school math benchmark, with GSM1k, a new counterpart, found accuracy drops of up to 8 percent in its current version, with several model families showing systematic overfitting, while the frontier models it tested showed minimal signs of overfitting. Decay is fastest for the models least able to pay for private tests.

Illustrative example: an insurer deploying a claims-handling agent builds 400 test tasks from closed claims, each with the adjuster's final decision as the reference answer. Six months later a change in policy wording makes a quarter of those tasks wrong under the new rules, and a new product line adds a workflow the set never covered. Both events create a purchase no public benchmark can satisfy, which is why this product type has the strongest recurrence logic and also the heaviest rights burden: every task comes from records someone must be allowed to use for testing.

Trust cuts both ways. A 2025 arXiv paper on private evaluation curators argues that keeping prompts unseen reduces contamination, but that a curator who also supplies training data to the labs it evaluates faces a conflict of interest, and that expert annotators' preferences can tilt results toward models trained on the curator's own data. An evaluation supplier's independence is therefore part of its product, not a footnote.

Static benchmarks, dynamic environments or customer test programs: which one recurs?
Evaluation productExampleWhy buyers come backHow it decaysTrust obstacle
Public static benchmarkSWE-bench Verified, which OpenAI stopped evaluating after finding contaminationNo purchase: it is free, which is why it anchors comparisonsFastest, because items are public from day oneTraining on the items inflates scores
Private static benchmarkScale's SEAL leaderboards, which the company says use private datasetsEach new model needs ranking on unseen itemsItems leak through repeated use and saturateThe curator may also sell training data to the labs it ranks
Dynamic benchmark or environmentLiveBench, which releases new questions monthly to limit contaminationFresh items arrive each cycle by designSlowly, but every refresh needs new source materialScores are harder to compare across releases
Customer-specific test programTasks built from one company's own records for one deploymentEvery new workflow, policy change or model swapWorkflows and policies change, so tasks go staleNeeds rights to the records and a neutral grader

How to judge the evaluation versus training market future without a market size

Judge the evaluation versus training market future by counting purchase events and their triggers, not dollars. Public dollar figures exist for a handful of training licenses and almost none for evaluation, so any size comparison today would compare a disclosed number with a guess.

At a glance

How to judge the evaluation versus training market future without a market size

The training side shows how thin even the best disclosures are. Reddit's S-1 (2024) reported $203.0 million of aggregate contract value for data licensing arrangements signed in January 2024, with terms of two to three years. Its annual report for 2024 then says that, so far, two partners are the source of substantially all of the contract value linked to its licensing revenue. One company, two buyers: precise facts, but not a market.

Evaluation leaves even fewer traces, so the evidence has to be built from events. The measures below can be collected from contracts, announcements and interviews, compared over time, and falsified. SourceX has not published these measures for its own business; it shares company and commercial metrics with investors directly.

Checklist · 6 items

  1. Repurchase interval: days between successive evaluation purchases by the same buyer, with open intervals counted rather than dropped.
  2. Trigger attribution: the share of evaluation purchases tied to a dated release, contamination finding, rule or deployment.
  3. Distinct buyers: how many separate organizations bought held-out test material in a period, and the share from the largest two.
  4. Self-build share: the share of a buyer's evaluations written in-house, from public engineering posts and interviews.
  5. Refresh ratio: refreshed items delivered as a share of the original set, by product type.
  6. Training renewal rate: training licenses renewed at term end, the comparison series for the same period.

0 of 6 ticked. Tick each item as you go.

Four scenarios, each with a signpost and a kill condition

Four scenarios cover the plausible paths, and none is assigned a probability. Each names what must hold, a signpost an investor can watch, and the evidence that would end it. Update by moving weight between scenarios when a signpost fires, not by inventing numbers.

Epoch AI's forecasts anchor the last row. Its 2024 study estimated that roughly 300 trillion tokens of quality, repetition-adjusted public human text make up the effective stock, and projected full use between 2026 and 2032, earlier with heavy overtraining. Its later outlook, What will AI look like in 2030?, finds enough public text to scale to at least 2027, notes that synthetic data can be generated in large quantities and calls a data bottleneck hard to rule out but surmountable.

Four scenarios, each with a signpost and a kill condition
ScenarioWhat must holdSignpost to watchWhat would end it
Evaluation-ledAt least four of the six triggers show paid repurchase, and unseen test material stays scarceModel builders retire public benchmarks faster than new ones gain trustOpen, refreshed benchmarks prove good enough for release decisions
Parallel growthEvaluation repurchases rise while each training generation still buys new dataFilings and announcements keep naming training licenses alongside evaluation programsConcentrated training contracts lapse without renewal
Commoditized evaluationOpen tooling, synthetic tests and model graders cover most needsFree evaluation suites expand and enterprises adopt them unchangedSynthetic tests fail to predict production failures, forcing tests built from real records
Both shrinkModels need far less outside data of either kindForecasts move further toward surmountable data limitsCapability gaps on private business work persist

What could make the evaluation thesis wrong

The thesis fails if buyers can meet evaluation triggers cheaply on their own, or if too few buyers exist to sustain repeat purchases. Each risk below has a public marker an investor can check.

At a glance

What could make the evaluation thesis wrong

None of these risks is hypothetical, and several already have strong evidence behind them. The honest reading is that evaluation demand is likely to keep growing as activity, while the share paid to outside data suppliers is the open question.

  • Free tooling: Inspect, the evaluation framework from the UK AI Security Institute, includes over 200 pre-built evaluations ready to run on any model (README, read October 26, 2026). Tooling lowers the cost of running tests, though not of writing unseen ones.
  • Synthetic and model-written tests: Epoch AI judges a data bottleneck surmountable, partly through synthetic data. If generated tasks predict production failures well enough, paid test material loses its premium.
  • Buyer concentration: two partners account for substantially all of Reddit's licensing contract value. Evaluation buyers may be just as concentrated among a few labs, making revenue lumpy.
  • Uneven contamination: GSM1k found minimal overfitting in the frontier models it tested, so the contamination trigger may be weakest for the buyers with the largest budgets.
  • Narrow regulation: the EU's systemic-risk evaluation duties apply to the most capable general-purpose models, with a presumption above 10^25 training FLOP in the Commission's guidelines, not to every deployment.
  • Conflicts of interest: if curators that sell training data also run evaluations, buyers may discount their results or build tests in-house, as the 2025 private-curator paper warns.

Questions to ask any company selling evaluation data, SourceX included

Ask for evidence of repurchase, triggers and independence, defined so the answers can be checked. The same questions apply to any company that licenses data for testing models, SourceX included.

On the supply side, SourceX handles each step for businesses licensing operational data to AI developers: it sources and qualifies the data, reviews the rights, does the contracting, prepares the records, delivers them and settles payment. It has not published how its business divides between training and evaluation uses; its company and commercial metrics go to investors directly rather than into articles. The company's current raise, under Rule 506(c), is for accredited investors, and investor materials are available on request through the investors page.

Checklist · 6 items

  1. By contract count, how many licenses permit evaluation only, training only, or both, and how is each use defined in the agreement?
  2. What dated event triggered each repeat evaluation purchase: a release, a contamination finding, a rule or a new deployment?
  3. How is held-out status protected after delivery, and what happens to a test set when a buyer reports leakage?
  4. How many distinct buyers made repeat evaluation purchases in the period, and what share came from the largest two?
  5. Who writes and grades the tasks, and does the seller also supply training data to the same buyer?
  6. Which customer-specific programs were renewed after a workflow or policy change, and which lapsed?

0 of 6 ticked. Tick each item as you go.

Methods note: how the flywheel triggers were chosen and checked

The Evaluation Demand Flywheel Test and the four scenarios are editorial frameworks built for this article. Triggers were chosen because each has a dated public signal, and the disconfirming evidence for each was written before any trigger was judged.

Sources were checked on October 26, 2026: OpenAI's GDPval and SWE-bench Verified posts, the European Commission's guidelines for general-purpose AI providers, two arXiv papers (GSM1k and the private-curator paper), the tau-bench, LiveBench and Inspect repositories, Scale's leaderboard announcement (a company claim, attributed as such), Epoch AI's data forecasts and Reddit's SEC filings.

Not measured: dollar spending on evaluation or training data, market shares, prices, growth rates, scenario probabilities, or any SourceX data. Repository statistics change; tau-bench's maintainers now point users to a newer version of the benchmark, and Inspect's evaluation count will move.

Frequently asked questions

Is evaluation data a bigger market than training data today?

No public data shows that, and neither market is measured cleanly. Training data licensing occasionally appears in filings, such as Reddit's 2024 S-1, while evaluation spending is usually bundled into research budgets and services. Any claim that one is larger should state its definitions and sources.

Why does evaluation data have to be refreshed?

Test items leak into training data or get solved, and then they stop separating models. OpenAI says SWE-bench Verified became increasingly contaminated, and LiveBench releases new questions monthly to limit contamination. Refresh is what turns a one-time test set into a repeat purchase.

Can the same dataset be used for training and evaluation?

Not for the same model at the same time: a test set that enters training stops measuring anything. That is why evaluation sets are kept apart from training data, and a license can permit evaluation use without permitting training.

What evidence would show the evaluation flywheel is real?

Paid repurchases tied to dated triggers across several distinct buyers, customer test programs renewed after workflow or policy changes, and private test material that keeps its value after free evaluation tooling becomes widespread. Activity growth alone does not count; outside purchases do.

Does SourceX publish how much of its business is evaluation data?

No. SourceX has not published how its business divides between training and evaluation uses and shares company and commercial metrics with investors directly. Investors can ask for materials through sourcex.si/investors and put the training-versus-evaluation question to the company; any offering is open solely to verified accredited investors.

Sources

  • OpenAI says SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress, and its audit found at least 59.4% of audited problems had flawed test cases that reject functionally correct submissions. Source
  • OpenAI's GDPval measures models on tasks from 44 occupations in the top nine industries contributing to U.S. GDP and is limited to one-shot evaluations that do not capture context building or multiple drafts. Source
  • The European Commission's guidelines for general-purpose AI providers list model evaluation with standardised protocols and state-of-the-art tools, including adversarial testing, among the obligations for models with systemic risk, presumed above 10^25 FLOP of training compute. Source
  • The GSM1k study (arXiv 2405.00332, current version) found accuracy drops of up to 8% for some models, with several families showing systematic overfitting, and minimal signs of overfitting in frontier models. Source
  • Reddit's S-1 disclosed data licensing arrangements entered in January 2024 with an aggregate contract value of $203.0 million and terms of two to three years. Source
  • Reddit's annual report for 2024 says substantially all contract value associated with its licensing revenue comes from two partners. Source
  • Scale says its SEAL leaderboards rank frontier models using curated private datasets that cannot be gamed. Source
  • A 2025 arXiv paper on private evaluation curators argues that private evaluations reduce contamination but raise conflict-of-interest risks when curators have business relationships with the labs they evaluate, and that annotator preferences can bias results. Source
  • LiveBench is designed to limit potential contamination by releasing new questions monthly. Source
  • Inspect, the evaluation framework created by the UK AI Security Institute, includes over 200 pre-built evaluations ready to run on any model. Source
  • The original tau-bench repository's airline leaderboard lists one tool-calling setup at 0.460 pass^1 and 0.225 pass^4. Source
  • Epoch AI estimated the effective stock of quality, repetition-adjusted public human text at about 300 trillion tokens and projected full use between 2026 and 2032, earlier with heavy overtraining. Source
  • Epoch AI's 2030 outlook finds enough public human text to scale to at least 2027, notes synthetic data can be generated in large quantities, and calls a data bottleneck difficult to rule out but surmountable. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies