AI data market
Saturated AI benchmarks and future data: who pays for the next test
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
When a public benchmark saturates, evaluation demand does not disappear; it moves to tests that are hidden, refreshed or tied to one buyer's task. Anthropic notes SWE-bench Verified scores climbed from about 30 percent to over 80 percent, and OpenAI stopped reporting the benchmark after finding contamination and flawed tests. Which replacement wins decides whether evaluation data becomes a broad market or stays inside a few labs.
Evaluation split
Saturated AI benchmarks
Key takeaways
- Saturation, contamination and broken items end a benchmark's usefulness, and each calls for a different replacement.
- Replacement tests have at least one of three properties: hidden items, refreshed items or items drawn from the buyer's own work.
- Four scenarios follow: living public benchmarks, lab-private holdouts, independent custodians and buyer-owned task tests.
- Only buyer-owned task tests turn business records from many companies into evaluation inputs at scale.
- Buyer breadth is the weakest signpost today: the licensing disclosure cited here shows concentration, not many buyers.
What happens when a public benchmark becomes too easy?
It stops telling models apart, and the people who relied on it start paying for a different test. Anthropic's agent eval guide defines saturation as an agent passing all of the solvable tasks, which leaves no room to show improvement, and warns that near that point large capability gains appear as small score increases.
The climb has been fast. Stanford's AI Index 2025 reported that scores on three benchmarks introduced in 2023, MMMU, GPQA and SWE-bench, rose by 18.8, 48.9 and 67.3 percentage points in a single year. Anthropic's agent eval guide, published in January 2026, says SWE-bench Verified scores started the year at 30 percent and that frontier models were nearing saturation above 80 percent.
For an investor the decision is narrower than whether AI is improving. It is where the attention and money that used to flow through free public tests go next: to refreshed public benchmarks, to private tests inside a few labs, to independent custodians, or to buyers testing their own tasks. Each path creates a different market for evaluation data, and only some of them need data that businesses hold.
Three ways a public benchmark stops working
Benchmarks fail by saturation, contamination or broken items, and the remedy differs for each. Saturation means the remaining items no longer separate strong models. Contamination means models have seen the test during training. Broken items mean the test marks right answers wrong, so the last points measure the test rather than the model.
OpenAI's explanation of why it no longer evaluates SWE-bench Verified shows all three at once. Its audit of problems that models often failed found that at least 59.4 percent had tests rejecting functionally correct solutions. It found models could reproduce verbatim problem details for some tasks, a sign of exposure in training. And it noted scores had moved only from 74.9 to 80.9 percent over six months, which raised the question of whether remaining failures reflected the models or the dataset.
Academic tests show the replacement cycle. MMLU-Pro rebuilt MMLU with harder, reasoning-focused questions and ten answer choices instead of four, and its authors report accuracy drops of 16 to 33 percent against the original. Humanity's Last Exam, 2,500 questions written by subject-matter experts, describes itself as designed to be the final closed-ended academic benchmark of its kind, and it ships a canary string so model builders can filter it out of training data. Each replacement buys headroom; none ends the cycle.
Where evaluation demand moves: hidden, refreshed or task-specific tests
Demand moves to tests with at least one of three properties: the items are hidden, the items are replaced on a schedule, or the items come from the buyer's own work. Each property answers a different failure, and each already has a public example whose maintainers document the design.
Refreshing has become routine maintenance for the benchmarks that last. LiveBench releases new questions monthly, its README notes that not all questions from the current release are public, and its changelog for November 2025 says an update refreshed tasks to resolve, to some extent, saturation and contamination. SWE-bench-Live adds 50 newly verified issues to its full test split each month but keeps its lite and verified splits frozen so leaderboard comparisons stay fair, a tradeoff every refreshed test faces.
Training teams apply the same logic in-house. NVIDIA's Nemotron FAQ describes dropping training prompts a model already solves at a 100 percent pass rate, and selecting 3,000 held-out science problems with pass rates between 0.25 and 0.80 for reinforcement learning evaluation. A test is useful only while it sits in the band where models sometimes fail, so every useful test has a shelf life.
The common thread is that every replacement costs more to maintain than the static test it replaced. A static benchmark is written once; a refreshed or hidden one needs a pipeline, a custodian or both. That recurring cost is what turns evaluation from a public good into something someone pays for, and it is why the question of who pays matters more than the question of which benchmark is next.
| Property | Failure it answers | Public example | What it costs to keep up |
|---|---|---|---|
| Hidden items | Contamination | ARC-AGI-2: 120 public evaluation tasks plus a semi-private set for hosted commercial models and a fully private competition set | Custody, access control and a trusted operator |
| Refreshed items | Contamination and saturation | LiveBench monthly releases; SWE-bench-Live adds 50 verified issues a month | A standing pipeline of new, verified items |
| Task-specific items | Mismatch between the test and the buyer's work | OpenAI's evals framework supports private evals built from a company's own data | Access to real work and experts to set pass or fail criteria |
| Expert-built work tasks | Academic tests that no longer track jobs | GDPval: 1,320 tasks from real work products, 220 of them released as a public gold subset | Practitioner time and repeated expert review per task |
Four scenarios for who pays for the next hard test
Two choices separate the scenarios: who holds the test, and where its items come from. Crossing them gives four plausible end states. They can coexist, so the investment question is which grows fastest and whether its buyers are many or few.
Living public benchmarks keep generic tests hard by refreshing them in the open, funded by sponsors and maintainers; they need a steady flow of new public material but little purchased data. Lab-private holdouts move hard tests inside a few model developers, which creates real demand for expert-written items but concentrates it in the buyers least likely to share. Independent custodians hold private test tiers outside any one model developer, as the ARC-AGI-2 maintainers do, and sell credibility as much as items.
Buyer-owned task tests are the scenario in which business records from many companies become evaluation inputs at scale, because the items have to come from the work the buyer is deciding about. Anthropic's guidance points teams that way: it suggests starting with 20 to 50 simple tasks drawn from real failures, and treats a task as good enough when two domain experts, working separately, would arrive at the same pass or fail verdict. The figure in the next section plots which scenario each signpost currently supports.
The scenarios also value an item differently. In the first, items are close to free and the scarce input is maintainer time. In the second and third, an item is a commissioned expert product whose usefulness decays as models catch up, so revenue depends on refresh cycles rather than a one-time sale. In the fourth, an item is evidence about one buyer's work, valued by the decision it informs, which is why it can support many smaller purchases instead of a few large ones.
| Scenario | Who holds the test | Where items come from | Who pays | What it means for evaluation data |
|---|---|---|---|---|
| 1. Living public benchmarks | Public maintainers | New public material: competitions, papers, fresh code issues | Sponsors and grants | Small paid market; free sources carry most of the load |
| 2. Lab-private holdouts | A few model developers | Experts commissioned by each lab | The lab, in-house | Real but concentrated demand; few buyers set prices |
| 3. Independent custodians | A trusted third party | Expert-written items kept off the public internet | Labs and sponsors paying for credible scores | Evaluation as a service; value rests on custody |
| 4. Buyer-owned task tests | The deploying company | The buyer's own work and licensed business records | Many buyers, each for its own task | Broad, fragmented demand for task-specific evidence |
The forecast scorecard: seven signposts and where they point
Each signpost is public and dated, and each status restates a cited source as checked on October 26, 2026. The column on the right is a reading of evidence, not a probability.
Read together, public evidence supports all four scenarios at once, with the clearest movement toward hidden and refreshed tests. The weakest row for the broad-market case is buyer breadth: the one public licensing disclosure cited here shows concentration, not many buyers. A change in that row would matter more to an investor than another benchmark retirement.
Three developments would move the reading decisively. A frontier lab stating that it buys evaluation items from outside suppliers at scale would strengthen the second and third scenarios. Procurement rules that require testing on the buyer's own tasks before deployment would strengthen the fourth. Public refreshes that keep pace with each model release without new expert input would strengthen the first at the expense of the rest.
| Signpost | Counts as evidence when | Points to scenario | Status on 2026-10-26 |
|---|---|---|---|
| Saturated benchmarks retired from reporting | A frontier lab stops reporting a widely used benchmark and names why | 2 or 3 | OpenAI stopped evaluating SWE-bench Verified after a contamination and test audit |
| Newest public items held back | Refreshed benchmarks delay releasing their latest questions | 1, with a private lag | LiveBench: not all questions from its current release are public |
| Comparison splits frozen | Maintainers freeze subsets while new items flow into others | 1 | SWE-bench-Live adds 50 issues a month; lite and verified splits frozen |
| Private tiers held by a custodian | Benchmarks keep semi-private or private test sets | 3 | ARC-AGI-2 keeps two private test sets outside its repository |
| Economic evals released only in part | A lab publishes a subset of an expert-built task set | 2 | GDPval: 220-task gold subset public out of 1,320 |
| Tooling for private, buyer-built evals | Frameworks and guides push teams to test on their own tasks | 4 | OpenAI evals supports private evals; Anthropic advises 20 to 50 tasks from real failures |
| Buyer breadth for licensed data | Filings show licensing revenue from more than a few buyers | 4 if broad, 2 if narrow | Reddit 10-K for 2024: substantially all licensing contract value from two partners |
The strongest case against a paid evaluation-data market
Three forces could keep evaluation cheap or captive even as public benchmarks saturate, and each favors a different scenario.
Automatic checks first. LiveBench scores every question against objective ground truth without an LLM judge and draws new items from public sources such as recent datasets, arXiv papers and news articles. If enough capabilities can be tested that way, living public benchmarks absorb most demand and little needs to be bought.
Generated tests second. SWE-bench-Live describes itself as powered by an automated curation pipeline and publishes its task-creation code so others can build custom tasks. Automation lowers the cost of fresh items for any task whose outcome a program can check, which is most of software engineering and little of claims handling or credit review.
Concentration third. The developers that most need hard tests are few, and they can build them in-house, as OpenAI did with GDPval. Reddit's annual report for 2024 shows how concentrated data licensing already is: substantially all contract value associated with its licensing revenue comes from two partners.
These forces bite hardest on generic capability tests. They bite least on task-specific tests, where items must come from someone's real work and the party that cares about the result is the buyer, not the model developer.
Risks that could make these scenarios wrong
Six risks could make the scenario reading wrong, and each should be priced before backing an evaluation-data business.
At a glance
Risks that could make these scenarios wrong
- Public replacements keep pace: MMLU-Pro, Humanity's Last Exam and monthly refreshes show that new public tests can arrive quickly, which would keep the paid market small.
- Leakage: a hidden test loses its value once it leaks, so custody is both the product and a single point of failure; ARC-AGI-2 describes its semi-private set as having a low leakage probability and its fully private set a near-zero one.
- Concentration: if a few labs buy most hard tests and build many in-house, prices and renewals follow their plans, as Reddit's two-partner disclosure illustrates for licensing.
- Small samples: a buyer's own test is often a few dozen tasks, enough to start but too small to separate close models, so the buyer may fall back on public scores.
- Rights: task-specific items built from business records need license terms that allow evaluation use and limit publication; records a company cannot license are not supply.
- Timing: saturation is event-driven, not scheduled, so the signposts can move in bursts around model releases rather than on a calendar.
Questions to ask an evaluation-data company, SourceX included
Ask questions that reveal which scenario a company's revenue depends on and how long its items stay useful. Each has a countable answer.
SourceX is a labeled application of the fourth scenario, not evidence that it is winning. It helps companies turn selected business records into licensed datasets for AI, running sourcing, rights review, contracting, preparation, delivery and payment, and it sources datasets for buyers on request from U.S. businesses rather than holding stock. Whether evaluation demand flows through that channel is the open question this scorecard tracks. SourceX has not published request mix or buyer counts and shares company and commercial metrics with investors directly. The company is raising from accredited investors under Rule 506(c) and shares investor materials on request through the investor page.
Checklist · 6 items
- What share of requests or revenue comes from replacing tests that saturated, leaked or broke, and how is a replacement counted?
- How long does an item stay useful from creation to retirement, and what share retires for each reason?
- Who holds the test, and what stops it leaking: access control, custody terms, canary strings?
- How many distinct buyers paid in the last four quarters, and what share of value came from the top two?
- How many expert minutes does one accepted item take, and who checks the graders?
- Can a buyer reproduce a published score, and is the uncertainty reported alongside it?
0 of 6 ticked. Tick each item as you go.
Methods note: how the benchmark-holder scenarios and signposts were built
The scenarios cross two observable choices: who holds a test and where its items come from. Examples were chosen because their maintainers document the design in primary sources. Repository READMEs and changelogs were read directly by git clone (LiveBench, SWE-bench-Live, ARC-AGI-2, Humanity's Last Exam, MMLU-Pro, OpenAI evals and NVIDIA's Nemotron FAQ), as was Anthropic's eval guide. OpenAI, Stanford HAI and SEC pages were verified through search excerpts on their own domains, and a reviewer should open them before publication.
Each status cell restates its source as checked on October 26, 2026. Not measured: spending on evaluation, the number of buyers, scenario probabilities, whether private tests are more accurate than public ones, and any SourceX data. The scenario names and the signpost selection are editorial.
Frequently asked questions
What does it mean when an AI benchmark is saturated?
A benchmark is saturated when top models pass nearly all of its solvable items, so scores stop separating them. Anthropic's eval guidance warns that near saturation, large capability gains show up as small score changes. A saturated test can still catch regressions, but it no longer measures progress.
Why did OpenAI stop reporting SWE-bench Verified?
OpenAI says the benchmark became contaminated and partly broken. Its audit found at least 59.4 percent of the often-failed problems it reviewed had tests that rejected correct solutions, and models could reproduce problem details seen in training. Scores had moved only from 74.9 to 80.9 percent in six months.
Do saturated benchmarks mean AI progress has stopped?
No. Saturation means the test ran out of headroom, not the models. Teams respond with harder or refreshed tests, such as MMLU-Pro, whose authors report accuracy 16 to 33 percent lower than on MMLU, or LiveBench's monthly question releases.
What is a good AI benchmark once public tests are easy?
One whose items models have not seen, that still has headroom and that matches the decision being made. In practice that means hidden or refreshed items, objective grading and, for a business decision, tasks drawn from the buyer's own work with uncertainty reported.
Does SourceX sell evaluation datasets from stock?
No. It holds no dataset stock: each dataset is sourced from U.S. businesses after a buyer describes scope, volume, language, format, timing and licensing needs. SourceX has not published request or buyer metrics and shares them with investors directly.
Sources
- Anthropic agent eval guide (January 2026): saturation is passing all solvable tasks; near saturation large gains appear as small score increases; SWE-bench Verified from 30 percent to above 80 percent; start with 20 to 50 tasks from real failures; two experts should reach the same verdict. Source
- Stanford AI Index 2025: scores on MMMU, GPQA and SWE-bench, introduced in 2023, rose by 18.8, 48.9 and 67.3 percentage points in one year. Source
- OpenAI no longer evaluates SWE-bench Verified: at least 59.4 percent of audited often-failed problems had tests rejecting correct solutions; models reproduced verbatim problem details; scores moved from 74.9 to 80.9 percent in six months. Source
- MMLU-Pro: harder reasoning-focused questions, ten answer choices instead of four, accuracy drops of 16 to 33 percent versus MMLU. Source
- Humanity's Last Exam: 2,500 expert-written questions, designed to be the final closed-ended academic benchmark of its kind, with a canary string for filtering from training data. Source
- ARC-AGI-2: 120 public evaluation tasks plus a semi-private set for remotely hosted commercial models (low leakage probability) and a fully private competition set (near-zero leakage probability), both outside the repository. Source
- LiveBench: new questions monthly, objective ground truth without an LLM judge, not all current-release questions public; November 2025 changelog refreshed tasks to address saturation and contamination. Source
- SWE-bench-Live: automated curation pipeline; 50 newly verified issues added monthly; lite and verified splits frozen for fair comparison; task-creation code published. Source
- OpenAI evals framework: companies can build private evals from their own data without exposing it publicly. Source
- OpenAI GDPval: 1,320 tasks based on real work products with multiple stages of expert review; a 220-task gold subset is released publicly. Source
- NVIDIA Nemotron training FAQ: RL prompts solved at a 100 percent pass rate are dropped; 3,000 held-out science problems chosen with pass rates between 0.25 and 0.80. Source
- Reddit Form 10-K for 2024: substantially all contract value associated with licensing revenue derives from two partners. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.