Skip to content

AI data market

Benchmark saturation data opportunity: 2 of 7 tests still rank models

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Benchmark saturation is a data opportunity only where a test still separates models and extra inference cannot close the gap. Across the 23 OpenAI model rows in OpenAI's simple-evals table, two of seven benchmarks pass that bar: GPQA and SimpleQA. MMLU's best score leaves 6.7 points, close to the 6.49% of questions one audit estimates are flawed. The tests OpenAI kept maintaining center on questions and rubrics written by trainers and physicians.

Evaluation split

Benchmark saturation data

This page covers benchmark saturation data opportunity: 2 of 7 tests still rank models.Illustrative

Key takeaways

  • Saturation says a test has stopped ranking models; it says nothing yet about which data the next test needs.
  • In OpenAI's simple-evals table, by a stated rule, only GPQA and SimpleQA still separate OpenAI's 23 model rows, and a wider band leaves SimpleQA alone; OpenAI itself labels MGSM and DROP saturated.
  • MMLU's remaining 6.7 points sit next to an estimated 6.49% answer-key error rate, so that headroom is not evidence of uncovered failure.
  • Reasoning effort alone moved GPQA by 4.8 to 9.6 points and SimpleQA by under 1 point, so data claims must hold the inference budget fixed.
  • The replacements OpenAI kept center on questions and rubrics written by trainers and physicians; business records matter only where the next task population lives inside companies.

What does benchmark saturation mean for data demand?

It means one test can no longer rank the models a buyer is choosing between. It does not mean the work the test sampled is solved, and it does not by itself create demand for any particular kind of data. Saturation moves evaluation spending to a new task population; whether that population needs records only businesses hold is a separate claim an investor has to test.

Three observable signals mark a saturated test: many models bunched within a few points of the best score, remaining headroom close to the measured error rate of the answer key, and the test's own publisher saying so. OpenAI's simple-evals repository supplies the first and third for its own models, and a published audit of MMLU supplies the second. A footnote on two of its columns reads: "We believe these evals are saturated for our newer models, but are reporting them for completeness." A notice dated July 2025 says the repository will no longer be updated for new models or benchmark results.

The commercial claim usually attached to saturation is bigger than the evidence. A data company that cites it is saying the next test needs that company's records. The public record supports something narrower: saturation reliably creates demand for newly written, expert-checked test items, and only part of that demand reaches records that sit inside companies.

Which tests in OpenAI's own table are used up?

Two of the seven still separate OpenAI's models: GPQA and SimpleQA. The rule, fixed in the analysis script and applied the same way to every column, counts a benchmark as still separating when its best score leaves more than 10 points of headroom and fewer than 4 scored rows sit within 3 points of that best score. Everything else reads as used up for these models. Widen the band to 5 points and only SimpleQA remains, which shows how close GPQA sits to the line.

We parsed the results table in the simple-evals README at commit 652c89d, dated April 22, 2026, kept the 23 rows for OpenAI's own models and dropped the rows OpenAI lists as reported by other developers. MATH is the plainest used-up case: 10 of 23 rows sit within 3 points of o4-mini-high's 98.2, and the column mixes two test sets, since a footnote says models from o1 onward were scored on MATH-500. MMLU has 7 rows within 3 points of o3-high's 93.3. GPQA, by contrast, spreads 43.2 points, from 40.2 to 83.4, and SimpleQA spreads 54.9 points, from 7.6 to 62.5.

DROP shows that a used-up test does not always look compressed. Only 2 rows sit within 3 points of its best, and its headroom is 9.8 points, yet OpenAI calls it saturated, and o3's scores run out of order with reasoning effort: 82.3 at low effort, 80.6 at the default and 89.8 at high. Spread on an exhausted test can be noise rather than signal, which is why the rule needs both conditions.

The reading for all seven columns follows; effort gaps are the high-effort score minus the low-effort score for o3, o4-mini and o3-mini.

Which tests in OpenAI's own table are used up?
BenchmarkBest score (row)Rows within 3 points of bestHeadroom to 100Effort gap: o3, o4-mini, o3-mini
MMLU93.3 (o3-high)7 of 236.7+0.5, +0.8, +2.0
GPQA83.4 (o3-high)3 of 2316.6+4.8, +7.7, +9.6
MATH, mixed with MATH-50098.2 (o4-mini-high)10 of 231.8+1.2, +2.0, +2.1
HumanEval99.3 (o4-mini-high)4 of 220.7+1.1, +3.4, +3.1
MGSM (OpenAI: saturated)93.7 (o4-mini)9 of 236.3+0.1, +0.5, +2.6
DROP (OpenAI: saturated)90.2 (o1)2 of 239.8+7.5, +2.1, +3.0
SimpleQA62.5 (gpt-4.5-preview)1 of 2137.5-0.8, -0.9, +0.8
Range bars for seven benchmarks in OpenAI's simple-evals table, from the lowest to the best score among 23 OpenAI model rows, with GPQA and SimpleQA highlighted as the two that still separate models.
Lowest to best score per benchmark, 23 OpenAI model rows. Source: OpenAI simple-evals README, commit 652c89d (April 2026); separation rule from this article.

Is the headroom left on MMLU uncovered failure or a flawed answer key?

Mostly a flawed answer key, on the best public estimate. o3-high's 93.3 leaves 6.7 points of headroom, and in Are We Done with MMLU?, Gema and colleagues estimate that 6.49% of MMLU questions contain errors, based on MMLU-Redux, a set of 5,700 re-annotated questions covering all 57 subjects. In the virology subset, 57% of the questions they analyzed were flawed.

The comparison does not prove that every remaining miss is a bad key; a model can also agree with a wrong key and be scored correct. It does mean an investor cannot treat the last 6.7 points as real-world failure waiting for new data. The same arithmetic governs any private evaluation set a data company sells: its useful life ends when top scores reach its own error floor, so a supplier that has not measured that floor cannot say how long its product will discriminate.

Researchers described the pattern before the current model generation. Introducing the Dynabench platform at NAACL 2021, Kiela and colleagues argued that models now pass human-performance estimates on new benchmarks within a few years rather than decades, while still failing simple challenge examples. Their response was to collect test data continuously, with people and models in the loop, rather than to publish another static set.

Did more inference, rather than better data, move the scores?

Often, and OpenAI's table measures it, because three model families appear at low, default and high reasoning effort. Moving from low to high effort raised GPQA by 4.8 points for o3, 7.7 for o4-mini and 9.6 for o3-mini, with the same trained model each time. The same moves added 0.5 to 2.0 points on MMLU and shifted SimpleQA by between minus 0.9 and plus 0.8.

That split is the most useful line in the table for a data thesis. GPQA's remaining gap responds to compute spent at answer time. SimpleQA's barely moves, which fits a test of what the model already knows rather than how long it thinks, though the table alone cannot prove that reading. OpenAI's BrowseComp paper reports the compute side for agents: accuracy rose smoothly as test-time compute increased for an early version of its Deep Research system.

The underwriting rule follows. Compare scores only at a fixed inference budget, prompt and grader, and ask any company that credits a gain to its data to show the gain with all three held constant. A data claim that cannot hold the budget fixed is a compute claim.

Where did evaluation demand go once the table was retired?

Into newly written test sets built by human experts and trainers, which is real demand for data work but not yet demand for licensed business records. The July 2025 notice says simple-evals will keep hosting reference implementations for three evals, HealthBench, BrowseComp and SimpleQA, and each was assembled from new human-authored material.

SimpleQA holds 4,326 short, fact-seeking questions graded correct, incorrect or not attempted, and its paper says the questions were collected adversarially against GPT-4 responses. BrowseComp's 1,266 questions were written by human trainers to be hard to find but quick to check; trainers working without AI assistance solved 29.2% of them, and OpenAI describes earlier retrieval benchmarks as saturated. HealthBench draws on 262 physicians who have practiced in 60 countries; they wrote conversation-specific rubrics for 5,000 conversations, 48,562 unique criteria in all, and GPT-4.1 grades responses against them.

Demand for this kind of work has outrun supply at least once on the record. Announcing funding for third-party evaluations in 2024, Anthropic wrote that high-quality, safety-relevant evaluations remain hard to develop and that demand for them is growing faster than supply.

Read the pattern with care. In each case the published description centers on newly authored material: questions written by trainers for SimpleQA and BrowseComp, rubrics written by physicians for HealthBench. That is demand for expert authorship, not evidence of demand for licensed operating records. Business records enter only where the next task population is work that happens inside companies and whose context an author would have to invent: a dispute history, an approval chain, an exception queue with its real outcomes.

Seven questions that separate a used-up test from uncovered failures

Ask these seven in order, and stop at the first that fails: a claim that saturation creates demand for a company's data survives only if the old test is truly exhausted, the gap is real and not compute, and the replacement needs records an author cannot write.

Each row names the evidence to request, what a pass looks like and the warning sign that should end the conversation or change the price.

Seven questions that separate a used-up test from uncovered failures
QuestionEvidence to requestPasses whenRed flag
Is the old test used up for the models in question?The full score table at a pinned version, every rowMany models within a few points of the best, by a stated ruleOnly the top score is shown
Is the remaining headroom answer-key error?A measured error rate for the answer key, by subjectHeadroom clearly exceeds the audited error rateNo audit, or headroom below the error rate
Does the gap survive a fixed inference budget?Scores at matched effort, prompt and graderThe gap holds at equal budgetGains appear only at higher effort
What task population replaces the test, and who wrote it?Item sources, author types and the collection protocolA named population with documented authorshipHarder items with no population defined
Does the replacement need records only businesses hold?Sample items and where their context came fromContext could not be written without operating recordsAn expert could write the items from public knowledge
Does the claimed data close the gap?An ablation with and without the data at a fixed budgetA gain on held-out items larger than run-to-run variationGains only on items resembling the training data
How soon will the replacement saturate?Refresh plan, private split and measured error floorA refresh or private holdout with a known floorA static public set with no refresh plan

Replicating this reading on any published score table

Anyone can repeat this reading on a published table in an afternoon, and the steps matter more than the particular numbers. Our version took one file and one short script; the list below is what made it reproducible.

Use these steps for any score table a company presents in diligence:

  • Pin the source: repository, commit hash, file path and the date read.
  • Separate first-party rows from rows a publisher copied from other developers, because prompts and graders differ.
  • Read every footnote. In simple-evals, footnotes record a test-set swap (MATH-500), an answer-parsing tweak for GPQA and the saturation note on MGSM and DROP.
  • Write the thresholds into the script before reporting results, and publish them with the numbers.
  • Pair rows that differ only in inference effort, and report high-minus-low gaps per benchmark.
  • Find an independent estimate of each answer key's error rate, and compare it with the headroom.
  • Report the number of scored rows per column and say which cells were missing.
  • Keep the script with the article so a reviewer gets the same numbers.

What could make the saturation-to-data thesis wrong

Five developments would weaken the thesis that saturation pulls evaluation demand toward business records. Each can be checked against public evidence as it appears.

  • Model graders could shrink the human share. HealthBench already grades with GPT-4.1 against physician-written rubrics; if that spreads, paid human work concentrates in writing rubrics rather than judging answers.
  • Labs could keep authoring in-house. The three evals OpenAI kept maintaining center on material written by trainers and recruited physicians; if that pattern holds, the next tests create contract work rather than licenses.
  • Inference could keep closing gaps. GPQA's effort gaps of 4.8 to 9.6 points in OpenAI's table show reasoning budget can recover headroom with no new data.
  • Public tests may be retired without visible successors. simple-evals stopped updating in July 2025; a lab can stop publishing a test without saying what replaced it internally, which hides demand from outside investors.
  • Error floors cap private sets too. A supplier's evaluation set with an answer-key error rate like MMLU's will stop discriminating at that floor, so its commercial life is shorter than its size suggests.

Questions to put to an evaluation-data supplier, SourceX included

Put the worksheet's questions to any company that sells evaluation data, and expect answers measured on your definitions. SourceX belongs on that list. It licenses operational records from companies for AI training and evaluation, and it carries each deal from qualifying the supplier and reviewing rights through contracting, preparing and delivering the data, to paying the company.

None of SourceX's commercial figures are public, and this article reports none. The company is raising capital under Rule 506(c), which limits purchasers to verified accredited investors, and it sends diligence materials on request from its investor page.

Questions that test whether a supplier's records address uncovered failures rather than a used-up test:

  • Which saturated test does your data address, and which task population replaces it?
  • What is the measured answer-key error rate of your evaluation items, and who measured it?
  • Show a gain attributable to your data at a fixed inference budget, prompt and grader.
  • Which of your items could an expert have written without a supplier's records?
  • How do you refresh or hold out items to delay your own saturation?
  • Who approved the use of the underlying records for evaluation, and under what agreement?

Methods note: how the simple-evals table was read

Source: README.md in github.com/openai/simple-evals at commit 652c89d0ca9df547706735883097e9537d40dc47 (April 22, 2026), read on October 27, 2026 with the script simple_evals_saturation.py, kept with this draft for reviewers. Rows: every row in the Benchmark Results table above the row headed Other Models (Reported), skipping model-family header rows with no scores; 23 rows remain. A cell counts when it parses as a number; dashes and n/a are missing, which leaves 22 scored rows for HumanEval and 21 for SimpleQA.

Measures: best score, rows within 3.0 points of the best inclusive, headroom to 100, and high-minus-low effort gaps for o3, o4-mini and o3-mini. Rule: a benchmark still separates when headroom exceeds 10.0 points and fewer than 4 rows sit within 3.0 points of the best. The thresholds are editorial choices, so the script also re-runs the rule with other bands: at 2 points the same two benchmarks pass, and at 5 points GPQA drops out, leaving SimpleQA alone. No sampling or random draws were used.

Not measured: run-to-run variation, since the table reports single numbers; rows from other developers; OpenAI's internal evaluations; and whether any replacement test drew on licensed records. The MMLU error estimate comes from the 5,700-question MMLU-Redux sample, not the full test, and the article does not compare it question by question with model answers.

Frequently asked questions

What is benchmark saturation?

A benchmark is saturated when it no longer separates the models people are choosing between: top scores bunch together near the ceiling, the remaining headroom is close to the answer key's own error rate, or the publisher says so. OpenAI, for example, notes in its simple-evals repository that it believes MGSM and DROP are saturated for its newer models.

Does benchmark saturation mean a model has solved the task?

No. It means the test's sample of the task is exhausted. Remaining misses can be answer-key errors rather than failures, as with MMLU, where one audit estimated 6.49% of questions contain errors, and models that top a static test can still fail simple challenge examples outside it.

Which AI benchmarks are saturated?

It depends on the models compared. In OpenAI's own published table, MMLU, MATH, HumanEval, MGSM and DROP no longer separate its recent models by the rule used in this article, while GPQA and SimpleQA still do. Repeat the reading for the models in your own decision rather than relying on a general list.

Does benchmark saturation increase demand for AI training data?

It increases demand for new evaluation items more clearly than for training data. The tests OpenAI kept maintaining, SimpleQA, BrowseComp and HealthBench, center on newly written questions and physician-written rubrics. Demand reaches business records only when the next test covers work whose context exists inside companies.

How should an investor test a company that cites benchmark saturation?

Ask for the full score table at a pinned version, an audited answer-key error rate, results at a fixed inference budget and an ablation showing the company's data closes a gap on held-out items. Then ask whether its items could have been written by an expert without any company's records.

Sources

  • OpenAI's simple-evals README publishes scores for its models on seven benchmarks, labels MGSM and DROP saturated for newer models, notes MATH-500 for o1 onward, and says the repository stopped updating in July 2025. Source
  • Gema and colleagues estimate 6.49% of MMLU questions contain errors, using MMLU-Redux (5,700 re-annotated questions across 57 subjects), with 57% of analyzed virology questions flawed. Source
  • Kiela and colleagues (NAACL 2021) argue models pass human estimates on new benchmarks within years yet fail simple challenge examples, and propose human-and-model-in-the-loop data collection. Source
  • OpenAI's SimpleQA has 4,326 short fact-seeking questions graded correct, incorrect or not attempted. Source
  • The SimpleQA paper says questions were collected adversarially against GPT-4 responses. Source
  • BrowseComp's 1,266 questions were written by human trainers; trainers without AI solved 29.2%; earlier retrieval benchmarks are described as saturated; accuracy scales with test-time compute. Source
  • HealthBench was built with 262 physicians from 60 countries: 5,000 conversations, 48,562 rubric criteria, graded by GPT-4.1. Source
  • Anthropic wrote in 2024 that high-quality safety-relevant evaluations remain hard to develop and demand is growing faster than supply, and announced funding for third-party evaluations. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies