AI data market
A 4.9-point swing with no new data: underwriting benchmark performance as a data moat
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Benchmark performance proves a data moat only after the gain survives five gates: attribution to the data, replication, substitution, rights durability and repeat paying buyers. The first gate fails often. On Aider's public coding leaderboard, nine models gained a median 4.9 points, about five places, from reasoning or thinking settings alone, with no new data; neighboring runs sit a median 0.9 points apart.
Evaluation split
Benchmark performance data
Key takeaways
- A benchmark win shows incremental utility at best; a moat also needs replication, substitution, rights and buyer evidence.
- In 69 Aider polyglot runs, changing only reasoning or thinking settings moved nine models a median 4.9 points and five places.
- Aider's single retry with test feedback lifted scores a median 29.4 points, a harness effect with no data involved.
- Data-only attribution needs matched model, harness, retries, inference budget and test set, as DataComp-style designs show.
- Curated public data can close general benchmark gaps, and weak rights can force a model to be deleted, so later gates matter.
When does benchmark performance count as evidence of a data moat?
Only when the gain has been tied to the data and has then survived four commercial tests: a rival cannot rebuild comparable data, no cheaper source substitutes for it, the right to use it will last, and buyers keep paying for it. A benchmark win by itself shows, at most, incremental utility on one test at one moment. A moat is a claim about years.
The distinction matters because a benchmark chart is easy to produce without any data advantage. Scores move with the base model, the harness wrapped around it, the prompt format, the inference budget and the test's own flaws. A diligence team that credits the dataset for a gain the harness produced is underwriting an engineering choice a competitor can copy.
This article sets out five gates in order, attribution, replication, substitution, rights durability and repeat paying buyers, with a worksheet for each and a recount of public leaderboard data that shows how large the attribution problem is before any data changes at all.
How far does a score move with the data held fixed? 69 leaderboard runs recounted
Far enough to reorder a leaderboard. Aider's polyglot benchmark asks models to solve 225 Exercism coding exercises in C++, Go, Java, JavaScript, Python and Rust, and its published results file records each run's model, settings and pass rates. At commit 5dc9490 the file holds 69 runs, and nine models appear more than once at different reasoning-effort or thinking-budget settings, with the weights and the tasks unchanged.
Changing only that setting raised a model's score by a median of 4.9 percentage points, from 1.3 to 14.6, and moved it up a median of five places among the 69 runs. Adjacent runs on the board sit a median 0.9 points apart, so every one of the nine setting changes was larger than the typical gap between neighbors. Grok 3 Mini Beta went from 34.7 to 49.3; a Gemini 2.5 Flash preview went from 44.0 to 55.1 and climbed 13 places while its reported run cost rose about 7.5 times.
The harness itself moves scores further. Aider gives a model a second try on any exercise whose unit tests fail, sending the failing test output back as the next instruction. That one retry with feedback lifted the score by a median of 29.4 points across all 69 runs, 20 points or more in 48 of them, and 52.4 points for o4-mini at high effort. Swapping the pipeline cuts both ways: pairing DeepSeek R1 with a separate editing model added 7.1 points, while pairing o3 at high effort with GPT-4.1 cost 3.1.
None of these changes touched training data. If a company shows a benchmark gain without fixing the model version, the harness, the retry policy and the inference budget between the two runs it compares, the gain cannot yet be credited to its data.
Counting rules, as the script applies them: score is pass_rate_2, the percent of exercises whose tests all pass within two tries; setting groups are an explicit list of model labels that name the same model and differ only in a reasoning-effort or thinking-budget setting, stated in the label or left at its default; spread is the highest minus the lowest score in a group; rank is the position among all 69 runs by score, ties sharing the best place; the gap between neighboring runs is the difference between consecutive scores in that ranked list, ties counting as zero; second-try lift is pass_rate_2 minus pass_rate_1; a cost ratio divides the file's total_cost for the higher-scoring setting by that of the lower one; pipeline pairs are listed in the script as a model alone against the same model with a second editing model. All 69 runs are used, with no sampling or randomness, and the script is filed with the review materials for this page.
The nine setting groups, lowest and highest setting:
| Model (setting changed) | Lower setting score | Higher setting score | Points gained | Places gained |
|---|---|---|---|---|
| Grok 3 Mini Beta (reasoning effort) | 34.7 | 49.3 | 14.6 | 10 |
| Gemini 2.5 Flash preview 05-20 (thinking budget) | 44.0 | 55.1 | 11.1 | 13 |
| GPT-5 (reasoning effort) | 81.3 | 88.0 | 6.7 | 4 |
| o3-mini (reasoning effort) | 53.8 | 60.4 | 6.6 | 9 |
| Claude Sonnet 4 (thinking budget) | 56.4 | 61.3 | 4.9 | 6 |
| Claude 3.7 Sonnet (thinking budget) | 60.4 | 64.9 | 4.5 | 4 |
| o3 (reasoning effort) | 76.9 | 81.3 | 4.4 | 5 |
| Gemini 2.5 Pro preview 06-05 (thinking budget) | 79.1 | 83.1 | 4.0 | 4 |
| Claude Opus 4 (thinking budget) | 70.7 | 72.0 | 1.3 | 3 |
Prompts, test-time compute and leaky tests: the other suspects
Four published findings explain most benchmark gains that have nothing to do with a dataset, and each gives a diligence team a control to demand. They are not exotic; they are the default ways scores change between two runs.
Prompt format alone can swing results. Sclar and coauthors report accuracy differences of up to 76 points from formatting-only changes in few-shot prompts on LLaMA-2-13B, and about 10 points on average across more than 50 tasks and several models, sensitivity that persisted with larger models, more examples and instruction tuning. They recommend reporting a range across plausible formats rather than one number.
Inference budget substitutes for scale. Snell and coauthors found that, in a FLOPs-matched comparison, a smaller model given more test-time compute can outperform a model 14 times larger on problems where the smaller one already has a reasonable success rate, and that allocating compute adaptively per prompt improved efficiency by more than four times over a best-of-N baseline. Davidson and coauthors surveyed post-training enhancements such as tool use, prompting, scaffolding and solution selection and found most worth more than a fivefold increase in training compute on the benchmarks studied, some more than twentyfold, with fine-tuning for them typically costing under 1 percent of the original training run.
Tests leak and accept wrong answers. Reviewing the patches SWE-Agent with GPT-4 got accepted on SWE-bench, the SWE-Bench+ authors judged 32.67 percent to involve solution leakage, the fix already present in the issue report or comments, and 31.08 percent to have passed because the tests were too weak to catch incorrect code. Filtering out the problematic issues cut that agent's resolution rate from 12.47 percent to 3.97 percent. A gain on a leaky test may measure memory or luck.
What a data-only comparison looks like
It holds the model architecture, training code, compute and evaluation fixed and lets only the training data vary, then reports results across many held-out tests. That design exists in public research, which means a company claiming a data advantage can be asked to meet it.
DataComp set the template for image-text models: a benchmark in which the training code is fixed and participants compete by proposing new training sets, drawn from a 12.8 billion pair candidate pool, scored on 38 downstream tests at four compute scales. Its curated DataComp-1B subset trained a CLIP ViT-L/14 to 79.2 percent zero-shot ImageNet accuracy, 3.7 points above OpenAI's CLIP ViT-L/14 trained with the same compute. DataComp-LM extended the design to language models, with fixed pretraining recipes, model scales from 412 million to 7 billion parameters and 53 downstream evaluations.
Applied to a commercial claim, the design asks for a matched pair: the same base checkpoint, harness, retry policy, inference budget and evaluation set, with and without the dataset, run over several seeds and reported with intervals. Anything less leaves the gain unattributed. Our post-training ablation plan sets out the arms, seeds and reading rules for testing which part of a dataset helped; this article starts where that plan ends, at whether a confirmed gain is defensible.
After attribution: can a rival rebuild it, substitute it, or lose the right to it?
Attribution proves incremental utility; the remaining three gates test whether that utility lasts. Each has a public example of how an apparently strong data advantage can erode.
Substitution is the gate most often skipped. DataComp-LM's authors report that model-based filtering of public Common Crawl text produced a dataset that trained a 7 billion parameter model to 64 percent five-shot MMLU accuracy on 2.6 trillion tokens, 6.6 points above the previous best open-data model with 40 percent less compute, and close to Llama 3 8B while using 6.6 times less compute. If careful curation of public data closes most of a gap, a proprietary dataset's edge on a general benchmark is a lead, not a moat. Private operating records escape this test only when the benchmark measures work that public text does not describe.
Replication asks how a competitor with money would rebuild comparable data: license it from other holders, generate it, or collect it through their own product. The answer depends on how many companies hold similar records and whether they are willing to license, which is a market question no benchmark answers.
Rights durability is the gate that can reverse a result after the fact. The FTC's 2021 settlement with Everalbum, a photo app developer accused of deceiving users about facial recognition, required the company to delete models and algorithms it had developed using users' photos and videos. A benchmark gain built on records whose use rests on weak consent or ambiguous contracts can be unwound with the model that produced it.
Repeat paying buyers are the last gate and the only commercial one. A moat shows up as renewals and as several unrelated buyers paying for the same capability, not as a single pilot.
The benchmark underwriting worksheet and replication checklist
Score each gate pass, partial or fail on documents, and stop at the first fail: a later pass cannot rescue an unattributed gain. The worksheet is written for a technical diligence team reviewing an AI data company or a model developer that credits proprietary data for its results.
Start with the replication package, the nine items another team needs to rerun the comparison without the company in the room, then score the five gates in the order shown:
Checklist · 9 items
- Exact base model identifiers and checkpoints for every compared run.
- Harness version and configuration, including retry policy and tool access.
- Inference settings: reasoning effort or thinking budget, sampling parameters, attempts per task.
- The evaluation set or a hashed manifest of it, with its creation date and the training-data cutoff of each model.
- A contamination check result for the evaluation set against the training data.
- Seeds, run counts and per-task results, not only averages.
- Cost and compute per run, so budget differences are visible.
- The dataset's version, size and the rights record behind it.
- A tested public or synthetic substitute and its score under the same setup.
0 of 9 ticked. Tick each item as you go.
| Gate | Question for the company | Passes when | Fails when |
|---|---|---|---|
| 1. Attribution | Which runs differ only in the data? | Matched base model, harness, retries, inference budget and test set, several seeds, intervals reported | Model version, prompt format or budget changed between the compared runs |
| 2. Replication | How would a funded rival rebuild comparable data? | Few holders, slow to license or collect, and the company can show why | Similar records are widely held or easy to generate |
| 3. Substitution | Does curated public or synthetic data close the gap? | A tested public baseline, curated at similar effort, stays clearly behind | No substitute was tested, or a filtered public set comes close |
| 4. Rights durability | What written right supports each use, and for how long? | Signed licenses or owned records covering training, with term and revocation stated | Rights rest on terms of service, implied consent or expired agreements |
| 5. Repeat paying buyers | Who paid again, and who else pays for the same capability? | Renewals and several unrelated buyers, shown in contracts | One pilot, one buyer, or letters of interest only |
Ways this underwriting test could mislead
The framework can understate a real data advantage in at least four ways, and an investor should weigh them before treating a failed gate as a verdict.
Public leaderboards are a poor proxy for some commercial tasks. A private evaluation built from a buyer's own work may track value better than any public benchmark, and the gates above apply to it just as well; the recount here is public coding exercises only. Inference gains also have limits: Snell and coauthors report very little benefit from more test-time compute on the hardest questions they studied, which is exactly where better data could matter most. And Davidson and coauthors caution that enhancement gains are often domain-specific, so a large swing on one benchmark says little about general capability.
Attribution is expensive. Matched runs over several seeds cost real compute, and many early companies cannot afford them; an absent ablation is missing evidence, not proof that the data does nothing. The leaderboard inputs have their own limits: setting labels are as published by the maintainers, each entry is a single run without run-to-run variance, and serving changes at model providers between runs are not visible in the file.
Rights and substitution can move against the bear case as well as for it. Courts and regulators can make public data harder to use, which would strengthen licensed sources, and the FTC example concerned deceptive practices rather than every licensing defect. Buyer behavior is the hardest gate to observe from outside; without contracts, the fifth gate stays unscored, and that should be stated as unknown rather than guessed.
Running the gates on SourceX and on any data supplier
Apply the same five gates to SourceX as to anyone else, and treat SourceX as a case to test rather than an example of a passed gate. SourceX licenses operating records to AI developers directly from the businesses that hold them and manages the work from qualifying those records through rights review, contracting, preparation, delivery and payment. Each company approves the scope and terms of a license and signs before any records are delivered, and buyer datasets are sourced against a stated request rather than held in stock.
That model speaks most directly to gates two and four, how hard comparable records are to obtain and how well the right to use them is documented, and it gives no benchmark gain of its own to attribute. SourceX has published no benchmark results, renewal figures or other commercial metrics; it provides company and commercial information to investors directly. SourceX is raising from accredited investors under Rule 506(c), and investor materials can be requested through its investors page.
Questions to put to SourceX or to any company that credits proprietary data for model results:
- Which matched runs isolate your data's contribution, and can we rerun them?
- Which public or synthetic substitute did you test, and how close did it come?
- How many other holders of comparable records exist, and what would licensing them take?
- What signed document grants each training use, for what term, and what happens on termination?
- Which buyers renewed, and how many unrelated buyers pay for the same capability?
- Which gate do you expect to fail first, and what would change that?
Methods note: the Aider recount and the worksheet
Data: the public Aider repository at commit 5dc9490 (May 22, 2026), file aider/website/_data/polyglot_leaderboard.yml, 69 entries dated December 2024 to October 2025, read with Python's YAML parser. The retry mechanism is taken from the benchmark harness in the same commit, which appends failing test output to the next instruction, and the task description from the leaderboard documentation page.
Definitions: setting groups are the nine listed in the table, chosen because their labels name one model at different reasoning-effort or thinking-budget settings; architect pipelines and labels naming different checkpoints were excluded from the groups and reported separately. Places are competition ranks among all 69 entries by pass_rate_2. Cost ratios use the file's total_cost field. Some runs completed 224 or 223 of the 225 exercises; pass rates are as the file reports them.
Not measured: run-to-run variance, any model's training data, provider-side serving changes, and any SourceX data. The five-gate worksheet and replication checklist are editorial frameworks drafted on October 27, 2026 from the sources cited; they are not a scoring standard, and no company was rated with them here.
Frequently asked questions
Can a benchmark prove that proprietary data is a competitive advantage?
Not alone. A benchmark can show that a dataset improved a model on one test if the comparison held everything else fixed. Whether that advantage lasts depends on whether rivals can rebuild or substitute the data, whether the rights hold, and whether buyers keep paying, none of which a score measures.
What should a dataset ablation include to be credible?
Two or more arms that differ only in the training data: the same base checkpoint, harness, retry policy, inference budget and evaluation set, several seeds per arm, per-task results and intervals. A contamination check of the evaluation set and a tested public substitute make the result far more persuasive.
How much can inference settings change a benchmark score?
In Aider's published polyglot results, changing only the reasoning-effort or thinking-budget setting moved nine models by 1.3 to 14.6 points, a median of 4.9, while neighboring runs on the board sit a median 0.9 points apart. Published research reports even larger effects from prompt formatting and test-time compute.
Is a data moat weaker if public data can be filtered to similar results?
On general benchmarks, yes. DataComp-LM showed that model-based filtering of public web text trained a 7 billion parameter model to results close to models trained with far more compute. Private operating records keep their edge only on tasks public text does not describe.
Does SourceX publish benchmark gains from its datasets?
No. SourceX has not published benchmark results or commercial metrics; it shares company and commercial information with investors directly. Investors can request materials through the SourceX investors page, and any offering is limited to verified accredited investors through SourceX's offering documents.
Sources
- Aider's polyglot leaderboard file at commit 5dc9490 lists 69 runs with model, edit format, command, pass_rate_1, pass_rate_2 and total_cost; the setting-group, rank and lift figures in this article are computed from it. Source
- Aider's benchmark harness runs up to two tries by default and, when unit tests fail, sends the failing test output back to the model as the next instruction. Source
- Aider's polyglot benchmark tests models on 225 Exercism coding exercises across C++, Go, Java, JavaScript, Python and Rust. Source
- Sclar and coauthors report performance differences of up to 76 accuracy points from prompt formatting changes on LLaMA-2-13B, about 10 points on average across 50+ tasks, persisting with model size, more examples and instruction tuning. Source
- Snell and coauthors find compute-optimal test-time scaling more than 4x more efficient than best-of-N and, FLOPs-matched, able to beat a 14x larger model on problems where the smaller model has a reasonable success rate, with little benefit on the hardest questions. Source
- Davidson and coauthors estimate that most surveyed post-training enhancements give compute-equivalent gains above 5x, some above 20x, with fine-tuning typically under 1% of original training cost, and caution that gains are often domain-specific. Source
- SWE-Bench+ found 32.67% of successful SWE-bench patches involved solution leakage and 31.08% passed because of weak tests; filtering them cut SWE-Agent+GPT-4's resolution rate from 12.47% to 3.97%. Source
- DataComp introduced a benchmark with fixed training code where participants propose training sets from a 12.8B-pair pool, evaluated on 38 downstream tests; DataComp-1B trained CLIP ViT-L/14 to 79.2% zero-shot ImageNet accuracy, 3.7 points above OpenAI's CLIP ViT-L/14 at the same compute. Source
- DataComp-LM provides fixed pretraining recipes at 412M to 7B parameters and 53 evaluations; DCLM-Baseline trained a 7B model to 64% 5-shot MMLU on 2.6T tokens, 6.6 points above MAP-Neo with 40% less compute, close to Llama 3 8B with 6.6x less compute. Source
- The FTC's January 2021 settlement with photo app developer Everalbum required it to delete models and algorithms developed using photos and videos uploaded by its users. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.