Deal economics
Hybrid training data economics: a complement-or-substitute test
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Hybrid training data economics turns on one sign: does adding synthetic data raise or lower the value of the next real record? Studies point both ways. Synthetic gains leveled off near 300 billion tokens in SynthLLM's 2025 math runs and a 2025 preprint found real data restoring tail learning, yet the Llama 2 team stopped human annotation at 27,540 examples in 2023. Measure the cross-effect in a costed mixture grid before pricing either input.
Value drivers
Hybrid training data
Key takeaways
- The investable question is the cross-effect: does synthetic data make the next real record worth more or less?
- Published evidence leans toward complementarity for knowledge-heavy and long-tail tasks and toward substitution for style alignment and verifiable tasks.
- A six-arm grid at matched preparation budgets measures the cross-effect directly, with every arm scored on the same untouched real tasks.
- Cost per held-out point, with preparation, rights review and verification counted on both sides, is a better unit than cost per row.
- If cheap verifiers spread into business tasks, the complement case narrows to tails and evaluation; that is the signal to watch first.
How should investors evaluate hybrid training data economics?
Measure one quantity before modeling anything else: the cross-effect between real and synthetic data on held-out real tasks, per unit of preparation cost. If adding synthetic data makes the next real record worth more, the two inputs are complements and real-data suppliers gain from cheaper generation. If it makes the next real record worth less, they are substitutes, and real-data prices face pressure wherever generation gets cheaper.
Evidence that can settle it has three properties. Arms are compared at matched budgets with every cost line counted: sourcing, rights review and preparation on the real side; seed writing, generation, filtering and verification on the synthetic side. Scores come from real tasks that no arm has seen. And results are reported per task family, because the sign can differ between a formatting task and a rare-exception task inside the same model.
Most investor models skip this and assume substitution at a falling price. That assumption is right in some regions and wrong in others. What follows is a substitution matrix for telling them apart, an equal-score reading of hybrid mixes, and the grid that would settle the question for a given company's task families.
The substitution story is too simple, and the published curves show why
Synthetic data does replace some real data, but its returns flatten in places where real data keeps paying. In SynthLLM, a 2025 study of synthetic data for mathematical reasoning, performance gains leveled off near 300 billion synthetic tokens, and larger models reached their best results with fewer tokens: an 8-billion-parameter model peaked at about 1 trillion tokens, while a 3-billion-parameter model needed about 4 trillion.
A November 2025 preprint by Wang and colleagues describes the shape behind that flattening in mixtures of real and synthetic data: a rapid-learning phase dominated by common knowledge, a plateau where extra data yields diminishing returns because synthetic data covers rare tail knowledge poorly, and a tail-learning phase where enough real data restores gains. The authors attribute the tail gap partly to truncation from sampling methods such as top-p sampling and temperature scaling.
The Phi-4 technical report (December 2024) adds a production-scale example. Its authors raised the share of synthetic tokens compared with earlier Phi models, yet reported that models trained only on synthetic data underperformed on knowledge-heavy benchmarks and showed increased hallucinations, while web data brought small benefits on reasoning-heavy benchmarks. A synthetic-first team still kept real data in its mix.
Where real and synthetic records substitute, and where they complement
The sign of the cross-effect depends on what the data has to teach. The matrix below gives each regime the relation the cited evidence suggests and the pricing consequence for a supplier of real records. It is a reading of published studies, most of them on public text, math or code, and every row should be re-tested on business task families before it is priced.
| Regime | Likely relation | Dated public evidence | Pricing consequence for real records |
|---|---|---|---|
| Style and format alignment | Substitute beyond a small anchor | Llama 2 paper, 2023: human fine-tuning annotation stopped at 27,540 examples | Price a quality anchor, not volume |
| Tasks with a cheap, reliable verifier | Substitute once verification works | Feng and colleagues, 2024: selected synthetic data avoided collapse even with imperfect verifiers | Value moves to whoever owns the verifier |
| Knowledge-heavy tasks | Complement | Phi-4 report, 2024: synthetic-only training underperformed and hallucinated more | Real records keep a floor price |
| Long-tail and rare cases | Complement past the plateau | Wang and colleagues, 2025 preprint: enough real data restores tail learning | Tail coverage is the priced asset |
| Repeated generation across model versions | Complement: keep real data in the pool | Gerstgrasser and colleagues, 2024: replacing real data tends toward collapse; accumulating avoids it | Retained real archives hold value |
| Held-out evaluation | Not substitutable by design | OpenAI: stopped reporting SWE-bench Verified after flawed tests and training exposure | Sealed real test sets are a separate product |
Reading a hybrid mix as an equal-score curve
Plot real examples on one axis and synthetic examples on the other, and every mix that reaches the same held-out score lies on one curve. Where the curve is shallow, a little synthetic data replaces a lot of real data. Where it turns steep, more synthetic data buys nothing, and no amount of it replaces the next real record; that steep part is the plateau the studies above describe. The cheapest mix for a target score is the point where a line of equal total cost just touches the curve.
Preparation cost decides where that point lands. In an illustrative case, suppose a prepared real example costs 20 cost units, counting sourcing, rights review, de-identification and quality checks, and an accepted synthetic example costs 1 unit after generation, filtering and verification. If 2,000 real examples reach the target score alone, that route costs 40,000 units. If 500 real examples plus 15,000 synthetic examples reach the same score, the mix costs 25,000 units and wins, even though synthetic rows dominate it. But if the target needs tail cases the generator cannot produce, the curve turns steep before that mix exists and the saving disappears.
For an investor, the consequence is that a falling synthetic price moves the cheapest mix toward synthetic rows only along the shallow part of the curve. On the steep part, cheaper generation changes nothing, and the real records there carry the price.
The six-arm grid that measures the cross-effect
Six arms are enough to estimate the sign of the cross-effect for one task family. All share the base model, training recipe, compute and a sealed set of real tasks drawn from records that no arm or generator has touched; only the data changes. R is a fixed number of prepared real examples and S a fixed number of accepted synthetic examples, sized so the two carry comparable preparation budgets.
The cross-effect is the hybrid arm's score minus the real-only score, minus the synthetic-only score, plus the base score: D minus C minus B plus A. A positive value means synthetic data made the real records more productive; a negative value means each input made the other partly redundant. Comparing E with C and F with D shows which input to buy next, and every comparison should be reported as cost per held-out point, with intervals across repeated training seeds.
This page publishes the design only. The grid has not been run on SourceX data, and no results from it exist or are reported here.
| Arm | Real examples | Synthetic examples | What it identifies |
|---|---|---|---|
| A: base | 0 | 0 | The starting score every gain is measured from |
| B: synthetic only | 0 | S | What generation achieves without real records |
| C: real only | R | 0 | What prepared real records achieve alone |
| D: hybrid | R | S | The combined gain that defines the cross-effect |
| E: double real | 2R | 0 | The value of the next batch of real records |
| F: hybrid, double synthetic | R | 2S | Whether more synthetic data still sits on the shallow part of the curve |
Preparation cost belongs on both sides of the ledger
A hybrid comparison is only as fair as its cost accounting, and the usual error is to price synthetic rows at generation cost alone. Synthetic examples also carry seed writing, filtering, verification and contamination checks; real examples carry sourcing, rights review, de-identification, linking and quality review. Counting cost per accepted example, and then per held-out point, removes most of the illusion that synthetic data is nearly free.
Quality adjusts volume on both sides. The Llama 2 team reported setting aside millions of third-party fine-tuning examples in favor of fewer, higher-quality vendor annotations, and stopping at 27,540, noting that tens of thousands were enough for a high-quality result. That is evidence that a well-prepared real anchor can be small, which cuts both ways for a supplier: the anchor is valuable, but nobody needs it in bulk.
In the grid, record every cost line per arm in the same units before training starts. If the synthetic arm needed an expert to write seeds or adjudicate outputs, that time belongs in its budget, not in overhead.
The strongest case for substitution
Substitution is strongest wherever checking an example is cheaper than producing one. Feng and colleagues argued in 2024 that telling good synthetic examples from bad ones is easier than generating high-quality ones, showed in a theoretical analysis that well-selected synthetic data can reach optimal performance, and reported that even imperfect verifiers prevented collapse in their experiments, which included news summarization with Llama 2.
Combine that with the small human anchor in Llama 2's fine-tuning and SynthLLM's finding that larger models need fewer synthetic tokens to peak, and a coherent bear case for real-data volume appears: as verifiers spread from math and code into business tasks, the regions where real records act as complements shrink to the tail and to evaluation.
That is a falsifiable forecast, which makes it useful. If a company's own grid shows the cross-effect near zero or negative for most of its task families, the bear case has arrived for that company, whatever the market narrative says.
Risks that would break the complement thesis
Each of these would push hybrid economics toward substitution or make real records uneconomic:
At a glance
Risks that would break the complement thesis
- Verifiers spread. If Feng and colleagues' result generalizes to business tasks, selected synthetic data can stand in for real records outside the tail.
- Generators reach the tail. Better sampling or seeding could narrow the tail gap Wang and colleagues attribute to truncation; their paper is a preprint, and its three-phase finding may not replicate.
- Results do not carry over. SynthLLM's plateau comes from mathematical reasoning and Phi-4's comparison from general benchmarks; business task families may behave differently.
- Preparation and rights costs stay high. If rights review and de-identification keep real examples expensive, the cheapest mix shifts toward synthetic data even where real data is more productive per example.
- Evaluation leaks. If sealed real tasks reach any training run, as OpenAI reported for SWE-bench Verified, the grid stops measuring anything.
- Small anchors suffice. If tens of thousands of examples are enough, as in Llama 2's fine-tuning, real-data demand may be valuable but thin.
Questions to put to a real-data supplier in a hybrid market, SourceX included
These questions fit any supplier whose pitch depends on real records staying valuable as generation gets cheaper. SourceX is one such supplier. It takes companies that want to license operating records to AI developers through qualification, rights review, contracting, preparation, delivery and payment, and it sources each dataset against a buyer's stated request instead of selling from inventory.
SourceX is raising from accredited investors under Rule 506(c); its materials go to investors who ask for them at sourcex.si/investors. It has not run or published the grid described here, and its commercial metrics are not public. Ask, and expect answers broken out by task family:
- For which task families have you measured the cross-effect, and what was its sign?
- What does a prepared real example cost per accepted unit once rights review and de-identification are included?
- Where on the equal-score curve do your buyers sit today: the shallow part or the steep part?
- Which of your products would lose value if a cheap verifier appeared for that task?
- How are sealed real evaluation sets kept out of every training run, including your buyers' runs?
- Is your pricing set per record, per anchor set or per evaluation set, and why?
Methods note: sources, dates and what the grid leaves out
The substitution matrix, the equal-score reading and the six-arm grid are editorial frameworks prepared on 2026-10-27 from the cited sources: SynthLLM (2025), Wang and colleagues (November 2025 preprint), the Phi-4 technical report (December 2024), Gerstgrasser and colleagues (2024), Feng and colleagues (2024), the Llama 2 paper (2023) and OpenAI's post on retiring SWE-bench Verified. The relations in the matrix are inferences from those studies, mostly on public text, math and code, not measurements on business records.
The cost figures in the worked example are illustrative units, not prices, and the figure's curves are drawn for illustration. Not measured: any SourceX data, deal or result; the cross-effect for any real business task family; the size of the tail gap outside the cited studies; and buyer willingness to pay.
Frequently asked questions
Is synthetic data a substitute or a complement for real training data?
It depends on the task. Published evidence points to substitution for style and format alignment and for tasks with reliable verifiers, and to complementarity for knowledge-heavy tasks, rare cases and repeated generation, where keeping real data in the pool avoids collapse. For one task family, the only way to know is to measure the cross-effect in a mixture grid.
What is the cheapest mix of real and synthetic training data?
It is the mix where a line of equal total cost touches the curve of mixes that reach your target score on held-out real tasks. That point moves toward synthetic rows as generation gets cheaper only where synthetic data can still replace real examples; past the plateau, extra synthetic data buys nothing.
Does model collapse make synthetic data unsafe to train on?
Not by itself. Gerstgrasser and colleagues reported in 2024 that replacing real data with each generation's synthetic output tends toward collapse, while accumulating synthetic data alongside the original real data avoided it. The economic lesson is to keep real data in the pool rather than swap it out.
How much real data does a hybrid training set need?
There is no general number. The Llama 2 team stopped human fine-tuning annotation at 27,540 examples in 2023, but that was alignment for a general chat model, and tail-heavy business tasks may need more real cases. Size it with a mixture experiment scored on untouched real tasks.
Why does cost per row mislead investors about synthetic data?
A generated row is not a usable example until it has been filtered and verified, and a real row is not usable until it has been rights-reviewed and prepared. Comparing cost per held-out point, with all of those lines counted, shows which input actually buys performance.
Sources
- SynthLLM (2025) found synthetic-data gains in math reasoning leveling off near 300 billion tokens, with larger models peaking on fewer tokens. Source
- Wang and colleagues (November 2025 preprint) describe a plateau in real-synthetic mixtures because synthetic data covers tail knowledge poorly, with real data restoring gains. Source
- The Phi-4 report (December 2024) found synthetic-only models underperformed on knowledge-heavy benchmarks and hallucinated more, while web data helped reasoning slightly. Source
- Gerstgrasser and colleagues (2024) found replacing real data with synthetic data tends toward collapse while accumulating both avoids it. Source
- Feng and colleagues (2024) argue verification is easier than generation and report that even imperfect verifiers prevent collapse. Source
- The Llama 2 paper (2023) reports stopping human fine-tuning annotation at 27,540 examples because tens of thousands were enough. Source
- OpenAI stopped reporting SWE-bench Verified scores, citing flawed tests and training exposure. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.