Skip to content

Leadership and readiness

Synthetic data volume is a vanity metric: what to underwrite instead

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Synthetic data volume is a vanity metric because generation is cheap and output counts can grow tenfold while usable value barely moves. Investors should underwrite two numbers instead: verified usable examples, which pass validity, deduplication, verification and a real-holdout test, and buyer acceptance against written criteria. Ask for both with fixed definitions, and test any growth claim with a same-size subsample control before crediting it.

Value drivers

Synthetic data volume vanity

This page covers synthetic data volume is a vanity metric: what to underwrite instead.Illustrative

Key takeaways

  • Generation cost puts no floor under volume: Stanford's Alpaca team produced 52K instruction examples for less than $500 in 2023.
  • Curated subsets can beat full sets: AlpaGasus, trained on 9k examples selected from Alpaca's 52k, outperformed the model trained on all of them.
  • A verified usable example clears four gates in order, valid, distinct, verified and useful, and counts only after the last.
  • Buyer acceptance against a written test is the growth figure only a customer can produce, and it needs both counts shown.
  • A same-size subsample control tells growth in quantity apart from growth in quality before any increase is credited.

Is synthetic data volume a growth metric at all?

Not on its own. Synthetic volume counts what a pipeline produced, and that can climb with compute and prompting alone, while the value a buyer pays for depends on how many outputs are correct, distinct and useful on real work. A company whose generated volume grows tenfold in a year may have grown its usable output by half, or not at all.

Generation is cheap enough that a count of it says little about effort or scarcity. Stanford's Alpaca team reported in 2023 that its 52K instruction-following examples cost less than $500 to produce, helped by generating 20 instructions per request. When output multiplies that cheaply, the count tracks spending more closely than progress.

Picture two readings of one growth slide. Read the generated column and the business compounds; read the accepted column and it grows at the pace its buyers set, which is the pace that predicts revenue quality. The gap between the columns is output that never reached a buyer. The figures below are invented to show that gap and describe no company. Index values, first quarter set to 100 (Illustrative):

Is synthetic data volume a growth metric at all?
QuarterGenerated examples (index)Verified usable examples (index)Buyer-accepted examples (index)
Q1100100100
Q2220140115
Q3480170130
Q41,000190150
Grouped bars for four quarters on an index where the first quarter is 100: generated examples climb to 1,000 while verified usable examples reach 190 and buyer-accepted examples reach 150.
One growth slide, three counts. Illustrative index values; the figures describe no company.

Three ways output counts outrun usable value

Repetition, filtering loss and selection effects each let volume grow faster than value, and each has public evidence behind it.

Repetition first. Muennighoff and colleagues found that repeating training data for up to about 4 epochs made little difference compared with unique data, but that the returns from further repetition eventually diminished to zero. Near-duplicate synthetic examples behave like repeats: they add tokens and almost no information.

Filtering loss second. In the AlpaGasus study, a model fine-tuned on 9k examples selected from Alpaca's 52k outperformed the original model trained on all 52k across four test sets, and also beat a model trained on a random 9k subset. The selected data was about 17.75% of the original set. Adding the other examples to the selected ones produced a weaker model on those tests.

Selection effects third. DataComp, a benchmark built on a candidate pool of 12.8 billion image-text pairs, found that smaller, more stringently filtered datasets can lead to models that generalize better than larger ones drawn from the same pool. The scarce input was the filter, not the pool, and a company that reports pool size is reporting the cheap part.

The metric substitution matrix for a synthetic data business

Swap each volume figure for a number that a buyer, an auditor or an untouched test can confirm. Each row gives the replacement, the denominator to insist on and the way the original figure inflates.

Read it from the left. Any figure in the first column that shows up on a growth slide without its replacement is a measure of activity, not of value:

The metric substitution matrix for a synthetic data business
Volume figure reportedUnderwrite insteadDenominator to requireHow the original inflatesWho can confirm
Tokens or rows generatedVerified usable examplesExamples generated in the same periodParaphrases, repeats and retries all countAn outside spot check plus the holdout test
Generation runs or API callsBuyer-accepted examplesExamples delivered against a written acceptance testInternal and test traffic count as usageThe buyer's acceptance record
Domains or templates coveredReal-holdout tasks passed per domainHoldout tasks per domain, frozen before generationA template counts once it exists, used or notWhoever froze the holdout
Model-graded quality scoreGrader agreement with expert reviewSize of the expert-reviewed sampleThe grader shares the generator's blind spotsNamed reviewers outside the pipeline
Cost per million tokens generatedCost per buyer-accepted exampleAccepted examples, not generated onesFalls with cheaper compute whether or not quality risesFinance records plus the acceptance record
Customers using the generatorBuyers who reordered after an accepted deliveryBuyers with an accepted delivery one period earlierTrials and pilots count as customersContracts and order history

Four gates a generated record must pass before it counts as usable

A verified usable example is a generated record that has cleared four gates in order and is counted only after the last: valid, distinct, verified and useful. Each gate strips out a different kind of empty volume.

At a glance

Four gates a generated record must pass before it counts as usable

Serious builders already count this way inside their own pipelines. DeepSeek's R1 report describes sampling several responses per prompt from an earlier checkpoint and keeping only the correct ones, ending with about 600k reasoning samples within roughly 800k fine-tuning samples. The reported figure is what survived the check, not what was sampled. A company that reports the sampled number is describing its compute bill.

The gates, in the order they should run:

  • Valid: the record parses, follows the schema and respects the business rules the task depends on, such as a refund never exceeding the order total.
  • Distinct: it is not a near-duplicate of a seed, of another generated record or of any holdout item, under a similarity rule written down in advance.
  • Verified: its answer or label was confirmed by an exact check where one exists, or by expert review on a sample with the agreement rate reported.
  • Useful: the batch it belongs to raised scores on an untouched real holdout compared with training without that batch.

Buyer acceptance: the growth figure only a customer can produce

Buyer acceptance counts examples a paying buyer accepted against criteria written before delivery. It is the nearest thing to a revenue-quality signal that a data company can report without disclosing prices, because it moves only when a customer's own test says yes.

Securities guidance already sets a standard for any metric a company leans on. The SEC's 2003 guidance on management's discussion and analysis told companies to identify and discuss the key performance indicators, including non-financial ones, that management uses to run the business. Its 2020 guidance on key performance indicators and metrics added that companies should consider what further material information a metric needs so it is not misleading, and what context an investor needs to understand it. Those rules govern public filers, but a fund partner can hold a private company's growth slide to the same test.

Three details make acceptance hard to game. Fix the acceptance test in writing before delivery and keep it unchanged when comparing periods. Report acceptance as a share of delivered examples with both counts shown. Separate first deliveries from reorders, since a buyer who accepts once and never returns has said something a single rate conceals.

Acceptance has a blind spot of its own: a buyer can accept examples that pass its checks and still see no change in its model. Where a buyer shares post-training results, pair acceptance with them. Where it does not, the reorder rate is the indirect signal, because buyers seldom reorder data that did nothing for them.

The growth attribution test: did a bigger quarter make a better dataset?

Credit volume growth only when a larger synthetic set beats a same-size random subsample of itself on an untouched real holdout. That single control separates growth in quantity from growth in quality, and the other arms price the alternatives at a matched budget.

Two conditions decide whether the test can run at all. The company must have kept each quarter's output as it was delivered, not regenerated it with today's recipe, and the holdout must predate both quarters; a holdout assembled afterward from cases the newer pipeline handles well will flatter the current quarter. If either condition fails, the honest conclusion is that growth in quality is unmeasured, not that it is zero.

Run it on a task a buyer funds, with the real holdout frozen before either quarter's data was generated and several training seeds per arm. Keep each quarter's generation recipe as it actually was, because the point is to test the company's real output rather than an idealized pipeline. The arms:

The growth attribution test: did a bigger quarter make a better dataset?
ArmTraining dataQuestion it answersHow to read it
A. Real seeds onlyThe real records the pipeline starts fromWhere the baseline sitsThe floor every other arm must clear
B. Last quarterSeeds plus last quarter's verified outputWhat the earlier output was worthThe reference for growth
C. This quarterSeeds plus this quarter's larger verified outputWhat the current output is worthMust beat B by more than the spread across training seeds
D. Same-size controlSeeds plus a random subsample of this quarter's output, cut to last quarter's sizeWhether the gain came from quality or quantityIf D matches C, the growth was in volume only
E. Real at matched budgetSeeds plus extra real records bought with arm C's generation and checking budgetWhether the money would have done more as real dataIf E beats C, generation is the costlier route for this task
F. Unfiltered volumeSeeds plus this quarter's raw output before any filterWhat filtering contributesIf F matches C, the filters are not doing work

When synthetic volume and value move together

Volume tracks value when every counted example has passed an exact, cheap check, so the count is already a count of verified records. Math with known answers, code that must pass its tests and reconciliations that must balance are the clearest cases, and there synthetic methods can stand in for licensed data outright.

DeepSeek's R1 pipeline shows volume done properly: rejection sampling against correctness converts sampled volume into verified volume before anything is counted. The claim here is not that large synthetic sets are useless. It is that the figure worth underwriting is the verified one, and that in business tasks judged by policy, discretion or outcomes that arrive months later, verification is costly, partial and the place where the margin sits.

In most business tasks synthetic data complements real inputs rather than replacing them. Seeds come from real records, and the holdout that proves usefulness has to be real. A company that grows its verified output and its access to real seeds and holdouts together is growing on both sides of that ledger, which is the pattern an investor should hope to find.

Risks that would make output counts more meaningful

Four developments would make raw output counts more informative, and each would weaken the argument above. Watch for them:

  • Verification gets cheap. If automated checks for business tasks become reliable, verified and generated counts converge and volume becomes a fair proxy.
  • Buyers pay per unit. Contracts priced per token or per example pay for volume directly, so revenue can track output even when usefulness does not, at least until renewal.
  • Acceptance tests go soft. A lenient acceptance test inflates acceptance the way a loose filter inflates volume; ask who wrote the test and whether it changed between periods.
  • Concentration hides in the denominator. If one or two buyers account for most accepted examples, acceptance growth measures a relationship rather than a market.

Growth-slide questions for any data company, SourceX included

Ask for the replacement metrics with definitions fixed before the period they describe, and for the growth attribution test on at least one funded task. A company that can produce both has measured its growth; one that offers only output counts has measured its spending.

SourceX is not a generator. It sits on the real side of the ledger, helping companies license selected business records to AI developers and sourcing each dataset against what a buyer asked for rather than from a shelf. The same discipline applies with records in place of generated examples: how many offered records became deliverable, and how many deliveries buyers accepted against written tests. None of those figures are public. SourceX keeps company and commercial metrics for direct conversations with investors, its Rule 506(c) raise admits only accredited investors, and the investors page is where to request materials and ask how each figure is defined.

Questions that do most of the work, for any company:

  • What share of generated examples became verified usable examples last quarter, under which written definition?
  • How many examples did buyers accept against written tests, out of how many delivered, and how many of those came from reorders?
  • Did a same-size subsample of this quarter's output score below the full set on a frozen real holdout?
  • What would the same budget have bought in real records, and was that arm run?
  • Which definitions changed during the year, and what do earlier periods look like restated on the new ones?

Methods note: sources, dates and limits of the substitution matrix

The metric substitution matrix, the four gates, the growth attribution test and the illustrative growth slide were prepared for this article on 2026-10-27 as editorial tools. They draw on the Stanford Alpaca repository, Muennighoff and colleagues on data-constrained scaling, the AlpaGasus and DataComp papers, DeepSeek's R1 report and the SEC's 2003 and 2020 guidance on MD&A metrics.

Not measured: the ratio of verified to generated output at any company, acceptance rates, the share of contracts priced per unit and the cost of verification by task. The index values in the growth slide are illustrative. Running the attribution test requires a funded task, a real holdout frozen before both quarters, retained copies of each quarter's output and several training seeds per arm.

Frequently asked questions

Why is synthetic data volume considered a vanity metric?

Because it measures pipeline output, which grows with compute and prompts whether or not the output is correct, distinct or useful. Stanford's Alpaca team generated 52K examples for less than $500, so large counts are cheap. Value depends on how many examples survive verification and how many buyers accept.

What should a synthetic data company report instead of volume?

Verified usable examples, meaning records that passed validity, deduplication, verification and a real-holdout test, and buyer-accepted examples counted against written acceptance criteria. Each should come with its denominator and a definition fixed before the period it describes.

How can an investor tell whether output growth improved a dataset?

Ask for a growth attribution test: train on this quarter's output, on last quarter's, and on a random subsample of this quarter's cut to last quarter's size, then score all of them on a frozen real holdout. If the subsample matches the full set, the growth was in volume only.

Do more synthetic training examples always improve a model?

No. Muennighoff and colleagues found returns from repeated data eventually diminish to zero, and AlpaGasus showed a 9k selected subset beating the full 52k Alpaca set. Selection and verification often matter more than count.

When is synthetic data volume a fair metric?

When every counted example has passed an exact, cheap check, such as a math answer or a code test, so the count is already a count of verified records. Pipelines that keep only correct samples, as DeepSeek's R1 report describes, report volume in that sense.

Sources

  • Stanford's Alpaca README reports 52K instruction-following examples generated for less than $500, with batch decoding of 20 instructions at once reducing cost. Source
  • Muennighoff and colleagues find that repeating data for up to about 4 epochs makes little difference versus unique data, while returns from further repetition eventually diminish to zero. Source
  • AlpaGasus, fine-tuned on 9k examples selected from Alpaca's 52k (about 17.75%), outperforms the 52k-trained Alpaca on four test sets and a random 9k subset. Source
  • DataComp, built on a 12.8 billion image-text pair pool, finds that smaller, more stringently filtered datasets can lead to models that generalize better than larger ones from the same pool. Source
  • DeepSeek's R1 report describes rejection sampling that keeps only correct responses, yielding about 600k reasoning samples within roughly 800k fine-tuning samples. Source
  • The SEC's 2003 MD&A guidance tells companies to identify and discuss key performance indicators, including non-financial ones, that management uses to manage the business. Source
  • The SEC's 2020 guidance on key performance indicators and metrics in MD&A (Release 33-10751) asks companies to consider what further material information a metric needs so it is not misleading, and what context investors need to understand it. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies