Deal economics
AI training dataset buyer value: count the cases that clear the gate
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
AI training dataset buyer value is the change a dataset makes to one buyer's deployment decision, priced in that buyer's cost units, net of integration, re-measured on the records the license permits and capped by the cheapest alternative. Measure it at the buyer's automation gate, not as average accuracy: in a public digit-reading test, a 2.7-point accuracy gain cleared 17.0 points more cases.
Value drivers
AI training dataset buyer
Key takeaways
- Buyer value is specific to one buyer: what a dataset changes in that buyer's deployment decision, in its cost units, net of integration and capped by the cheapest alternative.
- Deployed models act above a confidence gate, so value follows coverage at the buyer's error target rather than average accuracy.
- In a public digit-reading test, going from 100 to 200 training records added a median 2.7 points of accuracy but cleared 17.0 points more forms at a 1-in-100 error gate; going from 400 to 797 added 1.0 and 3.3 points.
- Rights normalization works best as a re-test: with 75 of 100 added records usable, the median coverage gain fell from 17.0 to 12.2 points.
- Asking prices and published ranges are offers, not transactions; ask any data company for gate-level evidence on untouched cases, SourceX included.
What is a dataset's buyer value, and what would prove it?
A dataset's buyer value is the difference it makes to one buyer's deployment decision, expressed in that buyer's own cost units, net of the work needed to use it, measured on the records the license actually permits, and capped by the cheapest other way to reach the same result. It is a buyer-side number, specific to one organization, one model and one workflow.
That sets it apart from fair value in accounting. IFRS 13 defines fair value as a market-based measurement that uses the assumptions market participants would use, in an orderly transaction between them, and treats an entity's own intentions as irrelevant. Investors need both views: buyer value bounds what a particular buyer can rationally pay, while executed comparables show what deals have actually cleared.
The evidence that proves buyer value is concrete: the buyer's written acceptance gate, coverage at that gate on cases neither training set has seen, coverage with the dataset on the same cases, the buyer's volume and cost per routed case, an integration estimate, and a map of which records the license permits for this purpose. A company that can show those items for a deal can explain its price. One that can show only a benchmark lift cannot.
The thesis to test is simple. Datasets that move a buyer's model across its automation gate will be worth far more to that buyer than their average-accuracy lift suggests, and data businesses that measure value at the gate will price more consistently than those that quote benchmark gains.
Why average accuracy is the wrong unit for a buyer
Many deployed models do not act on every case: they act on the cases they are confident about and route the rest to people, so the buyer's money sits in how many cases clear that bar. Amazon's Textract documentation describes exactly this design for form processing: human review loops are created only when the model's confidence meets activation conditions the customer sets, for example on specific form keys.
Research on selective classification formalizes the same idea. Geifman and El-Yaniv (arXiv, 2017) describe a method in which the user sets a desired risk level and the classifier abstains as needed to meet it, which makes coverage at that risk the quantity worth maximizing. For a buyer, each case that clears the gate is a review it no longer pays for, and each case below the gate costs what it cost before.
Average accuracy blends the easy cases the model already gets right with the hard ones that decide whether it can be trusted alone. The same accuracy gain can move many cases across the gate or almost none, depending on where the buyer's model starts.
A public test: two doublings, very different buyer outcomes
To see how large the gap can be, I trained a simple classifier on a public form-reading task: scikit-learn's copy of the UCI handwritten-digits test set, 1,797 small images of digits taken from preprinted forms. For each of 21 fixed random splits, 600 images were held out for testing, 400 were used to set the gate, and the model learned from 50 to 797 of the rest. The gate accepts a reading automatically only at or above the confidence that kept errors to at most 1 in 100 on the gate-setting images; everything below goes to a person.
Doubling the training records from 100 to 200 raised accuracy by a median 2.7 points but raised the share of forms cleared at the gate by a median 17.0 points, roughly 170 fewer reviews per 1,000 forms, and coverage rose by more than accuracy in all 21 splits. Each gain here is the median of the 21 per-split gains, so it is not the gap between two columns of the table below. Roughly doubling again, from 400 to 797 records, added 1.0 point of accuracy and 3.3 points of coverage, about 33 fewer reviews per 1,000. Doubling at the low end was worth about five times as many avoided reviews as doubling at the high end, although the second doubling added almost four times as many records.
The gate also leaked. Set on 400 images, the threshold held errors at or under 1 in 100 on the test images in only 8 of the 21 splits at 200 training records, with a median realized precision of 98.9%. A buyer that reads coverage without checking the gate on fresh cases will overstate what the data is worth.
This is a mechanism demonstration on a clean public task, not evidence about any business dataset, buyer or deal. Medians across the 21 splits, by training-set size:
| Training records | Accuracy | Cleared at the 1-in-100 gate | Reviews per 1,000 forms | Precision of cleared forms |
|---|---|---|---|---|
| 50 | 81.3% | 37.3% | 627 | 98.7% |
| 100 | 89.2% | 61.8% | 382 | 99.2% |
| 200 | 92.8% | 79.3% | 207 | 98.9% |
| 400 | 94.0% | 83.8% | 162 | 98.9% |
| 797 | 95.2% | 87.2% | 128 | 99.1% |
The buyer value test in six steps
Each step produces a number a second analyst can reproduce, and skipping one turns the result into a guess. Run them in this order:
Checklist · 6 items
- Name the decision and the gate: what the model's output triggers, and the error rate the buyer accepts on cases handled without a person.
- Measure coverage at that gate with the buyer's current data, on evaluation cases that neither the current data nor the dataset has touched.
- Retrain with the dataset and measure coverage at the same gate on the same cases, over several splits, reporting the range as well as the middle.
- Repeat step 3 using only the records whose permitted use covers this buyer's purpose.
- Convert the coverage gain into the buyer's cost units at its real volume, then subtract integration and preparation.
- Cap the result at the cost of the cheapest other route to the same coverage, and confirm the gate on fresh cases before relying on any of it.
0 of 6 ticked. Tick each item as you go.
Worksheet: buyer value, normalized for license rights
The worksheet carries the six steps into one page that a buyer, a seller or an investor can fill in and audit. Rights normalization happens in line 4 as a re-test, not as a haircut: in the digit-reading test, keeping only 75 of the 100 added records cut the median coverage gain from 17.0 to 12.2 points. A flat 75% haircut on the full gain would have said 12,750 cases a year at the illustrative volume; the re-test says 12,200. The difference is small here and could run either way on other data, which is the argument for measuring it.
The demonstration medians are measured; the volume of 100,000 cases a year, the cost of 1 review unit per routed case and the 4,000-unit integration estimate are illustrative assumptions in review units, not money. The worksheet, line by line:
| Line | What to enter | Illustrative entry | Evidence that settles it |
|---|---|---|---|
| 1. Decision and gate | What the model's output triggers and the error rate accepted on automated cases | Accept a reading automatically only at or above the confidence that keeps errors to 1 in 100 | The buyer's written acceptance criterion |
| 2. Coverage without the data | Share of cases clearing the gate with current data | 61.8% (demo median, 100 records) | The buyer's evaluation on untouched cases |
| 3. Coverage gain with the full dataset | Same gate and cases, model retrained with the dataset | +17.0 points (demo, 100 to 200 records) | Paired runs over several splits, with the range |
| 4. Coverage gain on licensed records only | Line 3 repeated with records whose permitted use covers this purpose | +12.2 points (demo, 75 of the 100 added records) | Rights review mapping each record to its permitted uses |
| 5. Cases moved per year | Line 4 gain times annual volume | 12,200 of 100,000 cases (volume illustrative) | The buyer's volume records |
| 6. Year-one value in review units | Cases moved times cost per routed case, minus one-time integration | 12,200 minus 4,000 = 8,200 review units (illustrative) | The buyer's review cost and integration estimate |
| 7. Ceiling | The lower of line 6 and the cost of the cheapest other route to the same coverage | Whichever is lower | Quotes or internal estimates for each alternative |
| 8. Gate check on fresh cases | Realized error rate at the chosen threshold on new cases | Precision 98.9% at 200 records; target missed in 13 of 21 splits | A holdout set or a shadow deployment |
Do asking prices or published ranges show buyer value? Testing the alternative
No: an asking price or a published range records what someone hopes to receive, not what a buyer paid or what the data changed for that buyer. That is the strongest alternative to a buyer-side test, the view that market signals such as listings, rate cards and payout ranges already reveal value, and it fails on both counts. IFRS 13 builds fair value on orderly transactions between market participants; a listing is an offer, and a range on a website is neither a transaction nor a measurement of anyone's gate.
Buyer purpose statements are evidence of direction rather than value. OpenAI's Data Partnerships page, for example, describes seeking large-scale datasets that reflect human society and are not already easily accessible online, prefers data that expresses human intention such as long-form writing or conversations, and offers a private track for training proprietary models. It says what kind of data a buyer wants; it does not say what any dataset changed for that buyer or what was paid.
The alternative is right about one thing: executed comparables, normalized for rights, scope, preparation and purpose, are the best evidence of what deals clear at. Used together, the two views bracket a price. A deal far above the buyer value it can demonstrate is unlikely to repeat, and one far below it leaves room for the next negotiation.
What could make the buyer value test wrong
Several things could make a gate-level valuation wrong, and an investor should ask how each one is handled before trusting a reported figure. The main risks:
At a glance
What could make the buyer value test wrong
- Gate drift: if the buyer changes its error target, volume or workflow, the coverage gain and the value move with it.
- Noisy gates: in the demonstration, a threshold set on 400 images missed its target on test images in 13 of 21 splits at 200 records, so small gate-setting sets overstate coverage.
- Contaminated evaluation: if evaluation cases overlap the licensed records, the measured gain is inflated.
- Rights shrinkage: restrictions found after measurement shrink the usable records, as the drop from 17.0 to 12.2 points of gain shows in miniature.
- Substitutes: synthetic data, the buyer's own collection or another licensor can cap value regardless of the lift.
- Task transfer: handwritten digits are far cleaner than business records, so the size of the gate effect will differ, and near a model's ceiling extra data can move few cases at all.
- Concentration: a given gate may exist at only a few buyers, so value demonstrated to one buyer may not carry to the next.
What to ask a data company about buyer value, SourceX included
SourceX licenses business data directly from companies for AI training and evaluation, and it does not publish how it prices offers. Its how-it-works page says licenses are exclusive for AI training, so the supplier commits not to grant the same selected records to another party for that use, with the term and any alternatives fixed in each deal's Dataset Deal Confirmation. Its investors page says it shares company and commercial metrics with investors directly. SourceX's capital raise is open only to accredited investors under Rule 506(c); requests for investor materials go through the investors page.
The questions below apply to any company that sells, licenses or brokers training data, SourceX included, and each has a checkable answer:
Checklist · 6 items
- Do you measure a dataset's value at the buyer's gate, or as an average benchmark gain?
- Whose evaluation cases were used, and were they untouched by the licensed records?
- How many splits or repeats stand behind a reported gain, and what is the range?
- Is the gain re-measured on the records the license actually permits for the buyer's purpose?
- How often did the buyer's gate hold on fresh cases after deployment?
- Which alternative routes did the buyer price, and which one set the ceiling?
0 of 6 ticked. Tick each item as you go.
Methods note: the digit-reading test and the worksheet
Data: scikit-learn's bundled copy of the UCI optical digits test set, sklearn/datasets/data/digits.csv.gz in the scikit-learn repository at commit 42981ef2fd9154420e96ca36d90bfddb6f4ff8af (2026-10-27), read on 2026-10-28: 1,797 rows of 64 pixel counts from 0 to 16 plus a digit label. scikit-learn's description calls the file a copy of the test set of the UCI optical recognition of handwritten digits data; the UCI page lists 5,620 instances from 43 writers, 13 of whom supplied the test set. The run used only that bundled copy; the UCI download was not reachable from this environment, so the copy was not byte-compared with it, and a rerun on the official UCI file is a release check. Only the data file was read; no scikit-learn code was run.
Design: seeds 1 to 21; for each, numpy's default_rng(seed).permutation of the 1,797 rows, with positions 0 to 599 as test, 600 to 999 for setting the gate and 1,000 to 1,796 as the training pool. Training sets are the first 50, 100, 175, 200, 400 or 797 pool rows, nested, so the 175 set is the 100 set plus 75 of the 100 rows the 200 set adds. The model is multinomial logistic regression on pixel counts divided by 16 plus a bias, with weights starting at zero and 500 full-batch gradient steps at learning rate 0.5 and an L2 term adding 0.001 times each non-bias weight to the gradient of the mean cross-entropy. Confidence is the top class probability. The threshold is the confidence of the k-th most confident gate-setting image, where k is the largest count whose top k contain at most floor(0.01 x k) errors; test images at or above it are cleared.
Accuracy uses all 600 test images; coverage is the cleared share; precision is the correct share of cleared images; reviews per 1,000 forms is 1,000 times the uncleared share. Gains and avoided reviews are computed within each split and then summarized, so a reported gain is the median of 21 paired differences rather than the difference between two medians; a median is the 11th of 21 sorted values, and values are rounded once, half up. Worksheet line 2 uses the 100-record median, lines 3 and 4 the median paired gains and line 8 the 200-record results; the annual volume of 100,000 cases, the cost of 1 review unit per routed case and the 4,000-unit integration estimate are illustrative; cases moved equal volume times gain, and year-one value equals cases moved times cost per case minus integration. Not measured: any business dataset, any real buyer's gate, prices, or the extra cost of errors that pass the gate. The scripts and their outputs are kept with this draft for recount.
Frequently asked questions
How do AI buyers decide what a training dataset is worth to them?
By what it changes in deployment: how many more cases their model can handle without a person at the error rate they accept, at their volume, net of integration work and limited to the records the license permits. The cheapest other route to the same result caps that value.
Why isn't benchmark accuracy enough to value a dataset?
Because deployed models act only above a confidence threshold, and accuracy averages easy and hard cases together. In a public digit-reading test, a median 2.7-point accuracy gain came with 17.0 points more forms cleared at a 1-in-100 error gate, while a later 1.0-point gain cleared only 3.3 points more.
What is a deployment gate for an AI model?
It is the acceptance rule that decides when the model's output is used without review, usually a confidence threshold tied to an error rate the buyer accepts. Amazon's Textract documentation, for example, describes creating human review loops only when confidence meets conditions the customer sets.
How should license restrictions enter a dataset valuation?
Re-run the measurement on the records the license actually permits for the buyer's purpose instead of applying a flat discount. In the digit-reading test, keeping 75 of 100 added records cut the median coverage gain from 17.0 to 12.2 points.
Does SourceX publish how it prices datasets?
No. SourceX does not publish how it prices offers; licenses are exclusive for AI training, and each deal's term is set in its Dataset Deal Confirmation. Investors can request company and commercial metrics through the investors page.
Sources
- scikit-learn's digits dataset description says the file holds 1,797 instances of 64 attributes (8x8 images, pixel counts 0-16) and is a copy of the test set of the UCI handwritten digits data; SourceX ran its gate test on this bundled copy (sklearn/datasets/data/digits.csv.gz) at this commit, not on the UCI download. Source
- The UCI Machine Learning Repository lists the optical recognition of handwritten digits dataset with 5,620 instances and 64 features, from 43 people, 30 contributing training and 13 test data; SourceX's test used scikit-learn's bundled copy of the test set, not a UCI download. Source
- Amazon Textract's documentation describes Amazon A2I human review loops created only when Textract's confidence meets activation conditions, including confidence thresholds for specific form keys. Source
- Geifman and El-Yaniv (2017) describe selective classification in which the user sets a desired risk level and the classifier rejects instances as needed to meet it. Source
- IFRS 13 measures fair value with the assumptions market participants would use in an orderly transaction between them, so an entity's own intentions are not relevant. Source
- OpenAI's Data Partnerships page describes seeking large-scale datasets not already easily accessible online, preferring data that expresses human intention, and offering a private track for proprietary models. Source
- SourceX's how-it-works page says licenses are exclusive for AI training and that the term and any alternatives are set in each Dataset Deal Confirmation. Source
- SourceX's investors page says it shares company and commercial metrics with investors directly and offers an investor materials request. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.