Deal economics
Dataset valuation utility pilot: what a small buyer test can and cannot tell you
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A small buyer pilot can establish whether your records are usable as delivered, whether they cover the cases the buyer cares about and whether a sample moves the buyer's own metric against a baseline. It cannot set a price on its own, predict the gain at full volume with precision, or speak for other buyers. Agree the question, sample, metric, permitted uses and what results come back before any records leave.
Value drivers
Dataset valuation utility
Key takeaways
- A pilot measures utility for one buyer on one question; value also depends on rights, exclusivity scope, preparation effort, timing and the buyer's alternatives.
- Agree before the pilot what the buyer will measure, what it compares against and which results it will share with you.
- No lift on a small sample is weak evidence that the data lacks value; lift on a small sample is weak evidence of how much value the full set has.
- The pilot terms decide whether anything the buyer trains on your sample stays usable after the pilot, which is itself a value question.
- Unique cases, outcome fields and coverage of rare situations usually explain a pilot result better than raw volume does.
What can a small buyer pilot establish about your dataset's value?
A pilot establishes three things well: whether the records survive the buyer's intake checks, whether they cover the situations the buyer is trying to teach or test, and whether a sample changes the buyer's own metric compared with a baseline. Those are evidence of utility and fit for one buyer. They are inputs to a value conversation, not a valuation.
What a pilot does poorly is extrapolate. A gain on a few thousand records does not tell you the gain on a few hundred thousand, and a buyer's result on its model says little about what a different buyer would see. Treat the pilot as the cheapest way to replace guesses about your data with observations, and be explicit about which guesses remain.
For the business that owns the records, the practical goal is narrower still: leave the pilot knowing which of your records the buyer found useful, which it could not use and why, so the next conversation is about specific records rather than about the dataset in general.
Why can the same records be worth more to one buyer than another?
Data value depends on context as much as content. The OECD's 2021 report on measuring the economic value of data names both as components of value and notes that the context in which data are collected, stored and used largely determines how much of their potential value is realized. A claims history is worth more to a team building a claims-triage model than to one building a general chat assistant, even though the records are identical.
Governance matters as well. A 2022 OECD paper argues that the value of data depends to a large extent on the governance framework that determines how it can be created, shared and used. For a pilot, that means the permitted uses you can offer, and those you cannot, shape the result before any model is trained.
This is why a pilot with one buyer should not be read as a verdict from the market. A second buyer with a different model, a different gap and different alternatives can reach the opposite conclusion on the same sample, and both can be right about their own situation.
The utility pilot scope and result interpretation worksheet
Agree each line below with the buyer before the sample leaves your systems. The third column explains why the line matters when you later talk about value; entries in brackets are illustrative.
Keep the signed worksheet with the pilot terms. When the result arrives, the first check is whether the buyer tested what was agreed: the same sample, the same metric and the same baseline. A result produced on a different question can still be informative, but it should not be read as the answer to the one you agreed.
| Input | What to agree before the pilot | Why it matters for value | Who supplies it |
|---|---|---|---|
| Buyer question | The one decision the pilot informs, in a sentence (does the sample cut routing errors on multi-site jobs) | A pilot without a question produces a result nobody can act on | Buyer, confirmed by you |
| Sample definition | How records are drawn, from which system and date range (random 2,000 closed work orders from 2023 to 2025) | A hand-picked sample makes any result unrepeatable at volume | You, with the buyer's agreement |
| Unique case count | How many distinct situations the sample contains, not just records | Repeated near-identical records add little beyond the first | You |
| Coverage of rare cases | Which uncommon situations the sample includes and how many of each | Rare, well-documented cases are often what a buyer cannot find elsewhere | You and the buyer |
| Baseline and comparison | What the buyer compares against (current model, or a random set of similar size) | A result without a baseline cannot be called a gain | Buyer |
| Metric and threshold | The buyer's metric and the change it would treat as meaningful | Tells you in advance what counts as success | Buyer |
| Permitted use of the sample | Evaluation only or training allowed, retention period and deletion | Determines whether the pilot itself transfers value | Both, through the pilot terms |
| Results shared back | Which results you receive: overall metric, per-slice results, rejected records and reasons | Without feedback you learn nothing about your own data | Buyer, written into the terms |
| Time box | Start date, end date and the decision date (eight weeks from delivery) | An open-ended pilot delays every other conversation | Both |
| Decision after the pilot | What each outcome leads to: proceed, narrow, expand the sample or stop | Turns the result into a next step instead of a debate | Both |
How should you read each possible result?
Read the result against the question you agreed, not against a hope. The table pairs the common patterns with what they suggest and, just as important, what they do not show.
Mixed results are the most common and the most useful. A gain concentrated in one slice points to a narrower, better-defined license; a gain that appears only after the buyer cleaned your records points to preparation work you could do once and offer to every later buyer.
| Result pattern | What it likely means | What it does not mean | Sensible next step |
|---|---|---|---|
| Clear gain over the baseline on the agreed metric | The records carry signal the buyer's current data lacks | That the full dataset will produce a proportionally larger gain | Discuss a larger, defined scope and how gains at volume will be checked |
| No measurable gain | Either the sample lacked the needed cases or the test was too small to detect a change | That the dataset has no value to anyone | Ask for per-slice results and whether the sample size could detect the agreed threshold |
| Gain on one slice only | Value is concentrated in specific case types | That the rest of the dataset is worthless | Offer a scope built around the slice that worked |
| Records failed intake checks | Fields, format or de-identification did not meet the buyer's needs | That the underlying cases lack value | Fix preparation and rerun a small sample |
| Buyer shares no results | You cannot tell which of the above happened | That the pilot went well or badly | Make shared results a condition of any future pilot |
Why does a small sample understate or overstate the full dataset?
Learning curves are not straight lines. Hestness and colleagues found that generalization error falls as a power law of training set size across several domains, but with exponents that had to be measured for each problem, and that very small training sets sit in a region dominated by best guessing before that curve takes hold. A pilot sample may sit in that early region and show little, or sit on a steep part of the curve and show more than later records will add.
Small comparisons also mislead in the other direction. Card and colleagues showed that underpowered experiments are common in language research and that significant results from small samples tend to overstate the true effect, or even get its direction wrong. So a striking gain on a small pilot deserves the same caution as a disappointing one.
The useful response is to ask the buyer to test at two sample sizes if the pilot budget allows, and to report a range rather than a single figure. Two points on a curve say more about value at volume than one point measured very precisely.
Which assumption moves your net proceeds and the buyer's fit the most?
For the business, the assumption that most often changes the outcome is preparation effort at full volume. A pilot sample is usually extracted and de-identified with care by hand; doing the same for the whole dataset can take far longer and require system work, and that effort comes out of what the licensing is worth to you. Ask your own team to estimate the full-volume preparation work before the pilot ends.
At a glance
Which assumption moves your net proceeds and the buyer's fit the most
For the buyer, the deciding assumptions are usually its alternatives and its timing. If the buyer can collect similar records itself, or already holds a comparable set, a good pilot result is worth less to it than the same result for a buyer with no other source. If the buyer's model release is close, records that arrive late may be worth little regardless of quality.
- Preparation effort: hours and system changes needed to deliver the full set in the piloted condition.
- Permitted uses: whether the full license would cover training, evaluation or both, as tested.
- Exclusivity scope: whether the buyer expects sole use, and for what field and period.
- Buyer alternatives: other sources the buyer could use for the same cases.
- Timing: when the buyer needs delivery for the result to matter.
- Coverage at volume: whether the rare cases in the sample stay as frequent in the full set.
Can the pilot show which of your records carry the value?
Sometimes, if the buyer reports results by slice. Research methods exist to attribute a model's behavior to individual training records; the open-source Datamodels toolkit trains many models on different subsets of a training set and fits a model of their outputs, which lets a team see which subsets drive which predictions. Few buyers will run anything that elaborate on a pilot, but the idea behind it is simple to request.
Ask for results broken down by the case types you defined in the worksheet, and for the list of records the buyer rejected with a reason for each. That feedback tells you which parts of your history are worth preparing for a larger license and which are not, which is often worth more to you than the headline metric.
What should the pilot terms say about the sample and anything built from it?
The pilot terms decide whether value leaves with the sample. If the buyer may train on the sample and keep the resulting model, part of what you would license later has already been transferred. Public data agreements show how easily this happens: the Computational Use of Data Agreement published by Microsoft limits the data itself to computational use but places no restriction on Results, and defines AI models trained on the data as Results.
For a utility pilot, state whether training is allowed or only evaluation, what happens to any model or derived output when the pilot ends, how long the sample may be kept and how deletion is confirmed. Counsel should review the wording, because the right answer depends on the records and the buyer's plans.
Feedback rights belong in the same terms. If the buyer agrees to share per-slice results and rejected-record reasons, write down the format and the date, and whether you may use that feedback when preparing records for other buyers. Without that clause, the most valuable output of the pilot for you is optional.
Worked example (illustrative): work orders from a field-service company
All figures here are invented. A field-service company with eight years of work orders agrees a pilot with a buyer building a scheduling assistant. The agreed question is whether the records reduce wrong-technician assignments on jobs that need two trades. The sample is 2,000 randomly drawn closed work orders, of which about 9 percent involve two trades.
The buyer reports a modest gain overall and a much clearer gain on the two-trade jobs, and shares that about 6 percent of records were rejected because the completion notes were empty. The company reads this as concentrated value: the two-trade jobs are what the buyer cannot easily find elsewhere.
The next step is not a price but a scope: a larger license built around multi-trade jobs, a preparation plan to fill or exclude empty completion notes, and a second test at a larger sample size to see whether the gain holds. One condition would have changed that reading: had the buyer trained a model on the sample under unrestricted terms, the second test would have been measuring records the buyer had already used.
Starting with SourceX before any sample leaves the building
SourceX begins with a short company assessment that asks for no data uploads, so the questions in the worksheet can be discussed before a single record is extracted. Records leave the company only after it has approved the scope and terms and signed, and that signed agreement, not a pilot score, is where exclusivity and term are set.
Frequently asked questions
How much is 100 GB of data worth to an AI buyer?
Size alone does not set value. What matters is how many distinct, well-documented cases the data contains, whether it includes outcomes, which uses you can license and what alternatives the buyer has. A pilot on a sample is one way to replace the size question with evidence about those factors.
Can a pilot result be used to set a price?
It can inform one but not settle it. A pilot shows utility for one buyer on one question, while price also depends on rights, exclusivity scope, preparation effort, timing and the buyer's alternatives. Use the result to agree scope first.
What if the buyer will not share the pilot results?
Then the pilot tells you very little about your data. Make shared results, at least by slice with rejected records and reasons, a written condition of any pilot, and decline pilots that offer only a yes or no.
Does a pilot with no gain mean the data has no value?
No. The sample may have lacked the cases the buyer needed, or been too small to detect a change. Ask whether the test could have detected the agreed threshold and which slices, if any, moved.
Should the buyer be allowed to train on the pilot sample?
Only if the terms say what happens to the resulting model afterward. Evaluation-only pilots keep the sample's value intact; training pilots need explicit terms on retention, deletion and derived models.
Sources
- The OECD names content and context as components of data value, with context largely determining how much potential value is realized. Source
- The value of data depends to a large extent on the governance framework that determines how it can be created, shared and used. Source
- Generalization error follows a power law in training set size with empirically measured exponents, after a small-data region dominated by best guessing. Source
- Underpowered comparisons on small samples tend to overstate or reverse true effects. Source
- Datamodels trains many models on subsets of a training set and fits their outputs to attribute predictions to training data. Source
- The C-UDA limits data to computational use, places no restriction on Results and treats AI models trained on the data as Results. Source
See if your company qualifies
A short company assessment. No data uploads are needed.