Skip to content

Deal economics

Dataset valuation utility pilot: what a small buyer test can and cannot tell you

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A small buyer pilot can establish whether your records are usable as delivered, whether they cover the cases the buyer cares about and whether a sample moves the buyer's own metric against a baseline. It cannot set a price on its own, predict the gain at full volume with precision, or speak for other buyers. Agree the question, sample, metric, permitted uses and what results come back before any records leave.

Value drivers

Dataset valuation utility

This page covers dataset valuation utility pilot: what a small buyer test can and cannot tell you.Illustrative

Key takeaways

  • A pilot measures utility for one buyer on one question; value also depends on rights, exclusivity scope, preparation effort, timing and the buyer's alternatives.
  • Agree before the pilot what the buyer will measure, what it compares against and which results it will share with you.
  • No lift on a small sample is weak evidence that the data lacks value; lift on a small sample is weak evidence of how much value the full set has.
  • The pilot terms decide whether anything the buyer trains on your sample stays usable after the pilot, which is itself a value question.
  • Unique cases, outcome fields and coverage of rare situations usually explain a pilot result better than raw volume does.

What can a small buyer pilot establish about your dataset's value?

A pilot establishes three things well: whether the records survive the buyer's intake checks, whether they cover the situations the buyer is trying to teach or test, and whether a sample changes the buyer's own metric compared with a baseline. Those are evidence of utility and fit for one buyer. They are inputs to a value conversation, not a valuation.

What a pilot does poorly is extrapolate. A gain on a few thousand records does not tell you the gain on a few hundred thousand, and a buyer's result on its model says little about what a different buyer would see. Treat the pilot as the cheapest way to replace guesses about your data with observations, and be explicit about which guesses remain.

For the business that owns the records, the practical goal is narrower still: leave the pilot knowing which of your records the buyer found useful, which it could not use and why, so the next conversation is about specific records rather than about the dataset in general.

Why can the same records be worth more to one buyer than another?

Data value depends on context as much as content. The OECD's 2021 report on measuring the economic value of data names both as components of value and notes that the context in which data are collected, stored and used largely determines how much of their potential value is realized. A claims history is worth more to a team building a claims-triage model than to one building a general chat assistant, even though the records are identical.

Governance matters as well. A 2022 OECD paper argues that the value of data depends to a large extent on the governance framework that determines how it can be created, shared and used. For a pilot, that means the permitted uses you can offer, and those you cannot, shape the result before any model is trained.

This is why a pilot with one buyer should not be read as a verdict from the market. A second buyer with a different model, a different gap and different alternatives can reach the opposite conclusion on the same sample, and both can be right about their own situation.

The utility pilot scope and result interpretation worksheet

Agree each line below with the buyer before the sample leaves your systems. The third column explains why the line matters when you later talk about value; entries in brackets are illustrative.

Keep the signed worksheet with the pilot terms. When the result arrives, the first check is whether the buyer tested what was agreed: the same sample, the same metric and the same baseline. A result produced on a different question can still be informative, but it should not be read as the answer to the one you agreed.

The utility pilot scope and result interpretation worksheet
InputWhat to agree before the pilotWhy it matters for valueWho supplies it
Buyer questionThe one decision the pilot informs, in a sentence (does the sample cut routing errors on multi-site jobs)A pilot without a question produces a result nobody can act onBuyer, confirmed by you
Sample definitionHow records are drawn, from which system and date range (random 2,000 closed work orders from 2023 to 2025)A hand-picked sample makes any result unrepeatable at volumeYou, with the buyer's agreement
Unique case countHow many distinct situations the sample contains, not just recordsRepeated near-identical records add little beyond the firstYou
Coverage of rare casesWhich uncommon situations the sample includes and how many of eachRare, well-documented cases are often what a buyer cannot find elsewhereYou and the buyer
Baseline and comparisonWhat the buyer compares against (current model, or a random set of similar size)A result without a baseline cannot be called a gainBuyer
Metric and thresholdThe buyer's metric and the change it would treat as meaningfulTells you in advance what counts as successBuyer
Permitted use of the sampleEvaluation only or training allowed, retention period and deletionDetermines whether the pilot itself transfers valueBoth, through the pilot terms
Results shared backWhich results you receive: overall metric, per-slice results, rejected records and reasonsWithout feedback you learn nothing about your own dataBuyer, written into the terms
Time boxStart date, end date and the decision date (eight weeks from delivery)An open-ended pilot delays every other conversationBoth
Decision after the pilotWhat each outcome leads to: proceed, narrow, expand the sample or stopTurns the result into a next step instead of a debateBoth

How should you read each possible result?

Read the result against the question you agreed, not against a hope. The table pairs the common patterns with what they suggest and, just as important, what they do not show.

Mixed results are the most common and the most useful. A gain concentrated in one slice points to a narrower, better-defined license; a gain that appears only after the buyer cleaned your records points to preparation work you could do once and offer to every later buyer.

How should you read each possible result?
Result patternWhat it likely meansWhat it does not meanSensible next step
Clear gain over the baseline on the agreed metricThe records carry signal the buyer's current data lacksThat the full dataset will produce a proportionally larger gainDiscuss a larger, defined scope and how gains at volume will be checked
No measurable gainEither the sample lacked the needed cases or the test was too small to detect a changeThat the dataset has no value to anyoneAsk for per-slice results and whether the sample size could detect the agreed threshold
Gain on one slice onlyValue is concentrated in specific case typesThat the rest of the dataset is worthlessOffer a scope built around the slice that worked
Records failed intake checksFields, format or de-identification did not meet the buyer's needsThat the underlying cases lack valueFix preparation and rerun a small sample
Buyer shares no resultsYou cannot tell which of the above happenedThat the pilot went well or badlyMake shared results a condition of any future pilot

Why does a small sample understate or overstate the full dataset?

Learning curves are not straight lines. Hestness and colleagues found that generalization error falls as a power law of training set size across several domains, but with exponents that had to be measured for each problem, and that very small training sets sit in a region dominated by best guessing before that curve takes hold. A pilot sample may sit in that early region and show little, or sit on a steep part of the curve and show more than later records will add.

Small comparisons also mislead in the other direction. Card and colleagues showed that underpowered experiments are common in language research and that significant results from small samples tend to overstate the true effect, or even get its direction wrong. So a striking gain on a small pilot deserves the same caution as a disappointing one.

The useful response is to ask the buyer to test at two sample sizes if the pilot budget allows, and to report a range rather than a single figure. Two points on a curve say more about value at volume than one point measured very precisely.

Which assumption moves your net proceeds and the buyer's fit the most?

For the business, the assumption that most often changes the outcome is preparation effort at full volume. A pilot sample is usually extracted and de-identified with care by hand; doing the same for the whole dataset can take far longer and require system work, and that effort comes out of what the licensing is worth to you. Ask your own team to estimate the full-volume preparation work before the pilot ends.

At a glance

Which assumption moves your net proceeds and the buyer's fit the most

For the buyer, the deciding assumptions are usually its alternatives and its timing. If the buyer can collect similar records itself, or already holds a comparable set, a good pilot result is worth less to it than the same result for a buyer with no other source. If the buyer's model release is close, records that arrive late may be worth little regardless of quality.

  • Preparation effort: hours and system changes needed to deliver the full set in the piloted condition.
  • Permitted uses: whether the full license would cover training, evaluation or both, as tested.
  • Exclusivity scope: whether the buyer expects sole use, and for what field and period.
  • Buyer alternatives: other sources the buyer could use for the same cases.
  • Timing: when the buyer needs delivery for the result to matter.
  • Coverage at volume: whether the rare cases in the sample stay as frequent in the full set.

Can the pilot show which of your records carry the value?

Sometimes, if the buyer reports results by slice. Research methods exist to attribute a model's behavior to individual training records; the open-source Datamodels toolkit trains many models on different subsets of a training set and fits a model of their outputs, which lets a team see which subsets drive which predictions. Few buyers will run anything that elaborate on a pilot, but the idea behind it is simple to request.

Ask for results broken down by the case types you defined in the worksheet, and for the list of records the buyer rejected with a reason for each. That feedback tells you which parts of your history are worth preparing for a larger license and which are not, which is often worth more to you than the headline metric.

What should the pilot terms say about the sample and anything built from it?

The pilot terms decide whether value leaves with the sample. If the buyer may train on the sample and keep the resulting model, part of what you would license later has already been transferred. Public data agreements show how easily this happens: the Computational Use of Data Agreement published by Microsoft limits the data itself to computational use but places no restriction on Results, and defines AI models trained on the data as Results.

For a utility pilot, state whether training is allowed or only evaluation, what happens to any model or derived output when the pilot ends, how long the sample may be kept and how deletion is confirmed. Counsel should review the wording, because the right answer depends on the records and the buyer's plans.

Feedback rights belong in the same terms. If the buyer agrees to share per-slice results and rejected-record reasons, write down the format and the date, and whether you may use that feedback when preparing records for other buyers. Without that clause, the most valuable output of the pilot for you is optional.

Worked example (illustrative): work orders from a field-service company

All figures here are invented. A field-service company with eight years of work orders agrees a pilot with a buyer building a scheduling assistant. The agreed question is whether the records reduce wrong-technician assignments on jobs that need two trades. The sample is 2,000 randomly drawn closed work orders, of which about 9 percent involve two trades.

The buyer reports a modest gain overall and a much clearer gain on the two-trade jobs, and shares that about 6 percent of records were rejected because the completion notes were empty. The company reads this as concentrated value: the two-trade jobs are what the buyer cannot easily find elsewhere.

The next step is not a price but a scope: a larger license built around multi-trade jobs, a preparation plan to fill or exclude empty completion notes, and a second test at a larger sample size to see whether the gain holds. One condition would have changed that reading: had the buyer trained a model on the sample under unrestricted terms, the second test would have been measuring records the buyer had already used.

Starting with SourceX before any sample leaves the building

SourceX begins with a short company assessment that asks for no data uploads, so the questions in the worksheet can be discussed before a single record is extracted. Records leave the company only after it has approved the scope and terms and signed, and that signed agreement, not a pilot score, is where exclusivity and term are set.

Frequently asked questions

How much is 100 GB of data worth to an AI buyer?

Size alone does not set value. What matters is how many distinct, well-documented cases the data contains, whether it includes outcomes, which uses you can license and what alternatives the buyer has. A pilot on a sample is one way to replace the size question with evidence about those factors.

Can a pilot result be used to set a price?

It can inform one but not settle it. A pilot shows utility for one buyer on one question, while price also depends on rights, exclusivity scope, preparation effort, timing and the buyer's alternatives. Use the result to agree scope first.

What if the buyer will not share the pilot results?

Then the pilot tells you very little about your data. Make shared results, at least by slice with rejected records and reasons, a written condition of any pilot, and decline pilots that offer only a yes or no.

Does a pilot with no gain mean the data has no value?

No. The sample may have lacked the cases the buyer needed, or been too small to detect a change. Ask whether the test could have detected the agreed threshold and which slices, if any, moved.

Should the buyer be allowed to train on the pilot sample?

Only if the terms say what happens to the resulting model afterward. Evaluation-only pilots keep the sample's value intact; training pilots need explicit terms on retention, deletion and derived models.

Sources

  • The OECD names content and context as components of data value, with context largely determining how much potential value is realized. Source
  • The value of data depends to a large extent on the governance framework that determines how it can be created, shared and used. Source
  • Generalization error follows a power law in training set size with empirically measured exponents, after a small-data region dominated by best guessing. Source
  • Underpowered comparisons on small samples tend to overstate or reverse true effects. Source
  • Datamodels trains many models on subsets of a training set and fits their outputs to attribute predictions to training data. Source
  • The C-UDA limits data to computational use, places no restriction on Results and treats AI models trained on the data as Results. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies