Leadership and readiness
How to build an AI data purchase internal business case you can test
By SourceX Editorial · Updated
9 sections
Short answer
A testable business case for an AI data purchase states, before any pilot runs, the model failure the data should fix, the measured change that would justify buying, the full acquisition cost and the result that would stop the purchase. Approvers then fund a pilot to test that hypothesis instead of a purchase on faith, and the larger commitment follows only if the agreed criteria are met.
Value drivers
AI data purchase internal
Key takeaways
- Write the purchase as a hypothesis that can fail: a named failure, a measurable change and a deadline.
- Fix success, stop and guardrail criteria before the pilot, and record any later change with its reason.
- Count acquisition cost in internal hours, compute and dependencies, and adjust estimates for optimism before approval.
- A pilot must prove the sample represents the full delivery, the rights are clear and acceptance works at volume.
- An ambiguous result is a decision too: narrow the scope, extend the test once, or stop.
What makes a data purchase business case testable?
A business case is testable when it names a result that would make the team stop. Most dataset proposals argue for a purchase; a testable one proposes a hypothesis, the evidence that would confirm or refute it, and the decision each outcome triggers, all written before anyone sees a pilot score.
Science has a name for this discipline: preregistration. The Center for Open Science describes it as fixing a study plan before data collection, and it is explicit that plans change. The point is that changes are recorded, with the original kept and the reasons stated, rather than quietly absorbed into the conclusion.
Procurement benefits from the same habit for a simple reason. Once a team has spent weeks on a supplier, any pilot result starts to look like a reason to proceed. A memo signed before the pilot removes that pressure, because the threshold was agreed when nobody had a stake in passing it.
This memo does not replace the return-on-investment case that finance will want later. It comes first: the approval it asks for is to run a bounded test, and the result of that test feeds the larger case.
Start from the model failure, not the dataset
The hypothesis should begin with a failure you can observe today, not with a dataset you would like to own. Write down the task, the slice of inputs where the model fails, how it fails, and the current baseline on a frozen evaluation set drawn from that slice.
The NIST AI Risk Management Framework gives a useful frame for the approver's side. Its MAP function asks organizations to examine and document the potential benefits of intended AI functionality and the potential costs, including non-monetary costs from expected errors, and to map risks and benefits for every component of a system, third-party data included.
A good failure statement is narrow enough to test with a few hundred held-out cases. Contract review that misses auto-renewal clauses in amended agreements is testable. Making the legal assistant better is not, because no single pilot result can confirm or refute it.
Include the alternatives you already tried. If prompt changes, retrieval or internal data did not fix the failure, say so with the measured result; if they were never tried, an approver will reasonably ask why buying data comes first.
State the business consequence of the failure in operational terms the approver already tracks, such as misrouted claims per week or hours of rework per case, without converting it into money yet. That conversion belongs in the later return case, once the pilot has measured a real gain.
The hypothesis, cost and success criteria memo
The memo fits on two pages and every field has an owner who signs it. Fill it in before the supplier sample arrives, and keep the signed version unchanged; amendments go in a dated change log underneath.
Three fields do most of the work. The failure statement makes the purchase specific, the success criterion makes it measurable, and the stop criterion makes it falsifiable. If a reviewer can only read three lines, those are the three, and a memo missing any of them is not yet ready to sign.
The illustrative entries show the level of detail that works: a named slice, a number with a unit, a person who owns it. Vague entries such as improved quality or relevant records cannot be checked after the pilot, which defeats the purpose of signing them before it.
| Memo field | What to write | Illustrative entry | Who signs |
|---|---|---|---|
| Failure statement | Task, input slice, failure mode, current baseline on a frozen evaluation set | Claims triage misroutes multi-policy claims; baseline 71% correct on 600 held-out cases | Model owner |
| Hypothesis | Which data, in what quantity, should change which metric, by when | Licensed triage records for multi-policy claims raise cohort accuracy within one training cycle | Research lead |
| Success criterion | Smallest worthwhile gain and the confidence rule for calling it | At least 8 points, with the 95% interval lower bound above 3 points | Research lead and business owner |
| Guardrail | What must not get worse, and by how much | No drop over 1 point on the 2,000-case general suite | Evaluation lead |
| Stop criterion | Result that ends the purchase, decided now | Interval includes zero, or any guardrail breach | Business owner |
| Data requirements | Fields, volume, period, language, format and exclusions | Claim, policy links, routing decision and outcome, two years, English, JSONL | Research lead |
| Acquisition cost | Internal hours, compute, review effort and dependencies, with an optimism adjustment | About 140 internal hours over 6 weeks, plus 2 training runs and 1 re-run | Finance partner |
| Rights and privacy | Who confirms license scope, personal data handling and permitted use | Counsel reviews the evaluation license; privacy reviews the de-identification method | Legal and privacy |
| Evidence requested | Documents the supplier must provide before and during the pilot | Datasheet, sampling method, schema, rights summary, delivery plan | Procurement lead |
| Decision and date | Who decides, on which date, using which result | Steering group decides within 5 working days of the final score | Executive sponsor |
Writing success and stop criteria a pilot can settle
A success criterion has to be both meaningful to the business and detectable with the evaluation set you have. If the smallest gain worth paying for is smaller than the noise in your evaluation, no pilot can settle the question, and the memo should say so before money is spent on samples.
At a glance
Writing success and stop criteria a pilot can settle
Treat the evaluation as an experiment. Evan Miller's work on error bars for language model evaluations gives formulas for analyzing evaluation data, measuring the difference between two models and planning how large an evaluation needs to be. The practical consequence is to compare the baseline and the pilot-trained model on the same held-out items and report the paired difference with an interval.
Guardrails deserve the same care as the success criterion. A purchase that lifts the target slice while degrading tasks the model already handles well can leave the business worse off, so name the general evaluation suite, its size and the largest drop you will tolerate.
Decide too how many training runs the comparison will use. A single run of each model can move by chance; if the budget allows, train each arm more than once with different seeds and report the spread alongside the paired difference.
Write three rules, not one, and decide them in advance:
- Proceed: the interval on the cohort gain clears the minimum worthwhile effect and no guardrail is breached.
- Narrow: the gain is clear on part of the cohort, such as one claim type or one region, but not on the rest; scope the full purchase to that part and restate the hypothesis.
- Stop: the interval includes zero, a guardrail is breached, or the sample turns out not to represent the delivery the supplier can actually make.
Counting acquisition cost in hours and dependencies
The memo should count every internal hour and dependency the purchase creates, because the license is rarely the only cost an approver is taking on. Express cost in units your organization tracks, such as reviewer hours, compute runs and weeks of calendar time, and leave pricing to the quote the finance partner attaches.
Typical lines include legal review of the evaluation and full licenses, privacy review of the preparation method, security review of the delivery channel, acceptance testing, schema mapping and integration, training and evaluation runs including at least one re-run, and ongoing documentation and retention handling.
Estimates made by the people who want the purchase tend to be optimistic. The UK Treasury's Green Book requires appraisers to make explicit optimism bias adjustments at the outset, raising estimated costs and timeframes and lowering estimated benefits. A data purchase memo can borrow the rule: state the adjustment you applied and the evidence behind it, such as how far the last two data projects overran their schedules.
Add continuity as a dependency, not a footnote. If the model will need refreshed data every quarter, the memo should say whether the supplier can repeat the delivery, and what happens to the model if it cannot.
After the pilot, compare the hours actually spent with the memo's estimate, line by line. The gap is the most reliable adjustment you have for the full purchase, and it is better evidence than any general rule, because it comes from your own reviewers, your own systems and this supplier's records.
What the pilot must prove besides a better score
The pilot has to prove more than a score. It must show that the sample was drawn the way the full delivery will be drawn, that the rights cover the intended use, and that the buyer can accept a delivery at the planned volume, because a strong result on an unrepresentative sample predicts nothing.
At a glance
What the pilot must prove besides a better score
Ask for documentation that lets an approver judge representativeness. Datasheets for Datasets proposes that every dataset document its motivation, composition, collection process and recommended uses, and Hugging Face's dataset card format records contents and context with metadata such as license, language and size. Either structure is a reasonable request to a supplier.
Use this checklist as the pilot's exit test:
- Task specification: the pilot task matches the failure statement, including the input slice and the output the model must produce.
- Sampling: the supplier explains how the sample was selected and confirms the full delivery will use the same method and period.
- Source references: each record traces to a system, export date and schema version.
- Rights: counsel has confirmed the evaluation license and the scope proposed for the full license.
- Acceptance: your acceptance checks ran on the sample at the speed the full volume will need.
- Total cost: the hours actually spent during the pilot are compared with the memo's estimate and the full-purchase estimate is updated.
- Continuity: the supplier has described how refreshes or follow-on deliveries would work.
- Approvals: every signer in the memo has seen the result against the criteria they signed.
Worked example (illustrative): a claims-triage team proposes a purchase
This example is illustrative; the figures are counts, hours and percentage points chosen to show the method, not observed results. A claims operations team finds its triage model misroutes claims that touch more than one policy. On a frozen set of 600 held-out multi-policy claims, baseline routing accuracy is 71%.
The memo hypothesizes that licensed triage records for multi-policy claims, with the routing decision and the eventual handling outcome, will raise cohort accuracy. Success is a gain of at least 8 points with the 95% interval's lower bound above 3 points. The guardrail is no drop of more than 1 point on a 2,000-case general suite. Internal effort is estimated at 140 hours over six weeks, raised by a stated adjustment from the 115 hours first proposed.
The pilot-trained model gains 9 points, with an interval from 4 to 14 points, and the general suite is unchanged. That meets the proceed rule. During the evidence review, though, the team learns the supplier's sample came from one region, while the full delivery would span four.
This is the exception that matters. The memo's sampling item fails even though the score passes, so the decision is narrow, not proceed: license the region the sample represents, and run a second short pilot on a sample drawn across all four before extending the purchase.
Reading an ambiguous pilot: narrow, extend or stop
An ambiguous result should trigger the rule you wrote, not a new debate. If the interval straddles your minimum worthwhile effect, you may extend the pilot once, with a larger held-out set sized in advance, but a second ambiguous result is a stop.
Some findings should halt the test regardless of the score and go to a specialist. Personal data appearing where the supplier said it had been removed goes to the privacy reviewer. A rights summary that does not match the records goes to counsel. A large gap between the sample's distribution and production traffic goes back to the research lead to restate the hypothesis.
Keep the original memo and the change log together in the purchase file. When the next team proposes a data purchase, the record of which hypotheses held and which did not is the best evidence your organization has about what data actually fixes.
Turning an approved memo into a sourcing request
A signed memo is also a good brief for a supplier, because it already states the failure, the fields, the volume, the period and the evidence you will ask for. Send the requirements and evidence sections, not the internal thresholds.
Through SourceX, the memo's data requirements field maps almost one to one onto the request form, which asks for scope, volume, language, format, timing and licensing needs; SourceX then looks for matching records among U.S. businesses. The rights line in the memo also gets a fixed checkpoint, because the supplying company approves scope and terms and signs before anything is delivered.
How do you word a sample request so the records can fail the claim?
Ask for a sample the supplier cannot curate: name the population and the draw rule yourself, take every record the draw returns, and send the record-level acceptance test with the request. A sample chosen to impress shows the best records a supplier holds; a sample drawn by rule shows what the delivery will contain, and only that one can refute the business case.
The pilot exit test above asks the supplier to explain how its sample was selected. The request is where that answer gets decided, because an open request for a representative sample invites the cleanest export a supplier has. Each line of the table below turns an illustrative request into one a pilot can settle.
Anthropic's engineering team advises testing the cases where a behavior should occur and the cases where it should not, warning that one-sided evaluations create one-sided optimization (January 9, 2026). A purchase sample works the same way: a draw of target cases alone can show a model learning a behavior, but not learning when to hold back.
Keep the split this page already uses. The supplier receives the population, the draw rule and the record-level acceptance test; the model-level success and stop thresholds stay in the signed memo. The supplier can then check its own delivery against your test without shaping the sample toward a number it can see.
| Request line | Illustrative version | Testable version | What it lets you conclude |
|---|---|---|---|
| Who picks the records | The supplier sends examples it considers representative | You define the population and a random draw with a recorded seed; the supplier runs it and returns the query | Whether results hold for the population rather than a showcase |
| Records with gaps | Incomplete records are swapped for complete ones | Every drawn record is delivered, and any withheld record is listed with its reason | The defect rate you measure is the rate you would buy |
| Hard and negative cases | Well-documented successes only | Minimum counts for the failure slice, ordinary cases and cases where the target behavior should not occur | Whether the data teaches the behavior without over-triggering it |
| Acceptance test | Quality judged by reading the records | Written record-level checks, such as required fields present and outcome recorded, sent with the request | A defect count both sides agreed to before delivery |
| Success rule | Chosen after the scores come back | The memo's thresholds, signed before the draw and kept internal | A result that can say stop as easily as proceed |
What should arrive with the sample to prove how it was drawn?
A short sampling manifest should arrive with the sample, detailed enough that someone outside the supplier could rerun the draw and get the same records. Without it, a passing pilot cannot be told apart from a well-chosen one.
- Frame: the system, table or export the draw ran against, and its total record count on the export date.
- Draw rule: the filter, the random method and the seed.
- Strata: each stratum's definition, its count in the frame and its count in the sample.
- Exclusions before the draw: records removed as outside the license scope or failing de-identification, with counts and reasons.
- Withheld after the draw: each drawn record that was not delivered, by identifier and reason.
Could an evaluation or access fix close the gap before any records are bought?
Often it can, so replay the failing cases with the information the model needed placed directly in its context, or with the broken tool, retrieval step or grader fixed, before scoring any purchased data. If that closes most of the gap, the claim under test is about access or measurement rather than training records, and the request should change before a sample is ordered.
Evaluation fixes alone can move scores a long way. Anthropic reports that its Opus 4.5 model first scored 42 percent on CORE-Bench, and 95 percent after its researchers fixed grading bugs, including rigid answer matching and ambiguous task specifications, and switched to a less constrained scaffold. A pilot that skips this control can credit purchased records with a gain the evaluation fix would have produced on its own.
What keeps a dataset pilot from expanding past its stop rule?
A written definition of done, a ceiling on extras and a rule for classifying every proposed change, all fixed before the draw. The pilot ends when the planned arms are scored against the signed criteria, and a change proposed after scores are visible is a new hypothesis that needs its own memo.
This page already allows one extension of an ambiguous result, with a larger held-out set sized in advance. Pilots also grow in quieter ways: another slice, a newer base model, an extra metric, more supplier records. Each looks small, and each gives the result another chance to cross the line. Johari and colleagues describe the general problem in online experiments: p-values and confidence intervals become unreliable when the people running a test choose its size by continuously watching the results.
Write three limits into the memo: the completion definition (arms, runs per arm, held-out set and grader version), a ceiling on added spend and calendar time for the one permitted extension, and the person who rules on change requests. Log every request against the table below; anything that lands in the third column ends this pilot or opens a separate one.
Before proceeding on a pass, rerun the comparison against the strongest inexpensive alternative you found, such as the context or retrieval fix from the control, across several runs; Anthropic's post notes that agent behavior varies between runs and uses multiple trials for that reason. A gain that shrinks against the better baseline was not the data's to claim. How much more of the same data to buy after a clear pass is a different question, covered by SourceX's separate stop-rule experiment for marginal data purchases.
| Proposed change | Allowed inside this pilot when | Becomes a new pilot when | Log entry |
|---|---|---|---|
| Enlarge the held-out set | The new size was set in the memo for the one permitted extension | The size is chosen after seeing the interval | Date, new size, approver, whether scores had been seen |
| Add an input slice | Never; the failure statement fixes the slice | Always; write a new failure statement first | Slice, reason, link to the new memo |
| Change the base model or checkpoint | A vendor retires the version and both arms move together | Only the treated arm moves, or the change follows a weak result | Old and new versions, date, baseline rerun |
| Add or swap a metric | It is reported as secondary and the signed metric still decides | The new metric would decide proceed or stop | Metric, definition, decision status |
| Request more supplier records | The same draw rule and seed extend the same sample | Records are hand-picked to cover the cases that failed | Count, draw rule, seed |
| Revise the grader or rubric | A grading bug is found and both arms are regraded | Criteria are loosened after failures are seen | Bug, fix, regrade date, effect on both arms |
| Extend the calendar | Within the ceiling set for the one extension | Beyond the ceiling, or a second time | New end date, spend to date, approver |
Sources for this section
- Anthropic's engineering post (January 9, 2026) recommends testing cases where a behavior should and should not occur, reports that Opus 4.5 rose from 42 to 95 percent on CORE-Bench after grading bugs were fixed and a less constrained scaffold was used, and runs multiple trials because agent behavior varies between runs. Source
- Johari, Pekelis and Walsh show that frequentist p-values and confidence intervals become unreliable when experimenters choose sample sizes by continuously monitoring their tests. Source
Frequently asked questions
What should the hypothesis, cost and success criteria memo include?
It should include a failure statement with a baseline, the hypothesis, a success criterion, a guardrail, a stop criterion, the data requirements, the acquisition cost in hours and dependencies, the rights and privacy owners, the evidence requested and the decision date. Each field should have a named signer.
How is a testable business case different from an ROI business case?
A testable case is written before the pilot and asks for permission to run a bounded test with agreed thresholds. An ROI case is written after the pilot and converts the measured gain into business value. The first keeps the second honest.
Who should set the stop criterion?
The business owner who would fund the full purchase should set it, with the research lead confirming it is measurable. Setting it before the pilot matters more than who drafts it, because a threshold chosen after seeing results is no longer a test.
What if the supplier cannot provide a representative sample?
Then the pilot cannot predict the full delivery, however good its score. Ask for a sample drawn by the same method and period as the full delivery, or narrow the purchase to the slice the sample does represent.
Can the memo change after the pilot starts?
It can, but the original stays on file and each change is dated with its reason, as in a preregistration. Changes made after scores are visible should be treated with suspicion and approved by the original signers.
Sources
- Preregistration fixes a study plan in advance; changes are expected and should be recorded with the original kept and the reasons stated. Source
- The NIST AI RMF asks organizations to examine and document potential benefits and costs of AI systems and to map risks and benefits for third-party data. Source
- Language model evaluations can be analyzed as experiments, with formulas for comparing two models and planning evaluation size. Source
- The Green Book requires explicit optimism bias adjustments that raise estimated costs and timeframes and lower estimated benefits. Source
- Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process and recommended uses. Source
- Dataset cards describe a dataset's contents and context, with metadata such as license, language and size. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.