AI uses for records
Dataset pilot procurement report: 10 locked fields, 4 decisions
By SourceX Editorial · Updated
9 sections
Short answer
A dataset pilot procurement report should end in one of four decisions: buy as scoped, buy a narrower slice, extend once with a named question, or decline, each traced to a rule written before results existed. Lock ten fields in advance, from the decision question and sample rule to the success, stop and inconclusive rules. Then translate the result into scope, limits and measured integration work.
Evaluation split
Dataset pilot procurement
Key takeaways
- Write and date ten protocol fields before any candidate record reaches the pilot; anything added later is exploratory.
- Report scope, lift, harm, limits, integration work and the decision as separate blocks, each tied to a pre-set rule.
- A gain counts only if it survives the strongest current baseline, a clean holdout and repeated runs.
- An inconclusive result is a legitimate outcome with its own pre-set response, not an invitation to keep testing.
- Integration work measured on the pilot sample belongs in the decision alongside the lift.
What should a dataset pilot procurement report decide?
It should decide exactly one of four things: buy the dataset as scoped, buy a narrower slice, extend the pilot once to answer one named question, or decline. Each decision must trace to a rule written before the first result, and the report must state what the purchase covers, what the evidence does not show and how much integration work the pilot measured.
Most dataset pilots end with a chart and a recommendation that reads like an opinion. Procurement cannot sign an opinion, and finance cannot budget one. The fix is to treat the pilot as a small confirmatory experiment with a protocol locked in advance, and the report as the translation from that experiment's result into the scope of a purchase.
This page is for the evaluation lead who runs the pilot and has to hand procurement a document it can act on. It covers the ten fields to lock, a result-to-purchase translation table, the four decisions and their rules, the checks that make a gain believable, integration work and an illustrative report. For comparing several AI agent vendors on one workload, the agent evaluation procurement report linked below is the closer guide.
Lock these ten fields before the first candidate record arrives
Lock and date the protocol before any candidate data reaches the pilot, so the analysis tests a prediction instead of finding one. The Center for Open Science describes preregistration as a time-stamped plan recorded before data are collected or analyzed, treats the analyses in that plan as confirmatory and anything added afterwards as exploratory, and suggests holding part of the data back so exploration and confirmation happen on different portions. Exploratory findings can still appear in the report, labeled as such, but they cannot drive the purchase.
At a glance
Lock these ten fields before the first candidate record arrives
Keep the protocol where changes are visible, such as a versioned document or a ticket, and record every deviation with its date and reason. These are the ten fields:
- Decision question: the purchase in one sentence, naming the task, the candidate dataset and the alternative of not buying.
- Sample rule: how the pilot sample was drawn from the candidate dataset, by whom and across which slices, so the sample cannot be a showcase.
- Model and version: the base model, checkpoint and training or prompting method, held fixed across every arm.
- Budget: training steps, tuning trials and reviewer hours per arm, equal for the candidate arm and the baselines.
- Baselines: the current production setup and the strongest alternative you could deploy without buying, tuned with the same budget.
- Frozen holdout: the evaluation cases, their source and date range, sealed from the supplier and from training, read once for the decision.
- Metrics: one primary metric, one severity metric that must not get worse, and the grader for each.
- Success rule: the minimum gain on the primary metric, with the interval it must clear, that triggers a purchase.
- Stop rule: results or events that end the pilot early, such as a rise in severe errors or a rights problem found in the sample.
- Inconclusive rule: what happens when the interval straddles the threshold, including whether one extension is allowed and its fixed size.
The result-to-purchase translation table
Every block of the report answers one procurement question and sets one part of the purchase. The table pairs what is locked in advance with what is reported afterwards, so a reviewer can check that the purchase follows from the protocol rather than from the mood in the room.
Fill the blocks in this order and keep each to a short paragraph plus its numbers. A block that cannot be filled is itself a finding: it moves to the limits block and usually rules out a full purchase. The figure shows the same blocks as a one-page report, filled with the illustrative pilot described further down.
| Report block | Locked before the pilot | Reported after | What it sets in the purchase |
|---|---|---|---|
| Scope | Slices the sample covers and how they were drawn | Gain per slice, each with an interval | Which slices, periods or record types are bought |
| Lift | Primary metric, baselines and success threshold | Paired gain over the strongest baseline across repeated runs | Whether the purchase case is met at all |
| Harm | Severity metric that must not get worse | Change in severe errors versus the baseline | Exclusions, or a stop |
| Limits | Claims the design cannot support | Untested suppliers, periods, model versions and tasks | Conditions and follow-up tests attached to the purchase |
| Integration | What will be measured on the sample | Mapping hours, rejected share and review findings | Preparation effort and delivery format requirements |
| Decision | The four outcomes and the rule for each | Which rule fired, with the evidence | Buy as scoped, buy narrower, extend once or decline |
Hold model, budget and evaluation fixed so the dataset is the only change
The candidate dataset must be the only difference between arms, or the report cannot attribute the gain to it. Use the same base model, training method, tuning budget, evaluation cases and grader in every arm, and write each into the protocol.
Budget is the condition most often left loose. In Show Your Work, presented at EMNLP in 2019, Dodge and colleagues showed that a model's expected performance depends on how much tuning computation it gets, and found published comparisons whose conclusion would have changed with more or less computation. If the candidate arm receives extra tuning, part of its gain is the tuning, not the data.
Baselines need the same care. Compare against the strongest setup you could deploy without the purchase, not a convenient weak one, and run a regression check beside the capability check. Anthropic's January 2026 guide to agent evaluations separates capability evals, which ask what a system does well and should start at a low pass rate, from regression evals, which ask whether it still handles what it used to and should pass nearly every time. A dataset that lifts the new task while breaking an old one fails the harm block, whatever its headline gain.
Four purchase decisions and the rule that triggers each
Write the four outcomes into the protocol with their triggers, so the report only has to say which one fired. An inconclusive result is a legitimate outcome, not a failed pilot; treating it as a reason to keep adding cases until something clears the bar is the most common way a pilot talks itself into a purchase. The thresholds and caps are yours to set, and the figure shows one filled report.
At a glance
Four purchase decisions and the rule that triggers each
Check the rules in this order and stop at the first that fires:
- Decline when the paired gain over the strongest baseline falls short of the success threshold with an interval that excludes it, when severe errors rise, or when measured integration work exceeds the cap set in the protocol.
- Buy narrower when the gain clears the threshold only in named slices that were defined before the pilot, and limit the purchase to those slices.
- Extend once when the interval straddles the threshold and the protocol allowed an extension: add the pre-set number of cases, rerun once and apply the same rules with no further extension.
- Buy as scoped when the gain clears the threshold overall, holds across repeated runs, shows no rise in severe errors and fits the integration cap.
Would the gain survive a stronger baseline, a clean holdout and a rerun?
Assume it might not, and build the checks that would show it into the protocol. Compute the comparison as paired differences on the same holdout cases rather than as two separate scores. Evan Miller's 2024 paper on error bars for evaluations treats evaluation questions as a sample from a larger population and gives formulas for comparing two models on shared questions and for planning how many questions a comparison needs.
Four ordinary causes can erase a positive pilot result, and each has a check that belongs in the protocol:
- Leakage: holdout cases, or near copies, reached training through the supplier's sample or a shared source. Kapoor and Narayanan's study of machine-learning-based science sets out eight types of leakage and shows, in a civil war prediction case, that claimed advantages of complex models over logistic regression disappeared once leakage was corrected. Check overlap between candidate data and holdout by entity and by date before reading results.
- Run-to-run variance: Bouthillier and colleagues (2021) found that data sampling, weight initialization and hyperparameter choice change benchmark results markedly, and recommend randomizing as many of those sources as possible. Train each arm more than once and report the spread, not one run.
- Inconsistent agents: for agentic tasks, report pass^k, the chance that all k trials succeed, beside the average. Anthropic's guide gives the arithmetic: a 75 percent success rate per trial becomes about 42 percent across three trials.
- Flawed graders: the authors of the Agentic Benchmark Checklist (2025) found that flaws in task setup or reward design can misstate agent performance by up to 100 percent in relative terms, and cite a benchmark that counted empty responses as successes. Score an empty output and a known-good answer before trusting the grader.
Integration work belongs in the report, measured on the pilot sample
Measure integration work on the pilot sample instead of estimating it later, because it often decides whether a real gain is worth buying. Count the hours to map the supplier's schema to your training format, the share of records rejected for missing fields or failed checks, the findings from privacy and rights review of the sample, and the work needed to repeat all of it for each refresh delivery.
Report each as a range with the sample size behind it, and say how it scales: schema mapping is mostly a fixed cost, while rejection and review effort grow with volume. A pilot that shows a strong lift and a high rejection rate is describing two purchases, the data and the cleanup, and the report should size both in hours before procurement compares offers. The acquisition cost worksheet linked below breaks these lines down further.
Delivery format belongs in this block as well. If the pilot needed a custom export, the purchase should require that format, a data dictionary and the same field-level checks on every delivery, so the full dataset arrives in the shape the pilot tested.
Illustrative report: an invoice-dispute routing pilot read to a decision
Every figure in this example is invented to show the method; it describes no real buyer, supplier or dataset. An evaluation lead tests whether a candidate set of 12,000 licensed invoice-dispute cases from three industries improves a model that routes disputes to the right resolution team.
The protocol, dated before any data arrived, fixed one base model and training recipe, equal tuning trials per arm, two baselines (the production model and the strongest prompted general model), a frozen holdout of 800 cases from the buyer's most recent quarter, routing accuracy as the primary metric with a 5-point success threshold, misroutes of disputes under legal hold as the severity metric, three named industry slices, and one allowed extension of 400 cases.
The results, from three training runs per arm, were mixed. Overall routing accuracy rose 4 points over the strongest baseline, with an interval that straddled the threshold. By slice, the two manufacturing industries gained 7 and 8 points with intervals clear of the threshold, while the retail slice showed no gain. Severe misroutes did not rise. Schema mapping took a measured range of hours, and 11 percent of sample records were rejected for missing resolution codes.
The decision was to buy narrower. The decline rule did not fire, and the slice rule fired before the extension rule, because the manufacturing slices meet the success threshold on their own and were named in the protocol. The report recommends buying those slices, lists retail, other time periods and the next model version as untested, and makes complete resolution codes a delivery requirement. The extension was never used.
Turning a buy-narrower result into the next data request
A buy-narrower decision is also a sourcing brief: the slices that gained, the slices that did not, the fields that caused rejections and the acceptance test the pilot used. Put all four into the next request, so the next supplier sources against evidence rather than a general description.
SourceX does not hold datasets in stock; it sources them for each AI buyer on request from businesses in the United States, and its request form asks for scope, volume, language, format, timing and licensing needs. A buyer can describe the pilot's acceptance test and intended use there. Whether suitable records exist, and whether any dataset passes that test, is established deal by deal, not assumed in advance.
Frequently asked questions
What is a dataset pilot procurement report?
It is the document that turns a dataset pilot into a purchase decision. It reports the gain over a fixed baseline on a frozen holdout, the harm check, the limits of the evidence and the measured integration work, then states which of four pre-set decisions applies. Procurement and finance can act on it without rerunning the analysis.
How is a dataset pilot report different from an AI vendor evaluation report?
A vendor evaluation compares several tools on the buyer's workload. A dataset pilot report asks whether adding one candidate dataset to a fixed model improves results enough to buy it, so the dataset must be the only difference between arms. The decision concerns the scope and volume of data, not which vendor wins.
How should a buyer handle a pilot result that is neither a pass nor a fail?
Follow the inconclusive rule written before the pilot. That usually means one extension with a fixed number of added cases and the same analysis, or a decline if no extension was allowed. Adding cases again and again until a result clears the bar turns a confirmatory test into a search.
How many training runs does a dataset pilot need?
More than one per arm. Data sampling, initialization and tuning choices change results, so train each arm several times and report the spread alongside the mean. For agent tasks, also report the share of cases solved on every trial, not only the average success rate.
Should the data supplier see the evaluation set?
No. Keep the holdout sealed from the supplier and from training, and check the candidate data for overlap with it by entity and date before reading results. A supplier that has seen the test can, deliberately or not, deliver data shaped to pass it.
Sources
- Anthropic's January 2026 evaluation guide distinguishes capability evals (low starting pass rate) from regression evals (nearly 100 percent pass rate) and defines pass^k, with 75 percent per trial becoming about 42 percent over three trials. Source
- The Center for Open Science describes preregistration as a time-stamped plan recorded before data collection or analysis, separating confirmatory from exploratory analyses, and suggests splitting data for exploration and confirmation. Source
- Dodge et al. (EMNLP 2019) report expected validation performance as a function of computation budget and find published comparisons whose conclusion would change with more or less computation. Source
- Bouthillier et al. (2021) find variance from data sampling, parameter initialization and hyperparameter choice markedly affects benchmark results, and recommend randomizing as many sources of variation as possible. Source
- Kapoor and Narayanan present a taxonomy of eight leakage types and show that claimed advantages of complex models over logistic regression in civil war prediction fail to reproduce once leakage is fixed. Source
- Miller (2024) treats evaluation questions as drawn from an unseen super-population and gives formulas for analyzing evals, measuring differences between two models and planning evaluation experiments. Source
- The Agentic Benchmark Checklist paper (2025) finds task setup or reward design flaws can under- or overestimate agent performance by up to 100 percent in relative terms; TAU-bench counted empty responses as successful. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.