AI data market
AI data vendor pilot milestones to agree before a full purchase
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Before buying a full dataset, an AI data buyer should agree pilot milestones, each tied to evidence: a signed task specification, rights and source documents, a sample drawn by an agreed method, a pass through the buyer's own pipeline, a score on evaluation data the supplier never saw, and a full-volume continuity check. Commit to the full purchase only when every milestone's evidence is signed off.
Value drivers
AI data vendor pilot
Key takeaways
- Tie every pilot milestone to named evidence, a pass condition and the decision it unlocks.
- Settle rights and source evidence before any records move; quality cannot fix missing authority.
- Declare the population first, then let the buyer control the random draw.
- Score the pilot on evaluation data the supplier never sees, and document the test set and metric.
- Check that pilot preparation scales to the full volume before signing.
- Narrow the purchase when a failure is confined to a clean segment; stop when it sits in the supplier's process or authority.
What should an AI data buyer agree in a pilot before a full purchase?
Agree a short schedule of pilot milestones, each tied to a named piece of evidence, a pass condition and the decision it unlocks. The full purchase is committed only after the last milestone closes. A pilot that ends with someone saying the sample looked fine has tested almost nothing, because the questions that sink a dataset purchase are rarely visible in a quick look at a few hundred rows.
The questions that matter are narrower and more checkable. Does the supplier have the authority to license these records for the use you intend? Was the sample drawn the way the full delivery will be drawn? Does the data survive your own pipeline? Does it move a metric on evaluation data the supplier never saw? Can the supplier produce the same thing again at the full volume and on the agreed cadence? Each of those becomes one milestone.
This page covers the evidence side of a pilot: what each milestone must produce and who signs it. The commercial side, such as pilot pricing, credits and option windows, is a separate negotiation, and so is the payment schedule for the full delivery once you buy.
Which decision does each milestone protect?
Each milestone exists to stop one specific bad commitment, so write the bad commitment next to it. If you cannot name what a milestone protects, drop it; if you can name a risk that no milestone covers, add one. The order matters because later milestones are wasted work when an earlier one fails.
- Task specification signed: protects against buying data for a task nobody wrote down, which makes every later test unscorable.
- Rights and source evidence received: protects against moving records the supplier cannot license for model training, the one failure that no amount of quality fixes.
- Sampling frame and draw agreed: protects against a showcase sample that is cleaner than the full population.
- Pipeline ingestion passed: protects against discovering format, schema or de-identification problems after the full volume lands.
- Held-back evaluation run: protects against paying for data that does not change the model on work you care about.
- Full-volume and continuity check: protects against a supplier who can hand-prepare a pilot slice but cannot produce the same thing at scale or on schedule.
- Go, narrow or stop decision recorded: protects against a pilot that drifts into an informal purchase with no written acceptance.
The milestone and evidence schedule
Use the schedule below as the pilot annex. Every row names evidence from both sides, because a pilot where only the supplier produces documents tends to measure the supplier's preparation rather than the data. Put a target date and a named owner next to each row in your own copy.
| Milestone | Supplier provides | Buyer produces | Pass condition | Unlocks |
|---|---|---|---|---|
| M0 Task specification | Comments on feasibility; the systems and record types that could serve the task | Written task statement, target record definition, required fields, exclusions, evaluation metric | Both sides sign one version; open questions listed with owners | Rights and sampling work starts |
| M1 Rights and source evidence | Source register by system; authority to license; restrictions from customer contracts; de-identification approach | Rights questions from legal; intended uses written as permitted activities | Every source system mapped to an authority document; no unresolved restriction on the intended use | Records may move under evaluation-only terms |
| M2 Sampling frame | Population counts by system, period, status and record type; the draw method | Random seed or record ID list; strata that matter for the task | Sample drawn by the agreed method from the declared population, with counts that reconcile | Supplier prepares the sample |
| M3 Sample delivery | Sample prepared exactly as the full delivery would be; data dictionary; documentation of collection and known limits | Intake log: counts, checksums, schema and de-identification spot checks | Counts match the draw; schema matches the dictionary; no direct identifiers found | Pipeline test |
| M4 Pipeline ingestion | Fixes for format or schema defects within the agreed turnaround | Run through the real training or retrieval pipeline; defect list by category | Defect rate inside the written threshold; no defect class that would recur at scale | Evaluation run |
| M5 Held-back evaluation | Nothing; the supplier does not see the evaluation set | Baseline and pilot results on buyer-held evaluation data, with the test set and metric documented | The agreed metric moves by the agreed margin, or the result is explained | Commercial terms for the full purchase |
| M6 Full-volume and continuity | Full population counts at a cutoff; production plan; refresh cadence; named contacts | Cost model at full volume; check that pilot preparation steps scale | Counts and plan support the full volume; no step depended on manual pilot-only effort | Go, narrow or stop decision |
| M7 Decision and approvals | Confirmation of final scope and exclusions | Written decision with approvals from research, security and legal reviewers | Signed decision record that cites the evidence from M0 to M6 | Full license and delivery schedule |
Why the evaluation set should never reach the supplier
The pilot proves value only if it is scored on data the supplier never saw. If the supplier can see or shape the test, the pilot measures how well the sample matches the test rather than how well the data serves your model.
U.S. federal acquisition policy states the same principle for AI systems. OMB memorandum M-25-22 tells agencies to run independent evaluations on data they define themselves, such as their own validation and test sets, and says that data should not be accessible to the vendor. Where the vendor runs the tests, the memo asks agencies to monitor closely and to require results detailed enough to be independently verified or reproduced where practicable. A private buyer is not bound by that memo, but the logic carries over directly to a dataset pilot.
Document the evaluation itself so the result can be reviewed later. The NIST AI Risk Management Framework lists, under its MEASURE function, that test sets, metrics and details about the tools used during testing are documented. For a pilot, that means recording which evaluation records were used, how they were selected, which baseline the pilot data was compared against, and which metric decided the outcome. A result without that record cannot support a purchase approval.
What rights evidence has to exist before any pilot records move?
Rights evidence comes before the sample, not after the evaluation. Four different things are often blurred together: system access, which says who can export the records; privacy permission, which says whether personal data in them may be processed for this purpose; ownership, which says who holds title to the records; and contractual licensing rights, which say whether the holder may grant you a license for model training. Removing identifiers does not create the fourth. A supplier who can export the data may still be barred by its own customer contracts from licensing it.
Ask for documentation in a form you can check. A datasheet in the sense proposed by Gebru and colleagues covers why a dataset was created, what it contains, how it was collected and what it is recommended for. The Croissant metadata format adds machine-readable places for the same material: its Responsible AI vocabulary includes fields for data collection, data limitations and personal or sensitive information, and the core specification attaches data use conditions to a dataset or part of one through the schema.org usageInfo property. A supplier who can fill those fields for the pilot slice is far more likely to be able to fill them for the full delivery.
Keep the pilot grant narrow. Records should move under evaluation-only terms that name the permitted activities, the teams and environments with access, and what happens to the records, derived artifacts and any trained checkpoints if you do not buy. Counsel should review the specific terms, because what evaluation-only permits differs from one agreement to the next.
How do you keep the pilot sample from being a showcase?
Agree the population and the draw method before the supplier prepares anything, and let the buyer control the randomness. A supplier asked for a sample will usually reach for the cleanest records, not out of bad faith but because clean records are quickest to prepare.
The supplier states the population first: counts by source system, period, record status and record type, taken at a stated cutoff. The buyer then chooses the strata that matter for the task and supplies a random seed or a list of record IDs drawn from the declared population. The supplier prepares those records through the same de-identification, mapping and export steps planned for the full delivery, and reports any drawn record it could not deliver, with the reason.
This mirrors the advice in the UK government's guidelines for AI procurement, which recommend data discovery before going to market so that the completeness and limits of the available data are understood, and which favor proofs of concept before a solution goes live. A pilot is that discovery step done with the actual supplier and the actual records.
- Population counts at a dated cutoff, broken down the same way the full delivery will be.
- The draw method in writing, including the seed or ID list and who generated it.
- A list of drawn records the supplier withheld, each with a reason such as a rights restriction or a failed de-identification check.
- Confirmation that the pilot used the production preparation steps, with any manual pilot-only step called out.
Worked example (illustrative): a support-ticket pilot for an agent-training team
The figures in this example are illustrative. A team building a customer-support agent wants resolved ticket threads with the agent's actions, macros applied and the final resolution from a mid-sized software company's help desk.
At M0 the team writes the task as resolving billing and access tickets end to end, and requires the full thread, status changes and the resolution code. At M1 the supplier maps two help desk instances and a CRM to authority documents. One instance holds tickets for customers on a reseller agreement that restricts secondary use of their data. That instance is excluded before any sampling, which shrinks the declared population by roughly a fifth.
At M2 the supplier declares the remaining population by quarter and resolution code; the buyer draws 600 ticket IDs stratified across eight quarters. At M3, 584 arrive; the supplier lists 16 withheld because free-text fields failed the de-identification check. At M4 the pipeline flags that one quarter uses a retired status field, which the supplier maps in the production script rather than by hand. At M5 the team fine-tunes on the pilot slice and scores against 300 of its own held-back tickets that the supplier never saw.
The exception: the M5 result is strong for billing tickets and flat for access tickets. Instead of buying the full population, the team records a narrowed decision at M7, limited to billing-related tickets, and asks for a second, smaller pilot on access tickets with additional fields from the identity system.
Which pilot findings justify a narrower purchase rather than walking away?
Narrow when the failure is confined to a segment you can cleanly cut; stop when the failure is in the supplier's process or authority; escalate when the finding turns on law or contract wording. The table gives the usual cases.
| Finding | Narrow the purchase | Stop | Send to specialist review |
|---|---|---|---|
| One source system lacks clear licensing authority | Exclude that system if the rest of the population still meets the task | If that system holds most of the records you need | Legal, when authority depends on contract interpretation |
| Evaluation gain in some record types only | Buy the record types that moved the metric | If no segment beats the baseline | Research lead, to rule out an evaluation set problem |
| Withheld records cluster in one segment | Accept with that segment excluded and documented | If the cluster is the segment the task depends on | Privacy reviewer, when withholding came from de-identification failures |
| Pilot preparation relied on manual effort | Accept a smaller first tranche while the supplier automates | If the supplier cannot describe a scalable process | Data engineering, to estimate the full-volume defect rate |
| Population counts do not reconcile with the draw | Not advisable until the counts reconcile | If the supplier cannot explain the difference | Data engineering, before any further sampling |
| Sensitive data found in delivered records | Not advisable until preparation is fixed and re-tested | If found again after a fix | Privacy and security reviewers immediately |
Running the same pilot on data sourced on request
The schedule works the same way when the dataset does not exist yet and is sourced from a business for your request. With SourceX, nothing is pulled from a shelf: the dataset is sourced from a U.S. business once a buyer has set out the scope, volume, language, format, timing and licensing terms the pilot is meant to test.
On the supplier side, the business approves the scope and terms and signs its agreement before any data is delivered, which is the point at which the rights evidence for M1 is settled. SourceX licenses the data directly from the business and supplies licensed datasets to AI developers, arranging rights review, contracting, preparation and delivery, and de-identification is performed by SourceX through approved processing routes. A buyer who brings a written task specification and its own held-back evaluation plan makes the remaining milestones much quicker to agree.
Frequently asked questions
How long should an AI data vendor pilot take?
Set the length by the milestones rather than the calendar. Give each milestone a target date and an owner, and leave the evaluation milestone enough time for a baseline and a pilot run in your own pipeline. A deadline that arrives before the held-back evaluation finishes forces a purchase decision without the one result that justifies it.
Can the data vendor run the pilot evaluation for us?
It can, but the result is weaker evidence. If the vendor runs the tests, monitor the work and require results detailed enough for your team to verify or reproduce them, which is the approach U.S. federal guidance takes for AI acquisitions. The stronger design keeps the evaluation set on your side so the vendor never sees it.
What if the vendor will not share rights documents before a contract?
Ask for a source register and a written statement of authority for each source system under an evaluation agreement or NDA. If even that is refused, do not let records move and do not commit to the full purchase, because no later milestone can repair missing authority.
Is a pilot the same as a free data sample?
No. A free sample usually shows the format and a few records chosen by the supplier. A pilot draws records by an agreed method from a declared population, runs them through your pipeline and scores them on your own evaluation data, so it can support a purchase decision.
Who should sign the go or no-go decision?
The procurement lead owns the decision record, with sign-off from the research lead on the evaluation result, security on the intake findings and legal on the rights evidence. The record should cite the evidence from every milestone so a later reviewer can see why the purchase went ahead or was narrowed.
Sources
- OMB M-25-22 directs agencies to evaluate AI systems on data they define themselves, keep that data away from the vendor, and require verifiable results when vendors run the tests. Source
- The UK government's AI procurement guidelines recommend data discovery before going to market on the completeness and limits of the data, and proofs of concept before go-live. Source
- NIST AI RMF 1.0 MEASURE 2.1: test sets, metrics and details about the tools used during testing, evaluation, verification and validation are documented. Source
- Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process and recommended uses. Source
- Croissant 1.1 attaches data use conditions to a dataset or part of one through schema.org usageInfo and records provenance with W3C PROV-O. Source
- The Croissant RAI vocabulary includes rai:dataCollection, rai:dataLimitations and rai:personalSensitiveInformation. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.