Skip to content

Deal economics

AI data procurement renewal decision: renew, expand or stop?

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Base the renewal on measured task utility, not usage or price. For each business task the dataset serves, withhold each slice, rerun a frozen evaluation at the same training budget, test whether a substitute closes the gap, and add the supplier's delivery and acceptance record. Then apply rules agreed in advance: renew, renew a narrower scope, expand through a pilot, stop, or hold for rights review.

License terms

AI data procurement renewal

This page covers AI data procurement renewal decision: renew, expand or stop.Illustrative

Key takeaways

  • Usage shows dependence, not value; renewal evidence should measure what each slice does for each business task.
  • Withhold one slice at a time at a fixed training budget, and test a substitute in the same pass.
  • Agree the decision rules before the scores arrive so the outcome cannot be argued backwards.
  • Expansion goes through a pilot with written criteria for utility, acceptance, rights, volume and exit.
  • Supplier performance and third-party contingency belong on the scorecard alongside utility.
  • Open rights, provenance or privacy gaps hold the decision whatever the utility score says.

How do you use evidence to decide whether to renew, expand or end a dataset contract?

Score the dataset on what it measurably did for the tasks your models actually perform, then let that evidence, not the renewal notice, choose among four outcomes: renew as is, renew a narrower scope, expand through a pilot, or stop. For each business task the data serves, record the change in a frozen evaluation when the dataset's slice is withheld, whether a cheaper or already-owned substitute closes the gap, and how the supplier performed on delivery and acceptance.

The point of the scorecard is that it is filled in before the commercial conversation starts. A renewal discussion that begins with price tends to end with a discount on the same scope. One that begins with slice-level utility often ends with a different scope: more of the slice that earns its keep, none of the slice that does not.

Everything below is general information. Contract terms, retained rights and privacy obligations vary by agreement, and counsel should read the specific contract before any decision is final.

Why the dataset that gets used most is not always the one worth renewing

Usage tells you who depends on a dataset; it does not tell you whether the dependency is earning anything. A large conversation corpus can sit in every training mix because it was added early and nobody tested removing it. A small set of hard cases can be read by one team and still decide whether a model passes its release gate.

Research on data valuation makes the same point at the level of individual examples. Ghorbani and Zou's Data Shapley work scores each training point by its contribution to model performance, and reports that low-value points tend to flag outliers and corrupted data while high-value points suggest what new data to acquire. Full per-example valuation is rarely practical in a procurement cycle, but the lesson carries: value is measured against a task and a metric, and it is uneven inside one dataset.

That unevenness is why the scorecard works at slice level. A slice is any subset the contract or the delivery can separate: a record type, a period, a product line, a language, a channel. If the supplier cannot deliver or price by slice, that is itself a finding for the renewal.

Measuring task utility one slice at a time

Measure each slice by withholding it and rerunning the same evaluation, with the training budget held constant so the comparison is about content, not quantity. Use the frozen evaluation suites that already gate releases, scored per business task rather than as one blended number, because a slice can help one task and do nothing for another.

At a glance

Measuring task utility one slice at a time

Run the substitute test in the same pass. Replace the withheld slice with the closest alternative you already hold or could obtain, such as internal records, a public set or synthetic cases, at the same example count. If the substitute recovers the score, the slice's utility is real but not unique, and the renewal decision becomes a comparison of cost and risk rather than of capability.

Record uncertainty with every figure. Evaluation suites for narrow tasks are often small, so a difference of a few points can be noise. Run more than one seed where training cost allows, report the number of evaluation cases behind each task score, and agree before the run what size of change the team will treat as meaningful.

  • Inputs: the dataset's slices as delivered, a frozen evaluation suite per business task, and a fixed training recipe.
  • Runs: full mix, mix minus each slice, and mix with each slice swapped for its best substitute.
  • Outputs: per task and per slice, the score change when withheld, the share recovered by the substitute, and the evaluation case count.

The renewal scorecard built on actual task utility

The scorecard has one row per business task and slice, and every cell cites a run, a document or a ticket rather than an opinion. The rows below show the format with illustrative content for a licensed support-conversation dataset; the figures are invented to show structure.

The renewal scorecard built on actual task utility
Task and sliceUtility when withheld (frozen eval)Substitute testDelivery and acceptance recordProvisional verdict
Refund resolution, billing conversationsIllustrative: task score drops by about 9 points without the slice, across 3 seeds and 400 eval casesInternal tickets recover about a third of the dropDelivered on schedule; one schema defect cured within the cure periodRenew; candidate for expansion
Order status, logistics conversationsIllustrative: change within run-to-run noiseNot needed; no measurable gapDelivered on schedule; no defectsDrop from scope at renewal
Escalation detection, all channelsIllustrative: score drops by about 4 points; 120 eval cases, wide intervalSynthetic escalations recover most of the dropTwo late batches; one batch rejected for missing timestampsRenew narrower or replace; rerun with a larger eval
Spanish-language billing conversationsIllustrative: score drops by about 12 points; 90 eval casesNo substitute availableVolume below the agreed level in two batchesExpand through a pilot with volume terms
Rights and permitted use (applies to all rows)Not a utility measureNot applicablePermitted-use metadata complete for most batches; gaps in one periodHold all verdicts until counsel confirms the gaps
Supplier continuity (applies to all rows)Not a utility measureAlternative suppliers identified for two slicesSingle point of contact; no written refresh scheduleAsk for a written refresh and notice plan

From scorecard to decision: renew, narrow, expand or stop

Map the rows to an outcome with rules agreed before the scores come in, so the decision cannot be reverse-engineered from a preferred answer. The table gives one workable set of rules; adjust the thresholds to the size of your evaluation suites and the cost of the data.

From scorecard to decision: renew, narrow, expand or stop
OutcomeWhat the scorecard showsWhat to do nextEvidence to keep in the file
Renew as isMost slices show utility beyond noise, substitutes recover little, and delivery was acceptableRenew on current scope; carry the scorecard into next term as the baselineRun IDs, eval versions and the signed acceptance records
Renew a narrower scopeUtility is concentrated in some slices; others are within noise or fully substitutedRemove low-utility slices from scope and adjust volume terms accordinglySlice-level results showing why each removed slice was dropped
Expand through a pilotA slice shows high utility, no substitute, and the task needs more coverage than the current scope givesRun a bounded pilot on the expansion slice before committing to volumePilot criteria written before the pilot starts
StopNo slice shows utility beyond noise, or substitutes recover it at lower cost or riskLet the term end; plan retained-rights handling and data deletion as the contract requiresThe ablation results and counsel's reading of post-term obligations
Hold the decisionRights, privacy or provenance gaps are open, whatever the utility scoreResolve the gap or route it to counsel before choosing an outcomeThe gap description, owner and resolution date

What must a pilot establish before the buyer commits to a larger purchase?

The pilot must show that the expansion slice delivers the utility the scorecard predicted, in the volume and quality the larger purchase assumes, under rights terms that fit the intended use. Write these criteria before the pilot starts, so a promising sample cannot turn into a commitment without meeting them.

  • Task specification: the business task, the evaluation suite and the minimum score change that would justify expansion.
  • Representative sample: pilot records drawn the way full deliveries will be drawn, not hand-picked, with the sampling method written down.
  • Acceptance: the same validators and sampled checks the full delivery will face, run on the pilot, with defects counted and cured.
  • Rights: the permitted-use terms for the pilot records match the terms proposed for the expansion, including any retention after the term.
  • Volume and cadence: written evidence that the supplier can deliver the expanded volume on the proposed schedule, such as a sourcing plan by period.
  • Total effort: internal hours for intake, validation, relabeling and legal review, so the expansion is compared with alternatives on the whole cost.
  • Exit: what happens to pilot data and any model trained on it if the expansion does not proceed.

Supplier performance evidence the scorecard should cite

Supplier performance belongs on the scorecard because a useful dataset delivered late, incomplete or with repeated defects carries a cost the utility score hides. Public procurement offers a tested structure. In the codified FAR, section 17.207 lets a U.S. federal contracting officer exercise a contract option only after determining, among other things, that the requirement still meets an existing need, that exercising the option is the most advantageous way to meet it, that the contractor's past performance evaluations have been considered, and that performance on the contract has been acceptable. The 2025 FAR overhaul's model deviation text, which agencies apply through class deviations, moves these conditions to a new section 17.204-1; it still requires acceptable performance on the current contract and adds a check that the option price is fair and reasonable under current market conditions.

FAR 42.1503 sets minimum evaluation factors for contractor performance assessments, including technical quality of the product or service, schedule and timeliness, and management or business relations, and asks that evaluations rest on objective facts supported by performance data. A private AI buyer is not bound by these rules, but the four questions map well onto a data renewal: is the need still there, is renewing the best way to meet it, how did the supplier perform, and is that performance documented.

The NIST AI Risk Management Framework adds the third-party angle. Its GOVERN 6 category addresses AI risks arising from third-party software, data and other supply chain issues, with GOVERN 6.1 calling for policies on third-party risks and GOVERN 6.2 for contingency processes when high-risk third-party data or systems fail. For a renewal, that means the scorecard should record what the team would do if the supplier stopped delivering mid-term.

Rights and privacy findings that override a good utility score

Stop scoring and route the file to counsel when the permitted use no longer matches how the data is used, when provenance for part of the delivery cannot be shown, or when deliveries contain personal information the contract did not anticipate. None of these can be offset by strong utility, because the risk sits in the models already trained, not only in the next term.

Provenance gaps are common enough to check for rather than assume away. The Data Provenance Initiative's audit of more than 1,800 text datasets reported license omission rates above 70% and error rates above 50% on widely used dataset hosting sites. Licensed business data is a different setting, but the audit is a reminder that license metadata attached to a file is not the same as a documented right.

Also confirm the version you are scoring. If the supplier changed existing records during the term, the evaluation may describe a dataset you no longer receive. The Croissant dataset metadata format treats edited, removed or reshuffled records as a major version change, distinct from minor versions that only add data, and asking the supplier to version deliveries the same way makes renewal evidence comparable across terms.

  • Access permission to the source systems is not the same as a right to license the records for AI training.
  • De-identification lowers privacy risk but does not by itself create a right to use or license data.
  • Rights to keep models trained during the term, and duties to delete data at the end, depend on the contract text.

Worked example: a support-conversation license near the end of its term (illustrative)

Suppose an AI team licenses support conversations from a software company for a twelve-month term, with a renewal notice due two months before the end. In month nine the procurement lead asks research for the scorecard. Every figure here is illustrative.

The ablation runs show the billing slice clearly helps refund resolution and that Spanish-language billing conversations help even more, with no substitute available. The logistics slice moves nothing, and synthetic escalation cases recover most of what the escalation slice provided. Delivery records show two late batches and one rejected batch, all in the escalation slice.

The team proposes renewing the billing slice, dropping logistics and escalation, and expanding Spanish-language billing through a pilot with written volume and acceptance criteria. Counsel reviews the permitted-use metadata gap from one period before anything is signed.

The exception: suppose the pilot returns fewer Spanish-language records than planned and half fail the timestamp validator. The expansion criteria are not met, so the team renews the narrower scope only and keeps looking for a second source for that slice. The scorecard did its job by preventing a volume commitment the evidence could not support.

Sourcing the expansion or the replacement

An expansion or a replacement usually needs records the current supplier cannot provide: another language, another industry, a longer history or a process the current source never ran. The scorecard's high-utility, no-substitute rows are the best brief for that search, because they say exactly what the data must do.

The search for a second source can go through SourceX, which handles sourcing, rights review, contracting, preparation and delivery for businesses licensing operational records to AI developers. It sources datasets on request from U.S. businesses rather than from inventory, so the buyer's request can carry the task, slices, volume, language, format, timing and licensing terms the renewal scorecard points to.

Frequently asked questions

What is task utility in a dataset renewal?

Task utility is the measured effect of a dataset, or a slice of it, on the business tasks your models perform. It is usually measured by withholding the slice, rerunning a frozen evaluation with the same training budget, and comparing scores per task. It differs from usage, which only shows who reads the data.

How long before the renewal date should the scorecard start?

Early enough to finish the ablation runs, any pilot and counsel's review before the notice deadline in the contract. For many teams that means starting several months before the notice date, not the expiry date. Check the notice terms first, because missing them can renew the contract automatically.

Can we expand a dataset contract without a pilot?

You can, but the risk is that the expansion slice does not deliver at the volume or quality the larger purchase assumes. A short pilot with criteria written in advance tests utility, acceptance, rights and delivery capacity on the actual expansion scope. Skip it only when the slice is already proven at comparable volume.

What if the evaluation set is too small to show a clear difference?

Report the result as inconclusive rather than forcing a verdict. Enlarge the evaluation for that task, run more seeds where cost allows, or hold that slice at its current scope for one more term while the evidence builds. A narrow, uncertain result should not drive an expansion.

Who should sign off on the renewal decision?

Research owns the utility evidence, procurement owns the commercial terms and supplier record, security reviews delivery and access, and counsel reviews rights, retention and privacy. The decision is stronger when each owner signs their own part of the scorecard rather than one person approving the whole file.

Sources

  • Ghorbani and Zou's Data Shapley values each training point by its contribution to model performance and report that low-value points flag outliers and corrupted data while high-value points suggest what data to acquire. Source
  • FAR 17.207 lets a contracting officer exercise an option only after determining that the requirement fulfills an existing need, exercise is the most advantageous method, past performance evaluations were considered and performance on the contract was acceptable. Source
  • FAR 42.1503 requires performance evaluations to rest on objective facts and to include, at a minimum, technical quality, cost control where applicable, schedule and timeliness, and management or business relations. Source
  • The NIST AI RMF 1.0 GOVERN 6 category addresses AI risks from third-party software, data and supply chain issues, with 6.1 on third-party risk policies and 6.2 on contingency processes for failures in high-risk third-party data or systems. Source
  • The Data Provenance Initiative audited 1,858 datasets across 44 collections and reported license omission of more than 70% and error rates of more than 50% on widely used dataset hosting sites. Source
  • The Croissant 1.1 specification treats edited, removed or reshuffled records as a major version change and added data with existing data still retrievable as a minor change. Source
  • The FAR overhaul model deviation text for Part 17 moves option-exercise conditions to 17.204-1, requiring acceptable performance on the current contract and an option price that is fair and reasonable under current market conditions. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies