Skip to content

Rights and contracts

What changes when an AI dataset evaluation license becomes a training deal?

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

An evaluation license becomes a training license only when the grant is rewritten, not relabeled. Restate the permitted training activities, the records in scope, which records stay held out for testing, which models and outputs the buyer may keep, who may receive copies, term and survival, the pricing basis and the privacy purpose. Training changes a model permanently, so each of those terms needs an explicit decision.

Evaluation split

Changes

This page covers what changes when an AI dataset evaluation license becomes a training deal.Illustrative

Key takeaways

  • Converting an evaluation license to training is a new grant, because training permanently changes a model and evaluation only measures it.
  • Decide which records stay held out; training on the evaluation sample removes the buyer's clean way to measure the result.
  • Name every lasting use explicitly: fine-tuning, pretraining, reward models, distillation, synthetic data, embeddings and future evaluations.
  • Check whether the evaluation paper borrowed a Results clause from a public data license that already leaves trained models unrestricted.
  • Settle the pricing structure, evaluation credit and privacy purpose before discussing amounts.

What has to change when an evaluation license becomes a training license?

Almost everything that defines the grant changes. The permitted use moves from measuring a model to changing it, so the agreement has to restate which records are in scope, which derived outputs the buyer may keep, whether any records stay held out for testing, who inside and outside the buyer may touch the data, how long rights last and what survives the end, and how the price is calculated. Swapping the word evaluation for training in the existing paper leaves most of those questions unanswered.

A successful evaluation tempts both sides to move fast. The buyer has seen useful results and wants to start training; the company that owns the records sees a deal close. That is exactly when an amendment drafted in an afternoon carries over limits written for a short test, such as a small sample, a narrow team and a deletion date, into a relationship where the records will shape a model that may be used for years.

Treat the conversion as a new license that can reuse parts of the evaluation agreement, not as a one-line amendment. The checklist below lists the issues in the order a reviewer usually needs them settled.

Why is training a different use rather than a larger evaluation?

Training changes the model and the change persists; evaluation only measures it. The EU AI Act draws the same line in its definitions: training data is used to train an AI system by fitting its learnable parameters, while testing data is used to provide an independent evaluation of the system. A license written for the second activity does not describe the first.

Persistence is the practical difference. Once records have shaped a model's parameters, deleting the records does not remove what the model learned. Regulators have treated models as something that can carry the consequences of how data was used: in its 2021 final order against the photo app developer Everalbum, the FTC required deletion of facial recognition models or algorithms developed with users' photos or videos, not only the photos themselves. A training license therefore needs terms about the models, which an evaluation license rarely has.

Training also undermines the evaluation that justified it. Research on benchmark contamination, including a 2023 Findings of EMNLP position paper by Sainz and coauthors, explains that a model exposed to test data during training scores higher on that test than it would otherwise, overstating its real performance. If the records that proved the dataset useful are folded into training, the buyer loses a clean way to measure what training achieved, and the supplier loses the reason a separate held-out set is worth licensing.

The scope conversion and pricing issues checklist

Work through the rows in order and record a decision for each, even when the decision is to keep the evaluation term unchanged. The middle columns show the usual gap between the two kinds of agreement; the last two say what to write and who settles it. Entries are typical patterns, not legal conclusions, and the specific wording belongs to counsel.

Two rows tend to be skipped because they feel settled by the evaluation. Records in scope looks obvious until the buyer asks for the full archive and the evaluation sample came from a different export with different fields. Affiliates and contractors looks obvious until the training run happens at a cloud provider or a labeling vendor the evaluation never involved. Give both rows a written answer even when the answer is to carry the evaluation term forward.

The scope conversion and pricing issues checklist
IssueEvaluation license usually saysTraining deal must decideClause or evidence to requireWho settles it
Permitted useRun tests and measure performance only; no trainingWhich training activities are allowed: fine-tuning, pretraining, reward models, distillationA defined list of permitted training activities and an explicit catch-all exclusionCounsel with the buyer's technical lead
Records in scopeA named sample, version and item countWhether training covers the sample, the full corpus or new batchesSchedule naming each dataset version, record count and delivery dateData owner
Held-out setThe whole sample is the test setWhich records stay evaluation-only and out of every training runRecord-level list of held-out IDs and a no-training covenant for themTechnical reviewer on both sides
Derived models and outputsUsually silent, because nothing should be trainedWhich models, checkpoints, embeddings and synthetic data the buyer may keep and useDefinition of covered models and successor versions, with rights on terminationCounsel
Results and publicationBenchmark scores may or may not be sharedWhether scores, examples or model cards may name or quote the dataPublication clause with an approval step and a ban on quoting recordsData owner and counsel
Affiliates and contractorsNamed evaluation team onlyWhich affiliates, cloud providers and labeling vendors may receive copiesNamed recipients, flow-down terms and an access list on requestCounsel
Term and survivalShort window with a deletion dateLicense length, what survives termination, and model rights after it endsTerm, survival clause and the treatment of models trained before terminationCounsel
Evaluation copiesDelete at the end of the windowWhether existing copies convert under the new license or are deleted and redeliveredWritten election plus a deletion certificate for any copy not convertedBoth data owners
Pricing basisFixed evaluation fee or no feeWhat the training price is keyed to and whether the evaluation fee is creditedPricing schedule that names its unit, refresh terms and any creditCommercial leads
Personal data basisAssessed for a short testWhether the new purpose is compatible or needs a new basis or noticeUpdated privacy assessment and de-identification method on recordPrivacy reviewer

Which proposed uses need an explicit scope decision rather than an assumption?

Any use that creates a lasting artifact needs a written decision: fine-tuning, pretraining, training reward or preference models, distillation into smaller models, generating synthetic data, building embeddings or retrieval indexes, and evaluating future model versions on the same records. Each produces something that outlives the delivery, and each is one a buyer may assume is included when the agreement just says training.

Check the evaluation paper before negotiating the training grant, because some standard data agreements already release models. The Community Data License Agreement Permissive 2.0 defines Results as any outcome of computational analysis of the data, including machine learning models, and imposes no restriction on Results. Microsoft's Computational Use of Data Agreement treats AI models trained on the data, if they contain no more than a de minimis portion of it, as Results it does not restrict. If an evaluation license borrowed either structure, the buyer may argue that a model built during evaluation is already unrestricted, and the conversion is the moment to close that gap.

  • Fine-tuning a named model family versus any model the buyer builds.
  • Pretraining or continued pretraining on the records.
  • Reward, preference or judge models trained on outcomes in the records.
  • Distillation into smaller student models and their onward release.
  • Synthetic data generated from the records or from models trained on them.
  • Embeddings, retrieval indexes and caches built from the records.
  • Using the same records to evaluate later model versions.

Which pricing questions reopen when the permitted use changes?

The pricing basis reopens, not just the amount. An evaluation fee usually pays for access to a sample for a fixed period; a training price has to say what it buys, which means naming its unit and the events that change it. Settle the structure before anyone discusses a number, because the structure is where most later disputes start.

At a glance

Which pricing questions reopen when the permitted use changes

Keep commercial questions separate from scope questions in the negotiation record. A buyer that asks for a broader grant in exchange for a credit has changed scope, and that change should appear in the permitted use row of the checklist, not only in the pricing schedule.

Watch for pricing proposals that quietly assume a scope answer. A flat price for unlimited use implies successor models and every derived artifact are included; a per-record price implies the held-out records are excluded or priced separately. Read each proposal back against the checklist and make the assumption explicit in the scope rows before agreeing the number.

  • Unit: whether the price is keyed to the records delivered, a dataset version, a period of access or a set of permitted uses.
  • Evaluation credit: whether anything paid for the evaluation counts toward the training license, and on what condition.
  • Refreshes: whether new batches are priced under the same schedule, and how a schema change affects them.
  • Exclusivity: whether the buyer asks for any exclusivity, for which use and for how long.
  • Successor models: whether rights extend to later model versions or require a renewal.
  • Held-out records: whether records kept for testing are priced separately from training records.

What happens to the evaluation copies, results and held-out records?

Decide them explicitly in the conversion document. The buyer usually holds three things from the evaluation: copies of the sample, results such as scores and error analyses, and sometimes intermediate artifacts like embeddings. Each needs an election: convert under the new license, keep under the old restrictions, or delete with a certificate.

The held-out set deserves its own treatment. If the evaluation sample will keep serving as a test set, the training license should list those record IDs, forbid their use in any training run, and allow the supplier or an agreed auditor to check that the training manifests exclude them. If the sample will become training data instead, the buyer needs a new held-out set, and that set is a scoping and pricing decision, not an afterthought.

Pin versions. The evaluation sample and the training corpus may come from different exports, so the training schedule should name the dataset version for each part, including how a record that appears in both is identified.

Results deserve a short clause too. Scores, error analyses and examples of failures produced during the evaluation can reveal the content of individual records if quoted. The conversion document should say whether the buyer may keep them, use them in model cards or publications, and whether quoting record text needs approval.

Do personal data in the records need a fresh purpose check?

Often, yes. Under the GDPR, when personal data are processed for a purpose other than the one they were collected for, and the new processing is not based on consent or a qualifying law, Article 6(4) asks the controller to assess compatibility, considering the link between the purposes, the context of collection, the nature of the data, the possible consequences and the safeguards in place.

Moving from evaluation to training is a change of purpose for the licensee and often for the company that holds the records, so the privacy assessment done for the evaluation should be reopened rather than reused. De-identification before delivery reduces the exposure, but it is a privacy safeguard, not a grant of rights; the company still needs the contractual and ownership position that lets it license the records for training at all.

Worked example: a claims archive moves from a test set to a training deal

Illustrative example. A freight company gave an AI developer an evaluation license over 2,000 de-identified claim files for 90 days. The developer's tests went well and it asked to train on the full archive of about 60,000 claims and to keep the original 2,000 as its test set.

Walking the checklist, the parties agree that the 2,000 evaluation files stay evaluation-only, listed by ID and excluded from every training manifest. The training schedule covers the remaining files as one dataset version, with later batches under the same schedule. Covered models are defined as fine-tuned versions of two named model families, with distillation allowed only into models the developer does not release publicly. Synthetic data generated from the records is kept internal. The evaluation fee is credited under a stated condition, and the pricing schedule names its unit.

The exception surfaced in the technical review. During the evaluation, an engineer had fine-tuned a small experimental model on part of the sample, which the evaluation license did not allow. The conversion could not simply proceed. Counsel for both sides had to decide whether that model would be deleted with a certificate or brought under the new license by an express grant, and the held-out set had to be redrawn because some of its files had already been trained on.

The limit of the checklist is visible here. It finds the issue and records the decision; it cannot decide the remedy for use outside the old license, which depends on the agreement's wording and the facts.

What should make you pause the conversion and call in counsel?

Pause when the conversion request reaches beyond the records or beyond the parties. These signals mean the decision depends on contract wording, ownership or law, and a qualified reviewer should see the request before anyone replies.

  • The buyer has already trained anything on evaluation records, even as an experiment.
  • The evaluation agreement uses a Results or output clause borrowed from a public data license.
  • The request covers public weight release, distillation for third parties or resale of synthetic data.
  • Records include personal data and the original privacy assessment assumed a short test.
  • Some records came from customers, vendors or platforms whose terms limit onward use.
  • The buyer wants affiliates, investors or labeling vendors to receive copies without named flow-down terms.

Turning an evaluation into a SourceX training agreement

Through SourceX, a training deal rests on its own signed agreement rather than on an extended test. No records move before the company has approved the scope and terms and signed; from then on, SourceX's use, sublicensing and distribution of the de-identified dataset is limited to what that agreement allows. Exclusivity and term are fixed in that signed agreement, and the first conversation is a brief company assessment with nothing uploaded.

Frequently asked questions

Can an evaluation license simply be amended to allow training?

It can be amended, but a one-line change usually leaves the important terms unwritten. Training creates lasting models, so the amendment has to cover derived outputs, held-out records, recipients, survival and pricing basis. Most reviewers find a new training license that reuses parts of the evaluation agreement cleaner.

Should the evaluation sample become training data?

Only if the buyer has another way to measure the trained model. Research on benchmark contamination shows that training on test data overstates performance on that test. Many deals keep the evaluation sample held out and license a separate training set.

What if the buyer already trained on the evaluation sample?

Stop and resolve it before converting. Counsel decides whether the model is deleted with a certificate or brought under the new license by an express grant, and the held-out set usually has to be redrawn.

Does the evaluation fee count toward the training price?

Only if the agreement says so. Write down whether a credit applies, on what condition and against which part of the training price, and keep that commercial term separate from the scope decisions.

Is de-identified data free of these issues?

No. De-identification lowers privacy risk but does not grant the right to license records for training, and it does not settle what happens to models built from them. Those questions are answered by ownership and contract terms.

Sources

  • The EU AI Act (Regulation 2024/1689, Article 3) defines training data as data used to train an AI system by fitting its learnable parameters and testing data as data used to provide an independent evaluation of the system. Source
  • The FTC's May 2021 final Everalbum order required the company to delete facial recognition models or algorithms it developed with users' photos or videos. Source
  • Sainz et al. (Findings of EMNLP 2023) explain that training-data contamination causes an overestimation of a contaminated model's performance on the affected benchmark. Source
  • The Community Data License Agreement Permissive 2.0 defines Results as any outcome of computational analysis of the data, including machine learning models, and imposes no restriction on Results. Source
  • Microsoft's Computational Use of Data Agreement 1.0 treats AI models trained on the data that include no more than a de minimis portion of it as Results, which it does not restrict. Source
  • GDPR Article 6(4) lists factors for assessing whether processing for a new purpose is compatible with the original one. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies