Skip to content

Rights and contracts

Evaluation vs training license scope: four things evaluate can mean

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

An evaluation training license scope decision starts with the verb. Evaluate can mean inspecting a sample, scoring a model on held-out records, using records to pick or tune a model, or training small models to measure a dataset's value; only the first two leave models unchanged. Name each permitted activity, default the rest to not permitted, bar carry-over of anything tuned or trained, and log what was shared, done and deleted.

Evaluation split

Evaluation

This page covers evaluation vs training license scope: four things evaluate can mean.Illustrative

Key takeaways

  • Evaluate covers four activities: inspect a sample, score a model, pick or tune, and train to measure; only the first two leave models unchanged.
  • The EU AI Act's validation data definition combines evaluation with tuning, so an unqualified evaluation grant can be read to allow tuning.
  • Model developers often evaluate data by training on it; DataDecide pretrained models on 25 corpora across 14 sizes to compare pretraining data.
  • Write permitted activities by name, default the rest to not permitted, and decide what may carry over: weights, settings, architecture and derived artifacts.
  • Log every evaluation grant, past or proposed, because an earlier grant that allowed training can affect a later confirmation that data was never licensed for AI training.

What does a permission to evaluate actually allow?

Only the activities the grant names, and nothing a reader has to infer from the word evaluate. Treat the evaluation training license scope decision as a question about verbs: list every activity the requester will perform on the records, decide each one, and write down what may not survive the evaluation.

The reason is that evaluate has no settled meaning in data deals. To a buyer doing diligence it can mean opening a sample to judge fit. To a model team it can mean scoring a finished model on records it never saw, using records to choose between checkpoints, or training small models on the data to see whether it improves them. Those activities differ in one respect that matters to a licensor: some leave the requester's models exactly as they were, and some change them.

A supplier, a privacy lead or a rights reviewer who signs an evaluation agreement without separating those meanings may find later that the counterparty read it as permission to do all four. This page sets out the separation, a permission matrix for common requests, the ways an evaluation grant expands after signature, and a log that records what was actually shared, done and deleted. It is general information; your counsel decides how your documents read.

Four activities hiding inside one verb

Read any evaluation request as one or more of four activities: inspect a sample, score a model, pick or tune, and train to measure. The first two leave models unchanged; the last two change models.

Inspect a sample is dataset review. People, or simple scripts, read records to check fields, coverage and quality before deciding whether to license. No model learns anything.

Score a model means running a model that already exists on held-out records and reporting how it performs. The EU AI Act's definitions describe testing data, in paraphrase, as data used for an independent evaluation that confirms a system's expected performance before it is placed on the market or put into service.

Pick or tune uses the records to make choices about a model: which checkpoint to keep, which hyperparameters, prompt or threshold to use. The Act's definition of validation data covers exactly this middle ground: "data used for providing an evaluation of the trained AI system and for tuning its non-learnable parameters and its learning process in order, inter alia, to prevent underfitting or overfitting". In the Act's own vocabulary, then, evaluating and tuning can be one activity, and a validation data set can, in paraphrase, be a separate set or part of the training set.

Train to measure is how model developers often judge a dataset's worth: they train models on it and compare the results. The DataDecide study (arXiv 2504.11393) reports pretraining models on 25 corpora, up to 100B tokens, across 14 model sizes from 4M to 1B parameters, in order to predict which pretraining data works best. For a team that evaluates data this way, evaluating a corpus means training on it.

A left-to-right line of four activities, inspect a sample, score a model, pick or tune, and train to measure, with a dashed boundary after score a model marked models change from here and EU AI Act data labels beneath the last three.
Four meanings of evaluate on one line; the boundary marks where records start to change models. Labels follow the EU AI Act's Article 3 definitions.

Where does evaluation turn into training?

Evaluation turns into training at the point where the records start to shape something that outlives the evaluation: learned weights, a tuned setting or an architecture choice. Models change from here, and the grant has to say whether that change is allowed and whether it may be kept.

Two careful vocabularies draw the line in different places, which is why a contract should not borrow either silently. The AI Act defines training data as "data used for training an AI system through fitting its learnable parameters", and puts tuning of non-learnable parameters under validation. The Montreal Data License paper (arXiv 1903.12262) defines a Trained Model as "An Untrained Model that has been modified by exposure to the Data", yet still treats as untrained a model whose structure was adjusted after variations of it were exposed to the data to find the best one, which the AI Act would call tuning.

The Montreal Data License resolves the gap with a carry-over rule rather than a definition. Its benchmark right includes "the right to use the dataset to measure the performance of a Trained or Untrained Model, without however having the right to carry-over weights, code or architecture". That is the practical test this page recommends: let the requester measure, and decide in writing which results of the measuring, if any, it may carry into its own models.

The Act's Article 10 data-governance duties apply to high-risk AI systems, so its definitions are used here as vocabulary, not as a claim that the Act governs every data deal.

The activity-by-activity permission matrix

Decide each requested activity on its own row, and default anything unnamed to not permitted. The middle columns record what changes and what the grant has to say; the last column names the evidence the licensor should hold when the evaluation ends.

The rows are patterns for discussion with counsel, not drafting. Your agreement's own definitions decide how each activity is classified.

The activity-by-activity permission matrix
Requested activityWhat changes on the requester's sideGrant neededWhat must not carry overEvidence at the end
Read a sample to judge fitNothing in any modelDataset review only, named people or teamsCopies of the recordsDeletion confirmation for every copy
Run an existing model on held-out records and report scoresNothing in the model; scores are createdModel testing on named modelsRecords, and any record text quoted in resultsList of models tested; results shared
Compare candidate models and keep the bestWhich model is usedSelection, stated as an evaluation use or as a training useSelection results used for later models, unless grantedWhich decisions the scores informed
Tune hyperparameters, prompts or thresholds against the recordsSettings of a model that will be usedValidation or tuning, named explicitlyTuned settings, unless grantedThe settings derived and where they went
Train small models to estimate the data's valueNew trained models existTraining for measurement onlyWeights, code and architecture from those runsDeletion of the runs' weights and checkpoints
Fine-tune or pretrain a production modelThe model the requester shipsA training license, not an evaluation grantNot an evaluation question; renegotiateA signed training agreement
Build embeddings, indexes or synthetic data from the recordsNew derived datasets existNamed derived artifacts, if anyEvery derived artifact, unless grantedInventory and deletion of derived artifacts

Five ways an evaluation grant grows without a new signature

Most evaluation grants expand through small, reasonable-sounding steps rather than a single breach. Each of the five routes below has a clause or a piece of evidence that closes it:

  • Copies outlive the decision. A sample shared for a two-week review is still on a laptop, a shared drive or a notebook environment months later. Close it with a deletion date tied to the decision, and a written confirmation that names each location.
  • Results become training signals. Test scores decide which checkpoint ships or which data mix is used next, which is the pick-or-tune activity in disguise. Close it by stating whether results may inform the requester's own development, and for which models.
  • Derived artifacts escape the definition of the records. Embeddings, retrieval indexes, error examples and synthetic records generated from the sample are not the sample, so a grant that governs only the records leaves them free. Close it by defining derived artifacts and giving each a fate.
  • New people join. Evaluation vendors, contractors and affiliated companies run tests for the requester. Close it by naming who may touch the records and requiring the same terms to pass down.
  • The same records serve successive model versions. Re-running a held-out set every release turns a one-off test into a standing feedback loop on your records. Close it with a cap on versions or a date, after which a new grant is needed.

Is a gated download or a de-identified sample protection enough?

No. A gate controls who receives the files, and de-identification changes the privacy position of the records; neither one states what the recipient may do with them.

Hugging Face's documentation for gated datasets shows the gap clearly. It says the feature exists "To give more control over how datasets are used", but what it controls is access: in the default automatic-approval mode, "any user can access your dataset once they've shared their personal information with you." The terms a user accepts at the gate are what carry the use restrictions, so the words of those terms still decide whether training is allowed.

De-identification is a privacy safeguard, not a scope term. The UK Information Commissioner's Office says the UK GDPR does not apply to personal data that has been anonymised. That can settle a data protection question about the sample; it says nothing about whether the company that holds the records has agreed to let them be trained on, and the records remain the company's records either way.

Did an earlier evaluation grant already license the data for training?

Possibly, and it is worth settling before any new license is signed. A grant labelled evaluation that in fact permitted tuning, training to measure, or keeping trained artifacts may be read as a license for AI training, which matters wherever a new licensee asks you to confirm the data has never been licensed for that purpose.

SourceX asks exactly that. Its how-it-works page lists, among the things a supplier confirms, that the data "hasn't been licensed for AI training before", and its FAQ says the agreement asks you to confirm the data has not already been licensed to anyone for AI training. So a supplier that once gave an AI developer a pilot sample should pull that agreement, read the activities it allowed against the matrix above, and record what the counterparty actually did and kept.

Classify each earlier grant in one of three ways: evaluation only, with deletion confirmed; training or tuning permitted, or trained artifacts kept; or unclear. The first is usually the easiest to explain; the second and third go to counsel with the documents, and the answer goes in the log before the new confirmation is given.

The evaluation review log, with one illustrative entry

Keep one log entry per evaluation grant, past or proposed, so the answer to the earlier-grant question is on file rather than reconstructed from memory. The illustrative entry below follows a fictional pilot; it describes no real company or deal.

Fill every field, including the ones that feel obvious, because the fields people skip are the ones counterparties later read broadly.

The evaluation review log, with one illustrative entry
Log fieldWhat to recordIllustrative entry (fictional pilot)
Counterparty and dateLegal entity, signatory, agreement date and versionAn AI developer; pilot agreement signed in March, version 2
Records sharedExport, version, selector and record count3,000 de-identified support tickets from one queue, one export
Activities grantedEach verb in the grant, quotedReview and score existing models; no training, no tuning
Activities performedWhat the counterparty confirms it didScored two models; one engineer tuned a prompt on 200 tickets
Artifacts keptModels, settings, embeddings, resultsScores and the tuned prompt; no weights
Deletion evidenceLocations and confirmation dateConfirmation names two storage locations; the tuned prompt was not covered
ClassificationEvaluation only, training or tuning, or unclearUnclear: a tuning step and a kept setting
Decision and ownerWho decided, when, and what followsSent to counsel before any new training license is confirmed

How evaluation requests sit next to a SourceX license

SourceX licenses business data directly from companies for AI training and evaluation, and its licenses are exclusive for AI training: the supplier agrees not to give anyone else an AI-training license to the selected records. How long that lasts, and any alternative to it, is written into the Dataset Deal Confirmation for that deal, a document the supplier itself signs.

That makes the matrix useful in both directions. Before signing, it tells you whether an earlier evaluation grant affects what you can confirm. After signing, an outside request to evaluate the same records should be checked against your signed Master agreement and Confirmation before you agree to anything, because an activity on the training side of the line could conflict with the exclusivity you gave; do not assume evaluation falls outside it.

The company approves the scope and terms and signs before any records move. To see whether a dataset fits, start with the short company assessment, which needs no data upload. An assessment is not an offer, and it does not promise any payout.

What this decision record cannot settle

The record shows what was granted, done and kept; it cannot decide how a court or regulator would read a particular clause, and it cannot undo use that already happened outside a grant. Where an evaluation went further than its terms allowed, the remedy depends on the agreement's wording and the facts, which is counsel's work.

It also cannot see inside a counterparty. The log depends on what the requester confirms it did, so the strongest entries rest on documents: lists of models tested, deletion confirmations that name locations, and written answers to the derived-artifact question. Jurisdiction matters too: the AI Act vocabulary used here is European, and the data protection point cites the UK regulator; other places may draw both lines differently.

Treat the matrix as a set of questions to answer before an agreement is signed, and the log as the record of the answers, not as a substitute for either the agreement or the lawyer who reads it.

Frequently asked questions

Is scoring a model on my records the same as training on them?

Not by itself. Running an existing model on held-out records and reporting the result leaves the model unchanged. It turns into something closer to training when the scores are used to choose or tune the model that will ship, so the grant should say whether results may inform the requester's own development.

Can a buyer tune hyperparameters or prompts on an evaluation sample?

Only if the grant says so. The EU AI Act's definitions put tuning of non-learnable parameters under validation data, alongside evaluation, so a grant that just says evaluate leaves the question open. Name tuning explicitly, and say whether tuned settings may be kept.

Does deleting the sample after evaluation undo the evaluation?

Deleting copies ends access to the records, but it does not remove settings tuned on them, trained runs or derived artifacts such as embeddings. Ask for a deletion confirmation that names locations, and handle trained or tuned outputs separately.

Do I need a separate agreement for a buyer's pre-purchase review of a sample?

It is safer to have one, even a short one. It should name the people who may see the sample, limit the activity to review or scoring, exclude training and tuning, set a deletion date and cover derived artifacts. A data room or gated download controls access but does not state these limits.

Could a past evaluation pilot affect licensing data through SourceX?

It could, depending on what the pilot allowed and what was kept. SourceX asks suppliers to confirm the data has not been licensed for AI training before, so pull the pilot agreement, classify it as evaluation only, training or tuning, or unclear, and take anything other than evaluation only to counsel.

Sources

  • Regulation (EU) 2024/1689 (the AI Act), Article 3, defines training data, validation data, validation data set and testing data; validation data covers evaluation and tuning of non-learnable parameters. Source
  • The Montreal Data License paper defines Trained and Untrained Models and a benchmark right that excludes carrying over weights, code or architecture. Source
  • The DataDecide paper reports pretraining models on 25 corpora across 14 model sizes, from 4M to 1B parameters, to predict the best pretraining data from small experiments. Source
  • Hugging Face's gated datasets documentation describes access requests and automatic approval, under which any user can access a dataset after sharing their information. Source
  • The UK Information Commissioner's Office says the UK GDPR does not apply to personal data that has been anonymised. Source
  • SourceX's how-it-works page says SourceX licenses business data for AI training and evaluation, that licenses are exclusive for AI training, and that suppliers confirm the data has not been licensed for AI training before. Source
  • SourceX's FAQ says the agreement asks the supplier to confirm the data has not already been licensed to anyone for AI training. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies