Rights and contracts
Evaluation vs training license scope: four things evaluate can mean
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
An evaluation training license scope decision starts with the verb. Evaluate can mean inspecting a sample, scoring a model on held-out records, using records to pick or tune a model, or training small models to measure a dataset's value; only the first two leave models unchanged. Name each permitted activity, default the rest to not permitted, bar carry-over of anything tuned or trained, and log what was shared, done and deleted.
Evaluation split
Evaluation
Key takeaways
- Evaluate covers four activities: inspect a sample, score a model, pick or tune, and train to measure; only the first two leave models unchanged.
- The EU AI Act's validation data definition combines evaluation with tuning, so an unqualified evaluation grant can be read to allow tuning.
- Model developers often evaluate data by training on it; DataDecide pretrained models on 25 corpora across 14 sizes to compare pretraining data.
- Write permitted activities by name, default the rest to not permitted, and decide what may carry over: weights, settings, architecture and derived artifacts.
- Log every evaluation grant, past or proposed, because an earlier grant that allowed training can affect a later confirmation that data was never licensed for AI training.
What does a permission to evaluate actually allow?
Only the activities the grant names, and nothing a reader has to infer from the word evaluate. Treat the evaluation training license scope decision as a question about verbs: list every activity the requester will perform on the records, decide each one, and write down what may not survive the evaluation.
The reason is that evaluate has no settled meaning in data deals. To a buyer doing diligence it can mean opening a sample to judge fit. To a model team it can mean scoring a finished model on records it never saw, using records to choose between checkpoints, or training small models on the data to see whether it improves them. Those activities differ in one respect that matters to a licensor: some leave the requester's models exactly as they were, and some change them.
A supplier, a privacy lead or a rights reviewer who signs an evaluation agreement without separating those meanings may find later that the counterparty read it as permission to do all four. This page sets out the separation, a permission matrix for common requests, the ways an evaluation grant expands after signature, and a log that records what was actually shared, done and deleted. It is general information; your counsel decides how your documents read.
Four activities hiding inside one verb
Read any evaluation request as one or more of four activities: inspect a sample, score a model, pick or tune, and train to measure. The first two leave models unchanged; the last two change models.
Inspect a sample is dataset review. People, or simple scripts, read records to check fields, coverage and quality before deciding whether to license. No model learns anything.
Score a model means running a model that already exists on held-out records and reporting how it performs. The EU AI Act's definitions describe testing data, in paraphrase, as data used for an independent evaluation that confirms a system's expected performance before it is placed on the market or put into service.
Pick or tune uses the records to make choices about a model: which checkpoint to keep, which hyperparameters, prompt or threshold to use. The Act's definition of validation data covers exactly this middle ground: "data used for providing an evaluation of the trained AI system and for tuning its non-learnable parameters and its learning process in order, inter alia, to prevent underfitting or overfitting". In the Act's own vocabulary, then, evaluating and tuning can be one activity, and a validation data set can, in paraphrase, be a separate set or part of the training set.
Train to measure is how model developers often judge a dataset's worth: they train models on it and compare the results. The DataDecide study (arXiv 2504.11393) reports pretraining models on 25 corpora, up to 100B tokens, across 14 model sizes from 4M to 1B parameters, in order to predict which pretraining data works best. For a team that evaluates data this way, evaluating a corpus means training on it.
Where does evaluation turn into training?
Evaluation turns into training at the point where the records start to shape something that outlives the evaluation: learned weights, a tuned setting or an architecture choice. Models change from here, and the grant has to say whether that change is allowed and whether it may be kept.
Two careful vocabularies draw the line in different places, which is why a contract should not borrow either silently. The AI Act defines training data as "data used for training an AI system through fitting its learnable parameters", and puts tuning of non-learnable parameters under validation. The Montreal Data License paper (arXiv 1903.12262) defines a Trained Model as "An Untrained Model that has been modified by exposure to the Data", yet still treats as untrained a model whose structure was adjusted after variations of it were exposed to the data to find the best one, which the AI Act would call tuning.
The Montreal Data License resolves the gap with a carry-over rule rather than a definition. Its benchmark right includes "the right to use the dataset to measure the performance of a Trained or Untrained Model, without however having the right to carry-over weights, code or architecture". That is the practical test this page recommends: let the requester measure, and decide in writing which results of the measuring, if any, it may carry into its own models.
The Act's Article 10 data-governance duties apply to high-risk AI systems, so its definitions are used here as vocabulary, not as a claim that the Act governs every data deal.
The activity-by-activity permission matrix
Decide each requested activity on its own row, and default anything unnamed to not permitted. The middle columns record what changes and what the grant has to say; the last column names the evidence the licensor should hold when the evaluation ends.
The rows are patterns for discussion with counsel, not drafting. Your agreement's own definitions decide how each activity is classified.
| Requested activity | What changes on the requester's side | Grant needed | What must not carry over | Evidence at the end |
|---|---|---|---|---|
| Read a sample to judge fit | Nothing in any model | Dataset review only, named people or teams | Copies of the records | Deletion confirmation for every copy |
| Run an existing model on held-out records and report scores | Nothing in the model; scores are created | Model testing on named models | Records, and any record text quoted in results | List of models tested; results shared |
| Compare candidate models and keep the best | Which model is used | Selection, stated as an evaluation use or as a training use | Selection results used for later models, unless granted | Which decisions the scores informed |
| Tune hyperparameters, prompts or thresholds against the records | Settings of a model that will be used | Validation or tuning, named explicitly | Tuned settings, unless granted | The settings derived and where they went |
| Train small models to estimate the data's value | New trained models exist | Training for measurement only | Weights, code and architecture from those runs | Deletion of the runs' weights and checkpoints |
| Fine-tune or pretrain a production model | The model the requester ships | A training license, not an evaluation grant | Not an evaluation question; renegotiate | A signed training agreement |
| Build embeddings, indexes or synthetic data from the records | New derived datasets exist | Named derived artifacts, if any | Every derived artifact, unless granted | Inventory and deletion of derived artifacts |
Five ways an evaluation grant grows without a new signature
Most evaluation grants expand through small, reasonable-sounding steps rather than a single breach. Each of the five routes below has a clause or a piece of evidence that closes it:
- Copies outlive the decision. A sample shared for a two-week review is still on a laptop, a shared drive or a notebook environment months later. Close it with a deletion date tied to the decision, and a written confirmation that names each location.
- Results become training signals. Test scores decide which checkpoint ships or which data mix is used next, which is the pick-or-tune activity in disguise. Close it by stating whether results may inform the requester's own development, and for which models.
- Derived artifacts escape the definition of the records. Embeddings, retrieval indexes, error examples and synthetic records generated from the sample are not the sample, so a grant that governs only the records leaves them free. Close it by defining derived artifacts and giving each a fate.
- New people join. Evaluation vendors, contractors and affiliated companies run tests for the requester. Close it by naming who may touch the records and requiring the same terms to pass down.
- The same records serve successive model versions. Re-running a held-out set every release turns a one-off test into a standing feedback loop on your records. Close it with a cap on versions or a date, after which a new grant is needed.
Is a gated download or a de-identified sample protection enough?
No. A gate controls who receives the files, and de-identification changes the privacy position of the records; neither one states what the recipient may do with them.
Hugging Face's documentation for gated datasets shows the gap clearly. It says the feature exists "To give more control over how datasets are used", but what it controls is access: in the default automatic-approval mode, "any user can access your dataset once they've shared their personal information with you." The terms a user accepts at the gate are what carry the use restrictions, so the words of those terms still decide whether training is allowed.
De-identification is a privacy safeguard, not a scope term. The UK Information Commissioner's Office says the UK GDPR does not apply to personal data that has been anonymised. That can settle a data protection question about the sample; it says nothing about whether the company that holds the records has agreed to let them be trained on, and the records remain the company's records either way.
Did an earlier evaluation grant already license the data for training?
Possibly, and it is worth settling before any new license is signed. A grant labelled evaluation that in fact permitted tuning, training to measure, or keeping trained artifacts may be read as a license for AI training, which matters wherever a new licensee asks you to confirm the data has never been licensed for that purpose.
SourceX asks exactly that. Its how-it-works page lists, among the things a supplier confirms, that the data "hasn't been licensed for AI training before", and its FAQ says the agreement asks you to confirm the data has not already been licensed to anyone for AI training. So a supplier that once gave an AI developer a pilot sample should pull that agreement, read the activities it allowed against the matrix above, and record what the counterparty actually did and kept.
Classify each earlier grant in one of three ways: evaluation only, with deletion confirmed; training or tuning permitted, or trained artifacts kept; or unclear. The first is usually the easiest to explain; the second and third go to counsel with the documents, and the answer goes in the log before the new confirmation is given.
The evaluation review log, with one illustrative entry
Keep one log entry per evaluation grant, past or proposed, so the answer to the earlier-grant question is on file rather than reconstructed from memory. The illustrative entry below follows a fictional pilot; it describes no real company or deal.
Fill every field, including the ones that feel obvious, because the fields people skip are the ones counterparties later read broadly.
| Log field | What to record | Illustrative entry (fictional pilot) |
|---|---|---|
| Counterparty and date | Legal entity, signatory, agreement date and version | An AI developer; pilot agreement signed in March, version 2 |
| Records shared | Export, version, selector and record count | 3,000 de-identified support tickets from one queue, one export |
| Activities granted | Each verb in the grant, quoted | Review and score existing models; no training, no tuning |
| Activities performed | What the counterparty confirms it did | Scored two models; one engineer tuned a prompt on 200 tickets |
| Artifacts kept | Models, settings, embeddings, results | Scores and the tuned prompt; no weights |
| Deletion evidence | Locations and confirmation date | Confirmation names two storage locations; the tuned prompt was not covered |
| Classification | Evaluation only, training or tuning, or unclear | Unclear: a tuning step and a kept setting |
| Decision and owner | Who decided, when, and what follows | Sent to counsel before any new training license is confirmed |
How evaluation requests sit next to a SourceX license
SourceX licenses business data directly from companies for AI training and evaluation, and its licenses are exclusive for AI training: the supplier agrees not to give anyone else an AI-training license to the selected records. How long that lasts, and any alternative to it, is written into the Dataset Deal Confirmation for that deal, a document the supplier itself signs.
That makes the matrix useful in both directions. Before signing, it tells you whether an earlier evaluation grant affects what you can confirm. After signing, an outside request to evaluate the same records should be checked against your signed Master agreement and Confirmation before you agree to anything, because an activity on the training side of the line could conflict with the exclusivity you gave; do not assume evaluation falls outside it.
The company approves the scope and terms and signs before any records move. To see whether a dataset fits, start with the short company assessment, which needs no data upload. An assessment is not an offer, and it does not promise any payout.
What this decision record cannot settle
The record shows what was granted, done and kept; it cannot decide how a court or regulator would read a particular clause, and it cannot undo use that already happened outside a grant. Where an evaluation went further than its terms allowed, the remedy depends on the agreement's wording and the facts, which is counsel's work.
It also cannot see inside a counterparty. The log depends on what the requester confirms it did, so the strongest entries rest on documents: lists of models tested, deletion confirmations that name locations, and written answers to the derived-artifact question. Jurisdiction matters too: the AI Act vocabulary used here is European, and the data protection point cites the UK regulator; other places may draw both lines differently.
Treat the matrix as a set of questions to answer before an agreement is signed, and the log as the record of the answers, not as a substitute for either the agreement or the lawyer who reads it.
Frequently asked questions
Is scoring a model on my records the same as training on them?
Not by itself. Running an existing model on held-out records and reporting the result leaves the model unchanged. It turns into something closer to training when the scores are used to choose or tune the model that will ship, so the grant should say whether results may inform the requester's own development.
Can a buyer tune hyperparameters or prompts on an evaluation sample?
Only if the grant says so. The EU AI Act's definitions put tuning of non-learnable parameters under validation data, alongside evaluation, so a grant that just says evaluate leaves the question open. Name tuning explicitly, and say whether tuned settings may be kept.
Does deleting the sample after evaluation undo the evaluation?
Deleting copies ends access to the records, but it does not remove settings tuned on them, trained runs or derived artifacts such as embeddings. Ask for a deletion confirmation that names locations, and handle trained or tuned outputs separately.
Do I need a separate agreement for a buyer's pre-purchase review of a sample?
It is safer to have one, even a short one. It should name the people who may see the sample, limit the activity to review or scoring, exclude training and tuning, set a deletion date and cover derived artifacts. A data room or gated download controls access but does not state these limits.
Could a past evaluation pilot affect licensing data through SourceX?
It could, depending on what the pilot allowed and what was kept. SourceX asks suppliers to confirm the data has not been licensed for AI training before, so pull the pilot agreement, classify it as evaluation only, training or tuning, or unclear, and take anything other than evaluation only to counsel.
Sources
- Regulation (EU) 2024/1689 (the AI Act), Article 3, defines training data, validation data, validation data set and testing data; validation data covers evaluation and tuning of non-learnable parameters. Source
- The Montreal Data License paper defines Trained and Untrained Models and a benchmark right that excludes carrying over weights, code or architecture. Source
- The DataDecide paper reports pretraining models on 25 corpora across 14 model sizes, from 4M to 1B parameters, to predict the best pretraining data from small experiments. Source
- Hugging Face's gated datasets documentation describes access requests and automatic approval, under which any user can access a dataset after sharing their information. Source
- The UK Information Commissioner's Office says the UK GDPR does not apply to personal data that has been anonymised. Source
- SourceX's how-it-works page says SourceX licenses business data for AI training and evaluation, that licenses are exclusive for AI training, and that suppliers confirm the data has not been licensed for AI training before. Source
- SourceX's FAQ says the agreement asks the supplier to confirm the data has not already been licensed to anyone for AI training. Source
See if your company qualifies
A short company assessment. No data uploads are needed.