AI uses for records
Same model, two corpora: a four-arm dataset contribution controlled experiment
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
To isolate what a dataset contributes, train one pinned model four ways with the same recipe and evaluation: no new data, a volume-matched filler, the candidate corpus, and the best permitted substitute. The contribution is candidate minus filler, not candidate minus the starting model. Fix adopt, reject and inconclusive rules before training, and report paired intervals from repeated runs on a holdout neither corpus touched.
Evaluation split
Same model, two corpora
Key takeaways
- Contribution is the candidate arm minus a volume-matched filler arm, not the candidate arm minus the model you started with.
- Freeze model snapshot, recipe and compute, evaluation and permission in a written ledger, and let only the corpus vary.
- A best-substitute arm shows whether public, synthetic or in-house data would deliver the same gain for less.
- Believe the buyer-owned holdout; a gain that appears only on the supplier's own records is supplier-specific learning.
- Write adopt, reject, inconclusive and void rules before training; an interval that includes zero is inconclusive, not a reject.
What decision does a dataset contribution controlled experiment settle?
It settles one purchase question: does adding this candidate corpus to your training mix improve your target task by at least the margin you would pay for, compared with adding the same volume of data you already have the right to use? Every other part of the design exists to stop a different cause from claiming that difference.
The evidence that supports a yes is narrow. It is a paired difference between the candidate arm and a volume-matched control arm, measured on held-out tasks that neither corpus touched, repeated across seeds, with an interval that clears a minimum written down before training. A single fine-tune that scores higher than last quarter's model is not evidence of contribution; it is evidence that something changed.
The design borrows from data benchmarks. DataComp, introduced in 2023, fixes the training code so participants compete only by proposing new training sets; its authors report that their DataComp-1B set trained a CLIP ViT-L/14 model to 79.2 percent zero-shot ImageNet accuracy, 3.7 percentage points above OpenAI's CLIP ViT-L/14 trained with the same procedure and compute. OpenAI's GDPval, launched in September 2025, runs the mirror image: a fixed set of 1,320 tasks across 44 occupations scores different models. A buyer's contribution experiment combines the two: fixed tasks, fixed model and fixed recipe, with the corpus as the only moving part.
It does not settle price, whether the gain transfers to another model family, or which fields inside the corpus did the work. Crediting parts of a corpus is a separate ablation experiment, and ranking two finished datasets head to head is a matched budget comparison; both are linked below.
Why does a with-and-without test overstate what a corpus adds?
Because adding a corpus also adds training, and more training helps on its own. A model trained on the current mix plus the candidate has seen more tokens, taken more optimizer steps and used more compute than the model trained on the current mix alone, so the comparison credits the corpus with the effect of volume.
Scaling research makes that effect predictable rather than incidental. Kaplan and colleagues reported in 2020 that language model test loss follows power-law trends in model size, dataset size and training compute, with some trends spanning more than seven orders of magnitude. An arm that gets more data and more compute is expected to look better before the content of that data matters at all.
Swapping the candidate in for part of the existing mix keeps volume fixed but creates the opposite problem: the result now blends what the candidate added with what the removed slice used to contribute. The cleaner fix is a filler arm, the current mix plus the same number of tokens of permitted, task-neutral data in the candidate's format. Contribution is the candidate arm minus the filler arm.
When data arrives is part of volume too. NVIDIA's Nemotron training FAQ describes two-phase pretraining curricula in which the first phase emphasizes diversity and coverage and the second shifts to primarily high-quality sources. If the candidate enters in a late, high-quality phase while the filler is spread evenly, the arms differ in timing as well as content. Insert filler, candidate and substitute at the same point in the schedule with the same sampling weight.
The isolation ledger: what stays frozen while only the corpus changes
Freeze four factors and let exactly one vary. The ledger below is the first page of the preregistration: every arm must match it, and a run that drifts from it is excluded under a rule written before the first result, not repaired afterward.
The model row fails more often than teams expect when the model sits behind an API. Anthropic's model documentation says aliases automatically point to the most recent snapshot, recommends specific model versions in production for consistent behavior and states that models with the same snapshot date are identical across platforms. Put the dated identifier in every arm and in the evaluation harness.
The recipe row needs exact values, not intentions. NVIDIA's Nemotron FAQ, for example, notes that the Super pretraining peak learning rate is high for fine-tuning and that its published full fine-tuning recipe for Super uses a constant 1e-5 after a 30,000-sample linear warmup. Whatever values you pick, write them once. In Hugging Face Transformers, the seed argument is set at the beginning of training and a separate data_seed controls data sampling independently of the model seed, so log both for every run.
The permission row is what makes the answer usable. A contribution measured on a sample you could not license for production, or against a filler nobody had the right to train on, cannot support a purchase. Confirm before the run that the sample's terms cover training a test checkpoint, evaluating it and deleting both afterward; pilot terms are usually narrower than the full license.
| Factor | How to freeze it | What to log per run | Drift that breaks isolation |
|---|---|---|---|
| Model | One checkpoint hash, or a dated API snapshot instead of an alias | Checkpoint or snapshot identifier and tokenizer version | An alias that moves to a newer snapshot between arms |
| Recipe and compute | One recipe tuned on a separate split, then frozen; equal added tokens per arm | Tokens, steps, epochs, learning rate schedule, warmup, seed and data seed | Retuning the learning rate after seeing one arm's curve |
| Evaluation | Frozen holdouts, grader, harness version and number of trials | Per-item scores for every trial, grader version and harness commit | Editing the rubric or swapping the judge model mid-study |
| Permission | Every arm's data cleared in writing for pilot training and evaluation | Terms reference for each corpus and its deletion date | A filler or substitute corpus nobody checked for training rights |
| Corpus, the one change | Candidate, filler and substitute in the same format and length profile | Corpus version, token count, share filtered and overlap with holdouts | A template difference that the grader happens to reward |
Four arms, three contrasts: which difference is the contribution?
The contribution is candidate minus filler, and the licensing case is candidate minus best substitute. Train four arms from the same checkpoint with the same recipe: A0 keeps the current mix with nothing added, A1 adds task-neutral filler, A2 adds the candidate corpus and A3 adds the best substitute. A1, A2 and A3 receive the same number of added tokens, so content is the only thing that differs among them.
Read the arms through three contrasts and treat no other comparison as confirmatory; the last two items set how to build the control arms. The table after the list defines each arm.
- A1 minus A0, more training: how much any added data helps. A large value warns that a naive with-and-without test would have overstated the candidate.
- A2 minus A1, contribution: what the candidate's content adds beyond volume. This is the number the experiment exists to measure.
- A2 minus A3, edge over the best substitute: whether the candidate beats the strongest source you could use instead. A corpus can contribute and still lose here, which makes the gain real but not worth licensing.
- Filler rule: draw it from the same pool as the current mix, matched on format and length, so it adds tokens without adding task signal.
- Substitute rule: build it before the experiment and build it to win. If a model-written corpus from the same task description is the obvious internal alternative, that is A3.
| Arm | Data added to the current mix | Added tokens | Why it is there |
|---|---|---|---|
| A0 Current mix | Nothing | None | Shows what you would ship today without buying anything |
| A1 Filler | Permitted, task-neutral data in the candidate's format | Equal to A2 | Absorbs the effect of more training and more compute |
| A2 Candidate | The corpus under evaluation, or its pilot sample | The planned test volume | The arm the purchase decision is about |
| A3 Best substitute | The strongest permitted alternative: public data, synthetic data written from the same task spec, or your own logs | Equal to A2 | Tests whether a cheaper source delivers the same gain |
Which holdout shows the gain carries over to your own tasks?
Use two holdouts and believe the one you own. An in-source holdout drawn from the supplier's records shows whether the model learned the supplier's world; a holdout drawn from your own tasks shows whether that learning carries over to the work you are paying for.
Split the in-source holdout by group and by time, never by random record. Records from one customer account, one case thread or one month leak into each other, so a random split rewards memorizing an account rather than learning the task. scikit-learn's GroupKFold keeps groups non-overlapping so each group lands in the test set exactly once across folds, GroupShuffleSplit splits by a provided group such as year of collection, and TimeSeriesSplit exists because other splitters would train on future data and evaluate on past data. Hold out whole accounts and the latest period.
Then check both holdouts mechanically against every training corpus, filler and substitute included. EleutherAI's evaluation harness documents a decontamination procedure that marks a test document as contaminated when it shares any 13-gram with a training document and reports a clean version of the benchmark. Report the flagged count per arm; uneven contamination across arms is a confound of its own.
Grade what happened, not how it was described. Anthropic's guide to agent evals separates a trial's transcript from its outcome, the final state of the environment, and its flight-booking example checks whether a reservation exists rather than what the agent said. Where human judgment is unavoidable, hide the arm from the grader, as GDPval does when its expert graders compare deliverables without being told which came from a model and which from a professional.
Adopt, reject, inconclusive or void: rules to write before the first run
Four verdicts, each with a rule and an action, written and dated before any arm trains. The Center for Open Science describes preregistration as specifying the research plan in advance so that planned, confirmatory work can be told apart from unplanned, exploratory work; in a data purchase, these rules are the plan.
The minimum useful difference is a business number, not a statistical one: the smallest gain on your task metric that would repay the license and the integration work. Set it with whoever owns the budget, in the units of the task metric, before anyone sees a training curve.
Treat no difference as something to test, not a conclusion. Lakens' 2017 primer on equivalence testing warns that a nonsignificant result does not show that an effect is absent and sets equivalence bounds from the smallest effect size of interest. Here that means a reject needs the interval to sit below the minimum useful difference; an interval that merely includes zero is inconclusive.
| Verdict | Rule fixed before training | What happens next |
|---|---|---|
| Adopt | On the buyer holdout, the lower interval bounds of A2 minus A1 and of A2 minus A3 both exceed the minimum useful difference, and no regression check fails | Move to licensing terms at the tested volume, with claims limited to the tested model and budget |
| Reject | The upper interval bound of A2 minus A1, or of A2 minus A3, falls below the minimum useful difference, or A2 fails a regression check | Do not license at this volume; if only A2 minus A3 failed, the substitute is the better buy |
| Inconclusive | An interval for A2 minus A1 or A2 minus A3 straddles the minimum useful difference | Run the one preplanned extension, more items or seeds, apply the same rules once more, then stop |
| Void | Contamination above the preset threshold, an arm short of its token budget, a grader or harness change, or a permission gap | Fix the cause and restart from the frozen plan; results from the void run are never read |
Does the gain survive a stronger baseline, a proper holdout and repeated trials?
A positive result earns belief only after it survives three attempts to make it disappear. Build them into the protocol rather than running them in reaction to a number someone dislikes.
Analyze per item, paired across arms. Miller's 2024 paper on error bars for evals recommends inference on question-level paired differences when comparing two models, because scores on the same questions are correlated, and clustered standard errors when questions come in related groups. Cluster by customer account or source document, and print the item count, cluster count and seed count beside every interval.
- Stronger baseline: compare against A3, then repeat A1 and A2 on the newest pinned model you could realistically ship. If a newer base model closes the gap without the corpus, the contribution belonged to the old model's weakness, and the purchase case shrinks with each release.
- Proper holdout: rescore on the buyer-owned holdout and the latest held-out period. A gain confined to the in-source holdout means the model learned the supplier's accounts, templates or vocabulary; that is real learning, but it is not what you are buying.
- Repeated trials: train every arm with the same seed list, varying both the model seed and the data seed, and run each evaluation task more than once. Anthropic's eval guide recommends multiple trials because model outputs vary between runs and notes that pass^k, which requires every one of k trials to succeed, drops as k grows; a corpus that lifts single-attempt success but not pass^k improved capability without improving reliability.
The preregistered result table, and what it leaves unmeasured
Fill the table only after every arm has finished, in the order the rows appear, and publish it alongside the plan it was preregistered against. The right-hand column is the reading rule written in advance; no values appear here because this article reports no run.
Methods note: this protocol was written for this article on 2026-10-26 from DataComp (2023), Kaplan and colleagues (2020), Miller (2024), Lakens (2017), the Center for Open Science's preregistration guidance, Anthropic's eval guide and model documentation, OpenAI's GDPval, NVIDIA's Nemotron FAQ, and the Hugging Face Transformers, EleutherAI and scikit-learn documentation. It reports no measured results from SourceX or anyone else.
Not measured: how large contributions typically are for business corpora, how often a substitute matches a licensed corpus, how many seeds or items a given task needs, and how long a measured gain lasts across model releases. Each needs runs that have not been published.
| Contrast | Holdout | Record after the run | Preset reading |
|---|---|---|---|
| A2 minus A1, contribution | Buyer-owned | Mean paired difference, interval, items, clusters and seeds | Primary result: decides adopt, reject or inconclusive |
| A2 minus A3, edge over substitute | Buyer-owned | Same fields as the first row | Must also clear the minimum useful difference for adopt |
| A2 minus A1, contribution | In-source, split by account and period | Same fields as the first row | A gain here but not on the buyer holdout means supplier-specific learning |
| A1 minus A0, more training | Buyer-owned | Same fields as the first row | Context only: shows how much a naive test would overstate |
| A2 minus A0, regression | Untouched capability suite | Score change per suite, pass or fail against its tolerance | Any failure blocks adopt |
| A2 minus A1 on a newer pinned model | Buyer-owned | Same fields as the first row | A gap that closes means the purchase case decays with releases |
Turning a tested requirement into a data request
The protocol's outputs, meaning the task, the holdout design, the minimum useful difference and the acceptance test, are what a supplier needs to source a corpus worth testing. Write them into the request before any sample moves.
SourceX sources datasets from U.S. businesses on request rather than holding them in stock, and its request form asks for scope, volume, language, format, timing and licensing needs. Add the intended use, whether the sample must cover pilot training as well as evaluation, and the acceptance test from your result table. The request can describe the data characteristics the experiment needs, such as complete case histories with recorded outcomes and pseudonymous account keys kept so holdouts can be split by group; describing them does not mean such data is available, will pass your test or will improve your model.
The ledger's permission row is closed on paper before the first arm trains, because the company holding the records must give its approval to the scope and terms and sign before delivering anything.
Frequently asked questions
What is a dataset contribution controlled experiment?
It is a training experiment in which one pinned model, one recipe and one evaluation stay fixed while only the added corpus changes. Contribution is the difference between the candidate arm and a filler arm that adds the same number of tokens of task-neutral data, so the effect of extra training is not credited to the corpus.
Why not just compare the fine-tuned model with the original model?
Because the fine-tuned model also received more training, and scaling research shows loss falls predictably as data and compute grow. That comparison mixes the corpus's content with its volume. A volume-matched filler arm separates the two.
What should the filler data be?
Data you already have the right to train on, drawn from the same pool as your current mix and matched to the candidate's format and length, so it adds tokens without adding task signal. Check it against your holdouts for overlap like any other corpus, and record its terms in the ledger.
Can a dataset contribute and still not be worth licensing?
Yes. If the candidate beats the filler but a public, synthetic or in-house substitute does about as well, the contribution is real and the licensing case is not. That is why the adopt rule requires the candidate to clear the minimum useful difference against the best substitute as well.
What if the result is inconclusive?
Run the single extension you preregistered, such as more holdout items or seeds, apply the same rules once and stop. Rerunning with new settings until a win appears turns a confirmatory test into an exploratory one, and any result found that way should be labeled exploratory.
Sources
- Anthropic's guide to agent evals separates transcripts from outcomes, recommends multiple trials because outputs vary between runs, and notes that pass^k drops as k grows. Source
- OpenAI's GDPval (September 2025) covers 1,320 tasks across 44 occupations and grades deliverables through blinded expert comparison. Source
- NVIDIA's Nemotron training FAQ describes two-phase pretraining curricula and a full fine-tuning recipe for Super at a constant 1e-5 after a 30,000-sample linear warmup. Source
- DataComp fixes the training code so participants innovate only on training sets; DataComp-1B trained CLIP ViT-L/14 to 79.2 percent zero-shot ImageNet accuracy, 3.7 points above OpenAI's CLIP ViT-L/14 with the same procedure and compute. Source
- Kaplan and colleagues (2020) report that language model test loss follows power-law trends in model size, dataset size and training compute, some across more than seven orders of magnitude. Source
- Miller (2024) recommends question-level paired differences when comparing models and clustered standard errors when questions come in related groups. Source
- Lakens (2017) explains that a nonsignificant result does not show an effect is absent and sets equivalence bounds from the smallest effect size of interest. Source
- The Center for Open Science describes preregistration as specifying the research plan in advance to distinguish planned confirmatory work from unplanned exploratory work. Source
- Anthropic's models overview says aliases point to the most recent snapshot, recommends specific model versions in production and states that same-date snapshots are identical across platforms. Source
- Hugging Face Transformers documents seed, set at the beginning of training, and data_seed, which controls data sampling independently of the model seed. Source
- EleutherAI's lm-evaluation-harness marks a test document as contaminated when it shares any 13-gram with a training document and reports a clean benchmark version. Source
- scikit-learn's GroupKFold keeps groups non-overlapping across test folds, GroupShuffleSplit splits by a provided group such as year of collection, and TimeSeriesSplit avoids training on future data. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.