Definitions and comparisons
How to measure dataset utility in a matched budget experiment: a protocol
By SourceX Editorial · Updated
10 sections
Short answer
To compare two training datasets under the same compute budget, hold everything except the data fixed: one base model, one training recipe, the same number of training tokens, several seeds per arm and one untouched holdout from the target task. Then compare paired per-item scores with intervals against a threshold set in advance. This page is a protocol for that experiment; it reports no results.
Value drivers
Dataset utility matched
Key takeaways
- Only the data may differ between arms: same base model, same frozen recipe and the same number of training tokens.
- Match tokens processed, not records, and log epochs because a smaller dataset will be repeated.
- Use the same seed list in both arms and compare per-item paired differences with clustered intervals.
- Deduplicate both datasets against the holdout and add regression checks on tasks neither arm targets.
- Write the minimum difference that would change the purchase into the preregistration before the first run.
How do you compare two training datasets under the same compute budget?
Treat the comparison as a controlled experiment in which the dataset is the only thing allowed to vary. Train the same base model with the same code, hyperparameters and number of training tokens on dataset A and on dataset B, repeat each arm with several random seeds, and score every run on the same held-out items that neither dataset has touched. The utility difference is the paired difference in holdout scores, reported with its uncertainty.
The matched budget is what makes the answer usable. A dataset that wins because it was trained for twice as many steps has not shown higher utility; it has shown that more compute helps. Matching the budget turns the question into the one a buyer actually faces: for the training run I can afford, which data gives the better model.
Everything below is a method. No measurements are reported on this page, and any numbers in the design are parameters you choose and write down before the first run.
Which budget unit should be held equal?
Match training tokens processed for the same model, and record steps and estimated compute alongside. Kaplan and colleagues estimate training compute as roughly six times the non-embedding parameter count times batch size times steps, so for one fixed model and sequence length, equal tokens means approximately equal compute. Matching records or files instead lets a dataset with longer examples receive more compute without anyone noticing.
Datasets of different sizes force a decision about repetition. If A has fewer tokens than the budget, it is repeated for several epochs while B may be seen once. Muennighoff and colleagues found that, with constrained data and a fixed compute budget, training for up to about four epochs of repeated data gave negligible loss changes compared with unique data, with returns from more repetition eventually diminishing to zero. That finding comes from pretraining, so the protocol records epochs per arm rather than assuming repetition is harmless for a fine-tuning task.
| Budget unit | When it is the right match | Trap if used alone |
|---|---|---|
| Training tokens | Default for the same model and sequence length | Packing and truncation can change effective tokens; log both |
| Optimizer steps | When batch size in tokens is fixed across arms | Different example lengths change tokens per step |
| Estimated FLOPs | When arms use different sequence lengths or model sizes | Estimates hide implementation differences; report the formula used |
| Number of examples | Only when examples are nearly equal in length | Longer examples quietly receive more compute |
| Acquisition or labeling effort | As a secondary cost lens after the compute-matched result | Mixes procurement cost with model utility |
Why do data benchmarks fix the recipe and vary only the data?
Because it is the only way to attribute a difference to the data. DataComp, a benchmark for multimodal training sets, holds the training code and compute budget fixed so that participants innovate only by proposing new training sets, and evaluates every candidate with the same standardized training run and downstream tests. DataComp-LM applies the same idea to language models, offering a testbed for controlled dataset experiments with fixed pretraining recipes at several model scales and a shared evaluation suite.
A buyer comparing two licensed datasets is running a small private version of the same benchmark. The lesson to copy is the discipline, not the scale: one frozen recipe, a budget declared in advance, and an evaluation that neither dataset supplier controls.
One more lesson comes from scaling work. Hoffmann and colleagues found that, for compute-optimal training, model size and training tokens should grow at equal rates, which means the best use of data depends on the budget. A ranking found at one budget may not hold at another, so the protocol tests at least two budgets when the purchase decision depends on the larger one.
The preregistered matched budget experimental protocol
Fill in every row before training starts, date the document and store it where it cannot be edited silently. The right-hand column shows what a complete entry looks like.
At a glance
The preregistered matched budget experimental protocol
| Element | What to specify | Complete entry looks like |
|---|---|---|
| Research question | One sentence naming the two datasets, the target task and the budget | Under a fixed token budget on base model M, does dataset B give higher holdout task success than dataset A? |
| Population and holdout | Where holdout items come from, how they were sampled, the date range and the exclusion rules | Items sampled from the buyer's own task logs after a cutoff date, deduplicated against both datasets |
| Arms | Each training condition and what it isolates | Base model untuned; A at budget T; B at budget T; optional A and B mixed at budget T |
| Budget matching | Unit, value, epochs per arm, packing and truncation rules | Equal training tokens T per arm; epochs logged; same sequence length and packing |
| Fixed recipe | Model checkpoint, optimizer, learning rate schedule, batch size, stopping rule | One recipe tuned on a separate validation split, then frozen for both arms |
| Replication | Number of seeds per arm and what each seed controls | Same seed list for both arms, controlling initialization of new layers and data order |
| Primary metric | One metric on the holdout, scored the same way for all arms | Task success rate graded by a fixed rubric or exact check |
| Regression checks | Untouched tasks that must not get worse | A general benchmark slice and an earlier internal task set |
| Analysis | Paired comparison, interval method, clustering, handling of failed runs | Per-item paired differences averaged over seeds, clustered by source document |
| Decision rule | The smallest difference that would change the purchase, and what happens otherwise | Prefer B only if the lower interval bound exceeds the stated minimum difference and no regression check fails |
Which variables must be logged for every run?
Log enough that someone else could rerun any arm and get the same budget, and enough to rule out the explanations in the next section. A run whose log is incomplete is excluded under a rule written in advance, not judged case by case.
- Dataset identifier, version, record count, token count after tokenization and the share removed by filtering.
- Tokens processed, optimizer steps, epochs over the dataset and the compute estimate with its formula.
- Base checkpoint identifier, code commit, hyperparameters and hardware type.
- Seed and what it controls: initialization of new parameters, data order or both.
- Overlap checks between each dataset and the holdout: exact and near-duplicate matches, with the method used.
- Mean and distribution of example length, and the share of examples truncated.
- Per-item holdout scores, not only the averages, so paired and clustered analysis is possible.
- Wall-clock time and failures, including runs that diverged and were restarted.
How many seeds and holdout items does a fair comparison need?
Seeds and holdout items are the two sources of noise, and the design needs enough of both. Dodge and colleagues varied only the random seed when fine-tuning BERT and found that weight initialization and training data order contributed comparably to the variance of out-of-sample performance, so a single run per arm cannot separate a data effect from luck. Use the same seed list in both arms so the comparison is paired by seed as well as by item.
For the holdout, Miller recommends treating evaluations as experiments: report standard errors, compare two models through paired differences on the same questions rather than two independent averages, and use clustered standard errors when questions come in related groups. Business holdouts are often clustered, for example several items drawn from one customer account or one long document.
Size both from a pilot rather than from a rule of thumb. Run two or three seeds per arm on a slice of the holdout, estimate the variance of the paired difference, and choose the item count and seed count that would detect the minimum difference written into the decision rule. The untuned base model is the baseline that shows whether either dataset helps at all.
What alternative explanations could produce an apparent improvement?
Several, and most of them make the dataset look better than it is. The protocol rules each one out by design or measures it so a reader can judge it.
| Alternative explanation | How it shows up | Control in this protocol |
|---|---|---|
| Holdout contamination | One dataset contains items identical or close to holdout items | Deduplicate both datasets against the holdout before training and log overlap |
| Unequal effective budget | Longer examples or different packing give one arm more tokens | Match tokens processed and log truncation and packing per arm |
| Format match to the grader | One dataset uses the same template or answer style as the scoring rubric | Normalize formats or score with a rubric that ignores surface style |
| Recipe tuned for one arm | Learning rate or stopping chosen after seeing one dataset's curves | Tune once on a separate split, freeze, then run both arms |
| Seed luck | A single favorable run drives the difference | Same seed list per arm; analyze seed-paired differences |
| Gains bought with regressions | Target task rises while general or earlier tasks fall | Untouched regression checks with their own pass thresholds |
| Repetition effects | The smaller dataset was repeated many times and overfits or underfits | Log epochs and, where possible, add an arm at a second budget |
Running the protocol once, step by step
This sequence shows the order of work without any values; the symbols stand for the numbers you fix in the preregistration. First, freeze the holdout H from the target task and deduplicate datasets A and B against it. Second, tune the recipe once on a validation split V that is separate from H, then freeze it.
Third, set the token budget T and the seed list S. Train the base model on A for T tokens with each seed in S, then on B for T tokens with the same seeds, logging epochs because A and B may need different numbers of passes to reach T. Fourth, score every run on every item of H and on the regression sets R.
Fifth, compute the per-item difference between B and A for each seed, average over seeds, and build the interval with clustering by source group. Apply the decision rule exactly as written. If the result depends on the budget, repeat the third to fifth steps at a second budget before drawing a purchase conclusion, and label any analysis you add after seeing results as exploratory.
Pre-registration checklist and what the protocol cannot show
The Center for Open Science describes preregistration as specifying the research plan before collecting data, so that what was planned can be told apart from what was discovered along the way. Use the checklist below as the minimum for this experiment.
At a glance
Pre-registration checklist and what the protocol cannot show
Checklist · 10 items
- Research question, both datasets with versions, and the target task named.
- Holdout source, sampling method, date range and deduplication method recorded.
- Budget unit and value, epochs allowed per arm, packing and truncation rules fixed.
- Recipe frozen, with the split used to tune it and the date it was frozen.
- Seed list and what each seed controls.
- Primary metric, grading method and regression checks with pass thresholds.
- Analysis method: paired differences, clustering variable, interval method.
- Minimum difference that would change the decision, and the decision for every outcome.
- Exclusion rules for failed or incomplete runs.
- Where the plan is stored, its date and who signed it off.
0 of 10 ticked. Tick each item as you go.
Using a matched budget test before you license data
The protocol answers a narrow question: which dataset gives the better model for this task at this budget on this base model. It does not show that a dataset will help a different model family, a much larger budget or a task the holdout does not represent, and a sample-based test cannot reveal problems that only appear in the full delivery. Evaluation rights for the sample also need to cover training a test model, which is a licensing question for the supplier agreement.
Where the data comes from businesses, SourceX sources it from U.S. companies on request rather than from stock; the request form takes scope, volume, language, format, timing and licensing needs, and the token budget and holdout plan from this protocol belong in that request.
Frequently asked questions
What is a matched budget experiment for training data?
It is a comparison in which two or more datasets are each used to train the same model with the same recipe and the same compute, usually measured in training tokens. Because only the data differs, any difference in held-out performance can be attributed to the data rather than to extra training.
Should I match the number of examples or the number of tokens?
Match tokens processed, and log examples, steps and estimated compute as well. Matching examples gives more compute to the dataset with longer examples. If the datasets use different sequence lengths, match estimated compute and state the formula used.
How many random seeds does each arm need?
There is no universal number. Run a pilot with a few seeds per arm, estimate how much the paired difference varies between seeds, and choose enough seeds to detect the minimum difference you wrote into the decision rule. Use the same seeds in both arms so the comparison can be paired.
What if one dataset is much smaller than the budget?
It will be repeated for several epochs while the larger one may be seen once. Record epochs per arm and treat repetition as part of what is being compared. If the purchase decision is about a larger budget, add a second budget so you can see whether the ranking holds.
Does this page report which kind of dataset wins?
No. It is a protocol for running the comparison on your own task and model. No results, effect sizes or recommendations about specific datasets are reported here.
Sources
- DataComp (Gadre et al., 2023) holds the entire training code and computational budget constant so participants innovate by proposing new training sets, evaluated with standardized training and downstream tests. Source
- DataComp-LM (Li et al., 2024) is a testbed for controlled dataset experiments with a standardized corpus, fixed pretraining recipes based on OpenLM at scales from 412M to 7B parameters, and 53 downstream evaluations. Source
- Hoffmann et al. (2022) trained over 400 models and found that, for compute-optimal training, model size and the number of training tokens should be scaled equally. Source
- Kaplan et al. (2020) estimate non-embedding training compute as C approximately equal to 6NBS, with N non-embedding parameters, B batch size and S training steps. Source
- Muennighoff et al. (2023) find that with constrained data and a fixed compute budget, up to about four epochs of repeated data yields negligible changes to loss compared with unique data, and returns from more repetition eventually diminish to zero. Source
- Dodge et al. (2020) vary only the random seed when fine-tuning BERT and find that weight initialization and training data order contribute comparably to the variance of out-of-sample performance. Source
- Miller (2024) recommends standard errors for evals, clustered standard errors for related groups of questions, and inference on question-level paired differences when comparing two models. Source
- The Center for Open Science describes preregistration as a plan made before the study that distinguishes what researchers set out to do (confirmation) from what they discovered along the way (exploration). Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.