Skip to content

Definitions and comparisons

How to measure dataset utility in a matched budget experiment: a protocol

By SourceX Editorial · Updated

10 sections

Short answer

To compare two training datasets under the same compute budget, hold everything except the data fixed: one base model, one training recipe, the same number of training tokens, several seeds per arm and one untouched holdout from the target task. Then compare paired per-item scores with intervals against a threshold set in advance. This page is a protocol for that experiment; it reports no results.

Value drivers

Dataset utility matched

This page covers how to measure dataset utility in a matched budget experiment: a protocol.Illustrative

Key takeaways

  • Only the data may differ between arms: same base model, same frozen recipe and the same number of training tokens.
  • Match tokens processed, not records, and log epochs because a smaller dataset will be repeated.
  • Use the same seed list in both arms and compare per-item paired differences with clustered intervals.
  • Deduplicate both datasets against the holdout and add regression checks on tasks neither arm targets.
  • Write the minimum difference that would change the purchase into the preregistration before the first run.

How do you compare two training datasets under the same compute budget?

Treat the comparison as a controlled experiment in which the dataset is the only thing allowed to vary. Train the same base model with the same code, hyperparameters and number of training tokens on dataset A and on dataset B, repeat each arm with several random seeds, and score every run on the same held-out items that neither dataset has touched. The utility difference is the paired difference in holdout scores, reported with its uncertainty.

The matched budget is what makes the answer usable. A dataset that wins because it was trained for twice as many steps has not shown higher utility; it has shown that more compute helps. Matching the budget turns the question into the one a buyer actually faces: for the training run I can afford, which data gives the better model.

Everything below is a method. No measurements are reported on this page, and any numbers in the design are parameters you choose and write down before the first run.

Which budget unit should be held equal?

Match training tokens processed for the same model, and record steps and estimated compute alongside. Kaplan and colleagues estimate training compute as roughly six times the non-embedding parameter count times batch size times steps, so for one fixed model and sequence length, equal tokens means approximately equal compute. Matching records or files instead lets a dataset with longer examples receive more compute without anyone noticing.

Datasets of different sizes force a decision about repetition. If A has fewer tokens than the budget, it is repeated for several epochs while B may be seen once. Muennighoff and colleagues found that, with constrained data and a fixed compute budget, training for up to about four epochs of repeated data gave negligible loss changes compared with unique data, with returns from more repetition eventually diminishing to zero. That finding comes from pretraining, so the protocol records epochs per arm rather than assuming repetition is harmless for a fine-tuning task.

Which budget unit should be held equal?
Budget unitWhen it is the right matchTrap if used alone
Training tokensDefault for the same model and sequence lengthPacking and truncation can change effective tokens; log both
Optimizer stepsWhen batch size in tokens is fixed across armsDifferent example lengths change tokens per step
Estimated FLOPsWhen arms use different sequence lengths or model sizesEstimates hide implementation differences; report the formula used
Number of examplesOnly when examples are nearly equal in lengthLonger examples quietly receive more compute
Acquisition or labeling effortAs a secondary cost lens after the compute-matched resultMixes procurement cost with model utility

Why do data benchmarks fix the recipe and vary only the data?

Because it is the only way to attribute a difference to the data. DataComp, a benchmark for multimodal training sets, holds the training code and compute budget fixed so that participants innovate only by proposing new training sets, and evaluates every candidate with the same standardized training run and downstream tests. DataComp-LM applies the same idea to language models, offering a testbed for controlled dataset experiments with fixed pretraining recipes at several model scales and a shared evaluation suite.

A buyer comparing two licensed datasets is running a small private version of the same benchmark. The lesson to copy is the discipline, not the scale: one frozen recipe, a budget declared in advance, and an evaluation that neither dataset supplier controls.

One more lesson comes from scaling work. Hoffmann and colleagues found that, for compute-optimal training, model size and training tokens should grow at equal rates, which means the best use of data depends on the budget. A ranking found at one budget may not hold at another, so the protocol tests at least two budgets when the purchase decision depends on the larger one.

The preregistered matched budget experimental protocol

Fill in every row before training starts, date the document and store it where it cannot be edited silently. The right-hand column shows what a complete entry looks like.

At a glance

The preregistered matched budget experimental protocol

The preregistered matched budget experimental protocol
ElementWhat to specifyComplete entry looks like
Research questionOne sentence naming the two datasets, the target task and the budgetUnder a fixed token budget on base model M, does dataset B give higher holdout task success than dataset A?
Population and holdoutWhere holdout items come from, how they were sampled, the date range and the exclusion rulesItems sampled from the buyer's own task logs after a cutoff date, deduplicated against both datasets
ArmsEach training condition and what it isolatesBase model untuned; A at budget T; B at budget T; optional A and B mixed at budget T
Budget matchingUnit, value, epochs per arm, packing and truncation rulesEqual training tokens T per arm; epochs logged; same sequence length and packing
Fixed recipeModel checkpoint, optimizer, learning rate schedule, batch size, stopping ruleOne recipe tuned on a separate validation split, then frozen for both arms
ReplicationNumber of seeds per arm and what each seed controlsSame seed list for both arms, controlling initialization of new layers and data order
Primary metricOne metric on the holdout, scored the same way for all armsTask success rate graded by a fixed rubric or exact check
Regression checksUntouched tasks that must not get worseA general benchmark slice and an earlier internal task set
AnalysisPaired comparison, interval method, clustering, handling of failed runsPer-item paired differences averaged over seeds, clustered by source document
Decision ruleThe smallest difference that would change the purchase, and what happens otherwisePrefer B only if the lower interval bound exceeds the stated minimum difference and no regression check fails

Which variables must be logged for every run?

Log enough that someone else could rerun any arm and get the same budget, and enough to rule out the explanations in the next section. A run whose log is incomplete is excluded under a rule written in advance, not judged case by case.

  • Dataset identifier, version, record count, token count after tokenization and the share removed by filtering.
  • Tokens processed, optimizer steps, epochs over the dataset and the compute estimate with its formula.
  • Base checkpoint identifier, code commit, hyperparameters and hardware type.
  • Seed and what it controls: initialization of new parameters, data order or both.
  • Overlap checks between each dataset and the holdout: exact and near-duplicate matches, with the method used.
  • Mean and distribution of example length, and the share of examples truncated.
  • Per-item holdout scores, not only the averages, so paired and clustered analysis is possible.
  • Wall-clock time and failures, including runs that diverged and were restarted.

How many seeds and holdout items does a fair comparison need?

Seeds and holdout items are the two sources of noise, and the design needs enough of both. Dodge and colleagues varied only the random seed when fine-tuning BERT and found that weight initialization and training data order contributed comparably to the variance of out-of-sample performance, so a single run per arm cannot separate a data effect from luck. Use the same seed list in both arms so the comparison is paired by seed as well as by item.

For the holdout, Miller recommends treating evaluations as experiments: report standard errors, compare two models through paired differences on the same questions rather than two independent averages, and use clustered standard errors when questions come in related groups. Business holdouts are often clustered, for example several items drawn from one customer account or one long document.

Size both from a pilot rather than from a rule of thumb. Run two or three seeds per arm on a slice of the holdout, estimate the variance of the paired difference, and choose the item count and seed count that would detect the minimum difference written into the decision rule. The untuned base model is the baseline that shows whether either dataset helps at all.

What alternative explanations could produce an apparent improvement?

Several, and most of them make the dataset look better than it is. The protocol rules each one out by design or measures it so a reader can judge it.

What alternative explanations could produce an apparent improvement?
Alternative explanationHow it shows upControl in this protocol
Holdout contaminationOne dataset contains items identical or close to holdout itemsDeduplicate both datasets against the holdout before training and log overlap
Unequal effective budgetLonger examples or different packing give one arm more tokensMatch tokens processed and log truncation and packing per arm
Format match to the graderOne dataset uses the same template or answer style as the scoring rubricNormalize formats or score with a rubric that ignores surface style
Recipe tuned for one armLearning rate or stopping chosen after seeing one dataset's curvesTune once on a separate split, freeze, then run both arms
Seed luckA single favorable run drives the differenceSame seed list per arm; analyze seed-paired differences
Gains bought with regressionsTarget task rises while general or earlier tasks fallUntouched regression checks with their own pass thresholds
Repetition effectsThe smaller dataset was repeated many times and overfits or underfitsLog epochs and, where possible, add an arm at a second budget

Running the protocol once, step by step

This sequence shows the order of work without any values; the symbols stand for the numbers you fix in the preregistration. First, freeze the holdout H from the target task and deduplicate datasets A and B against it. Second, tune the recipe once on a validation split V that is separate from H, then freeze it.

Third, set the token budget T and the seed list S. Train the base model on A for T tokens with each seed in S, then on B for T tokens with the same seeds, logging epochs because A and B may need different numbers of passes to reach T. Fourth, score every run on every item of H and on the regression sets R.

Fifth, compute the per-item difference between B and A for each seed, average over seeds, and build the interval with clustering by source group. Apply the decision rule exactly as written. If the result depends on the budget, repeat the third to fifth steps at a second budget before drawing a purchase conclusion, and label any analysis you add after seeing results as exploratory.

Pre-registration checklist and what the protocol cannot show

The Center for Open Science describes preregistration as specifying the research plan before collecting data, so that what was planned can be told apart from what was discovered along the way. Use the checklist below as the minimum for this experiment.

At a glance

Pre-registration checklist and what the protocol cannot show

Checklist · 10 items

  1. Research question, both datasets with versions, and the target task named.
  2. Holdout source, sampling method, date range and deduplication method recorded.
  3. Budget unit and value, epochs allowed per arm, packing and truncation rules fixed.
  4. Recipe frozen, with the split used to tune it and the date it was frozen.
  5. Seed list and what each seed controls.
  6. Primary metric, grading method and regression checks with pass thresholds.
  7. Analysis method: paired differences, clustering variable, interval method.
  8. Minimum difference that would change the decision, and the decision for every outcome.
  9. Exclusion rules for failed or incomplete runs.
  10. Where the plan is stored, its date and who signed it off.

0 of 10 ticked. Tick each item as you go.

Using a matched budget test before you license data

The protocol answers a narrow question: which dataset gives the better model for this task at this budget on this base model. It does not show that a dataset will help a different model family, a much larger budget or a task the holdout does not represent, and a sample-based test cannot reveal problems that only appear in the full delivery. Evaluation rights for the sample also need to cover training a test model, which is a licensing question for the supplier agreement.

Where the data comes from businesses, SourceX sources it from U.S. companies on request rather than from stock; the request form takes scope, volume, language, format, timing and licensing needs, and the token budget and holdout plan from this protocol belong in that request.

Frequently asked questions

What is a matched budget experiment for training data?

It is a comparison in which two or more datasets are each used to train the same model with the same recipe and the same compute, usually measured in training tokens. Because only the data differs, any difference in held-out performance can be attributed to the data rather than to extra training.

Should I match the number of examples or the number of tokens?

Match tokens processed, and log examples, steps and estimated compute as well. Matching examples gives more compute to the dataset with longer examples. If the datasets use different sequence lengths, match estimated compute and state the formula used.

How many random seeds does each arm need?

There is no universal number. Run a pilot with a few seeds per arm, estimate how much the paired difference varies between seeds, and choose enough seeds to detect the minimum difference you wrote into the decision rule. Use the same seeds in both arms so the comparison can be paired.

What if one dataset is much smaller than the budget?

It will be repeated for several epochs while the larger one may be seen once. Record epochs per arm and treat repetition as part of what is being compared. If the purchase decision is about a larger budget, add a second budget so you can see whether the ranking holds.

Does this page report which kind of dataset wins?

No. It is a protocol for running the comparison on your own task and model. No results, effect sizes or recommendations about specific datasets are reported here.

Sources

  • DataComp (Gadre et al., 2023) holds the entire training code and computational budget constant so participants innovate by proposing new training sets, evaluated with standardized training and downstream tests. Source
  • DataComp-LM (Li et al., 2024) is a testbed for controlled dataset experiments with a standardized corpus, fixed pretraining recipes based on OpenLM at scales from 412M to 7B parameters, and 53 downstream evaluations. Source
  • Hoffmann et al. (2022) trained over 400 models and found that, for compute-optimal training, model size and the number of training tokens should be scaled equally. Source
  • Kaplan et al. (2020) estimate non-embedding training compute as C approximately equal to 6NBS, with N non-embedding parameters, B batch size and S training steps. Source
  • Muennighoff et al. (2023) find that with constrained data and a fixed compute budget, up to about four epochs of repeated data yields negligible changes to loss compared with unique data, and returns from more repetition eventually diminish to zero. Source
  • Dodge et al. (2020) vary only the random seed when fine-tuning BERT and find that weight initialization and training data order contribute comparably to the variance of out-of-sample performance. Source
  • Miller (2024) recommends standard errors for evals, clustered standard errors for related groups of questions, and inference on question-level paired differences when comparing two models. Source
  • The Center for Open Science describes preregistration as a plan made before the study that distinguishes what researchers set out to do (confirmation) from what they discovered along the way (exploration). Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies