Skip to content

Deal economics

When to stop buying more of the same data: a stop rule experiment protocol

By SourceX Editorial · Updated

10 sections

Short answer

Stop buying more of the same data when a preregistered rule says the next tranche is unlikely to pay: buy in equal tranches, retrain with several seeds, score each step on an untouched holdout, and stop once the upper confidence bound of the marginal gain falls below the gain you said one tranche is worth. This page is a protocol for running that experiment; it reports no results.

Value drivers

Dataset acquisition stop

This page covers when to stop buying more of the same data: a stop rule experiment protocol.Illustrative

Key takeaways

  • Fix the smallest gain one more tranche must buy before any data arrives, and write the stop rule down with a date.
  • Buy in equal tranches of usable records and train nested arms, so each step measures exactly one tranche's marginal gain.
  • Add a repeat-compute control and an equal-volume different-data arm, or you cannot tell new information from more training or from the wrong slice.
  • Decide from paired, seed-averaged intervals on an untouched holdout, with a stricter level per look because every tranche is another look.
  • If the holdout cannot detect your threshold, the default decision is stop, not buy more.
  • This is a protocol only; it reports no results.

How do you decide when to stop buying more examples of the same task?

Decide before the data arrives. Write down the smallest improvement one more tranche must buy to be worth its total cost, then run a tranche-by-tranche experiment whose stop, continue and switch conditions are fixed in advance. The decision then follows from measured marginal gain and its uncertainty, not from how the last chart looks.

The protocol below is built for a buyer who already has a working pilot and is choosing whether to license another block of the same kind of records: the same task, the same source, the same labeling scheme. It treats each extra tranche as a treatment, compares it with the previous step on a holdout the supplier never sees, and adds two control arms that separate more data from more compute and from different data.

This is a method page. It names the question, arms, variables, metrics, uncertainty rules and limits, and ends with a pre-registration checklist. No results are reported, and every number shown is a design parameter or an illustrative placeholder, not a finding.

Why a flat-looking learning curve is not yet a stop decision

Learning curves are real, but reading them by eye mixes up four different things: the true return on more data, run-to-run noise, extra compute and duplicated content. A purchase decision needs those separated.

Hestness and colleagues found that generalization error falls as a power law in training set size across machine translation, language modeling, image processing and speech recognition, with an exponent that had to be measured for each domain. A power law can look flat on a linear axis long before it stops improving, so the shape alone does not say when to stop.

Muennighoff and colleagues report that, for a fixed compute budget, up to about four epochs of repeated data produced negligible changes in loss compared with unique data, while returns from further repetition shrank toward zero. A tranche full of near-duplicates of what you already own can behave like a repeated epoch rather than new information, which is exactly the case a stop rule should catch.

Run-to-run variance matters as much. Dodge and colleagues showed that, with everything else held fixed, changing only the random seed during fine-tuning moved out-of-sample results, with weight initialization and data order contributing comparably. A single run per tranche cannot tell a real gain from a lucky seed.

What is tested, and which arms isolate the value of more of the same data?

State the research question as one sentence a second analyst could test: for task T, scored by metric M on holdout H, does adding tranche k of records from source S improve M by at least the threshold θ over the model trained on tranches 1 to k minus 1?

The population is the stream of records the supplier would deliver under the current specification. Define it by the same fields you will use in the license: task family, source system, date range, label scheme and exclusions. If the supplier changes any of these between tranches, the experiment has a new population and the comparison breaks.

The unit of purchase is the tranche, a fixed count of usable records after deduplication, de-identification and quality filtering. Fix tranche size in usable records, not raw rows, because yield can differ between deliveries. Draw tranches at random from the delivered pool where you can, so later tranches are not systematically easier or harder than earlier ones.

Use nested training sets so each step adds exactly one tranche, and add three controls. All arms share the base model, hyperparameters, holdout and evaluation code; only the training data or the step count changes.

What is tested, and which arms isolate the value of more of the same data?
ArmTraining dataHeld constantWhat it answers
A0 baselineNo purchased data, or the pilot onlyModel, hyperparameters, holdout, scorerWhere the curve starts
A1 to Ak nested tranchesTranches 1 to k of the same source, cumulativeEverything except the added trancheMarginal gain of tranche k
R repeat controlTranches 1 to k minus 1, trained for extra steps to match the compute of AkData contentWhether gain came from more compute rather than new records
B different-data armTranches 1 to k minus 1 plus an equal-sized block from a different slice or sourceVolume and computeWhether the next tranche should buy different data instead
Seed replicatesEach arm retrained with at least three seedsData and settingsThe noise floor that any gain must clear

Which variables and metrics must be recorded at each step?

Record the inputs that drive cost and the outputs that drive value at the same grain, the tranche. A metric that is not tied to the business outcome you are paying for cannot set a stop threshold.

Which variables and metrics must be recorded at each step?
VariableDefinitionHow it is measuredRecorded when
Primary metric MVerified task success on the untouched holdout, defined before any trainingSame scorer and rubric for every armAfter each training run
Marginal gain ΔkMean paired difference in M between Ak and Ak minus 1, averaged over seedsItem-level pairing on identical holdout itemsAfter each tranche
Threshold θSmallest Δk worth the full cost of one tranche, in metric pointsSet by the business owner from cost and value of one pointBefore the first tranche
Tranche costLicense, preparation, review hours and compute for one trancheInternal cost units or hours, logged per trancheOn delivery
Usable yieldShare of delivered records surviving dedup, de-identification and quality checksCounts before and after each filterOn delivery
NoveltyShare of tranche records that are near-duplicates of earlier tranchesSame similarity method and cutoff every timeOn delivery
Protected slicesPer-slice M for slices that must not regressSame slice definitions as the holdoutAfter each training run

The preregistered marginal gain and cost stop rule

Write the rule as text before tranche 1 is delivered and file it with a date and version. Every term in it must already have a definition in the variable table, so nobody can reinterpret it after seeing the curve.

Compute Δk as the paired difference on each holdout item, average over seeds, and build the interval with a resampling method that respects the holdout's structure, for example resampling whole task groups rather than single items when items share a document, customer or case. Choose the interval level and the maximum number of tranches k max in advance, and, because every tranche is another look at the data, use a stricter level per look than you would for a single comparison.

The preregistered marginal gain and cost stop rule
Observed after tranche kConditionDecisionWhat to record
Clearly below the barUpper bound of the Δk interval is below θStop buying this source for this taskΔk, interval, θ, cost to date
Clearly above the barLower bound of the Δk interval is above θBuy tranche k plus 1 if k is below k maxΔk, interval, updated cost per point
AmbiguousInterval contains θBuy one more tranche only if fewer than the preset number of ambiguous steps have occurred; otherwise stopCount of ambiguous steps so far
Different data winsLower bound of B minus Ak at equal volume is above zeroSwitch the next tranche to the different slice or sourcePaired difference B minus Ak and interval
Compute explains itΔk interval overlaps the R minus Ak minus 1 differenceTreat the gain as compute, not data; do not credit the trancheBoth differences and intervals
RegressionAny protected slice falls by more than its preset tolerancePause purchases and inspect the trancheSlice, size of drop, suspected cause
Budget capk reaches k maxStop and report, whatever the curve showsFinal curve and all intervals

How large must the holdout and the replicate count be?

The holdout must be able to detect a change of size θ; if it cannot, no outcome of the experiment can justify continuing, and the rule should default to stop. Size it before buying anything, using the variance of paired item-level differences from the pilot.

Miller sets out how to treat evaluations as experiments: report a standard error for each score, use paired differences when two models answer the same questions, and use clustered standard errors when questions come in related groups. Those three choices map directly onto this protocol, since every arm answers the same holdout and business tasks often cluster by customer, document or case.

Seed replicates set the noise floor. Plan at least three seeds per arm as a starting design and increase the count if the pilot shows seed spread close to θ. Report the spread across seeds next to every Δk so a reader can see whether the gain clears it.

Keep the holdout untouched in the strict sense: drawn before any tranche is delivered, never shown to the supplier, never used for checkpoint selection or hyperparameter tuning, and checked for near-duplicates against every incoming tranche. Use a separate validation split for model selection.

What else could explain an apparent improvement?

Most false continue decisions come from a handful of confounders. The design controls each one with a specific arm or check, listed here so a reviewer can tick them off.

What else could explain an apparent improvement?
Alternative explanationWhat it looks likeControl in this protocol
Seed luckA one-off jump that does not repeatSeed replicates and paired intervals
More compute, not more dataGain also appears when old data is trained longerRepeat control arm R at matched compute
Holdout leakageLarge gains on items that resemble new tranche recordsNear-duplicate check of every tranche against the holdout
Selection on the holdoutGains shrink on a fresh sampleModel selection on a separate validation split only
Drifting tranche compositionLater tranches easier, newer or from a different subgroupRandom tranche assignment and per-tranche composition log
Label scheme driftMetric moves while task performance does notFrozen rubric, label audits on a sample of each tranche
Ceiling effectsGains vanish because the holdout is nearly solvedReport headroom; add harder held-out items before the first tranche, not mid-study

A worked walk-through with placeholder values

The following is illustrative only and reports no results. Suppose a team fine-tuning a model to classify support tickets by resolution path sets tranche size at 2,000 usable records, k max at 6 tranches, θ at 1 percentage point of verified routing accuracy, three seeds per arm and a holdout drawn from the most recent quarter of tickets before any purchase.

After each tranche the team trains A_k, R and B with three seeds, scores all of them on the same holdout and computes paired differences clustered by customer account. The table in the stop-rule section is applied mechanically: the analyst fills in the interval, reads the matching row and records the decision with the date, before anyone discusses whether the curve looks promising.

Two choices in this example carry most of the weight. The threshold θ comes from the business owner's estimate of what one point of accuracy is worth against one tranche's cost, expressed in the team's internal units, so it is the place where finance and the model team must agree before the experiment starts. The holdout window is chosen to match how the model will be used, so a gain on old ticket types cannot pass for a gain on current ones.

Pre-registration checklist and the limits of this protocol

The Center for Open Science describes preregistration as recording the plan before data collection so that hypothesis-testing analyses are kept apart from exploratory ones. In a purchase experiment that separation is what stops a team from choosing the convenient reading after the money is spent. Anything not on the filed plan is reported as exploratory.

At a glance

Pre-registration checklist and the limits of this protocol

The protocol has limits. It estimates the value of more records from one source for one task on one model; it does not show that other data would not do better, beyond the single different-data arm. Sorscher and colleagues show that choosing which examples to keep can change scaling behavior, so a stop decision on raw tranches says nothing about what a curated subset could achieve. Results also do not transfer to a different base model, objective or holdout definition without rerunning.

Checklist · 10 items

  1. Research question in one testable sentence, with task, metric, holdout and source named
  2. Population definition matching the license specification, including date range and exclusions
  3. Tranche size in usable records, k max and the random assignment method
  4. All arms, their training data and what each holds constant
  5. Primary metric, scorer version and rubric, frozen before tranche 1
  6. Threshold θ with the owner who set it and the cost basis behind it
  7. Interval method, clustering unit, level per look and the number of permitted ambiguous steps
  8. Holdout source, size, power to detect θ and the leakage check
  9. Protected slices and their regression tolerances
  10. Dated, versioned copy of the plan stored where the supplier cannot edit it

0 of 10 ticked. Tick each item as you go.

Writing the stop rule into a data request

A stop rule only works if the purchase can actually stop, so the license should be structured in tranches with the option, not the obligation, to buy the next one, and with delivery specifications that keep each tranche comparable. Share the population definition and tranche size with the supplier; keep the holdout and θ to yourself.

Teams that need operational records for a tranche plan can ask SourceX, which holds no stock and gathers each dataset from U.S. businesses once a request arrives; its request-data form takes the scope, volume, language, format, timing and licensing needs the plan depends on.

Frequently asked questions

What is a dataset acquisition stop rule?

It is a decision rule, written before the purchase starts, that says when to stop licensing more of the same data. It compares the measured gain from each new tranche, with its uncertainty, against the smallest gain that would justify one tranche's cost. Because it is fixed in advance, nobody can reinterpret the result after seeing it.

Is a flat learning curve enough reason to stop buying?

Not on its own. A power-law curve can look flat on a linear axis while still improving, and run-to-run noise can hide or fake a gain. Check the paired interval for the latest tranche against your threshold, and compare with a different-data arm before deciding.

How many training runs per tranche does the experiment need?

Plan at least three seeds per arm as a starting design, then increase the count if the pilot shows seed-to-seed spread close to your threshold. The point is to measure the noise floor, so the right number depends on how variable your model is on your task.

Should the supplier see the holdout set?

No. The holdout should be drawn before any tranche is delivered, kept away from the supplier and never used for model selection. Check every incoming tranche for near-duplicates of holdout items, because leakage is one of the most common reasons a tranche appears to help.

What if the holdout is too small to detect the threshold?

Then the experiment cannot justify buying more, and the preregistered default should be to stop or to enlarge the holdout first. Size the holdout from the variance of paired differences in the pilot before committing to any tranche.

Does this page report results from running the protocol?

No. It publishes the method only: question, arms, variables, metrics, uncertainty rules, confounders and a pre-registration checklist. Any numbers shown are illustrative design parameters, not findings.

Sources

  • Hestness et al. (2017) report that generalization error falls as a power law in training set size across machine translation, language modeling, image processing and speech recognition, with exponents measured per domain. Source
  • Muennighoff et al. (2023) find that, with constrained data and fixed compute, up to about four epochs of repeated data yields negligible changes to loss compared with unique data, and that returns from repetition eventually diminish to zero. Source
  • Sorscher et al. (2022) show that data pruning with a good pruning metric can in principle beat power-law error scaling, with supporting experiments on image benchmarks. Source
  • Dodge et al. (2020) fine-tune BERT varying only the random seed and find that weight initialization and training data order contribute comparably to variance in out-of-sample performance. Source
  • Miller (2024) treats language model evaluations as experiments and gives methods for standard errors, paired comparisons of two models on the same questions, and clustered standard errors. Source
  • The Center for Open Science describes preregistration as specifying the research plan before data collection, separating hypothesis-testing from hypothesis-generating research. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies