Deal economics
Post-training data selection economics: cost per measured improvement
By SourceX Editorial · Updated
10 sections
Short answer
Compare dataset purchases by dividing each option's full acquisition cost by the lower bound of its net measured gain on one model failure: score a pilot slice against a matched random draw on untouched held-out business tasks, subtract any regression elsewhere, and rank options on cost per point of that conservative gain. Records priced cheaply per row can still cost the most per real improvement.
Value drivers
Post training data selection
Key takeaways
- Rank dataset purchases by cost per point of net measured gain on a named failure, not by price per record.
- Measure each option against a matched random draw at the same training budget, on untouched held-out business tasks.
- Use the lower bound of the gain interval and subtract severity-weighted regressions before dividing cost by gain.
- Options without a demonstrated gain are unproven, not cheap; blocking regressions disqualify an option outright.
- Nested pilot subsets show whether the next tranche will still pay back.
- Show which assumption would flip the ranking instead of presenting a single answer.
How do you compare dataset purchases by cost per measured improvement?
Rank candidate purchases by cost per point of net gain, where the gain is the lower end of the measured improvement on a specific failure cohort, after regressions, from a pilot run at a matched training budget. That number answers the question procurement actually faces: which option buys the most reliable fix for this failure, per unit spent.
It replaces two habits that mislead post-training teams. Price per record ignores that records differ wildly in how much they move the model, and headline benchmark gains ignore both uncertainty and what got worse. The worksheet below makes every input explicit, so finance, research and legal can argue about the same numbers.
The method has one precondition. You need a named failure, such as a model that misroutes invoice exceptions with partial receipts, measured on held-out real cases. Without a cohort and a baseline error rate, there is no improvement to price.
Why does price per record say little about post-training value?
The value of an additional example depends on what the model already knows and on which example it is, so equal record counts rarely buy equal improvements. Comparing quotes per row treats training data like a commodity, and post-training data does not behave like one.
Large empirical studies found that generalization error falls as a power law in training set size, with exponents that have to be measured for each domain. Each new slice of similar data buys less than the last. Work on data pruning goes further: with a good metric for ranking which examples to keep, error could in principle fall faster than a power law, which means selection can matter more than volume.
Post-training adds its own evidence. The LIMA study fine-tuned a 65-billion-parameter model on 1,000 carefully curated prompts and responses and reported strong results in human preference comparisons. Whatever one makes of the specific findings, the point for buyers is that a small, well-chosen set can outperform a much larger generic one, so the unit that matters is measured gain, not rows.
What goes into the cost and marginal gain worksheet?
Fill in one column per purchase option. Every row has a definition and a named place the number comes from; a cell without a source is a guess and should be marked as one.
| Input | Definition | Where the number comes from | Common mistake |
|---|---|---|---|
| Total acquisition cost | License or production cost plus preparation, review, compute for training and evaluation, and procurement and legal time | Quotes, internal hour estimates converted at a fixed internal rate, compute logs from the pilot | Counting only the quoted price and ignoring preparation and review hours |
| Target failure cohort | The held-out real cases where the current model fails in the way the purchase is meant to fix | Error analysis on the untouched evaluation set | Defining the cohort after seeing which option did well |
| Measured gain | Per-item improvement on the cohort versus a random draw of equal size from a comparison pool, at matched training budget | Pilot runs on a sample of the candidate data | Comparing against the untouched base model instead of a matched random draw |
| Gain interval | Lower and upper bounds on the measured gain from paired resampling across items and seeds | Bootstrap over cohort items, several seeds per arm | Reporting a single point estimate from one run |
| Regression penalty | Score losses on a fixed suite of capabilities that must not get worse, weighted by business severity | Regression suite run on every pilot checkpoint | Checking only the target cohort |
| Net gain lower bound | Lower bound of measured gain minus the regression penalty | Computed from the two rows above | Using the mean instead of the lower bound |
| Marginal gain per tranche | Additional gain from the next slice of the same source, estimated from pilot subsets | Learning curve over nested subsets of the pilot | Extrapolating a curve far beyond the measured range |
| Durability | Whether the gain is expected to survive the next base model or policy change | Re-run on a second base model, or a stated assumption | Assuming gains transfer without testing |
| Rights fit | Whether the license covers the planned training, retention and model distribution | Contract review | Comparing options with different usage rights as if equal |
Which formulas and gates decide the ranking?
Compute cost per point on the conservative gain, then apply gates before ranking. An option that fails a gate is not cheaper; it is ineligible until the gate is cleared.
At a glance
Which formulas and gates decide the ranking
- Net gain lower bound = lower bound of measured gain on the target cohort minus the severity-weighted regression penalty.
- Cost per point = total acquisition cost divided by the net gain lower bound, in points per hundred cohort cases or another unit tied to the business task.
- Marginal cost per point for the next tranche = cost of the next tranche divided by its estimated additional net gain.
- Gate 1: the net gain lower bound is above zero. If not, record the option as no demonstrated improvement; cost per point is undefined, not large.
- Gate 2: no regression on a capability marked blocking, regardless of the target gain.
- Gate 3: the net gain clears the minimum improvement the business decided in advance was worth acting on.
- Gate 4: the rights in the license cover the intended use. Options with narrower rights are compared only against each other.
- Decision: among options that pass all gates, prefer the lowest cost per point; if two options sit within each other's ranges, prefer the one with better durability or broader rights.
Why measure the gain at a matched budget rather than against the bare model?
Score each candidate against a random draw of the same size and training budget, from the same base checkpoint, on the same untouched cohort. Comparing to the bare model credits the purchase with whatever any additional fine-tuning would have achieved, and that inflates every option equally while hiding which data actually helped.
Keep the training recipe identical across arms: steps, tokens, learning rate schedule and hyperparameter search budget. The comparison pool for the random draw should be data you already have or could get cheaply, because the question is what this purchase buys beyond the easy alternative.
Uncertainty belongs in the number from the start. Benchmark research found that variance from data sampling, parameter initialization and hyperparameter choice noticeably affects results, so run several seeds per arm. Paired bootstrap resampling over the same cohort items, described for comparing machine translation systems, gives an interval for the per-item difference; the lower end of that interval is what goes into the worksheet.
Check overlap before trusting a large gain. Deduplication research found that train-test overlap affects over 4% of the validation sets of standard datasets. If a supplier's records include cases that also appear in your evaluation set, or near copies of them, the measured gain partly reflects memorization.
Which cohort improved, and did an existing capability regress?
Report the gain for the target cohort and for every regression check separately, then net them. A purchase that fixes invoice exceptions while degrading the model's handling of credit memos may still be worth buying, but only if that trade is visible and priced.
Regressions are not hypothetical. An empirical study of continual instruction fine-tuning observed catastrophic forgetting in models from 1 billion to 7 billion parameters, covering domain knowledge, reasoning and reading comprehension. A narrow post-training set can pull a model away from behavior it previously had.
Build the regression suite before the pilot and weight each item by business severity. Mark a short list as blocking, for example refusing valid requests or inventing account numbers, so that no amount of target gain can offset them. Everything else enters the regression penalty at its weight.
How do you estimate what the next tranche of data will add?
Train on nested subsets of the pilot data, such as a quarter, a half and all of it, and fit the gain against size to estimate what the next tranche adds. If the curve is already flat at the pilot's full size, buying ten times the volume of the same source is unlikely to pay back.
Use the fitted curve only near the measured range. Scaling research shows power-law improvement with data size but with exponents that must be found empirically; a curve fitted on three small points carries wide uncertainty once extended far beyond them. Report the marginal estimate with its range and mark extrapolations as such.
Marginal analysis also separates sources. A licensed set of operational records and a commissioned annotation project may show similar gains at pilot size but very different curves, because one keeps adding new case variety and the other repeats the same templates. The worksheet's marginal row exists to catch that before the full purchase.
How do three illustrative options for an invoice-exception failure rank?
The figures below are invented to show the arithmetic and are labeled illustrative. Costs are expressed as an index where option A equals 100, so no prices are implied. Gains are points of correct handling per 100 cases in the target cohort, with the interval from paired resampling.
- Ranking on mean gain alone would favor B, which shows the largest; ranking on cost per point of conservative net gain favors A.
- C looks cheapest per row but fails gate 1, so it is not a cheaper option, just an unproven one.
- The second tranche of A costs about six times more per point than the first, which argues for a smaller purchase or a different source for the next step.
| Option (illustrative) | Cost index | Measured gain and interval | Net gain lower bound after regressions | Cost index per point |
|---|---|---|---|---|
| A. Licensed operational exception records | 100 | 9 points (6 to 12) | 6 points; no regressions found | About 17 |
| B. Commissioned expert annotations | 140 | 10 points (7 to 13) | 5 points; 2-point loss on credit memo handling | 28 |
| C. Synthetic exceptions from a small real seed | 45 | 4 points (-1 to 9) | Below zero | Undefined; no demonstrated improvement |
| A, second tranche of equal size | 100 | 3 points (1 to 5) additional | 1 point | 100; curve flattening |
Which assumption changes the decision, and how should it be shown?
Show the decision under the assumptions most likely to flip it, rather than one ranking. In practice four inputs move rankings most: the width of the gain interval, the weight given to regressions, the internal hours for preparation, and whether the gain survives a change of base model.
Present a small sensitivity table beside the worksheet: the ranking with lower-bound gains, with mean gains, with the regression weight doubled, and with preparation hours raised by an amount the team considers plausible. If the same option wins in every row, the decision is robust. If the winner changes, the report should say which assumption decides it and what evidence would settle it, such as more seeds or a second base model.
Durability deserves its own line. Gains from data that teaches a rare, real decision pattern tend to be re-tested easily on the next model; gains that come from matching a grader's format may not survive a new base model. Re-running the pilot on a second base model is the cheapest durability test available.
How does a priced failure become a data request?
The worksheet's output is a precise request: the failure cohort, the case types that would fix it, the volume the marginal curve supports and the rights the intended use needs. That is the information a supplier needs to quote for a pilot slice rather than a catalog.
That request maps directly onto SourceX's request-data form, which asks for scope, volume, language, format, timing and licensing needs; SourceX then sources the pilot slice from U.S. businesses rather than quoting from an inventory. On sizing, SourceX's general guide is that useful sources hold two or more years of consistent history and tens of thousands of records in one system, while rare work can qualify with less, a range that fits a pilot-first purchase.
Frequently asked questions
Why use the lower bound of the gain instead of the average?
The lower bound reflects the improvement you can reasonably count on given run-to-run and item-level variance. Ranking on averages rewards noisy options that happened to score well once. Reporting both is fine, but the decision should rest on the conservative number.
What counts in total acquisition cost besides the license?
Preparation and cleaning hours, review and labeling, compute for pilot training and evaluation, and procurement and legal time. These often differ more between options than the quoted price does, especially between licensed records and commissioned annotation.
How large should the pilot be?
Large enough that the gain interval on the target cohort is narrower than the differences you need to detect between options. Nested subsets inside the pilot also give the marginal curve, so plan at least three sizes.
What if no option clears the minimum improvement?
Then the failure may not be fixable with more data of these kinds, or the cohort definition is wrong. Revisit the error analysis before buying; a prompt, tool or model change may be cheaper than any dataset.
Can this worksheet compare synthetic data with licensed records?
Yes, as long as both are scored at matched budget on the same untouched real cohort with the same regression suite. Synthetic options often look cheap per row, so the gates on demonstrated gain and regressions matter most for them.
Sources
- Deep learning generalization error improves as a power law in training set size, with exponents that must be measured empirically for each domain. Source
- With a high-quality metric for ranking which examples to discard, error could fall faster than power-law scaling. Source
- LIMA fine-tuned a 65-billion-parameter model on 1,000 carefully curated prompts and responses. Source
- Catastrophic forgetting was observed in language models from 1 billion to 7 billion parameters during continual instruction fine-tuning. Source
- Paired bootstrap resampling can estimate significance when comparing two systems on the same test items. Source
- Variance from data sampling, parameter initialization and hyperparameter choice markedly affects machine learning benchmark results. Source
- Train-test overlap affects over 4% of the validation sets of standard datasets, and deduplication reduces it. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.