AI data market
How to test whether 200 targeted tasks beat 20,000 generic ones
By SourceX Editorial · Updated
10 sections
Short answer
Whether 200 carefully chosen business tasks can beat 20,000 generic ones is an empirical question for each model and failure, and published work on small curated sets does not settle it for your case. This small targeted dataset experiment protocol shows how to test it: a pre-registered, matched-budget comparison with size-matched random controls, untouched held-out business tasks, regression checks and seed-repeated, paired uncertainty. No results are reported.
Value drivers
Small targeted dataset
Key takeaways
- Published results on small curated sets motivate the test but do not answer it for a specific business failure.
- Compare targeted and generic arms at a matched budget in loss tokens or steps, not only example counts.
- Size-matched random controls separate the selection rule from the data source.
- Fixed regression and safety suites catch fixes that break existing capabilities.
- Repeat every arm across seeds and report paired, clustered intervals per error cohort.
Can 200 carefully selected business tasks outperform 20,000 generic tasks?
They can in principle, and the research literature gives reasons to test it, but nothing published tells you whether they will for your model and your failure. The answer has to come from a controlled experiment on held-out business tasks, and this page gives the design for that experiment rather than a result.
Several papers report small, selected datasets matching or beating much larger ones under their conditions. LIMA fine-tuned a 65B-parameter LLaMA model on 1,000 curated prompts and responses. AlpaGasus filtered the 52,000-example Alpaca set down to about 9,000 examples using a model-based quality score and reports that the filtered model outperformed the model trained on the full set, and also outperformed a random 9,000-example subset. LESS reports that training on a 5% subset selected for a target capability can often outperform training on the full dataset.
Those findings are for general instruction following and public benchmarks. A business failure, such as a model that misroutes invoices with split shipments or applies an outdated refund rule, is narrower and has its own regression risks. The protocol below is a small targeted dataset experiment built for that situation. The 200 and 20,000 in the question are the illustrative sizes used throughout; substitute your own.
What question does the matched-budget experiment answer?
The experiment asks whether a small set of tasks selected to fix a specific model failure improves held-out performance on that failure more than a large generic set does, at a matched training budget, without degrading capabilities the model already had. All three parts of that sentence are measured; a gain on the target that comes with a regression elsewhere is reported as a trade-off, not a win.
At a glance
What question does the matched-budget experiment answer
Start from a specific failure, not from a dataset. Build error cohorts from evaluation runs or production logs, write a definition for each cohort that a person can apply consistently, and set aside held-out examples of each cohort before any selection happens. The selection rule then picks training tasks that match the cohort definitions from a separate pool.
- Primary estimand: difference in held-out success on the target error cohorts between the targeted arm and the generic arm at matched budget.
- Selection estimand: difference between the targeted arm and a random sample of the same size from the same business pool, which isolates the selection rule from the data source.
- Regression estimand: change on a fixed general capability suite and a safety suite relative to the base model.
- Interaction estimand: whether adding the targeted set to the generic set beats either alone.
Preregistered matched-budget experiment plan: the arms
Every arm starts from the same base checkpoint with the same hyperparameter search space, and every arm is scored on the same untouched held-out tasks. The arms differ only in the training data, and two of them exist purely as controls.
AlpaGasus shows why the size-matched random control matters: comparing a filtered subset with the full set alone cannot separate the effect of selection from the effect of training on less data, so the paper also trained on a random subset of the same size. The same logic applies here with business tasks.
| Arm | Training data | Budget rule | What it isolates |
|---|---|---|---|
| A. Base | None | No training | Starting point for every comparison |
| B. Targeted | 200 business tasks chosen by the registered selection rule | Matched training budget, with validation-based early stopping | Effect of selected business tasks |
| C. Generic | 20,000 generic instruction tasks | Matched training budget | Effect of large generic data |
| D. Random business | 200 tasks sampled at random from the same business pool as B | Same as B | Whether the selection rule beats unselected business data |
| E. Random generic | 200 tasks sampled at random from C's pool | Same as B | Whether business data beats generic data at equal size |
| F. Combined | C plus B | Matched training budget | Whether targeted tasks add value on top of generic data |
| G. Natural budget | B and C each trained to their own validation optimum | Unmatched, reported separately | Practical outcome when each set is trained the way a team would actually train it |
How do you match the budget fairly?
Match the training budget in tokens that receive loss, or in optimizer steps at a fixed batch size, and state which one before training. Matching only the number of examples hides large differences: 200 long business tasks and 200 short generic prompts are not the same amount of training.
A matched budget creates a known tension. To spend the same budget, the 200-task arm must see its data for many more epochs than the 20,000-task arm, which risks memorization. The protocol handles this by registering two readings: the matched-budget comparison (arms B, C, F) with early stopping on a validation split, and the natural-budget comparison (arm G) where each set is trained to its own validation optimum. If the two readings disagree, report both.
Acquisition cost is a separate axis. Curating 200 business tasks from real records may cost more per task than generating generic instructions, so a team that cares about total cost should record hours and spend per arm alongside the training budget, without folding it into the primary estimand.
What makes the selection rule and evaluation trustworthy?
Write the selection rule before seeing any held-out results, apply it mechanically and keep the held-out tasks out of its reach. A rule that is adjusted after looking at test performance turns the experiment into tuning on the test set.
At a glance
What makes the selection rule and evaluation trustworthy
Selection can be manual, rule-based or model-based. LESS is a model-based example: its pipeline runs a short warmup training on a small portion of the data, builds a gradient datastore for the training pool and selects the training examples whose gradients are most similar to those of examples from the target task. In LESS those target examples come from a validation set. In this protocol the equivalent must come from a validation split of the error cohorts, never from the held-out evaluation tasks.
- Pools: separate business pool, generic pool, validation split and held-out evaluation set, with record-level lineage for each.
- Selection rule: written, versioned and applied by script or by a reviewer who cannot see evaluation results.
- Leakage control: near-duplicate checks between every training pool and the held-out set, with the method and threshold recorded.
- Outcome evidence: for each held-out business task, the recorded correct outcome from the source system, such as the approved invoice routing or the final refund amount, rather than a judge's opinion where an outcome exists.
- Lineage: every training task in every arm traceable to its source record, selection score and arm assignment.
Which regression checks catch a hidden cost?
Run a fixed general capability suite and a safety suite on every arm, because targeted fine-tuning can fix one failure while breaking something else. Two lines of research make these checks non-optional.
Luo and colleagues studied continual instruction fine-tuning of models from 1B to 7B parameters and report that catastrophic forgetting of domain knowledge, reasoning and reading comprehension is generally observed. Qi and colleagues report that fine-tuning aligned models on benign, commonly used datasets can degrade safety alignment, though less than fine-tuning on adversarial examples. A small, many-epoch targeted arm is exactly the setting in which both risks deserve measurement.
Choose the regression suites before training and keep them fixed across arms. Include at least one cohort of business tasks that the base model already handles well, since that is where a narrow fine-tune most often causes visible damage to users.
How many seeds and which intervals separate a real win from noise?
Repeat every arm across several seeds and report paired, clustered intervals on item-level differences. Dodge and colleagues found that changing only the random seed when fine-tuning pretrained language models produced substantial swings in results, which matters even more when the training set has only 200 examples.
At a glance
How many seeds and which intervals separate a real win from noise
Miller recommends inference on question-level paired differences when comparing two models evaluated on the same questions, and clustered standard errors when questions come in related groups. Business tasks are usually clustered: several held-out tasks may come from the same customer, vendor or ticket thread, so cluster by that unit.
Size the held-out set so each error cohort has enough tasks to estimate a difference at the precision the decision needs, and record the denominator per cohort. If a cohort is too small to support a conclusion, say so rather than merging it into the overall metric.
- Baselines: arm A for absolute change; arms D and E as size-matched controls; arm C as the comparator named in the question.
- Seeds: the same seed list for every arm, with seed spread reported alongside intervals.
- Confirmatory comparisons: B versus C on target cohorts, B versus D, and regression change for B; everything else is exploratory.
- Reporting: per-cohort results, including cohorts where the targeted arm lost.
Worked setup: an invoice routing failure (illustrative)
This setup is illustrative and describes no real company, model or result. An accounts payable assistant routes most invoices correctly but misroutes invoices that arrive with a split shipment, where one purchase order is received in two goods receipts. The team writes one error cohort for split-shipment invoices and a second for invoices whose approver changed after the purchase order was raised.
Before any selection, the team samples held-out invoices for both cohorts from the most recent quarter, plus a third cohort of ordinary invoices the base model already routes well, and locks them. A validation split of each cohort is drawn from the previous quarter. The selection rule picks 200 closed invoices from earlier records whose receipts and approval history match either cohort definition, with the final routing from the source system as the target.
The generic arm uses 20,000 instruction examples from a commercially usable general pool, and the random controls draw 200 examples from each pool. All arms are trained from the same checkpoint over the same seed list, with the budget matched in loss tokens. The team records the result per cohort, including the ordinary-invoice cohort, whatever it shows, and treats a gain on split shipments that comes with a loss on ordinary invoices as a trade-off to decide on, not a success.
Pre-registration checklist for a small targeted dataset experiment
Register the plan before selecting data or training any arm. The Center for Open Science describes preregistration as posting a time-stamped, read-only version of a study plan before data collection or analysis begins; a private, dated and access-controlled document serves the same purpose inside a company.
At a glance
Pre-registration checklist for a small targeted dataset experiment
Checklist · 10 items
- The failure being targeted, its error cohort definitions and how cohort membership is decided.
- Held-out evaluation set, validation split and their denominators per cohort, sampled before selection.
- Selection rule for the targeted arm, with version, inputs and the reviewer or script that applies it.
- Sources and sampling for the generic pool and the business pool.
- Full arm list with sizes, including random controls D and E and combined arm F.
- Budget definition (loss tokens or optimizer steps), early stopping rule and hyperparameter search space.
- Primary metric and scoring script version; outcome evidence used as ground truth.
- Regression and safety suites, fixed before training.
- Seed list, pairing, clustering unit, interval method and confirmatory comparisons.
- Exclusion rules for failed runs and the record of any deviation from the plan.
0 of 10 ticked. Tick each item as you go.
What are the limits, and where do real business tasks come from?
A result from this protocol applies to one base model, one failure and one selection rule. It does not show that small targeted data beats large generic data in general, and a positive result at 200 tasks does not predict the result at 2,000 without testing that size. The published studies cited here are motivation for running the experiment, not evidence about its outcome.
The targeted arm depends on real business tasks with recorded outcomes: closed tickets with their resolution, invoices with their final routing, claims with their decision. Those records need rights that cover training, and access to them is a separate question from privacy review and from licensing.
For the targeted arm, SourceX can source closed business tasks with recorded outcomes from U.S. companies; it works request by request rather than from a ready-made catalog. The request names scope, volume, language, format, timing and licensing terms, for example a few hundred split-shipment invoices with their final routing, and the supplier approves scope and terms and signs before anything is delivered.
Frequently asked questions
Does LIMA prove that small datasets beat large ones?
No. LIMA showed that a 65B-parameter model fine-tuned on 1,000 curated examples performed well in the authors' evaluation of general assistant behavior. It does not establish the result for a specific business failure, a different model or a different selection rule, which is why the protocol tests it directly.
What should the preregistered matched-budget experiment plan include?
It should name the target failure and its error cohorts, the held-out set, the selection rule, all arms including size-matched random controls, the budget definition, the primary metric, the regression and safety suites, the seed list and the uncertainty method. Every item is fixed before data selection or training.
Why include a random 200-task arm?
A random sample of the same size from the same business pool separates the effect of the selection rule from the effect of using business data at all. Without it, a win for the targeted arm could come from the data source rather than the selection.
How do you know whether an existing capability regressed?
Run the same fixed general capability suite and safety suite on the base model and on every arm, and report the paired change with intervals. Include a cohort of business tasks the base model already handled well, since narrow fine-tuning often damages those first.
Should budget be matched by examples or by tokens?
Match by tokens that receive loss or by optimizer steps at a fixed batch size, and register the choice. Matching by example count alone gives longer examples more training signal and can decide the comparison by accident.
Does this page report results?
No. It publishes the design of the experiment only. The sizes 200 and 20,000 are illustrative and should be replaced with the sizes a team can actually source and train.
Sources
- LIMA fine-tuned a 65B-parameter LLaMA model with a standard supervised loss on 1,000 carefully curated prompts and responses, without reinforcement learning or human preference modeling. Source
- AlpaGasus filtered the 52k Alpaca data to about 9k examples with a model-based quality score; the authors report that the 9k model outperformed the 52k model and a random 9k subset. Source
- LESS reports that training on a 5% subset of data selected for a target capability can often outperform training on the full dataset. Source
- The LESS repository describes a pipeline of warmup training on a small portion of the data, building a gradient datastore, and selecting training examples by influence scores computed against validation data for the target task. Source
- Qi and colleagues report that fine-tuning aligned language models on benign, commonly used datasets can degrade safety alignment, though to a lesser extent than adversarial fine-tuning. Source
- Luo and colleagues report that catastrophic forgetting is generally observed when models from 1B to 7B parameters undergo continual instruction fine-tuning, across domain knowledge, reasoning and reading comprehension. Source
- Miller recommends clustered standard errors when questions come in related groups and inference on question-level paired differences when comparing two models. Source
- Dodge and colleagues found that changing only the random seed when fine-tuning pretrained language models produced substantial variation in results. Source
- The Center for Open Science describes preregistration as posting a time-stamped, read-only version of a study plan to a repository before data collection or analysis. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.