AI uses for records
Undersized dataset buyer decision: focus, complement, grow or pass
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
An undersized dataset buyer decision starts with one question: undersized for what? Clean records below a training threshold can still clear the floor for an evaluation set, a narrow fine-tune or a gap in data the buyer already holds. Rule out rights, terms and task mismatch first, then test four routes in order: focus the use, complement existing data, grow the same source, or pass with a logged reason.
License terms
Undersized dataset buyer
Key takeaways
- A dataset is undersized only relative to a use and a unit; the same records can fail a training brief and pass as an evaluation set.
- Separate the statistical floor, too few examples to work or measure, from the contracting floor, too small to justify fixed deal costs.
- Published reference points span orders of magnitude: 20 to 50 tasks to start an agent eval, 1,000 curated examples in LIMA, tens of thousands of records for broad training.
- Judge a small dataset as a complement by its gap share, the fraction of records landing where the buyer's own data is thin.
- Ask for more of the same source only when a learning curve is still climbing; at shallow power-law exponents, halving error takes thousands of times the data.
Undersized for which use? The question behind a size rejection
A dataset is never undersized in general; it is undersized for a particular use, counted in a particular unit. The same 3,000 clean, well-documented claim decisions can be far too few to train a broad claims model, comfortably enough to build an evaluation set, and exactly what a buyer lacks in one segment of data it already holds.
That is why a size rejection deserves a second look before anyone walks away. Technically good here means the records already passed the agreed checks: the schema is consistent, required fields are filled, labels or outcomes are present and de-identification held. The open question is narrower. Does any use the buyer cares about have a floor this dataset clears?
Four answers are possible, and each needs different evidence. Focus the purchase on a smaller use the records can carry. Complement data the buyer already has, where a small set fills an empty cell. Grow the same source when more records would still improve the result. Or pass, with a logged reason that tells the supplier what would reopen the deal. The evidence that settles it is short: the intended use, the unit being counted, a floor derived from that use and the dataset's count in that unit.
Statistical floor or contracting floor: which one did the dataset miss?
A size rejection hides one of two different floors, and the remedy depends on which one the dataset missed. The statistical floor is the fewest examples a use needs to work or to be measured. The contracting floor is the smallest deal worth the fixed cost of rights review, contracting, security review, preparation setup and integration, costs that barely change whether the dataset holds 3,000 records or 300,000.
The two floors fail for different reasons. A dataset below the statistical floor cannot teach or test the behavior, so the fixes are a narrower use or more data. A dataset below only the contracting floor might help the model, but the paperwork costs more than the help is worth, so the fixes are commercial: bundle it with other records under one agreement, add it to a contract that already exists, or choose a lighter scope such as evaluation-only use.
One question separates them quickly. Ask the buyer whether it would take these exact records today if price and paperwork were negligible. A yes means the shortfall is economic, and more records will not change the answer as much as a cheaper route to signature. A no means the records are too few for the job, or that size is not the real objection at all.
Published minimums by use: where a clean dataset clears the bar
Published guidance puts useful floors orders of magnitude apart, depending on the use. Anthropic's January 2026 guide to agent evals says 20 to 50 simple tasks drawn from real failures is a great start, because early changes have large, visible effects. OpenAI's GDPval benchmark uses 1,320 expert-built tasks across 44 occupations, which works out to 30 per occupation, and releases a 220-task gold set. An OpenAI cookbook example on fine-tuning GPT-4o for visual question answering recommends at least 10 examples and says noticeable improvements typically come with 50 to 100.
At the other end, the LIMA paper (Zhou et al., 2023) fine-tuned a 65-billion-parameter model on only 1,000 carefully curated prompts and responses, and its authors argue that such tuning mostly teaches format while knowledge comes from pretraining. SourceX's own general guide for operational records is far higher: buyers typically want at least two years of consistent history and record counts in the tens of thousands from one system, though smaller sets can qualify when the work is rare, and every case is assessed deal by deal.
Read the table as reference points, not thresholds. The units differ, since tasks, examples and records are not interchangeable, and none of these figures says what a particular buyer will accept. Methods note: the figures come from the cited pages as checked on 2026-10-26; SourceX has not measured acceptance rates by dataset size, and no result here is a SourceX measurement. The table below lists each use, what gets counted, the published reference point and the sign that a dataset is truly undersized for it:
| Use | Unit counted | Published reference point | Undersized when |
|---|---|---|---|
| Capability or regression eval for an agent | Tasks with unambiguous success criteria | Anthropic, January 2026: 20 to 50 simple tasks from real failures is a great start | The change you must detect is smaller than the eval's confidence interval |
| Occupation-level benchmark | Expert-built tasks with graded deliverables | OpenAI GDPval, 2025: 1,320 tasks in 44 occupations, 220 in the gold set | A slice you plan to report has too few tasks for its interval |
| Narrow supervised fine-tune | Prompt and response examples | OpenAI cookbook: at least 10 examples, noticeable gains typically at 50 to 100 | The behavior varies more than the examples cover |
| Format or style alignment | Curated demonstrations | LIMA, 2023: 1,000 curated examples tuned a 65B model | The task needs new knowledge, not a new format |
| Broad training on operational records | Records from one system | SourceX general guide: 2+ years and tens of thousands of records; rare work can qualify smaller | The learning curve is still steep at the full dataset size |
Focus: re-scope the purchase to a use the records can carry
If the dataset clears the floor of a smaller use, re-scope the purchase to that use instead of rejecting it. The common landing spots are an evaluation set, a regression suite that guards behavior the model already has, a narrow fine-tune on one behavior and grounding material for retrieval.
Size the evaluation option with arithmetic, not instinct. On 300 tasks with a 50 percent pass rate, the standard error is about 2.9 points, so a 95 percent interval runs roughly 5.7 points either side. Comparing two unpaired runs, the difference has a standard error near 4.1 points: a 10-point change stands out, a 3-point change does not. Anthropic's statistical guidance for evals recommends paired comparisons, which narrow that interval, and power analysis to choose the number of questions in advance. It also warns that related questions cluster: clustered standard errors on popular evals can be more than three times naive ones, so 300 tasks drawn from 40 customers carry less information than 300 independent cases.
Focusing changes the license as well as the use. An evaluation-only grant is a different permission from a training grant, with its own limits on copies, retention and model use, so the new scope belongs in writing before records move. Check that the supplier can grant the narrower use at all; an evaluation-only use is usually narrower than training, but the agreement has to say so.
Complement: measure the gap the records fill, not their count
A small dataset can be worth buying when its records land in cells the buyer's existing data leaves nearly empty, so judge it by marginal coverage rather than standalone count. Three thousand records are undersized as a corpus and generous as a patch for a segment where the buyer holds 300.
The test needs no record-level exchange. The buyer cross-tabulates its own data by the dimensions that matter for the task, such as segment, decision type, region, language and period, and marks the cells below its floor. The supplier reports counts in the same cells from a manifest. The gap share is the fraction of the supplier's records that fall in below-floor cells. Illustrative: a buyer with 40,000 claims but only 300 commercial auto claims is offered 3,000 records, 1,800 of them commercial auto, for a gap share of 60 percent. A gap share near zero means the records mostly duplicate what the buyer has.
Then confirm the gap matters. Evaluate on the gap slice with and without the new records, using the paired method above, before paying for the whole set. Pooling with other suppliers to reach a floor is the other complement route, and it works only when field definitions, outcome labels and license terms line up across sources; otherwise the pool is three small datasets that disagree.
Grow: ask for more of the same source only if the curve still climbs
Ask the same supplier for more records, such as more years, more sites or ongoing deliveries, only when a learning curve shows the metric still improving at the full dataset size. A flat curve means more rows from the same process buy little, and the money is better spent on a complement or a different source.
Build the curve cheaply. Train or evaluate at 25, 50 and 100 percent of the dataset using nested subsets, repeat each with three random seeds and fit the slope of log error against log size. Hestness et al. (2017) found that generalization error improves as a power law of training set size, with exponents between about -0.07 and -0.35 across machine translation, language modeling, image classification and speech, and that architecture changes shifted the curves without changing the exponent.
Those exponents make the stakes concrete. At -0.35, doubling the data cuts error by about 21 percent and halving error takes roughly 7 times the data. At -0.07, doubling cuts error by under 5 percent and halving it would take about 20,000 times the data. Repetition is a partial substitute for modest shortfalls: in pretraining experiments, Muennighoff et al. (2023) found up to four epochs of repeated data changed loss negligibly compared with unique data, with returns fading toward zero after that. That is a pretraining result rather than a fine-tuning rule, and it does not rescue a dataset ten times too small.
Is size the real objection? Rights, terms and task fit come first
Sometimes too small is the politest reason available, and no number of extra records will fix a rights gap, a price disagreement or a task the records do not represent. Test that possibility before spending effort on any of the three routes.
Three signals point away from size. The buyer never stated a count or a unit. The floor moves when price or scope moves. The sample review raised missing fields, missing outcomes or unclear provenance rather than volume. Rights documentation deserves particular suspicion: when the Data Provenance Initiative went through more than 1,800 text datasets, license information was missing for about 70 percent of those on popular hosting sites and incorrect for more than half, so weak paperwork is common even in widely used data.
Two written questions settle most cases. What count, in what unit, would have passed? And would the same records at twice the size pass the rest of your review? If the second answer is no, log the rejection under rights, terms or task fit and route it to the people who own those problems; remediating size would solve nothing.
The undersized-dataset decision tree and size reason log
Run the decision as five ordered questions, then log one coded reason so the next offer starts from evidence rather than memory. The order matters: settling whether size is the real issue first prevents a team from enlarging a dataset that was never going to clear rights or terms.
Each code names the route and the event that would reopen the case. A size rejection logged without its unit and floor cannot be reopened by anyone, because no one can tell whether new records would change it. The decision tree and the size reason log follow:
- 1. Is size the real objection? If the same records at twice the size would still fail, stop and log the rights, terms or task-fit reason instead.
- 2. Which floor was missed? If the buyer would take the records with negligible price and paperwork, the shortfall is the contracting floor: bundle, amend an existing agreement or narrow the scope.
- 3. Does a smaller use clear its floor? If an evaluation set, regression suite or narrow fine-tune fits the count, re-scope and re-paper the license for that use.
- 4. Do the records fill a gap the buyer has? If the gap share is material and a paired test on the gap slice improves, buy the records as a complement.
- 5. Is the learning curve still climbing at full size? If yes, ask the supplier for more history, sites or deliveries; if no, pass and log the reason.
| Code | Evidence that triggers it | Route | What would reopen it |
|---|---|---|---|
| S1 Below floor for the intended use, clears a smaller one | Count in the agreed unit versus the floor for each candidate use | Focus: re-scope and re-paper the grant | A new use whose floor the dataset clears |
| S2 Below floor, fills a buyer gap | Gap share from manifest counts; paired test on the gap slice | Complement: buy for the gap only | The buyer's own coverage in that cell changes |
| S3 Below floor, curve still climbing | Slope across 25, 50 and 100 percent nested subsets | Grow: more history, sites or deliveries | The supplier can add records under the same rights |
| S4 Above statistical floor, below contracting floor | Buyer would take the records with negligible price and paperwork | Bundle, amend an existing agreement or narrow scope | A cheaper route to signature appears |
| S5 Size cited, another cause found | Twice the records would still fail review | Reroute to the rights, terms or task-fit log | That cause is resolved, not more records |
Writing a size floor and its reason into the data request
The cheapest size rejection is the one avoided by stating the floor up front. A request that names the use, the unit, the floor and why the floor exists lets a supplier say early whether its records can meet it, or which smaller use they could serve.
SourceX has no inventory to draw from; it sources each dataset from U.S. businesses when a request arrives, and the request-data form asks the buyer for volume, scope, language, format, timing and licensing needs. A useful request adds four lines to those fields: the acceptance test, the minimum count in its unit, whether that floor is statistical or contractual, and whether an evaluation-only scope or a complement to existing data would also be acceptable. Nothing in that request implies that matching records exist, that a dataset will be accepted or that it will improve a model; those questions are answered deal by deal.
For suppliers, the same logic runs in reverse. When a buyer says a dataset is too small, ask for the unit and the count that would have passed, then check the decision tree above before offering more records, a narrower use or a different scope.
Frequently asked questions
Is there a minimum dataset size for AI training?
No single minimum exists, because the floor depends on the use and on what is counted. An OpenAI cookbook example asks for at least 10 fine-tuning examples, Anthropic suggests 20 to 50 tasks to start an agent eval, and SourceX's general guide for broad training on operational records is tens of thousands of records over two or more years. Agree the use and the unit before arguing about the number.
Can a small dataset be worth more to a buyer than a large one?
Yes, when its records land where the buyer's existing data is thin. Measure the gap share, the fraction of offered records falling in segments below the buyer's floor, then confirm with a paired test on that slice. A large dataset that repeats what the buyer already holds can add less than a small one that fills a gap.
How many tasks does an evaluation set need?
Enough that the confidence interval is narrower than the change you need to detect. On 300 tasks at a 50 percent pass rate, the 95 percent interval is about plus or minus 5.7 points, which can show a 10-point improvement but not a 3-point one. Anthropic recommends paired comparisons and power analysis, and warns that clustered questions widen intervals.
Should we repeat a small dataset for more epochs instead of buying more records?
Repetition helps only with modest shortfalls. In pretraining experiments, Muennighoff et al. (2023) found up to four epochs of repeated data changed loss negligibly compared with unique data, with returns fading after that. It is not a fine-tuning rule and does not make up for a dataset many times too small.
What should a supplier do when a buyer says its dataset is too small?
Ask which unit was counted and what count would have passed, then ask whether twice the records would pass the rest of the review. If not, size is not the real reason. If it is, offer a narrower use, more history or sites under the same rights, or a complement to the buyer's existing data.
Sources
- Anthropic's guide to agent evals (January 9, 2026) says 20 to 50 simple tasks drawn from real failures is a great start, because early changes have large effects, and that a good task is one two domain experts would grade the same way. Source
- Anthropic's statistical guidance for model evals recommends paired-difference analysis and power analysis, notes that small evals give wide confidence intervals, and reports clustered standard errors on popular evals more than three times naive ones. Source
- An OpenAI cookbook example on fine-tuning GPT-4o for visual question answering recommends at least 10 examples and says noticeable improvements typically come with 50 to 100 training examples. Source
- OpenAI's GDPval spans 44 occupations with 1,320 expert-built tasks and an open-sourced gold set of 220 tasks. Source
- LIMA (Zhou et al., 2023) fine-tuned a 65B LLaMa model on 1,000 carefully curated prompts and responses and proposed that alignment mainly teaches format while knowledge comes from pretraining. Source
- Hestness et al. (2017) found generalization error improves as a power law of training set size, with exponents of about -0.07 to -0.35 across four domains, and that architecture changes shift learning curves without changing the exponent. Source
- Muennighoff et al. (2023) found that in data-constrained pretraining, up to four epochs of repeated data yield negligible changes to loss compared with unique data, with returns diminishing toward zero after that. Source
- The Data Provenance Initiative audited more than 1,800 text datasets and found license information frequently missing (about 70 percent) and wrong (more than half) on popular dataset hosting sites. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.