AI uses for records
Dataset examples versus outcome labels: a 2x2 pilot before you buy
By SourceX Editorial · Updated
9 sections
Short answer
Choose between more dataset examples and better outcome labels with a pilot, not instinct. Once a failure is confirmed as a data problem, adjudicate the test labels so any gain is visible, then run a matched-budget 2x2: current data, added examples, repaired labels and both, scored on untouched tasks over repeated trials. Buy volume where the learning curve still climbs; repair labels where errors follow a pattern.
Grader rubric
Dataset examples
Key takeaways
- Added examples and repaired labels fix different defects: volume helps while the learning curve still rises, label repair when the recorded outcome is wrong in a pattern.
- Random label noise can be outvoted by more data at a cost; systematic noise, such as a status that records workflow rather than outcome, is reinforced by more data.
- Repair the test set first: Northcutt and colleagues found label errors averaging at least 3.3 percent across ten benchmark test sets, enough to change model rankings.
- A matched-budget 2x2 pilot isolates the marginal value of each fix and shows whether they interact.
- The task-to-data requirements brief turns the pilot's answer into a data request a supplier can accept or decline quickly.
More examples or better labels: what exactly are you deciding?
You are deciding where the next unit of data budget goes: more records of the same kind, or corrected outcomes on records you already hold. Both can raise accuracy, they cost different things and they fail for different reasons, so the choice should rest on a measured comparison rather than on whichever fix a vendor happens to sell.
Exact definitions keep the comparison honest. An example is one task instance with its inputs, such as a disputed invoice with its order, contract price schedule and customer emails. An outcome label is the result that instance is trained or graded against, such as credit issued, dispute rejected or escalated. Business systems record outcomes as statuses, and a status often records that a workflow step ended rather than what actually happened to the customer or the money.
Three measurements settle dataset examples versus outcome labels for a given task: the slope of the learning curve at current volume, the rate and pattern of label errors in an audited sample, and a pilot that spends the same budget on each fix. This article sets out how to collect all three and reports no results of its own.
First rule out tooling, retrieval and an unwritten spec
Confirm that the repeat failure is a data problem before buying any records. Many agent failures come from a missing tool, a stale retrieval index or a task nobody has defined precisely, and neither more examples nor cleaner labels repair those. The post-training task-selection guide covers that triage and how to choose which examples to buy; this page starts once the failure is known to be a data problem and asks whether the next budget should go to volume or to label repair.
Anthropic's guidance on agent evaluations makes two points that apply directly. Everything a grader checks should be clear from the task description, so an agent does not fail because the spec was ambiguous; and success should be judged on the final state of the environment rather than on the agent's claim, as in its example of a flight agent whose booking counts only if a reservation exists in the database. If a failure disappears once the spec is rewritten or the outcome is checked in the system of record, it was never a data shortage.
Reviewer disagreement is the quickest test for an unwritten spec. Scikit-learn's documentation describes Cohen's kappa as a statistic for comparing labelings by different human annotators, running from -1 to 1, with scores above 0.8 generally considered good agreement and zero or lower meaning no agreement; it compares two annotators at a time.
Sort the failing cases with the table before spending anything on data.
| Symptom on failing cases | Likely bottleneck | Quick test | Will buying data help? |
|---|---|---|---|
| The agent never sees the record it needs | Tooling or permissions | Replay the case with the record supplied by hand | No: fix access first |
| It cites an outdated policy or the wrong account | Retrieval | Swap in the correct document and rerun | No: fix the index or ranking |
| Qualified reviewers disagree on the right answer | Unwritten spec | Two reviewers label a sample blind; compute Cohen's kappa | Not yet: write the rule, then relabel |
| Right inputs, wrong decision on a recurring pattern | Too few examples of that pattern | Learning curve on that cohort | Yes, more examples, if the curve still rises |
| Right inputs, decision matches a wrong recorded outcome | Label errors | Audit a sample of labels against outcome evidence | Yes, repaired labels before more volume |
Why added volume and repaired supervision pay off on different curves
Added examples follow a curve of diminishing returns, while repaired labels remove an error that extra volume can only dilute when the error is random. Knowing which shape you are on is most of the decision.
Volume first. Hestness and colleagues reported in 2017, across machine translation, language modeling, image processing and speech recognition, that generalization error falls as a power law in training set size, with an exponent that has to be measured empirically for each problem. On a power law each doubling of data buys less than the one before. Scikit-learn's documentation calls the learning curve a tool for finding out how much a model benefits from adding training data, and in its naive Bayes example concludes that when training and validation scores converge at a low value, more data will probably not help much.
Label noise second, where the pattern matters more than the rate. Rolnick and colleagues showed in 2017 that deep networks can learn from training sets in which randomly labeled examples heavily outnumber clean ones, keeping MNIST test accuracy above 90 percent with 100 random labels for every clean one, but only with a dataset that grows with the dilution. Random noise can be outvoted by volume, at a price.
Systematic noise behaves differently. When the wrong label is the most common one for a type of case, as when a help desk rule marks silent tickets solved or a CRM records deals as won before late cancellations, every added example of that type repeats the wrong answer, so more volume reinforces the error instead of diluting it.
Repairing labels has its own evidence. Sheng, Provost and Ipeirotis found in their 2008 KDD paper that repeated labeling can improve both label quality and model quality, though not always; that when labels are noisy, relabeling can be preferable to labeling new examples even when labels are not particularly cheap; and that relabeling a selectively chosen subset does better still. In instruction tuning, the LIMA paper fine-tuned a 65-billion-parameter LLaMa model on 1,000 carefully curated examples and reported that scaling quantity without scaling prompt diversity brought diminishing returns, while improving quality brought major gains.
Adjudicate the test set before comparing either fix
Repair the evaluation labels first, because a noisy test set can hide a real gain or reward a wrong one. If the scoring key is wrong, the pilot measures agreement with old mistakes.
The size of the risk is documented. Northcutt, Athalye and Mueller estimated in 2021 that label errors make up at least 6 percent of the ImageNet validation set and average at least 3.3 percent across the test sets of ten widely used datasets, and showed that on corrected labels model rankings can change. Business statuses are unlikely to be cleaner than curated benchmarks, because they were recorded to move work along, not to grade a model.
Adjudication needs qualified people and blind review. OpenAI's GDPval shows a standard a buyer can borrow: tasks built by professionals averaging 14 years of experience, and expert graders who compared deliverables without knowing which came from a model. For a business task, that means reviewers who do the work, who see the outcome evidence but not the original status, with disagreements settled by a third reviewer and logged.
Keep the test tasks untouched: drawn from a later period than the training records, never used to tune either arm, and labeled before the pilot starts. A task that leaks into training in one arm invalidates the comparison for every arm.
Write the task-to-data requirements brief before buying
Write one task-to-data requirements brief per repeat failure, fixing the task, its evidence boundary, the label and the decision rule before any data is bought or relabeled. Writing it first stops a purchase from being sized by whatever a supplier happens to hold.
A brief like this is also the most useful data request a buyer can send. It states the record type, the outcome field and its evidence, the volume question still open and the acceptance test, which lets a supplier say quickly whether it holds records that fit.
The right-hand column is an illustrative entry for an invoice dispute triage assistant; it describes no real company, model or result.
| Field | What to write | Illustrative entry | Evidence that must exist |
|---|---|---|---|
| Repeat failure | The failing cohort, its share of cases and its cost | Disputes that cite contract pricing are routed to the wrong queue | A frozen list of failing case IDs with dates |
| Task inputs | Every record the decision may use | Invoice, order, contract price schedule, customer emails | Field list and the source system of each field |
| Evidence boundary | What the model may see at decision time, and what it may not | No credit memos or later emails, which reveal the outcome | Timestamps proving each input existed before the decision |
| Outcome label and source | The label definition and where it is recorded | Credit issued, dispute rejected or escalated, from the billing system | A written definition plus status history |
| Label audit | Error rate and pattern in a blind, adjudicated sample | 9 of 60 sampled rejections later reversed, most on pricing disputes | Adjudicated sample with reviewer agreement |
| Learning curve | Score at 25, 50 and 100 percent of current volume | Score flattens between 50 and 100 percent | Curve with intervals over repeated runs |
| Success test | Metric, threshold and untouched test set | Routing accuracy on 300 adjudicated later-period disputes | Test set frozen before the pilot |
| Decision rule | Which pilot result triggers which purchase | Buy volume only if the added-examples arm beats repair at equal hours | A pre-registered rule with its date |
Run the matched-budget 2x2 pilot
Vary the two factors independently at equal cost: example count, current or increased, and label quality, as recorded or repaired. Four arms then give the effect of each fix alone and show whether the two interact.
Matching the budget is the point of the design. Price both fixes in one unit, such as reviewer hours, and size the added records so that acquiring and labeling them costs the same as adjudicating the existing ones. The combined arm costs more than either single fix and answers a different question, whether the two together beat the sum of their parts, so compare the single-fix arms with each other and read the combined arm for interaction.
Repeat every arm across several training runs with different random seeds, and run each test task more than once when the system is stochastic. Anthropic's evaluation guidance calls each attempt at a task a trial and runs multiple trials because outputs vary. Report each arm's mean with an interval across runs, and fix the minimum gain that would justify a purchase before the first run.
Illustrative arithmetic, with invented numbers: if adjudicating an existing label takes a reviewer 4 minutes and acquiring and labeling a comparable new record takes 12, a budget of 200 reviewer hours repairs 3,000 labels or adds 1,000 records. The pilot then asks which spend moved the untouched score more.
The repair arm need not fix labels at random. In line with Sheng and colleagues' finding that selective relabeling works best, spend the repair budget on the labels most likely to be wrong: cases whose status changed late, cases where reviewers disagreed and cases where the current model confidently contradicts the label. Record that selection rule in the brief, because it is part of what the arm measures.
The four arms are set out in the table.
| Arm | Examples | Labels | What it isolates |
|---|---|---|---|
| A. Baseline | Current set | As recorded | The starting score and its spread across runs |
| B. More examples | Current set plus new records bought with the budget | As recorded | The marginal value of volume at current label quality |
| C. Repaired labels | Current set | Adjudicated with the same budget | The marginal value of supervision at current volume |
| D. Both | Current set plus the new records | Adjudicated, new records included | Whether the fixes add up, overlap or amplify each other |
Reading the pilot: four results and the purchase each justifies
Each result pattern points to a different next purchase, and one points away from data altogether. Decide which pattern counts as which before the runs, using the minimum gain fixed in the brief.
Treat a difference inside the run-to-run spread as no difference. The pilot measures one task, one model family and one data source, so repeat it when any of the three changes, and rerun it at the next volume step before a second large purchase.
The table maps each result to the purchase it supports and the risk to watch.
| Result | What it means | Next purchase | Watch for |
|---|---|---|---|
| B beats C beyond the agreed margin | Volume is still the constraint and labels are good enough | More records with the same label definition | The curve flattening at the next volume step |
| C beats B beyond the agreed margin | Systematic label errors cap the model | Label repair, or records with stronger outcome evidence | Adjudicator bias replacing status bias |
| D beats both by more than their sum | The fixes amplify each other | Both, in order: fix the label definition, then add volume | Budget creep between pilot steps |
| No arm beats A | The bottleneck is not training data | None; return to tooling, retrieval or the spec | Pressure to buy anyway |
What a matched-budget pilot cannot settle
The pilot shows which fix pays at today's volume for one task. It cannot show where the curve goes next, whether a supplier's records behave like yours or whether your adjudicators share a blind spot, and each of those needs its own check.
- Supplier records may differ. Field names, outcome codes and workflow rules change across companies, so a gain on your own logs is a hypothesis about a supplier's records until a delivered sample passes the same test.
- Adjudicators can carry bias. Repaired labels reflect reviewers' reading of the evidence, so report their agreement and keep the original status beside each repaired label.
- Small pilots are noisy. With few failing cases, run-to-run spread can swamp a real gain; widen the untouched test set before concluding that nothing worked.
- Results are task-specific. This article describes the protocol and reports no results; a finding on dispute routing says little about claims, tickets or contracts.
Turning the brief into a data request
Send the filled-in brief as the request. It tells a supplier whether you need volume, outcome evidence or both, and how delivered records will be judged.
SourceX does not sell from inventory: its search of U.S. businesses for suitable records begins only when a request-data form arrives, and that form takes licensing needs alongside scope, volume, language, format and timing. For an outcome-label question, add the label definition, the status history and linked evidence you need, the intended use and the acceptance test from your brief.
Fit is assessed request by request. A request does not mean matching records exist, and no dataset can promise that a model will improve; the pilot you run on a delivered sample is what shows whether it did.
Frequently asked questions
How do I know if my model needs more training data?
Plot a learning curve: train on increasing fractions of your current data and score each on the same untouched test set over repeated runs. If the score is still rising at full volume, more examples are likely to help; if it has flattened, look at labels, tooling or the task spec instead.
How much label noise is acceptable in a training dataset?
It depends on the pattern more than the rate. Random errors can be outvoted by more data, as Rolnick and colleagues showed with heavily noisy MNIST training sets, while errors that follow a pattern teach the model the wrong rule. Measure both the rate and the pattern on an adjudicated sample before deciding.
Should we relabel existing data or label new examples?
Relabel when audited errors are frequent or systematic, and label new examples when labels are clean and the learning curve still rises. Sheng, Provost and Ipeirotis found that when labels are noisy, relabeling, especially of a selected subset, can beat labeling new examples. A matched-budget pilot settles it for your own task.
Why fix the test set before the training set?
Because the test set is the scoring key. Northcutt and colleagues found label errors averaging at least 3.3 percent across ten benchmark test sets, enough to change model rankings, so an unadjudicated test set can hide a real gain or reward a wrong one.
Can SourceX supply records with verified outcomes?
Not from stock. SourceX looks for each dataset at U.S. businesses after a buyer describes scope, volume, format, timing and licensing needs and, for outcome questions, the label definition and evidence required; fit is assessed per request, with no promise that matching records exist.
Sources
- Anthropic's agent-evaluation guidance says everything a grader checks should be clear from the task description, grades outcomes on the environment's final state, as in a flight-booking example, and runs multiple trials per task because outputs vary. Source
- Scikit-learn's documentation describes Cohen's kappa as comparing labelings by different human annotators, from -1 to 1, with scores above 0.8 generally considered good agreement and zero or lower no agreement, for two annotators at a time. Source
- Scikit-learn's documentation describes the learning curve as a tool to find out how much a model benefits from adding training data, and says a model whose training and validation scores converge low will probably not benefit much from more data. Source
- Hestness and colleagues (2017) found across machine translation, language modeling, image processing and speech recognition that generalization error falls as a power law in training set size, with exponents measured empirically. Source
- Rolnick and colleagues (2017) showed deep networks keep MNIST test accuracy above 90 percent with 100 randomly labeled examples per clean example, given a dataset that grows with the dilution. Source
- Sheng, Provost and Ipeirotis (KDD 2008) found repeated labeling can improve label and model quality but not always, can be preferable to single labeling of new examples when labels are noisy, and works better when a subset is selected for relabeling. Source
- The LIMA paper fine-tuned a 65B LLaMa model on 1,000 curated prompts and responses and reported that scaling data quantity without prompt diversity gave diminishing returns while improving quality gave major gains. Source
- Northcutt, Athalye and Mueller (2021) estimated label errors of at least 6 percent in the ImageNet validation set and at least 3.3 percent on average across ten test sets, and showed corrected labels can change model rankings. Source
- OpenAI's GDPval tasks were built by professionals averaging 14 years of experience, and expert graders blindly compared model and human deliverables. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.