Skip to content

AI uses for records

Dataset examples versus outcome labels: a 2x2 pilot before you buy

By SourceX Editorial · Updated

9 sections

Short answer

Choose between more dataset examples and better outcome labels with a pilot, not instinct. Once a failure is confirmed as a data problem, adjudicate the test labels so any gain is visible, then run a matched-budget 2x2: current data, added examples, repaired labels and both, scored on untouched tasks over repeated trials. Buy volume where the learning curve still climbs; repair labels where errors follow a pattern.

Grader rubric

Dataset examples

This page covers dataset examples versus outcome labels: a 2x2 pilot before you buy.Illustrative

Key takeaways

  • Added examples and repaired labels fix different defects: volume helps while the learning curve still rises, label repair when the recorded outcome is wrong in a pattern.
  • Random label noise can be outvoted by more data at a cost; systematic noise, such as a status that records workflow rather than outcome, is reinforced by more data.
  • Repair the test set first: Northcutt and colleagues found label errors averaging at least 3.3 percent across ten benchmark test sets, enough to change model rankings.
  • A matched-budget 2x2 pilot isolates the marginal value of each fix and shows whether they interact.
  • The task-to-data requirements brief turns the pilot's answer into a data request a supplier can accept or decline quickly.

More examples or better labels: what exactly are you deciding?

You are deciding where the next unit of data budget goes: more records of the same kind, or corrected outcomes on records you already hold. Both can raise accuracy, they cost different things and they fail for different reasons, so the choice should rest on a measured comparison rather than on whichever fix a vendor happens to sell.

Exact definitions keep the comparison honest. An example is one task instance with its inputs, such as a disputed invoice with its order, contract price schedule and customer emails. An outcome label is the result that instance is trained or graded against, such as credit issued, dispute rejected or escalated. Business systems record outcomes as statuses, and a status often records that a workflow step ended rather than what actually happened to the customer or the money.

Three measurements settle dataset examples versus outcome labels for a given task: the slope of the learning curve at current volume, the rate and pattern of label errors in an audited sample, and a pilot that spends the same budget on each fix. This article sets out how to collect all three and reports no results of its own.

First rule out tooling, retrieval and an unwritten spec

Confirm that the repeat failure is a data problem before buying any records. Many agent failures come from a missing tool, a stale retrieval index or a task nobody has defined precisely, and neither more examples nor cleaner labels repair those. The post-training task-selection guide covers that triage and how to choose which examples to buy; this page starts once the failure is known to be a data problem and asks whether the next budget should go to volume or to label repair.

Anthropic's guidance on agent evaluations makes two points that apply directly. Everything a grader checks should be clear from the task description, so an agent does not fail because the spec was ambiguous; and success should be judged on the final state of the environment rather than on the agent's claim, as in its example of a flight agent whose booking counts only if a reservation exists in the database. If a failure disappears once the spec is rewritten or the outcome is checked in the system of record, it was never a data shortage.

Reviewer disagreement is the quickest test for an unwritten spec. Scikit-learn's documentation describes Cohen's kappa as a statistic for comparing labelings by different human annotators, running from -1 to 1, with scores above 0.8 generally considered good agreement and zero or lower meaning no agreement; it compares two annotators at a time.

Sort the failing cases with the table before spending anything on data.

First rule out tooling, retrieval and an unwritten spec
Symptom on failing casesLikely bottleneckQuick testWill buying data help?
The agent never sees the record it needsTooling or permissionsReplay the case with the record supplied by handNo: fix access first
It cites an outdated policy or the wrong accountRetrievalSwap in the correct document and rerunNo: fix the index or ranking
Qualified reviewers disagree on the right answerUnwritten specTwo reviewers label a sample blind; compute Cohen's kappaNot yet: write the rule, then relabel
Right inputs, wrong decision on a recurring patternToo few examples of that patternLearning curve on that cohortYes, more examples, if the curve still rises
Right inputs, decision matches a wrong recorded outcomeLabel errorsAudit a sample of labels against outcome evidenceYes, repaired labels before more volume

Why added volume and repaired supervision pay off on different curves

Added examples follow a curve of diminishing returns, while repaired labels remove an error that extra volume can only dilute when the error is random. Knowing which shape you are on is most of the decision.

Volume first. Hestness and colleagues reported in 2017, across machine translation, language modeling, image processing and speech recognition, that generalization error falls as a power law in training set size, with an exponent that has to be measured empirically for each problem. On a power law each doubling of data buys less than the one before. Scikit-learn's documentation calls the learning curve a tool for finding out how much a model benefits from adding training data, and in its naive Bayes example concludes that when training and validation scores converge at a low value, more data will probably not help much.

Label noise second, where the pattern matters more than the rate. Rolnick and colleagues showed in 2017 that deep networks can learn from training sets in which randomly labeled examples heavily outnumber clean ones, keeping MNIST test accuracy above 90 percent with 100 random labels for every clean one, but only with a dataset that grows with the dilution. Random noise can be outvoted by volume, at a price.

Systematic noise behaves differently. When the wrong label is the most common one for a type of case, as when a help desk rule marks silent tickets solved or a CRM records deals as won before late cancellations, every added example of that type repeats the wrong answer, so more volume reinforces the error instead of diluting it.

Repairing labels has its own evidence. Sheng, Provost and Ipeirotis found in their 2008 KDD paper that repeated labeling can improve both label quality and model quality, though not always; that when labels are noisy, relabeling can be preferable to labeling new examples even when labels are not particularly cheap; and that relabeling a selectively chosen subset does better still. In instruction tuning, the LIMA paper fine-tuned a 65-billion-parameter LLaMa model on 1,000 carefully curated examples and reported that scaling quantity without scaling prompt diversity brought diminishing returns, while improving quality brought major gains.

Adjudicate the test set before comparing either fix

Repair the evaluation labels first, because a noisy test set can hide a real gain or reward a wrong one. If the scoring key is wrong, the pilot measures agreement with old mistakes.

The size of the risk is documented. Northcutt, Athalye and Mueller estimated in 2021 that label errors make up at least 6 percent of the ImageNet validation set and average at least 3.3 percent across the test sets of ten widely used datasets, and showed that on corrected labels model rankings can change. Business statuses are unlikely to be cleaner than curated benchmarks, because they were recorded to move work along, not to grade a model.

Adjudication needs qualified people and blind review. OpenAI's GDPval shows a standard a buyer can borrow: tasks built by professionals averaging 14 years of experience, and expert graders who compared deliverables without knowing which came from a model. For a business task, that means reviewers who do the work, who see the outcome evidence but not the original status, with disagreements settled by a third reviewer and logged.

Keep the test tasks untouched: drawn from a later period than the training records, never used to tune either arm, and labeled before the pilot starts. A task that leaks into training in one arm invalidates the comparison for every arm.

Write the task-to-data requirements brief before buying

Write one task-to-data requirements brief per repeat failure, fixing the task, its evidence boundary, the label and the decision rule before any data is bought or relabeled. Writing it first stops a purchase from being sized by whatever a supplier happens to hold.

A brief like this is also the most useful data request a buyer can send. It states the record type, the outcome field and its evidence, the volume question still open and the acceptance test, which lets a supplier say quickly whether it holds records that fit.

The right-hand column is an illustrative entry for an invoice dispute triage assistant; it describes no real company, model or result.

Write the task-to-data requirements brief before buying
FieldWhat to writeIllustrative entryEvidence that must exist
Repeat failureThe failing cohort, its share of cases and its costDisputes that cite contract pricing are routed to the wrong queueA frozen list of failing case IDs with dates
Task inputsEvery record the decision may useInvoice, order, contract price schedule, customer emailsField list and the source system of each field
Evidence boundaryWhat the model may see at decision time, and what it may notNo credit memos or later emails, which reveal the outcomeTimestamps proving each input existed before the decision
Outcome label and sourceThe label definition and where it is recordedCredit issued, dispute rejected or escalated, from the billing systemA written definition plus status history
Label auditError rate and pattern in a blind, adjudicated sample9 of 60 sampled rejections later reversed, most on pricing disputesAdjudicated sample with reviewer agreement
Learning curveScore at 25, 50 and 100 percent of current volumeScore flattens between 50 and 100 percentCurve with intervals over repeated runs
Success testMetric, threshold and untouched test setRouting accuracy on 300 adjudicated later-period disputesTest set frozen before the pilot
Decision ruleWhich pilot result triggers which purchaseBuy volume only if the added-examples arm beats repair at equal hoursA pre-registered rule with its date

Run the matched-budget 2x2 pilot

Vary the two factors independently at equal cost: example count, current or increased, and label quality, as recorded or repaired. Four arms then give the effect of each fix alone and show whether the two interact.

Matching the budget is the point of the design. Price both fixes in one unit, such as reviewer hours, and size the added records so that acquiring and labeling them costs the same as adjudicating the existing ones. The combined arm costs more than either single fix and answers a different question, whether the two together beat the sum of their parts, so compare the single-fix arms with each other and read the combined arm for interaction.

Repeat every arm across several training runs with different random seeds, and run each test task more than once when the system is stochastic. Anthropic's evaluation guidance calls each attempt at a task a trial and runs multiple trials because outputs vary. Report each arm's mean with an interval across runs, and fix the minimum gain that would justify a purchase before the first run.

Illustrative arithmetic, with invented numbers: if adjudicating an existing label takes a reviewer 4 minutes and acquiring and labeling a comparable new record takes 12, a budget of 200 reviewer hours repairs 3,000 labels or adds 1,000 records. The pilot then asks which spend moved the untouched score more.

The repair arm need not fix labels at random. In line with Sheng and colleagues' finding that selective relabeling works best, spend the repair budget on the labels most likely to be wrong: cases whose status changed late, cases where reviewers disagreed and cases where the current model confidently contradicts the label. Record that selection rule in the brief, because it is part of what the arm measures.

The four arms are set out in the table.

Run the matched-budget 2x2 pilot
ArmExamplesLabelsWhat it isolates
A. BaselineCurrent setAs recordedThe starting score and its spread across runs
B. More examplesCurrent set plus new records bought with the budgetAs recordedThe marginal value of volume at current label quality
C. Repaired labelsCurrent setAdjudicated with the same budgetThe marginal value of supervision at current volume
D. BothCurrent set plus the new recordsAdjudicated, new records includedWhether the fixes add up, overlap or amplify each other

Reading the pilot: four results and the purchase each justifies

Each result pattern points to a different next purchase, and one points away from data altogether. Decide which pattern counts as which before the runs, using the minimum gain fixed in the brief.

Treat a difference inside the run-to-run spread as no difference. The pilot measures one task, one model family and one data source, so repeat it when any of the three changes, and rerun it at the next volume step before a second large purchase.

The table maps each result to the purchase it supports and the risk to watch.

Reading the pilot: four results and the purchase each justifies
ResultWhat it meansNext purchaseWatch for
B beats C beyond the agreed marginVolume is still the constraint and labels are good enoughMore records with the same label definitionThe curve flattening at the next volume step
C beats B beyond the agreed marginSystematic label errors cap the modelLabel repair, or records with stronger outcome evidenceAdjudicator bias replacing status bias
D beats both by more than their sumThe fixes amplify each otherBoth, in order: fix the label definition, then add volumeBudget creep between pilot steps
No arm beats AThe bottleneck is not training dataNone; return to tooling, retrieval or the specPressure to buy anyway
A decision tree from a repeat failure through two gates, whether it is a data problem and whether test labels are adjudicated, to a matched-budget 2x2 pilot with four outcomes and the purchase each supports.
From repeat failure to purchase: two gates, one pilot, four outcomes. A decision method, not a measured result.

What a matched-budget pilot cannot settle

The pilot shows which fix pays at today's volume for one task. It cannot show where the curve goes next, whether a supplier's records behave like yours or whether your adjudicators share a blind spot, and each of those needs its own check.

  • Supplier records may differ. Field names, outcome codes and workflow rules change across companies, so a gain on your own logs is a hypothesis about a supplier's records until a delivered sample passes the same test.
  • Adjudicators can carry bias. Repaired labels reflect reviewers' reading of the evidence, so report their agreement and keep the original status beside each repaired label.
  • Small pilots are noisy. With few failing cases, run-to-run spread can swamp a real gain; widen the untouched test set before concluding that nothing worked.
  • Results are task-specific. This article describes the protocol and reports no results; a finding on dispute routing says little about claims, tickets or contracts.

Turning the brief into a data request

Send the filled-in brief as the request. It tells a supplier whether you need volume, outcome evidence or both, and how delivered records will be judged.

SourceX does not sell from inventory: its search of U.S. businesses for suitable records begins only when a request-data form arrives, and that form takes licensing needs alongside scope, volume, language, format and timing. For an outcome-label question, add the label definition, the status history and linked evidence you need, the intended use and the acceptance test from your brief.

Fit is assessed request by request. A request does not mean matching records exist, and no dataset can promise that a model will improve; the pilot you run on a delivered sample is what shows whether it did.

Frequently asked questions

How do I know if my model needs more training data?

Plot a learning curve: train on increasing fractions of your current data and score each on the same untouched test set over repeated runs. If the score is still rising at full volume, more examples are likely to help; if it has flattened, look at labels, tooling or the task spec instead.

How much label noise is acceptable in a training dataset?

It depends on the pattern more than the rate. Random errors can be outvoted by more data, as Rolnick and colleagues showed with heavily noisy MNIST training sets, while errors that follow a pattern teach the model the wrong rule. Measure both the rate and the pattern on an adjudicated sample before deciding.

Should we relabel existing data or label new examples?

Relabel when audited errors are frequent or systematic, and label new examples when labels are clean and the learning curve still rises. Sheng, Provost and Ipeirotis found that when labels are noisy, relabeling, especially of a selected subset, can beat labeling new examples. A matched-budget pilot settles it for your own task.

Why fix the test set before the training set?

Because the test set is the scoring key. Northcutt and colleagues found label errors averaging at least 3.3 percent across ten benchmark test sets, enough to change model rankings, so an unadjudicated test set can hide a real gain or reward a wrong one.

Can SourceX supply records with verified outcomes?

Not from stock. SourceX looks for each dataset at U.S. businesses after a buyer describes scope, volume, format, timing and licensing needs and, for outcome questions, the label definition and evidence required; fit is assessed per request, with no promise that matching records exist.

Sources

  • Anthropic's agent-evaluation guidance says everything a grader checks should be clear from the task description, grades outcomes on the environment's final state, as in a flight-booking example, and runs multiple trials per task because outputs vary. Source
  • Scikit-learn's documentation describes Cohen's kappa as comparing labelings by different human annotators, from -1 to 1, with scores above 0.8 generally considered good agreement and zero or lower no agreement, for two annotators at a time. Source
  • Scikit-learn's documentation describes the learning curve as a tool to find out how much a model benefits from adding training data, and says a model whose training and validation scores converge low will probably not benefit much from more data. Source
  • Hestness and colleagues (2017) found across machine translation, language modeling, image processing and speech recognition that generalization error falls as a power law in training set size, with exponents measured empirically. Source
  • Rolnick and colleagues (2017) showed deep networks keep MNIST test accuracy above 90 percent with 100 randomly labeled examples per clean example, given a dataset that grows with the dilution. Source
  • Sheng, Provost and Ipeirotis (KDD 2008) found repeated labeling can improve label and model quality but not always, can be preferable to single labeling of new examples when labels are noisy, and works better when a subset is selected for relabeling. Source
  • The LIMA paper fine-tuned a 65B LLaMa model on 1,000 curated prompts and responses and reported that scaling data quantity without prompt diversity gave diminishing returns while improving quality gave major gains. Source
  • Northcutt, Athalye and Mueller (2021) estimated label errors of at least 6 percent in the ImageNet validation set and at least 3.3 percent on average across ten test sets, and showed corrected labels can change model rankings. Source
  • OpenAI's GDPval tasks were built by professionals averaging 14 years of experience, and expert graders blindly compared model and human deliverables. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies