Skip to content

Systems and records

Business outcome label reliability study: how status fields skew AI scores

By SourceX Editorial · Updated

10 sections

Short answer

A business outcome label reliability study compares the status a system recorded, such as solved, approved or paid, with the outcome independent evidence shows, at fixed maturity windows. Publish a status-to-outcome confusion table that keeps censored and unverifiable cases visible, code every mismatch, then rescore the AI evaluation on corrected labels. Independent label errors shrink gaps between systems; errors that match a learned shortcut can reverse rankings.

Grader rubric

Business outcome label

This page covers business outcome label reliability study: how status fields skew AI scores.Illustrative

Key takeaways

  • A status records that a workflow step ended; the audit tests whether it also records the outcome an evaluation needs.
  • Keep censored and unverifiable cases in the confusion table, or recent cases will look cleaner than they are.
  • Mismatch codes such as premature closure and delayed reversal turn an error rate into a repair list.
  • Independent label errors compress score gaps by a factor of 1 minus 2e; errors aligned with a shortcut can flip rankings.
  • A status field is an adequate proxy only when rescoring on corrected labels leaves the ranking of candidate systems unchanged.

What does a business outcome label reliability study actually test?

It tests whether a status a business system recorded, such as solved, approved, paid or complete, can stand in for the outcome a task is supposed to produce, for one specific evaluation. The study compares each status with independently verified evidence at fixed maturity windows, counts every disagreement against a stated denominator, and then measures how much the evaluation result changes when corrected labels replace the status field.

That last step is the point. A status field can be wrong in a tenth of cases and still rank two AI systems correctly, or be wrong far less often and reverse the ranking, depending on where its errors fall. An audit that reports only an error rate leaves the buyer guessing which of those happened.

Agent evaluation guidance draws the same line. Anthropic's guide to agent evals separates what an agent says from the outcome: a flight-booking agent may say the flight is booked, while the outcome is whether a reservation exists in the environment's database. A business status field is the recorded version of that claim. This page is the audit protocol; no audit results have been collected or published.

Which status fields deserve an audit first?

Start with fields that already serve as labels, the ones a buyer will grade against, train on or use to pick a winning system. Then rank them by how easily the status can move after it is recorded, because a status that can change later is a label that can be wrong later.

Two documented behaviors show how a recorded status and the real outcome drift apart. In Zendesk, a closed ticket cannot be reopened; a later reply creates a new follow-up ticket linked to the original, so the original keeps its closed status even when the problem came back. In card payments, Stripe's documentation says networks typically allow cardholders to dispute a charge within 120 days of the original payment, and for prepaid services the window starts on the service date, so a payment recorded as successful can still be reversed months later.

Which status fields deserve an audit first?
Field familyStatus used as a labelIndependent evidenceWhat sets the maturity window
Support ticketsSolved or closedNo linked follow-up or recontact on the same issue; customer confirmationHow long problems take to resurface, and how follow-ups are linked
Card paymentsPayment succeededSettlement with no dispute, refund or reversal recordedThe dispute window, typically 120 days for cards and longer in some cases
Claims and approvalsApproved at first decisionFinal payment or determination after any appeal or auditAppeal and audit periods in the governing policy
Invoices and receivablesMarked paidCash applied in the ledger with no reversal or credit noteWhen reversals and credit notes can still be posted
Field service work ordersCompletedNo return visit for the same asset; parts and inspection recordsThe return-visit or warranty period for the work type

The status-to-outcome confusion table, with the two columns audits drop

Cross what the status recorded with what the evidence shows, and keep two columns most audits drop: cases whose outcome is not mature yet and cases that cannot be verified. Following the convention in the scikit-learn documentation, treat the verified outcome as the true class and the status as the prediction, so each cell counts records with one verified outcome and one recorded status.

Four rates come out of the table, each with its denominator written beside it. The false-success rate is status successes that verification shows failed, divided by all matured, verifiable success statuses. The missed-success rate runs the other way. The censored share and the unverifiable share are reported as shares of everything sampled. Publish counts, not only percentages, so a reader can recompute every rate.

Sample within each field family and each closing actor, because rule-driven closures and person-driven closures fail differently, and size each stratum from the interval width the evaluation needs rather than from a percentage of the file. The supplier-side acceptance protocol on this site gives that sizing arithmetic; the audit reuses it and adds the censored and unverifiable columns that sizing alone does not cover.

The status-to-outcome confusion table, with the two columns audits drop
CellStatus recordedVerified outcomeHow it enters the ratesExample
Confirmed successSuccessSuccess, window closedNumerator of status precisionSolved ticket with no follow-up after the window
False successSuccessFailureNumerator of the false-success rateClosed ticket whose linked follow-up reports the same fault
Missed successNon-successSuccessNumerator of the missed-success rateClaim denied at first review, paid after appeal
Confirmed failureNon-successFailureCompletes the matured denominatorPayment declined and never retried
Not yet matureEitherWindow still open on the audit dateCensored: in the sample count, never scoredPayment recorded 30 days before the audit
UnverifiableEitherNo independent evidence existsIts own share, excluded from scoringTicket closed by a bulk job with no later contact data

Mismatch codes: why a status and an outcome disagree

Give every false success and missed success one primary mismatch code, assigned from the evidence rather than from the status. Codes turn an error rate into a repair list: premature closures call for a later cut date, delayed reversals for a longer window, administrative closures for a filter, and definition changes for splitting the period.

The corrected-case examples below are illustrative and describe no real company. In an audit report, each code would carry its count, its share of all mismatches and two or three real corrected cases with their evidence references.

Mismatch codes: why a status and an outcome disagree
CodePatternEvidence that reveals itCorrected-case example (illustrative)
PC: premature closureStatus set before the work was finishedLater activity on the same case or assetWork order completed on day 1, technician back on day 9 for the same unit: relabel as a failed first visit
RF: reopened or followed upThe problem returned after the status was setReopen event or a linked follow-up recordClosed ticket with a linked follow-up on the same fault: relabel as unresolved
PA: provisional approvalFirst decision later overturned or reducedAppeal, audit or final determination recordClaim approved, then reduced on audit: relabel as partially approved
DR: delayed reversalMoney or goods moved back after a success statusDispute, refund, chargeback or credit notePayment succeeded, disputed 70 days later and lost: relabel as a failed payment
AC: administrative closureA rule, script or migration set the statusClosing actor is a system user or bulk jobTickets bulk-closed during a migration: move to unverifiable
DM: duplicate or mergeThe outcome lives on another recordMerge log or duplicate linkDuplicate ticket solved while the original stayed open: take the outcome from the original
SD: status definition changeThe meaning of the status changed mid-periodConfiguration or code-set change logApproved meant pre-approved before a policy date: split the period

Maturity windows: when a status can be scored at all

Score a status only after the window in which it can still change has closed, and audit at more than one window so the disagreement curve is visible. Choose windows from the mechanics of reversal, such as a dispute period, an appeal period or a warranty period, not from convenience. An audit at closure and at two later windows shows how fast false successes surface and when they stop.

Cases whose window is still open on the audit date are censored, not successes. The lifelines documentation describes right-censored cases as ones whose event has not yet been observed and calls ignoring them a common mistake, because it biases the estimate. In a label audit the bias has a predictable direction: recent cases look cleaner than they are, because their reversals have not happened yet.

Report the censored share beside every rate. An evaluation set drawn mostly from recent months sits inside the window and carries more hidden error than an audit of older cases suggests. Whether waiting for matured labels improves a model trained on them is a separate question with its own experiment; this audit measures the label and its effect on scoring, not the training effect.

How label corrections move an evaluation score

Corrections move scores in two ways, and the audit has to show which one is happening. When label errors are independent of the systems under test, they compress differences: on a two-class task with label error rate e, a system's measured accuracy equals e plus its true accuracy times (1 minus 2e), so a true gap shrinks by the factor 1 minus 2e but keeps its sign. When errors line up with a shortcut a system learned, they can reverse the ranking.

Illustrative arithmetic for compression: with 10 percent independent label error, systems with true accuracy of 90 and 85 percent measure 82 and 78 percent. The gap narrows from 5 points to 4, and the order holds.

Illustrative arithmetic for reversal: take 1,000 test cases, 80 of which carry a success status that verification shows failed. System Y matches the verified outcome on 900 cases, including 70 of the 80. System X matches it on 870 but predicts success on all 80, the shortcut a status field teaches. Against status labels, X scores 95.0 percent and Y 84.0 percent; against verified outcomes, Y leads 90.0 to 87.0. The ranking flips.

Public benchmarks show the flip is not hypothetical. Northcutt, Athalye and Mueller estimated an average of at least 3.3 percent label errors across the test sets of 10 widely used datasets, and found that on ImageNet with corrected labels, ResNet-18 outperforms ResNet-50 if the prevalence of originally mislabeled test examples increases by just 6 percent.

So rescore rather than only report. Score every candidate system on the same items against status labels and against corrected labels, report each system's paired difference and its change in rank, and attach intervals from a paired bootstrap. SciPy's bootstrap function, for example, resamples one set of indices and applies it to every array when samples are paired, which keeps each case's status label, corrected label and system outputs together.

Two side-by-side bar panels for the same 1,000 illustrative test cases: scored on status labels, System X leads 95.0 to 84.0 percent; scored on verified outcomes, System Y leads 90.0 to 87.0 percent.
Illustrative arithmetic from this article: 80 false successes are enough to reverse the ranking. Not measured data.

Who verifies outcomes, and how their disagreement is reported

Two verifiers check each sampled case independently, without seeing the status field, any mismatch code or any system's output, and each cites the evidence record behind the verdict. Disagreements go to an adjudicator, and the reason for the final verdict is logged with the case.

Report agreement before adjudication. The scikit-learn documentation describes Cohen's kappa as intended for comparing labelings by different human annotators, with scores above 0.8 generally considered good agreement. Low agreement on a field family means the outcome definition is unclear; rewrite the definition and verify again rather than averaging the verifiers.

Keep three verdicts: verified success, verified failure and insufficient evidence. Pressing a verifier to choose on a thin record converts unverifiable cases into confident labels, which manufactures the very error the audit exists to measure.

When is a status field an adequate proxy?

A status field is an adequate proxy for a specific evaluation when every condition below holds on that evaluation's own cases. If any one fails, the corrected label or a narrower scope is the safer choice.

The last condition decides most cases and is the one audits most often skip. A status with a modest error rate can still be unusable when its errors sit exactly where one candidate system's shortcut lives.

  • A system event sets the status, such as settlement or a completed inspection, rather than a person's judgment that the work is done.
  • The status definition stayed stable across the evaluation period, with any code-set change dated and handled.
  • Every scored case is past its maturity window, and the censored share is reported.
  • The false-success and missed-success rates sit inside a tolerance agreed with the buyer before the audit began.
  • Mismatches are not concentrated in the case types the evaluation weighs most heavily.
  • Rescoring on corrected labels leaves the ranking of candidate systems unchanged and moves no score by more than the agreed margin.

What a published audit must disclose

A credible audit report lets a reader recompute every number from the tables it publishes. No results have been published under this protocol yet, so the list below is the specification an audit would have to meet, including the results it would rather not show.

At a glance

What a published audit must disclose

  • The fields audited, the systems and export versions they came from, and the period covered.
  • The sampling frame, strata, sample sizes and random seed.
  • The maturity windows, the reason for each and the audit date.
  • The full confusion table for each field family, in counts, including censored and unverifiable cases.
  • Mismatch code counts with corrected-case examples and their evidence references.
  • Verifier agreement before adjudication, and the adjudication log.
  • Each candidate system's score on status labels and on corrected labels, the paired difference, rank changes and intervals.
  • Negative results: fields that passed as adequate proxies, and fields where corrections changed nothing.

Requesting outcome-labeled records for an evaluation

An evaluation built on outcomes needs more than a final status column. It needs status history with timestamps, the linked records that confirm or contradict each status, and enough elapsed time for the maturity windows to close. Name those in the request, along with the intended use and the acceptance test the labels must pass.

SourceX works from requests, not stock: it sources each dataset for an AI buyer from U.S. businesses, and the buyer's scope, volume, language, format, timing and licensing needs go into the request form. From sourcing and qualification through rights review, contracting and data preparation to delivery and payment, SourceX runs each step, and no data moves before the supplying company approves the scope and terms and signs. The audit above is how a buyer can check, rather than assume, what the outcome labels in a delivery support.

Frequently asked questions

What is a business outcome label reliability study?

It is an audit of whether status fields recorded in business systems, such as solved, approved or paid, match the outcomes independent evidence shows. It reports a confusion table with denominators, codes each mismatch, and measures how much an AI evaluation changes when corrected labels replace the status.

Is a closed or resolved status a reliable outcome label?

Sometimes, for a given evaluation. It is reliable when a system event sets it, its definition stayed stable, its maturity window has passed and rescoring on verified outcomes does not change the result. Closure on its own records only that a workflow ended.

How does label noise change AI evaluation results?

Errors independent of the systems under test shrink measured differences between them without changing their order. Errors that line up with a shortcut one system learned, such as predicting success whenever a ticket is closed, can reverse the ranking, which is why the audit rescores every candidate system.

What is a maturity window for an outcome label?

It is the period after a status is recorded during which it can still change, for example the time a cardholder has to dispute a payment or a customer has to report that a fix failed. Cases inside the window are censored, not counted as successes.

Have results from this outcome label audit been published?

No. This page publishes the audit design, the confusion table structure, the mismatch codes and the rescoring method. The numbers in the examples are illustrative arithmetic, not measurements.

Sources

  • Anthropic's agent evaluation guide defines the outcome as the final state in the environment, using a flight-booking agent whose claim of a booking differs from whether a reservation exists in the database. Source
  • Zendesk documents that closed tickets cannot be reopened and that a response to a closed ticket creates a new follow-up ticket linked to the original. Source
  • Stripe's documentation says card networks typically allow cardholders to initiate disputes within 120 days of the original payment, with more time in some situations, and that for future services the window starts on the event date. Source
  • The scikit-learn documentation defines confusion matrix entry i, j as observations actually in group i but predicted in group j, and describes Cohen's kappa as comparing labelings by different human annotators, with scores above 0.8 generally considered good agreement. Source
  • The lifelines documentation defines right-censored individuals as those whose event has not been observed and calls ignoring them a common mistake that biases estimates. Source
  • Northcutt, Athalye and Mueller estimate an average of at least 3.3 percent label errors across 10 widely used test sets and find that on corrected ImageNet labels ResNet-18 outperforms ResNet-50 if the prevalence of originally mislabeled examples increases by 6 percent. Source
  • SciPy's bootstrap function computes a two-sided bootstrap confidence interval and, when paired is True, resamples one array of indices and uses the same indices for all arrays. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies