AI uses for records
AI data pilot severity metrics: when a better pass rate hides worse failures
By SourceX Editorial · Updated
9 sections
Short answer
Score a dataset pilot on two axes fixed before training: how often the model succeeds and how badly it fails when it does not. Grade every failure on a written four-tier scale, compare baseline and candidate task by task in a transition table, and adopt only if success rises while severe failures do not. A pass-rate gain that pushes errors into the top tier is a failed pilot.
Agent run
AI data pilot severity
Key takeaways
- A success rate counts failures without weighing them; GDPval's expert graders rated roughly 3 percent of GPT-5's failures catastrophic, which no win rate shows.
- Anchor severity tiers on the intervention an outcome forces, as clinical adverse-event grades do, not on the type of error.
- Compare the same tasks under baseline and candidate in a tier transition table; the cells that move into the top tier decide the pilot.
- Sign adopt, narrow, stop and inconclusive rules, including a margin for severe failures, before the first training run.
- Zero severe failures in 300 tasks still allows a true rate near 1 percent, so size the high-severity strata on purpose.
What should an AI data pilot measure besides the success rate?
It should measure how bad each failure is, on a scale written before training, and decide on both numbers together. A success rate tells you how often the model trained on the candidate data got the task right. It says nothing about the tasks it got wrong, and in business workflows that is where the cost sits: a clumsy reply and a refund paid outside policy each count as one failure.
GDPval, OpenAI's benchmark built from real work products in 44 occupations, shows the gap in public data. When expert graders sorted GPT-5's losing deliverables by severity, the most common label was acceptable but subpar, roughly 29 percent of the ratings were bad or catastrophic, and roughly 3 percent of failures were marked catastrophic (GDPval paper, October 2025). A single win rate folds all of those into one number.
For a dataset pilot the consequence is direct. Training data changes how a model behaves, not only how often it is right, so a candidate set can lift the success rate and, in the same run, make the remaining failures more expensive. The protocol on this page is built to catch that case before procurement signs.
Harmless imperfection or unacceptable outcome: where is the line?
Draw the line by what the outcome forces someone to do next, not by what kind of error it is. An imperfection that leaves the output usable as delivered counts as a success with a note; a failure is graded by the intervention it requires and by whether that intervention can still undo the damage.
Clinical trials settled this problem long ago. The National Cancer Institute's Common Terminology Criteria for Adverse Events grade every event from 1 to 5 by consequence and required response: grade 1 calls for observation only, grade 2 for a minimal or noninvasive intervention, grade 3 is severe but not immediately life-threatening, grade 4 needs urgent intervention and grade 5 is death related to the event (CTCAE v5.0). The anchors describe what happened to the patient, not the mechanism, which is why different trials can compare them.
The four tiers in the table apply the same idea to business tasks. Tau-bench's error tooling shows why error type alone is not enough: it separates a goal that was only partially completed from an unintended action the agent took (tau-bench repository), and an unintended action can be trivial or catastrophic depending on what it touched. The business owner of the workflow signs the tier definitions; the evaluation lead only applies them.
| Tier | Anchor: what the outcome forces | Example in a refund-handling assistant (illustrative) | Default pilot rule |
|---|---|---|---|
| Tier 0: harmless imperfection | Nothing; the output is usable as delivered | Correct refund decision with an awkward greeting | Counts as success; logged for style work |
| Tier 1: correctable | A routine reviewer edit before use | Right decision, but the reply omits the order number the template requires | Counted and tracked as rework minutes |
| Tier 2: harmful if used | Rework, a customer contact or a reversal that normal review could miss | Escalates a routine refund, or quotes last year's return window | Must not rise beyond the preset margin |
| Tier 3: unacceptable | Money, access or data reaching the wrong party, or an action outside authority | Issues a refund outside policy on a subscription order | Must not rise; named classes stop the pilot at one case |
How can a dataset raise the pass rate and still make the model worse?
It can shift the model's behavior so that it fails less often but more decisively. Resolved tickets, approved claims and completed orders teach a model what finishing looks like. If the candidate set holds few cases where the right move was to stop, ask or escalate, the model learns to finish those too, and some become Tier 3 failures where the old model produced a cautious Tier 1.
Anthropic's guide to agent evaluations describes the mechanism from the test side: check both the cases where a behavior should occur and the cases where it should not, because one-sided evals create one-sided optimization (Anthropic, January 2026). Training data works the same way. A pilot task set that contains only should-act cases cannot see the damage, so the protocol oversamples should-not-act cases in every stratum where a Tier 3 outcome is possible.
Consistency is the second mechanism. A model that succeeds on 75 percent of trials passes all of three trials only about 42 percent of the time, the same guide notes. Tau-bench's public leaderboard shows that decay on tool-use tasks: the top retail entry falls from 0.692 at pass^1 to 0.462 at pass^4 (tau-bench repository). A severe failure that appears once in four runs is a severe failure for the customer who gets that run, so this protocol grades each task by its worst trial.
The preregistered pilot protocol: what is fixed before the first run?
Everything that could be tuned after seeing results is fixed in a dated document before training: the decision, the frozen conditions, the task strata, the severity scale and graders, the number of trials, the metrics and the rules that turn them into adopt, narrow, stop or inconclusive. The Center for Open Science describes preregistration as specifying a research plan in advance so that planned, confirmatory work stays distinct from unplanned, exploratory work. A pilot that changes its thresholds afterwards has become exploratory, whatever its slides say.
The severity margin borrows from drug trials. FDA's 2016 guidance on non-inferiority trials asks sponsors to fix, before the study, the largest loss of effect that would still be clinically acceptable, a margin it calls M2. The pilot equivalent is the largest increase in Tier 2 and Tier 3 outcomes the business owner would accept in exchange for the success gain, written in percentage points with the reason beside it.
Fill in each row, sign it and store it with the pilot record:
| Element | What is fixed in writing | Example entry (illustrative) | Failure it prevents |
|---|---|---|---|
| Decision and owner | The purchase question and who signs the result | License the refund-ticket set for the support assistant; owner: head of support operations | A pilot that answers an easier question than the purchase |
| Frozen conditions | Base checkpoint, recipe, seeds and a training budget matched in examples | Same checkpoint and 4,000 examples per arm: current mix versus candidate | Gains bought with extra training rather than better data |
| Task set and strata | Held-out tasks drawn before data selection, with should-act and should-not-act cases in every high-severity stratum | 300 tasks, with payment strata oversampled to 120 | A test set that cannot produce the costly failure |
| Severity scale and graders | Tier anchors, two graders blind to the arm, an adjudication rule and an agreement target | Tiers 0 to 3 signed by the owner; disagreements go to a third grader | Severity drifting toward the preferred arm |
| Trials and task scoring | Trials per task and how trials combine | Three trials; a task succeeds only if all three do and takes its worst tier | One lucky run hiding an unreliable behavior |
| Metrics | Success rate, rate per tier, tier transitions and Tier 3 count by stratum | Paired task-level differences with intervals clustered by task family | A headline average that hides where the damage moved |
| Decision rules and margins | Adopt, narrow, stop and inconclusive thresholds | Adopt only if success rises 5 points or more, Tier 3 does not rise and Tier 2 plus Tier 3 rises 1 point or less | Thresholds rewritten after the result is known |
| Stop rule | Outcome classes that halt the pilot at one case | Any refund sent to the wrong account in a shadow test | Collecting severe failures for a cleaner chart |
Reading the transition table: which tasks moved, and to which tier?
Put each task's baseline tier on the rows and its candidate tier on the columns, then read the cells off the diagonal. Every task appears once. Cells below the diagonal are tasks the candidate data fixed or softened, cells above it are tasks it made worse, and the Tier 3 column shows every task that ended in the unacceptable tier and where it came from.
Two equal totals can mean opposite things. A 10-point success gain built from 57 improvements and 35 regressions describes a different dataset from one built from 30 improvements and none. Evan Miller's statistical guide for language model evaluations makes the structural point: when two models answer the same questions, inference should run on question-level paired differences, with standard errors clustered when questions come in related groups (Miller, arXiv, November 2024). The transition table is that paired view, kept at the level of severity instead of collapsed into one score.
Two pieces of arithmetic keep the reading honest. A severe-failure count needs its own uncertainty: with zero Tier 3 outcomes in 300 tasks, the one-sided 95 percent upper bound on the true rate is 1 minus 0.05 to the power of 1/300, about 1 percent, the familiar rule of three. And the test belongs on the tasks that changed status, not on the totals: if 11 tasks entered Tier 3 and 2 left it, an exact sign test on those 13 tasks gives a two-sided p-value of about 0.02.
Illustrative result: 10 points better, 9 more unacceptable failures
The numbers in this section are invented to show the arithmetic and describe no real pilot. A support team tests a licensed set of resolved refund tickets against its current training mix on 300 held-out tasks, three trials each, with the protocol above signed in advance.
Success rises from 165 tasks to 195, from 55 to 65 percent, and the dashboard reports a 10-point win. The transition table tells the rest: 57 tasks improved, 35 got worse and 208 stayed in the same tier. Eleven tasks entered Tier 3 and 2 left it, so Tier 3 grew from 6 tasks to 15, from 2 to 5 percent. Tier 2 and Tier 3 together rose from 45 tasks to 51, 2 points against a preset margin of 1.
Under the signed rules the decision is stop. The success gate passed, both severity gates failed, and the sign test says the Tier 3 increase is unlikely to be noise. Had the new Tier 3 cases clustered in one stratum, such as subscription refunds, the rules would allow narrow instead: drop that stratum from the license scope or add examples of correct refusals, then rerun on a fresh holdout rather than the one already seen.
| Metric (300 tasks, worst of 3 trials) | Baseline mix | Candidate set | Preset rule | Result |
|---|---|---|---|---|
| Success (clean or Tier 0) | 165 (55%) | 195 (65%) | Rise of 5 points or more | Pass |
| Tier 1: correctable | 90 (30%) | 54 (18%) | Reported, no gate | Improved |
| Tier 2: harmful if used | 39 (13%) | 36 (12%) | Gated together with Tier 3 | Improved |
| Tier 3: unacceptable | 6 (2%) | 15 (5%) | No increase | Fail |
| Tier 2 plus Tier 3 | 45 (15%) | 51 (17%) | Rise of 1 point or less | Fail |
| Tasks changing Tier 3 status | 2 left Tier 3 | 11 entered Tier 3 | Sign test on the 13 changed tasks | p about 0.02 |
Could the severity shift be noise, grader drift or a weak baseline?
It could, so the protocol tests each explanation before accepting the result in either direction. A positive pilot result can disappear under a stronger baseline, a proper holdout or repeated trials, and a severity increase can come from the graders as easily as from the data. These five checks run before anyone reads the decision row.
At a glance
Could the severity shift be noise, grader drift or a weak baseline
- Grader drift: graders see outputs without the arm label, two grade each failure, and their agreement is reported. In GDPval, expert graders agreed with each other 71 percent of the time and the automated grader agreed with experts 66 percent of the time (GDPval paper), so disagreement about severity should be expected and measured.
- Broken tasks: Anthropic's guide warns that a task no run ever passes usually signals a broken task, not an incapable model, and fixing grading bugs and ambiguous specifications, then loosening the harness, moved one model's CORE-Bench score from 42 percent to 95 percent. Audit every task where both arms land in Tier 3.
- Weak baseline: the comparison arm receives the same number of examples from the current mix, never nothing, or the candidate gets credit for extra training.
- Holdout reuse: a narrowed dataset is retested on fresh tasks, because the first holdout has already informed a decision.
- Too few trials: if many tasks flip in and out of Tier 3 between trials, the pilot is inconclusive on severity until more trials run.
What can a severity protocol not settle?
It cannot set the tiers for you, see failures that surface after the pilot window, or rule out rare catastrophes with a few hundred tasks. Tier definitions are business judgments, and two owners can defensibly disagree about whether a delayed refund is Tier 1 or Tier 2. Record the dissent next to the decision.
Late harm needs a different clock. A wrong answer that a customer reopens a week later looks like a success in a same-day pilot, which is why rework windows belong in any evaluation of fast agents. Weighted scoring is a separate choice as well: this protocol uses gates because a pilot ends in a yes or a no, while severity weights suit ranking several systems against each other.
Rare events set the sample size. To show that Tier 3 outcomes stay below 1 in 1,000, roughly 3,000 clean tasks in the relevant strata are needed before a zero means much. That is why high-severity strata are oversampled, and why some pilots are inconclusive on severity by design rather than by accident.
Turning the severity rule into a data request
Write the tiers and strata into the request itself, so the supplier knows which cases the data must contain. A request for resolved refund tickets invites more of the should-act cases that caused the problem above. A request that names the should-not-act strata, the outcome fields that record a reversal or escalation, and the acceptance test the data will face describes data that has a chance of passing.
SourceX has no dataset inventory to draw on: it starts looking for records at U.S. businesses once a buyer has made a request, and the buyer's request form lists its licensing needs, scope, volume, language, format and timing, with the intended use and a severity-based acceptance test added to the same request. Delivery comes last on the supply side as well: the business supplying the records first approves what is in scope and on what terms, then signs, and only after that is anything delivered. Whether a given company's records hold enough should-not-act cases is something only the pilot can show, so treat no candidate set as accepted until it passes the rules you signed.
Frequently asked questions
What are severity metrics in an AI data pilot?
They are counts and rates of failures by consequence tier, reported beside the success rate on the same tasks. A practical set is the rate per tier, the number of tasks entering and leaving the top tier, and the top-tier count by stratum. Together they show whether the candidate data changed how badly the model fails, not only how often.
Is a severity gate the same as a severity-weighted score?
No. A weighted score multiplies each error by a weight and sums them, which suits ranking several systems. A gate fixes the largest acceptable increase per tier in advance and fails the pilot when it is crossed, which suits a yes-or-no purchase and cannot be offset by gains elsewhere.
How many test tasks do you need to measure rare severe failures?
More than most pilots plan for. With zero severe failures in n tasks, the 95 percent upper bound on the true rate is about 3 divided by n, so 300 clean tasks still allow a rate near 1 percent. Oversample the strata where severe outcomes are possible, or accept that the pilot can only be inconclusive on severity.
Should harmless imperfections count as failures?
Count them as successes with a note. Treating a stylistic flaw as a failure inflates the failure count with cases that cost nothing and dilutes the signal from the ones that do. Track them separately so style work can still use them.
What if the candidate data cuts severe failures but leaves the success rate flat?
That can be the most valuable result a pilot produces, if the rules allow for it. Write a severity-only adopt rule in advance, for example adopt when Tier 3 falls by a set number of tasks and success does not drop, so a safety gain is not rejected for missing a headline target.
Sources
- In the GDPval paper, expert graders rated GPT-5 failures by severity: acceptable but subpar was most common, roughly 29 percent of ratings were bad or catastrophic and roughly 3 percent of failures catastrophic; expert graders agreed 71 percent of the time and the automated grader 66 percent. Source
- GDPval covers 44 occupations from the top nine industries by contribution to U.S. GDP, with tasks based on real work products and graders who blindly compare model and human deliverables. Source
- NCI's CTCAE v5.0 grades adverse events from 1 (mild, observation only) through 4 (life-threatening, urgent intervention) to 5 (death related to the event). Source
- The tau-bench repository classifies agent faults, including goal_partially_completed and took_unintended_action, and its leaderboard shows the top retail entry falling from 0.692 at pass^1 to 0.462 at pass^4. Source
- Anthropic's January 2026 evals guide recommends testing where a behavior should and should not occur, shows a 75 percent per-trial rate giving about 42 percent across three trials, treats persistent zero pass rates as a sign of broken tasks and reports a CORE-Bench score moving from 42 to 95 percent after grading fixes. Source
- The Center for Open Science defines preregistration as specifying a research plan in advance and submitting it to a registry, separating planned confirmatory work from unplanned exploratory work. Source
- FDA's guidance on non-inferiority trials describes a prespecified margin, M2, usually smaller than M1, set to preserve a clinically acceptable portion of the control's effect. Source
- Evan Miller's paper recommends question-level paired differences when comparing two models and clustered standard errors when questions come in related groups. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.