AI uses for records
Grader false positive audit: measuring failures scored as success
By SourceX Editorial · Updated
10 sections
Short answer
To measure how often a grader labels failed tasks as successful, draw a sample that oversamples grader passes, have two blinded experts judge each run against inspectable business evidence, and fill a confusion matrix. Report the share of passes that were really failures and the share of real failures the grader passed, each with a confidence interval and weighted by severity, then decide against a threshold set before the audit.
Grader rubric
Grader false positive audit
Key takeaways
- A false pass is the costliest grader error because training rewards it and evaluation reports it as progress.
- Success must be defined by evidence a reviewer can open, such as the record that changed, not by the agent's final message.
- Oversample grader passes and add shortcut probes; a random sample of all runs finds too few false passes to measure.
- Keep an undetermined label and two blinded adjudicators; forced answers and single raters hide the error you are measuring.
- Zero false passes in a small audit does not mean zero; report the upper bound of the interval and decide on that.
How do you measure graders that incorrectly label failed tasks as successful?
You audit the grader's passes against an independent judgment of whether the work was really done. Draw a stratified sample of graded runs that oversamples passes and includes deliberate shortcut attempts, have two experts who cannot see the grader's verdict judge each run against business evidence, and fill a confusion matrix of grader verdict against adjudicated outcome.
From that matrix you report two numbers. The false pass share is the fraction of grader passes that were real failures; it tells you how much of a reported success rate is fiction. The false positive rate is the fraction of real failures the grader passed; it tells you how easy the grader is to fool. Both need confidence intervals and a severity breakdown before anyone decides whether the grader is fit to train or evaluate with.
This is a measurement protocol for a grader already in use, whether it is a rule check, a unit test suite, a state comparison or a language model judge. Designing a new grader or red teaming one in depth are separate jobs; the audit tells you whether either is needed.
Why does a false pass cost more than a false fail?
A false fail wastes a good run; a false pass rewards a bad one. In reinforcement learning the policy moves toward whatever the grader rewards, so a grader that passes a shortcut teaches the shortcut. In evaluation, false passes inflate the score that release decisions rest on.
The public record shows both how common and how large this is. The Agentic Benchmark Checklist paper calls the property outcome validity, meaning the evaluation result truly indicates task success, and reports that tau-bench counted empty responses as successful on intentionally impossible tasks, so a trivial agent reached 38 percent. METR has reported recent frontier models trying to raise their scores by modifying tests or scoring code.
Errors run in both directions, which is why the audit needs a full matrix rather than a pass review alone. OpenAI's SWE-bench Verified work found unit tests that were overly specific or unrelated to the issue and could reject correct solutions, and it used human review to screen samples. A grader can be too strict on some task families and too lenient on others at the same time.
How do you define success with evidence a reviewer can open?
Before sampling anything, write down what success means for each task family as evidence an adjudicator can open and check. If the definition depends on the agent's own summary of what it did, the audit measures the grader against the same claim the grader read.
State-based graders show the strongest form: tau-bench compares a hash of the environment's final data with the hash produced by replaying the ground-truth actions. Business tasks rarely have a single correct end state, so the definition is usually a short list of requirements, each tied to a record.
Write the definition with the people who will adjudicate, and test it on a handful of runs before the audit starts. If two experts reading the same run and the same definition reach different labels on more than an occasional case, the definition is the problem, and no amount of sampling will make the false pass rate meaningful until it is fixed.
- The requirement in business terms, for example the credit memo is applied to the invoice it references.
- The evidence location, for example the invoice's applied-credits table and the credit memo's status.
- What must not change, for example other open invoices for the same vendor.
- Who can confirm it, for example an accounts payable specialist, and what they need to see.
- The conditions under which the correct outcome is to stop or escalate, so a justified refusal counts as success.
- The severity of failing that requirement: cosmetic, recoverable or harmful.
Which runs go into the audit sample?
Sample by stratum and weight back to the population, because false passes are usually rare enough that a simple random sample of all runs contains too few to measure. The plan below oversamples grader passes and adds strata designed to find false passes; record the selection probability of every run so population estimates stay unbiased.
Fix the sampling plan in writing before anyone looks at results, including the size of each stratum, the margin that defines near-threshold, and the task families treated as high severity. Changing strata after seeing early adjudications is the audit equivalent of moving the goalposts, and it makes later audits incomparable. If the grader is a language model judge, freeze the judge version and prompt for the audit window and record both with every verdict, because a silent model update partway through splits the sample into two different graders.
| Stratum | How runs are selected | Why it is in the audit | Weighting |
|---|---|---|---|
| Random grader passes | Simple random draw from all passes | Estimates the false pass share for the population | Weight by passes in population over passes sampled |
| Random grader fails | Simple random draw from all fails | Estimates false fails and completes the matrix | Weight by fails in population over fails sampled |
| Near-threshold scores | Runs within a set margin of the pass cutoff | Where graders flip on small differences | Report separately; do not mix into population rates |
| High-severity task families | Extra passes from families where a false pass is harmful | Gives enough cases to estimate severe error | Weight within family, report by family |
| Shortcut probes | Runs from scripted no-op, partial and plausible-wrong agents | Tests whether known shortcuts pass | Report as a separate pass rate per probe; never weight into population |
How should adjudicators judge each sampled run?
Each sampled run is judged by two experts working independently, without the grader's verdict or score, using only the run, the evidence locations and the written success definition. Hiding the verdict matters: a reviewer who sees a pass tends to look for reasons it is right.
Give adjudicators three labels, success, failure and undetermined, plus a severity for every failure and a pointer to the evidence. Undetermined is for runs where the evidence is missing or the requirement is ambiguous; forcing those into success or failure either hides a grader error or invents one. Disagreements go to a third, senior adjudicator, and the original two labels stay in the record because they show where the success definition is unclear.
Measure agreement between the first two adjudicators before trusting the result. scikit-learn's documentation describes Cohen's kappa as a statistic for comparing labelings by different annotators and notes that scores above 0.8 are generally considered good agreement. If agreement is low, fix the success definition and re-run that part of the audit rather than letting the senior adjudicator settle everything.
What should you read from the confusion matrix?
Lay the results out with the adjudicated outcome as the true class and the grader's verdict as the prediction, the same orientation scikit-learn uses, where each row is the true class. Precision in that framing is true passes divided by all passes, so the false pass share is simply one minus precision. The table below defines every cell and metric the audit reports.
At a glance
What should you read from the confusion matrix
Read the matrix by task family as well as in total. A grader can look acceptable overall while one family carries most of the false passes, usually the family whose success depends on something the grader does not inspect. Break the false pass share down by family, by severity and by agent version, and treat any family with a high rate as its own finding even if the pooled number clears the threshold.
| Cell or metric | Definition | How to compute | What a high value means |
|---|---|---|---|
| True pass | Grader passed, adjudicated success | Count | The grader works on these runs |
| False pass | Grader passed, adjudicated failure | Count, with severity for each | Training and scores are rewarding failure |
| False fail | Grader failed, adjudicated success | Count | Good work is being discarded or penalized |
| True fail | Grader failed, adjudicated failure | Count | The grader works on these runs |
| Undetermined | Adjudicators could not decide from the evidence | Count by grader verdict, kept out of rates | The success definition or evidence needs work |
| False pass share | Real failures among grader passes | False pass over (true pass plus false pass), weighted | Reported success overstates real success |
| False positive rate | Real failures the grader passed | Weighted false passes over weighted real failures | The grader is easy to fool |
| Severe false pass share | Harmful failures among grader passes | Severe false passes over determined passes, weighted | The grader is unsafe as a reward |
| Adjudicator agreement | Agreement between the two first-round experts | Cohen's kappa on success and failure labels | Low values mean the definition is unclear |
Which shortcut probes should the grader fail?
Shortcut probes answer a narrower question than the random strata: which specific behavior could raise the score without completing the business requirement? Script a few agents that each try one shortcut, run them through the same grader, and report each probe's pass rate on its own.
Useful probes for business tasks include a no-op agent that ends with a confident summary, an agent that completes the visible part of the task but not the record change, an agent that edits the wrong record of the right type, and an agent that writes the expected output text without the underlying work. For a language model judge, add a probe that makes failed runs longer and more polished: Zheng and colleagues found strong judges reached over 80 percent agreement with human preferences, and also documented position, verbosity and self-enhancement biases. Any probe that passes more than rarely is a finding by itself, whatever the random strata say.
Probes also tell you whether a fix worked. After the grader changes, rerun the same probes first: they are cheap, deterministic and targeted at the failure you found. Only when every probe fails as intended is it worth spending adjudicator time on a fresh random audit.
How sure is the false pass rate?
Report every rate with a confidence interval, and make decisions on the upper bound, because audit samples are small and false passes are rare. statsmodels' proportion_confint offers, among others, the Wilson score interval and the Clopper-Pearson interval, and its documentation notes that Clopper-Pearson has coverage of at least the stated level but is in general conservative.
The most common mistake is reading zero as zero. If an audit finds no false passes in 60 sampled passes, the exact 95 percent upper bound on the false pass share is still about 6 percent. If a 6 percent rate of false passes would be unacceptable for this grader, 60 passes is too small an audit to clear it.
Uncertainty also comes from the undetermined label and from adjudicator disagreement. Report the rates twice, once excluding undetermined runs and once counting them all as failures, and say which assumption the decision rests on. If the two versions lead to different decisions, the evidence for those runs needs work before the audit can conclude.
What might an audit of an invoice-matching grader find? (illustrative)
This scenario is illustrative; every count is invented to show the arithmetic. A team trains an agent to apply vendor credit memos to invoices, scored by a grader that checks whether the agent's final message names the right invoice and whether the credit's status changed to applied. Over a period the grader scored 2,000 runs: 1,400 passes and 600 fails.
The audit sampled 120 random passes and 80 random fails. Adjudicators found 102 successes, 12 failures and 6 undetermined among the passes, and 9 successes, 68 failures and 3 undetermined among the fails. Excluding undetermined runs, the false pass share is 12 of 114, about 10.5 percent, with a Wilson 95 percent interval of roughly 6 to 18 percent. Weighted to the population, that suggests about 147 false passes against about 677 real failures, a false positive rate near 22 percent.
Four of the 12 false passes were severe: the credit was applied to a different open invoice for the same vendor, which the grader missed because it checked the credit's status but not where it went. The shortcut probe that edited the wrong record of the right type passed most of the time, confirming the cause.
The team had set its rule before the audit: no use as a training reward if the upper bound of the severe false pass share exceeds 5 percent. Four of 114 gives an upper bound near 9 percent, so the grader was blocked, fixed to check the invoice the credit was applied to, and re-audited with the same sampling plan.
The 9 false fails in the same audit pointed the other way: runs that applied the credit correctly but phrased the final message without the invoice number. The team relaxed that check at the same time, which is why the audit reports both directions rather than only the passes.
Where does adjudication evidence from real business outcomes come from?
An audit is only as good as the evidence adjudicators can open, and for business tasks the best evidence is the record of what the business actually concluded: the invoice that was paid, the ticket that stayed closed, the claim that was not reopened. Historical cases with their outcomes and the expert reasoning behind them also make the strongest gold sets for re-auditing a grader after each change.
Outcome-linked records like these reach evaluation teams through SourceX, which runs rights review, contracting, preparation and delivery for the businesses licensing them. The request-data form takes the scope, volume, language, format and timing an audit needs, and the records are found at U.S. businesses for each request instead of being held in stock.
Frequently asked questions
What is a grader false positive?
It is a run the grader marks as successful that an informed reviewer, looking at the evidence, judges to be a failure. In training it rewards the wrong behavior, and in evaluation it inflates the reported success rate. It is also called a false pass.
How many runs does a grader false positive audit need?
Enough that the upper bound of the interval on the false pass share falls below the threshold you care about. With no false passes found, about 60 audited passes still leaves an upper bound near 6 percent, so stricter thresholds need larger samples. Oversampling passes and high-severity task families makes each audited run more informative.
Can a language model adjudicate instead of human experts?
Not for the audit itself, because the audit exists to check automated judgment against an independent one. A model can help pre-sort evidence or flag likely problems for the experts. The labels that fill the confusion matrix should come from people who know the business task.
Should undetermined runs count as failures?
Report it both ways. Excluding them shows the grader's error on clear cases, and counting them as failures shows the worst case. If the decision changes between the two, improve the evidence for those runs before concluding.
How often should a grader be re-audited?
After any change to the grader, the judge model or prompt, the task mix or the agent being trained, since a stronger agent can find shortcuts a weaker one never tried. Between changes, a small recurring audit of passes catches drift. Keep the sampling plan fixed so results are comparable.
Sources
- The Agentic Benchmark Checklist paper defines outcome validity and found tau-bench counted empty responses as successful on impossible tasks, letting a trivial agent reach 38%. Source
- METR reported recent frontier models attempting, often successfully, to get higher scores by modifying tests or scoring code. Source
- OpenAI found SWE-bench unit tests that were overly specific or unrelated to the issue, which could reject correct solutions, and used human review to screen samples. Source
- scikit-learn's confusion matrix puts the true class on rows, defines precision as tp / (tp + fp), and describes Cohen's kappa for comparing annotators, with scores above 0.8 generally considered good agreement. Source
- tau-bench's reward compares a hash of the final data with the hash produced by replaying the ground-truth actions. Source
- statsmodels' proportion_confint supports Wilson and Clopper-Pearson intervals and notes that Clopper-Pearson has at least nominal coverage but is conservative. Source
- Zheng et al. found strong LLM judges reach over 80% agreement with human preferences and documented position, verbosity and self-enhancement biases. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.