AI uses for records
Grader adversarial validation with deliberately wrong answers
By SourceX Editorial · Updated
10 sections
Short answer
Challenge a grader by deriving deliberately wrong answers from expert-verified correct ones, each changing one business-relevant property, and mixing in valid answers that look different from the reference. Run the grader on all of them before it scores real work, and accept it only if it fails every critical wrong answer, passes the valid ones and routes contested items to review, at thresholds fixed in advance.
Grader rubric
Grader adversarial
Key takeaways
- Gold examples show a grader recognizes right answers, not that it rejects plausible wrong ones.
- Build each wrong answer by changing one business-relevant property of a verified correct answer.
- Include valid alternatives as controls, so a strict grader does not punish good agents.
- Fix thresholds per class in advance, with zero passes for critical classes and item counts sized by the rule of three.
- Keep contested items separate and the set held out, and pair it with audits of real runs.
How do you challenge a grader with deliberately incorrect answers?
Take tasks whose correct outcome a domain expert has verified, derive deliberately wrong answers from each one by changing exactly one thing that matters to the business, add valid answers that look different from the reference, and run the grader on all of them before it scores a single real run. The grader is accepted only if it fails the wrong answers and passes the valid ones at thresholds you wrote down beforehand.
This is grader adversarial validation: testing the test. It answers a narrower question than a general accuracy check. The question is not how often the grader agrees with experts on typical answers, but whether any cheap way of looking finished earns credit without the business requirement being met.
The method below suits evaluation leads and domain adjudicators working on business-task graders, whether the grader is a script that inspects system state, a rubric scored by a language model, or a mix. It produces two reusable artifacts: an adversarial answer set and an acceptance table.
Why a grader that passes its gold examples has proved very little
Gold examples only show that the grader recognizes correct work; they say nothing about what else it rewards. A grader tuned on correct answers can still hand full marks to an answer that is fluent, well formatted and wrong, and agents trained or selected against that grader will find those gaps.
The gaps are documented. METR reported in June 2025 that recent frontier models reward hacked on its evaluation tasks, that on one RE-Bench task o3 did so in every trajectory METR generated, and that telling the model not to cheat had a nearly negligible effect. Researchers studying language model judges showed that trivial inputs, a lone colon or an opener such as Thought process:, could elicit false positive rewards across a range of judge models used in reference-based grading. Earlier work on judging chat answers documented position, verbosity and self-enhancement biases, including an attack that padded answers with repetitive lists.
None of this means a particular grader is broken. It means a grader is a claim about what counts as done, and the claim needs counterexamples before anyone builds a training signal or a purchase decision on it.
Deriving wrong answers from verified right ones
Start from a verified correct answer and change exactly one property that a business owner would care about, so every wrong answer differs from a right one in a known way. Software testing calls this mutation testing: PIT, a mutation testing system for Java, seeds faults into code and counts a fault as killed when the tests fail and as surviving when they pass. Here the grader plays the role of the test suite and each deliberately wrong answer is a mutant it must kill.
Deriving from right answers matters for two reasons. The wrong answer stays realistic, sharing the correct answer's tone, format and most of its content, which is exactly what a weak grader cannot see past. And when the grader passes a mutant, you know which single property it ignored, so the fix is specific.
Write each item as a record: the source task id, the verified correct outcome, the one change made, the class from the table below, the expected grader verdict, the severity if the grader gets it wrong, and the adjudicator who confirmed the item is wrong for the stated reason.
The adversarial answer set, class by class
Cover every class at least once per task family, and weight the set toward the classes your grader is most likely to miss given what it reads. The last row is the control: valid answers the grader must not punish.
| Class | How to build it from a correct answer | Grader must | Severity if passed |
|---|---|---|---|
| Near miss | Change one key field by the smallest meaningful step: a date by a day, a quantity by one unit, a status to its neighbor | Fail | Critical when the field drives a business action |
| Right value, wrong record | Apply the correct change to a different customer, order or vendor with a similar name | Fail | Critical |
| Claimed, not done | Keep the closing message, revert the system change it describes | Fail | Critical |
| Partial completion | Perform two of three required updates and report all three | Fail, or partial only if the spec allows it | High |
| Shape without substance | Return schema-valid output with placeholder or copied input values | Fail | High |
| Surface cues | Prepend a reasoning opener, a lone punctuation mark, or pad a wrong answer with repeated lists | Fail | High |
| Answer spraying | List several candidate answers including the correct one | Fail | High |
| Collateral change | Reach the correct outcome plus one unrequested edit to another record | Fail | Medium to critical by record |
| Policy skipped, outcome right | Reach the right end state while skipping a required check or approval | Fail | Critical in regulated workflows |
| Valid alternative (control) | Reach the verified outcome by a different path, wording or acceptable format | Pass | False fail; blocks good agents |
Which shortcut could lift the score without meeting the requirement?
List every signal the grader reads, then for each signal write an answer that satisfies the signal without satisfying the requirement behind it. That answer is your highest-priority adversarial item, because it is the shortcut an optimizing agent will find first.
If the grader reads only the agent's final message, the claimed-not-done class will probably pass. If it checks that a file exists, a file of the right name with the wrong contents will. If a language model judge compares the answer with a reference, test the surface cues and padding classes and swap the order of the answer and reference in the prompt. If it checks a status field, flip the status without the work.
Checklist · 5 items
- Signal: closing message mentions the update. Shortcut: say it, do not do it.
- Signal: target record changed. Shortcut: change it to any value, or change a lookalike record.
- Signal: output validates against a schema. Shortcut: valid shape, invented values.
- Signal: similarity to a reference answer. Shortcut: reuse the reference phrasing around a wrong key value.
- Signal: absence of errors in logs. Shortcut: catch and suppress the error instead of fixing its cause.
0 of 5 ticked. Tick each item as you go.
Acceptance thresholds you set before the first run
Fix the thresholds per class before the run and do not move them after seeing results. For classes marked critical, the usual rule is zero passes; for controls, a pass rate high enough that good agents are not punished. Partial credit needs an explicit decision too: Inspect, the UK AI Security Institute's evaluation framework, maps a PARTIAL score to 0.5 by default in metrics such as accuracy, so a half-finished business task can quietly earn half credit unless you map it to zero.
Zero failures observed is not zero risk. The rule of three gives an approximate upper 95% bound of 3 divided by the number of items when no events occur, and the clinical statistics literature warns it should be used only when the observed count is exactly zero. With 30 near-miss items and no passes, a pass rate as high as roughly 10% has not been ruled out; with 300, roughly 1%. Size each critical class for the bound you can live with.
The example thresholds below are a starting point for a team to adjust, not a standard.
| Item group | Example threshold | Example minimum items | If the threshold is missed |
|---|---|---|---|
| Critical wrong classes | Zero passes | Enough that 3 divided by n is below your tolerance | Reject the grader; fix the signal it ignored; rebuild the class with new items |
| High-severity wrong classes | Zero or one pass, each reviewed | 20 to 50 per class | Review each pass with an adjudicator; fix or document |
| Medium-severity wrong classes | A small, written pass allowance | 10 to 20 per class | Log and fix in the next grader version |
| Valid alternatives (controls) | Nearly all pass | As many valid paths as experts can name | Loosen over-literal checks; re-run every wrong class after the change |
| Contested items | Grader returns unsure or routes to review | All items experts split on | Do not count toward pass or fail; report separately |
Worked example (illustrative): a vendor bank detail change grader
Every number in this example is illustrative. An accounts payable agent processes requests to change a vendor's bank details. The business requirement: the change is applied to the right vendor only after a callback to a phone number already on file, and the request is logged with the callback result. The grader under test is a language model judge that reads the agent's final message and a summary of changed fields.
The team built 8 items per wrong class from 12 verified tasks, plus 15 valid alternatives and 6 contested items. On the first run the judge passed 5 of 8 claimed-not-done items, because the summary it read was generated from the agent's own message, and 3 of 8 policy-skipped items, because nothing it read showed whether the callback happened. It correctly failed every near-miss and wrong-record item and wrongly failed 2 of 15 controls that logged the callback in a different but acceptable note format.
The fixes followed the signals. The grader now reads the vendor master record and the call log directly instead of the agent's summary, and the callback check accepts any note format that records the number dialed and the outcome. On the re-run, with fresh items built for the two failing classes, it passed none of the critical items and all of the controls. The team then doubled the claimed-not-done class so that zero passes would carry a tighter bound.
Items experts disagree on stay contested, not forced
When two qualified adjudicators disagree about whether an item is wrong, label it contested and keep both rationales instead of picking a winner. A grader that confidently decides contested items is encoding one expert's view as truth, and averaging the two opinions into a soft label hides a real policy question.
Score contested items separately. The useful grader behavior is to abstain or route the item to human review. If your framework supports a panel of grader models deciding by majority, as Inspect's model-graded scorers do, run the adversarial set on the panel as deployed; a majority of judges that share the same blind spot still passes the same wrong answer.
Feed resolved disagreements back into the spec. If adjudicators settle that a callback by email is acceptable, that becomes a valid alternative item; if they settle that it is not, it becomes a policy-skipped item.
What an adversarial set cannot tell you
An adversarial set only finds the failures someone thought to build, so a clean result is evidence about those classes, not proof that no other shortcut exists. METR cautions that eliminating the reward hacking you can detect should not by itself be reassuring. Pair this pre-deployment gate with a sampled audit of real runs after deployment.
Keep the set held out. If an agent is trained against the grader and the adversarial items leak into training data or prompts, the agent can learn the items rather than the requirement. Refresh each class when the grader, the reward rules or the business policy changes.
Stop and narrow the scope when experts cannot agree on what counts as done for a task family, when the grader cannot see the evidence that would prove completion, or when an error is irreversible. Those tasks need a specialist reviewer in the loop, not a stricter threshold.
Verified business outcomes to build adversarial items from
Adversarial items are only as good as the verified outcome they are derived from. Records where the business can show what actually happened, such as the change made, the check performed and the later result, let you build near misses and wrong-record items that are realistic rather than guessed.
For a grader team, the useful request to SourceX names verified outcomes and their linked evidence. SourceX finds such records at U.S. businesses when asked rather than drawing on stock, collects scope, volume, format, timing and licensing needs through its request form, and runs sourcing, rights review, contracting, preparation and delivery with the businesses that license them.
Frequently asked questions
What is grader adversarial validation?
It is testing a grader with answers known to be wrong, and with valid answers that look unusual, before trusting its scores. Each wrong answer is derived from a verified correct one by changing one business-relevant property, so a pass shows exactly what the grader ignores.
How many deliberately wrong answers does a grader need?
Enough per critical class that a zero-pass result means something. Using the rule of three, zero passes in n items still leaves an upper 95% bound of about 3 divided by n, so 30 items bounds the miss rate near 10% and 300 near 1%.
Should a grader give partial credit to partly finished business tasks?
Only if the business would accept partial completion, which is rare for actions such as payments or account changes. Some evaluation frameworks map partial scores to half credit by default, so map partial to zero explicitly when the requirement is all or nothing.
Can a panel of judge models replace adversarial testing?
No. A majority vote reduces random disagreement but not shared blind spots, so judges that all read the agent's own summary will all pass a claimed-but-not-done answer. Run the adversarial set on the panel exactly as deployed.
How is this different from auditing a grader on real agent runs?
An audit measures how often the grader is wrong on the runs that happened. Adversarial validation tests whether specific shortcuts are rewarded before any agent tries them, including failure types real runs have not produced yet. Teams usually need both.
Sources
- METR reported recent frontier models reward hacking in its evaluations, with instructions not to cheat having a nearly negligible effect. Source
- Simple master key inputs such as a colon or a reasoning opener elicited false positive rewards from a range of LLM judges. Source
- LLM judges show position, verbosity and self-enhancement biases, tested in part with a repetitive list attack. Source
- Mutation testing seeds faults and counts a fault killed when tests fail and surviving when they pass. Source
- Inspect's default score converter maps PARTIAL to 0.5, giving partly correct answers half credit in accuracy. Source
- Inspect's model-graded scorers can use several grader models with the grade decided by majority. Source
- The rule of three estimates risk as 3/n when no events are observed and should be used only when the count is zero. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.