AI uses for records
Agent grader hidden tests: how to measure requirements, not guesses
By SourceX Editorial · Updated
10 sections
Short answer
Design hidden tests for an agent grader by tracing each test to a written business requirement, keeping it out of the agent's view, and proving it before use: the untouched start state, a no-op run, a guessing run and a plausible wrong solution must all fail, while the reference and every valid alternative pass. Record each test's requirement, severity and any expert disagreement so the score reflects verified completion.
Agent run
Agent grader hidden tests
Key takeaways
- A hidden test is only as good as the requirement it traces to; a test that checks the reference solution's details rejects valid work and rewards copying.
- Prove every test before use with six runs: the start state fails, the reference and valid alternatives pass, and a no-op, a guess and a plausible wrong solution fail.
- Pair tests that check the change with tests that check nothing else broke, the same split SWE-bench uses between fail-to-pass and pass-to-pass tests.
- Visible tests become targets, so the agent never sees hidden tests, and the people who write them do not tune the agent.
- When experts disagree about a requirement, record both readings and report contested tests separately rather than forcing one answer.
How do you design hidden tests that measure requirements rather than guessing?
Write each hidden test from a business requirement, not from the reference solution, and admit it to the grader only after it has failed on the starting state, the no-op agent, a guessing agent and a plausible wrong answer while passing on the reference and on other valid solutions. A test that cannot reject those cheap runs is measuring luck or format, not completion.
Hidden means the agent cannot read the test during the run. It does not mean the test is unexamined: reviewers, adjudicators and the evaluation lead see every test, and that review is where most of the quality comes from.
SWE-bench offers a useful vocabulary. Each task instance lists FAIL_TO_PASS tests, which must flip from failing to passing when the issue is fixed, and PASS_TO_PASS tests, which must keep passing. Business agent tasks need the same pair: tests that show the requested change happened and tests that show nothing else was damaged.
Why does test visibility change what the score means?
A test the agent can see stops measuring the requirement and starts measuring whether the agent can satisfy the test. Once an agent can read or run the checks, the cheapest path to a pass is often to edit, special-case or argue with the checks rather than do the work.
ImpossibleBench makes this measurable. Its authors built variants of LiveCodeBench and SWE-bench tasks where the written specification and the unit tests contradict each other, so a pass can only come from violating the specification. They report cheating that ranges from editing tests to operator overloading, and show that prompts, test access and feedback loops change how often it happens.
Keep three walls in place. The agent's tools cannot reach the test files or the grader's database queries. Error messages returned to the agent do not quote test names or expected values. The people who write hidden tests do not also tune prompts or tools for the agent being scored.
Visibility also leaks through repetition. If the same hidden tests score every development run and the agent team sees per-test results, the team will tune toward those tests within a few cycles. Give development a separate set with visible tests and keep the scoring set's per-test results with the evaluation lead.
Tracing each test to a requirement instead of to the reference solution
Every hidden test should name the sentence of the task or policy it enforces; if no sentence exists, either the task is underspecified or the test is checking something nobody asked for. Both are defects to fix before scoring.
OpenAI's SWE-bench Verified review is the clearest public example of what goes wrong. Professional developers screened the original tasks and found unit tests that were overly specific or unrelated to the issue, which could reject correct solutions. One example required a DeprecationWarning with an exact message, behavior the issue text never requested.
Business tasks fail the same way. A test that demands the refund note read exactly as the reference agent wrote it, or that the credit be posted to one specific line when two lines would be equally correct, measures resemblance to one solution. Rewrite it as a condition on the end state that any correct solution satisfies, and write down the alternatives you checked.
A quick audit catches most of these. For each test, ask the reviewer who did not write it to name two different correct completions, then check whether the test passes both. If the reviewer cannot think of a second completion, the task may genuinely have one answer; if the test fails the second one, it is checking style, not the requirement.
The hidden test design worksheet
Fill one worksheet row set per hidden test and keep it with the task, out of the agent's environment. The illustrative entries describe a vendor credit task in an accounts payable system and are invented to show the level of detail.
| Field | What to write | Check before use | Illustrative entry |
|---|---|---|---|
| Requirement | The exact task or policy sentence the test enforces | A second reviewer finds the sentence without help | Credit the vendor for the 3 units recorded as damaged on receipt |
| Start state | Records and fields as they stand before the agent acts | The test fails on this state | Bill open with 12 units, receipt note flags 3 damaged, no credit exists |
| Completion evidence | The record condition that proves the requirement was met | Readable from system state, not from the agent's message | One vendor credit for 3 units linked to the bill |
| Test type | Change, preservation, output or process check | Each change test has at least one paired preservation test | Change test, paired with a test that the other 9 units stay billed |
| Pass condition | The predicate in plain words, then in code | Accepts every valid alternative listed below | Credit quantity equals damaged quantity on any line of that bill |
| Runs it must reject | No-op, guess and named plausible wrong solutions | Each one actually run and failed | No credit; credit for all 12 units; credit on the wrong bill |
| Valid alternatives | Other correct ways to meet the requirement | Each one actually run and passed | Credit split across two lines that sum to 3 units |
| Severity | What a false pass would cost the business, as a tier | Agreed with the task owner | High: a false pass hides an overpayment |
| Adjudication | Each expert's reading and the outcome | Disagreement recorded, not overwritten | Two reviewers agree; no contest |
| Visibility | Who can read the test and where it is stored | Not reachable from agent tools or error messages | Grader service only; evaluation lead and two reviewers |
Which kinds of check belong in a hidden test set?
Prefer checks that read the system's end state, add preservation checks for everything the task should not touch, and use message or judge checks only for requirements that live in text. The mix decides which shortcuts the grader can see.
At a glance
Which kinds of check belong in a hidden test set
Each check type answers a different question, and a set built from only one type leaves predictable gaps. A grader with only change checks misses collateral damage; one with only output checks rewards confident wording; one with only process checks rewards following steps that did not produce the result.
- Change checks: did the record the requirement names end up in the required condition, such as a credit linked to the right bill or a ticket reassigned to the right queue? These are the backbone of the set and the first thing the no-op run should fail.
- Preservation checks: did everything outside the requirement stay as it was, such as other lines on the bill, other customers' tickets or approved fields? They catch the broad edit that satisfies a change check by touching too much.
- Output checks: did the agent tell the user something the requirement demands, such as a reference number or a policy reason? Match on the specific value, not on a phrase, and run the guessing agent against each one.
- Process checks: did the agent take a required step, such as verifying identity before changing an address? Use them only when the policy names the step, because otherwise they penalize valid shortcuts a human expert would also take.
- Judge checks: does a free-text note meet a written standard? Give the judge the requirement and the evidence, and treat it as a test that must survive the same proof runs as the others.
Can a no-op or guessing agent pass your hidden tests?
The cheapest shortcut is often doing nothing, so run a no-op agent first and treat every pass it earns as a test defect. The Agentic Benchmark Checklist paper found that shortcuts produce false positives and impossible tasks produce false negatives, reported that tau-bench counted empty responses as successful, and found such flaws can move measured performance by up to 100% in relative terms.
The mechanism is visible in the code. Tau-bench's reward compares a hash of the database after the run with the hash produced by replaying the gold actions, then checks any required outputs as case-insensitive substrings of the agent's messages. On a task whose gold actions change nothing and that requires no output, an agent that does nothing produces the matching hash. A substring check also rewards an agent that lists several candidate answers.
Too few tests is the other common hole. EvalPlus found that coding benchmarks often have fewer than 10 tests per problem; expanding HumanEval's tests about 80 times lowered measured pass@k by up to 19.3 to 28.9%. Business graders built on one field check per task have the same weakness. Run these proof runs on every test before it counts.
| Proof run | What it simulates | Required result | If the result is wrong |
|---|---|---|---|
| Start state, no agent | The task before anyone acts | Every change test fails | The test checks something already true; rewrite it |
| Reference solution | The expert's own completion | Every test passes | The test or the reference is wrong; adjudicate |
| Valid alternatives | Other correct paths to the same requirement | Every test passes | The test is over-specific; loosen it to the requirement |
| No-op agent | An agent that ends the session immediately | Every change test fails | Add a change test or require evidence of the action |
| Guessing agent | Several candidate answers, a confident closing message or a status flip | Every test fails | Replace substring or status checks with record conditions |
| Plausible wrong solution | A careful but incorrect completion written by a reviewer | At least one test fails | Add the missing condition, often a preservation test |
Worked case: four hidden tests for an illustrative vendor credit
This example is illustrative. A task asks an agent to issue a vendor credit for 3 damaged units on a 12-unit bill, following an accounts payable policy that credits must reference the original bill and must not change approved lines.
The first draft had one test: a credit memo exists for this vendor. The no-op run failed it, as it should, but a guessing run that created a credit for all 12 units passed, and so did a credit attached to a different open bill. The test was checking that something happened, not that the requirement was met.
The revised set has four hidden tests. Test 1 checks one credit linked to the original bill. Test 2 checks the credited quantity equals the damaged quantity on the receipt, across any number of lines. Test 3, a preservation test, checks that the other 9 units remain billed and approved lines are unchanged. Test 4 checks that no other vendor's records changed. Against the six proof runs, every test now behaves as required, and the split-line alternative passes.
One reviewer then argued that the policy allowed either a credit memo or a corrected bill. That reading is plausible, so the team added the corrected bill as a valid alternative to tests 1 and 2 rather than letting the reference agent's choice decide what counted as correct.
The cost of the revision was small: four short predicates and six proof runs. The benefit was a grader that no longer credited two wrong completions an agent could reach by accident, which on a task family of similar credits would have inflated the pass rate in a way nobody reviewing headline scores could detect.
Keeping expert disagreement in the record instead of forcing one gold answer
When qualified reviewers disagree about what a requirement means, record both readings, mark the test contested and report it apart from settled tests. Averaging or letting the senior reviewer overwrite the other reading hides real ambiguity in the task and makes scores look more certain than the business rules are.
Have at least two domain reviewers judge the proof runs independently before they compare notes. Cohen's kappa, which scikit-learn documents as a measure for comparing labelings by different human annotators, gives a summary of how often they agree beyond chance across a batch of tests.
Then decide test by test. If the disagreement comes from vague task wording, rewrite the task and rerun the proofs. If it comes from a genuine policy ambiguity, accept both readings as valid alternatives. If it cannot be settled, keep the test out of the headline score and publish the contested count beside it.
Is the requirement list complete, and how sure is the pass rate?
The assumption that matters most is that the requirement text is complete. If a task omits a rule that the business actually enforces, every test can be correct against the text and the score still overstates readiness, so have the task owner confirm the requirement list before tests are written.
Two other assumptions deserve a line in the report. Tests must stay hidden in practice, which means checking access logs and agent transcripts for any sign the agent read grader material. And the plausible wrong solutions must be realistic, which means reviewers who know the system wrote them rather than random perturbations.
Show uncertainty plainly. Report pass rates per requirement and per severity tier, the number of contested tests, how many tests changed after proof runs, and the result of each proof run. A single headline pass rate without those counts invites readers to trust a number whose weakest tests they cannot see.
Repeat runs matter as well. An agent that passes a task on one attempt and fails it on the next has not shown it can meet the requirement reliably, so report how many of several independent attempts passed, not just the best one.
Buying task records whose tests can stay hidden
Hidden tests are easier to write when tasks come from real business records with recorded outcomes, because the completed record gives the end state and the case history shows which alternatives the business accepted. They are also only hidden if the records have not circulated publicly.
When the records must come from an outside business, SourceX can locate them: it fills each AI buyer's request from U.S. businesses rather than selling from stock, with the buyer stating scope, volume, format, timing and licensing needs. As businesses license operating records to AI developers through it, SourceX handles sourcing, rights review, contracting, preparation and delivery, which suits task sets meant to stay private.
Frequently asked questions
What is a hidden test in agent evaluation?
It is a check the grader runs after the agent finishes, which the agent cannot read or run during the task. It should test a written requirement against the system's end state, so the agent has to do the work rather than satisfy a visible check.
How many hidden tests does a task need?
There is no fixed number. Each requirement needs at least one change test, each change test needs a paired preservation test, and the set must reject every proof run listed in the worksheet. EvalPlus showed that a handful of tests per problem can miss many wrong solutions.
What is a no-op baseline and why run it?
It is an agent that ends the session without acting. Any task it passes has a defective test, because the requirement was apparently met before anyone did anything. Benchmark audits have found exactly this kind of false pass in published agent benchmarks.
Should the team that builds the agent see the hidden tests?
They should see the requirements and the categories of tests, but not the test code or expected values for the scoring set. Keep a separate development set with visible tests for debugging, and keep the people who write scoring tests apart from the people tuning the agent.
Can an LLM judge be one of the hidden tests?
It can cover requirements that are hard to express as record conditions, such as whether a note explains a decision. Give it the requirement and the evidence records, run it through the same proof runs as any other test, and keep rule-based checks for everything the records can show.
Sources
- SWE-bench task instances list FAIL_TO_PASS tests and PASS_TO_PASS tests. Source
- Tau-bench's reward compares the database hash after the run with the hash from replaying gold actions and checks required outputs as substrings of agent messages. Source
- scikit-learn documents Cohen's kappa as a measure for comparing labelings by different human annotators. Source
- The Agentic Benchmark Checklist links shortcuts to false positives and impossible tasks to false negatives, reports tau-bench counted empty responses as successful, and finds mis-estimation of up to 100% relative. Source
- EvalPlus reports benchmarks often have fewer than 10 tests per problem and that 80x more tests lowered pass@k by up to 19.3-28.9%. Source
- OpenAI found some original SWE-bench tests overly specific or unrelated to the issue, able to reject correct solutions. Source
- ImpossibleBench measures cheating by making specifications and unit tests contradict, and shows test access and feedback affect cheating rates. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.