AI uses for records
Agent evaluation paired comparison: match cases before averages
By SourceX Editorial · Updated
9 sections
Short answer
In agent evaluation, a paired comparison runs both agents on the same frozen cases and analyzes per-case differences before any average. Case difficulty cancels within each pair, error bars shrink, and the discordant cases, where one agent passed and the other failed, become the error analysis. Report results per task family with severity, repeated-trial consistency and clustered uncertainty.
Evaluation split
Agent evaluation paired
Key takeaways
- Run both agents on the same frozen cases and look at per-case differences before any average.
- Pairing cancels case difficulty and shrinks the error bar without adding cases.
- The discordant cases, where one agent passed and the other failed, are the evidence and the error analysis.
- Decide per task family with business severity, not on a single overall winner.
- Repeat trials per case and keep escalation separate from failure.
Why compare agents on matched cases before comparing averages?
Because two averages computed on different cases, or on the same cases without pairing, mix up the agents' skill with the cases' difficulty. When both agents attempt exactly the same cases, each case's difficulty cancels in the per-case difference, and what remains is the evidence about which agent is better and where.
A paired comparison asks a sharper question than a leaderboard. Not 'which agent averaged higher', but 'on the cases where the two agents disagreed, which one was right, how often, on which task families, and with what business consequence'. That question has a smaller error bar for the same number of cases, and its answer comes with the list of cases you need for error analysis.
This page gives the paired outcome comparison and error analysis as a template, walks an illustrative example through it, and marks where pairing is not enough.
What does the statistics literature say about paired evaluation?
The statistical case is old and well documented. NIST's Engineering Statistics Handbook describes paired observations as measurements matched one to one across two samples, analyzed by taking the difference within each pair and testing whether the mean difference is zero. Applied to agents, the pair is one case attempted by agent A and by agent B.
Miller's paper on adding error bars to language model evaluations makes the same recommendation for model comparisons: run inference on question-level paired differences rather than on each model's summary score, because scores on the same questions are correlated across models and pairing reduces variance without adding questions. It also recommends clustered standard errors when questions come in related groups, which applies to agent cases that share an account, a document set or a scenario template.
A later analysis by Wang of the noise in language model evaluations reports smaller standard errors for paired than for unpaired comparisons, and notes that pairwise comparisons are often more informative than a separate error bar for each model. For binary outcomes, pass or fail, the matched-pairs test is McNemar's: the statsmodels implementation uses the exact binomial distribution on the discordant cases, those where one agent passed and the other failed.
The cases where both agents passed or both failed carry no information about which agent is better, however many there are. They still matter for the business, since a family both agents fail is a gap neither choice fixes, but they drop out of the head-to-head test. For graded scores rather than pass or fail, the same logic applies through a paired t-test on per-case score differences.
Where do averages mislead in agent comparisons?
Averages mislead in three common ways, and each one is invisible until cases are lined up side by side.
At a glance
Where do averages mislead in agent comparisons
- Different case sets: a vendor reports a success rate on its own suite, another on a different suite. The higher number may only reflect an easier mix of tasks.
- Different case mixes inside one set: agent A attempted every case, while agent B timed out or was excluded on the hardest ones and is averaged over the rest. Unattempted cases must count as failures or be removed from both agents.
- Offsetting errors: two agents with near-equal averages can disagree on many cases, each winning a different task family. The average hides a routing decision worth making.
- Severity blindness: a one-point gap in average success can come from minor formatting misses on one side and irreversible wrong actions on the other.
- Run-to-run noise: an agent that passes a case once may fail it on the next trial. Without repeated trials per case, part of any gap is luck.
What goes into a paired outcome comparison and error analysis template?
The asset has two parts. The first is a per-case record that every comparison fills in, one row per case per agent pair. The second is the summary those rows produce. Both are designed so a business task owner can audit any conclusion back to individual cases.
Fill the case record from evidence, not from the agents' own reports: end-state checks, system records, test results or adjudicated labels. Fields below are the minimum.
- Summary 1: the paired table per task family, with counts of both pass, A only, B only and both fail.
- Summary 2: the difference in success rate with a paired or clustered confidence interval, and McNemar's exact p-value on the discordant cases.
- Summary 3: the discordant cases listed, grouped by error type and severity, for the error analysis.
- Summary 4: a coverage matrix of task families against the target workload, marking families with too few cases to judge.
- Summary 5: repeated-trial consistency per agent where trials were run, such as the share of cases passed on every trial.
| Field | What it holds | Why it is needed |
|---|---|---|
| case_id and case_version | Stable identifier and the version of inputs both agents saw | Proves both agents attempted the same case |
| task_family and cluster_id | Business task type, plus the account, document set or template the case shares with others | Supports per-family results and clustered standard errors |
| outcome_a and outcome_b | Pass, fail, abstained or escalated, per trial if trials are repeated | Builds the paired table; keeps abstention separate from failure |
| completion_evidence | Pointer to the end-state check, record or test that decided the outcome | Lets a reviewer verify any disputed case |
| error_type_a and error_type_b | Category from a fixed list: wrong action, missing step, policy breach, tool error, hallucinated fact, timeout | Turns disagreements into an error analysis |
| severity | Business severity of each failure on a 1 to 4 scale set by the task owner | Separates cosmetic from costly failures |
| adjudicator_note | Free text when a human resolved a disputed outcome, with adjudicator id | Preserves disagreement and its resolution |
In what order should the comparison run?
The procedure below turns the template into a decision. It assumes a frozen case set and both agents run under the same tools, permissions and time limits.
- Freeze the case set and its version, and write down the target workload it is meant to represent.
- Run both agents on every case, ideally several trials each, with the same environment state at the start of every trial.
- Score each trial from completion evidence; route disputed outcomes to an adjudicator and record the note.
- Build the paired table per task family before looking at any overall average.
- Compute the paired difference with clustered standard errors where cases share a cluster, and McNemar's exact test on discordant counts.
- Read every discordant case in the high-severity families and label the error type for both agents.
- Write the decision as a statement per task family, for example 'B for disputes, no reliable difference for address changes', not a single winner.
- Record which important task families or failure types the set does not cover, so the decision is not stretched beyond them.
What happens when two support agents have nearly identical averages?
All numbers in this example are illustrative. A team compares two customer support agents on 300 frozen cases from three task families: 150 address changes, 100 refund requests and 50 escalated complaints. Agent A succeeds on 61% of cases and agent B on 60%. On averages alone, A looks slightly better.
The paired table tells a different story. Both agents pass 150 cases and both fail 87. A alone passes 33 and B alone passes 30, so the two agents disagree on 63 cases, and the near-even split of discordant cases gives no reliable overall difference; an exact McNemar test on 33 against 30 is nowhere near conventional significance. By family, A wins most discordant address changes, while B wins most discordant escalated complaints.
Error analysis on the discordant cases finds that A's complaint failures are mostly issuing a refund without the required approval, rated severity 4 by the task owner, while B's address change failures are mostly a missing confirmation message, rated severity 1. The decision becomes a routing one: B for complaints, either agent for address changes pending more cases. The averages would have chosen A for everything.
When are matched cases not enough?
Pairing removes case difficulty from the comparison. It does not fix a case set that misses the workload, a grader that rewards the wrong behavior, or an environment that behaves differently between runs.
Repeated trials matter for agents more than for single-turn models. The tau-bench paper defines pass^k, the chance that all k independent trials of a task succeed, and reports that agents which solve a task once often fail to solve it consistently. Compare agents on the same cases and the same number of trials, and report consistency alongside one-trial success.
Scope differences also break naive pairing. If one agent may escalate to a human and the other may not, an escalation is neither a pass nor a fail. Record it as its own outcome, decide with the task owner whether it is acceptable for that family, and compare rates of acceptable escalation separately.
The grader must also be held fixed. If outcomes are judged by a language model or by people, both agents' transcripts for a case should be scored by the same judge version in the same session, ideally without the judge knowing which agent produced which transcript. A judge that changes between runs, or that has learned one vendor's phrasing, can create a paired difference that has nothing to do with completing the task. Spot-check a sample of judged discordant cases against completion evidence before trusting the totals.
Finally, a frozen case set can leak. If one agent's builder had seen the cases, its paired advantage measures memory. Keep the set private, version it, and retire cases that have been exposed.
Which task families and failures might your evaluation set be missing?
A paired comparison is only as good as its coverage. SWE-bench is a useful model of a shared, inspectable case set: every model attempts the same task instances drawn from real GitHub issues, and each patch is checked against real tests. Business agents rarely have anything that clean, so the coverage check has to be explicit.
Compare the task family counts in your frozen set with the live workload, and list failure types the task owner fears most, then check whether any case in the set could reveal them. Common gaps include cases that need a policy exception, cases with conflicting records, multi-step cases where an early mistake surfaces late, and cases where the right action is to do nothing.
Stop and narrow the claim when a family has too few discordant cases to read, when adjudicators disagree on outcomes more often than the agents do, or when the decision depends on statistical modeling beyond a paired test. Those calls belong to an evaluation statistician working with the business task owner.
What should a request for matched case sets specify?
A good paired comparison needs cases with recorded outcomes, task families that match the workload, and clusters that are labeled, such as account, site or template. When requesting records for an evaluation set, specify the families and counts, the evidence that decides completion, and the cluster fields to keep.
For teams that lack such records, SourceX prepares licensed datasets from selected business records, found at U.S. businesses for each request rather than drawn from stock. The request states scope, volume, language, format, timing and licensing needs, so the family mix and cluster fields are fixed before sourcing begins.
Frequently asked questions
What is a paired comparison in agent evaluation?
It is a comparison in which both agents attempt exactly the same cases and the analysis works on the per-case difference in outcome. Case difficulty cancels within each pair, so the remaining difference is better evidence about the agents than two separate averages.
Which statistical test fits pass or fail outcomes on matched cases?
McNemar's test is the standard choice for paired binary outcomes. It uses only the discordant cases, where one agent passed and the other failed. For scores rather than pass or fail, a paired t-test on per-case differences, with clustered standard errors where cases are related, is the usual starting point.
How many cases does a paired agent comparison need?
Enough discordant cases in each task family you want to decide on, not just enough cases overall. If the agents agree on almost everything, even a large set gives few discordant cases. Plan the set size per family from the disagreement rate you expect.
Should unfinished or timed-out runs be dropped?
Not from one agent alone. Count an unattempted or timed-out case as a failure for that agent, or remove the case for both agents and report that you did. Dropping it for one side silently gives that agent an easier set.
Can I compare vendor agents on their own published benchmarks?
Only as a first filter. Different suites have different case mixes, graders and trial counts, so their averages are not comparable. Run both agents on your own frozen cases under the same conditions before making a decision.
Sources
- NIST's Engineering Statistics Handbook describes paired observations and analyzes them through the differences within each pair, tested against a mean difference of zero. Source
- Miller recommends comparing models on question-level paired differences, which reduces variance, and using clustered standard errors when questions come in related groups. Source
- Wang reports smaller standard errors for paired than unpaired model comparisons and notes that pairwise comparisons are often more informative than per-model error bars. Source
- The statsmodels McNemar test works on a 2x2 table of paired outcomes and, in exact mode, uses the binomial distribution on the discordant counts. Source
- The tau-bench paper defines pass^k as the probability that all k trials of a task succeed and reports that agents are inconsistent across repeated trials. Source
- SWE-bench builds task instances from real GitHub issues and evaluates each generated patch against real tests. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.