AI uses for records
Agent evaluation sample size planning for comparing two agents
By SourceX Editorial · Updated
9 sections
Short answer
To compare two agents, count tasks rather than runs and size the set from four inputs: the smallest success gap that would change your decision, the spread of per-task differences measured in a pilot, a minimum per task family, and the lowest severe failure rate you must rule out. Compute each requirement, take the largest, add attrition, and freeze the plan before running.
Evaluation split
Agent evaluation sample size
Key takeaways
- Tasks are the sample; repeated trials measure consistency but do not add coverage.
- A paired design, where both agents run the same tasks, cancels task difficulty and needs far fewer tasks.
- The spread of per-task differences comes from a pilot and is the assumption most likely to be wrong.
- Family floors and a severe failure guard often set the total, not the average success comparison.
- Report an interval on the difference, and call a result inconclusive when it spans both no change and the decision gap.
- Register the decision rule, task list and analysis method before either agent runs on the main set.
How many business tasks do you need to compare two agents?
Count tasks, not runs, and size the set from four inputs: the smallest difference in task success that would change your decision, how much the two agents' per-task results differ from one task to the next, how many task families each need their own read, and how rare a severe failure you must rule out. A pilot of a few dozen tasks run on both agents gives the second input. The largest of the resulting counts is your sample.
Most teams start from a round number, such as 100 tasks, and discover afterward that the interval on the difference is wider than the gap they cared about. Planning reverses that order. You write down the decision, the assumptions and the formula, compute the count, and only then build or buy tasks. The worksheet below makes each step explicit so an evaluation statistician and a business task owner can check it together.
This page is a planning method and worksheet, not a report of any agent's results. The figures in the worked scenario are illustrative assumptions chosen to show the arithmetic.
The worksheet's input dictionary
Every input has an owner and a source. The business task owner sets the decision inputs; the pilot supplies the variance inputs; the evaluation statistician checks the formula and the uncertainty method. Write the values down before the main run and do not change them after seeing results.
| Input | Symbol | Who sets it and how | If it is wrong |
|---|---|---|---|
| Decision gap | delta | Task owner: the smallest difference in mean task success that would justify switching agents | Too small inflates the set; too large misses a gap that matters |
| False alarm rate | alpha | Statistician: usually 0.05, two-sided | Looser values switch agents on noise more often |
| Power | 1 minus beta | Statistician: usually 0.80 | Lower power misses real gaps more often |
| Spread of per-task differences | s_d | Pilot: standard deviation across tasks of agent B's minus agent A's mean success | Underestimating it is the most common reason a study comes out inconclusive |
| Trials per task | k | Team: repeated runs of the same task per agent, often 3 to 5 | Too few hides inconsistency; too many spends budget that new tasks would use better |
| Task families and floors | F, m | Task owner: families that need their own read and the minimum tasks in each | A missing family means the result does not cover that work |
| Severe failure ceiling | r_max | Task owner: the highest severe failure rate the business can accept | Often the input that drives the total |
| Attrition | a | Pilot: share of tasks lost to environment errors or unverifiable outcomes | Ignoring it leaves the final set short |
Why tasks, not trials, are the sample
Repeated runs of the same task are not independent evidence about new tasks. Miller's paper on adding error bars to evaluations treats questions as draws from a larger population and recommends clustered standard errors when questions come in related groups; the same logic applies to several trials of one task, which share everything about that task. Resampling answers reduces variance, but it does not add new task coverage.
Trials still matter, because agents are inconsistent. tau-bench computes pass^k for each task as the number of ways to choose k successful trials divided by the number of ways to choose k trials, then averages across tasks. The leaderboard in the original tau-bench repository, whose tasks its maintainers now mark as outdated, shows the score falling as k rises: one tool-calling configuration on the retail domain drops from 0.604 at k of 1 to 0.383 at k of 4. If your business needs an agent that works every time, measure consistency with several trials per task.
A simple way to see the trade: the variance of a task's mean success is the between-task variance plus the within-task variance divided by k. More trials shrink only the second term. Once within-task noise is small next to between-task differences, extra budget buys more precision as new tasks than as more trials.
| Trials per task | What it buys | What it cannot fix | When to choose it |
|---|---|---|---|
| 1 | Most tasks for the budget | No read on consistency; single-run noise stays in every task | Screening many candidate agents cheaply |
| 3 | A per-task success fraction and a basic consistency signal | Coverage of task families missing from the set | Default for comparing two agents |
| 5 or more | pass^k at higher k and tighter per-task estimates | Between-task differences, which dominate once tasks are few | Deployments where one failure in several runs is unacceptable |
| Adaptive | Extra trials only on tasks where the agents disagree | Bias if the rule for adding trials is not fixed in advance | Tight budgets, with the rule written before the run |
Working through the sample size, step by step
The core formula is the normal approximation for a paired comparison: tasks needed equals the square of the sum of the two z values, times the variance of the per-task differences, divided by the square of the decision gap. At an alpha of 0.05 two-sided and 80 percent power the z term is about 7.85. Because each task is scored for both agents, task difficulty cancels in the difference, which is why paired designs need far fewer tasks than two separate samples.
Steps
Working through the sample size, step by step
For a single pass or fail trial per task, the McNemar test is the paired alternative; the statsmodels implementation uses the binomial distribution on the tasks where exactly one agent succeeded when the exact option is on. A companion page on eval set size and statistical power covers that single-trial case and its discordance tables. The steps below extend it to multi-trial business tasks, task families and a severe failure guard.
Checklist · 8 items
- Step 1. Write the decision: switch to agent B if its mean task success is higher by at least delta, with no increase in severe failures beyond r_max.
- Step 2. Run a pilot of 40 to 60 tasks drawn from the same families as the main set, with k trials per agent per task. Compute each task's success fraction for each agent and the difference.
- Step 3. Take s_d as the standard deviation of those differences. Use the upper end of a reasonable range rather than the point estimate, because a small pilot understates spread as often as it overstates it.
- Step 4. Compute base tasks: 7.85 times s_d squared, divided by delta squared. Round up.
- Step 5. Apply family floors: if F families each need at least m tasks, the total is at least F times m, allocated in proportion to the real workload mix above the floor.
- Step 6. Apply the severe failure guard: find the number of tasks for which zero observed severe failures would put the upper confidence bound below r_max.
- Step 7. Take the largest count from steps 4 to 6 and divide by one minus the attrition rate.
- Step 8. Freeze the plan, the task list and the analysis method before running either agent on the main set.
0 of 8 ticked. Tick each item as you go.
Coverage gaps a bigger sample cannot fix
A sample size answers how precise the estimate is, not what it covers. Build a coverage matrix before counting: rows are task families from the real workload, columns are the facts each family needs recorded. A family with a share of real work but no tasks in the set is a blind spot that no amount of precision repairs.
The matrix below is an illustrative layout for a finance operations agent. Shares and floors are assumptions for the example; replace them with your own workload counts from ticket queues or system logs.
| Task family | Share of workload (illustrative) | Floor in the set | Severe failure to watch | How completion is verified |
|---|---|---|---|---|
| Invoice matching and coding | 35% | 30 tasks | Payment to the wrong supplier | Posted invoice matches purchase order and receipt |
| Vendor master changes | 10% | 30 tasks | Bank details changed without verification | Change record and approval log |
| Customer refunds | 20% | 30 tasks | Refund above the captured amount | Refund record against original charge |
| Period-end accruals | 10% | 30 tasks | Entry posted to a closed period | Journal entry date and period status |
| Collections notes and follow-ups | 15% | 30 tasks | Message sent to the wrong customer | Activity record on the right account |
| Exception escalations | 10% | 30 tasks | Escalation skipped when policy requires it | Handoff record and policy reference |
Worked scenario (illustrative): the severe failure guard sets the total
Assume a team will switch from agent A to agent B if B's mean task success is at least 5 points higher. A pilot of 50 tasks with 3 trials per agent gives a standard deviation of per-task differences of about 0.30, and roughly 1 task in 10 is lost to environment errors. Six families each need at least 30 tasks. All of these figures are illustrative.
Step 4 gives 7.85 times 0.09, divided by 0.0025, which is 283 tasks. The family floor of 180 is lower, so it does not bind. Adding attrition gives about 315 tasks.
Now the severe failure guard. If agent B shows zero severe failures on 315 tasks, the two-sided 95 percent Clopper-Pearson upper bound on its severe failure rate is about 1.2 percent; on 300 tasks it is also about 1.2 percent, and on 150 tasks about 2.4 percent. If the business can only accept a rate below 0.5 percent, about 736 tasks with zero severe failures would be needed before attrition. The guard, not the success comparison, sets the size of the study. The team can either fund it, accept a looser ceiling, or test severe failures in a separate targeted set built around known failure triggers.
| Decision gap | s_d of 0.20 | s_d of 0.30 | s_d of 0.40 |
|---|---|---|---|
| 5 points | 126 tasks | 283 tasks | 503 tasks |
| 10 points | 32 tasks | 71 tasks | 126 tasks |
| Single agent rate within plus or minus 5 points, for contrast | 385 tasks at a 50 percent rate | 385 tasks at a 50 percent rate | 385 tasks at a 50 percent rate |
| Zero severe failures, upper bound below 0.5 percent | About 736 tasks | About 736 tasks | About 736 tasks |
The two assumptions that swing the task count, and reporting the uncertainty
The spread of per-task differences and the decision gap move the answer most, because the count grows with the square of one and the inverse square of the other. Doubling the gap you care about cuts the tasks needed by about four times; doubling the spread multiplies them by four. Show the sensitivity table with the plan so readers can see how fragile the count is.
Report the result as an interval on the difference between agents, not two separate scores, and say how it was computed. Below a few hundred tasks, intervals based on the central limit theorem tend to be too narrow; a 2025 position paper argues that these methods fail to give valid uncertainty on small, specialized benchmarks and recommends alternatives. A paired bootstrap that resamples tasks, keeping each task's trials together, respects both the pairing and the clustering. For rates near zero, statsmodels documents that the Clopper-Pearson interval has coverage of at least the nominal level but is conservative.
Before the main run, register the plan in writing:
- The decision rule, the gap, alpha, power and the severe failure ceiling.
- The task list with family labels, frozen before either agent is run.
- Trials per task and any rule for adding trials.
- The success definition and how completion is verified from records.
- The interval method and the unit of resampling.
- What happens if the interval includes both no difference and the decision gap: an inconclusive result, not a win for either agent.
- No peeking at interim results unless a sequential method was chosen in advance.
Where this plan breaks and a statistician should step in
The worksheet assumes tasks are drawn from a stable workload, each task's outcome can be verified, and the two agents face the same conditions. Each assumption fails in common situations. Workloads drift by season or policy change, so a set built from one quarter may not represent the next. Simulated users and LLM judges add their own variance, which shows up as extra within-task noise and can bias one agent if it suits the simulator's style. Public tasks may be contaminated; a fixed benchmark such as SWE-bench Verified, a subset of 500 problems confirmed solvable by engineers, is useful for comparison but cannot stand in for a firm's own workload.
Bring in an evaluation statistician when you compare more than two agents, when you need per-family conclusions rather than descriptive family results, when outcomes are graded rather than pass or fail, or when you want to stop early. Each of these changes the formula, and a wrong correction for multiple comparisons can turn noise into a decision.
Sourcing enough real business tasks for a powered comparison
The counts in this worksheet often exceed what a team can write by hand, especially once family floors and a severe failure guard apply. Real business records are the usual source: historical tickets, transactions and case files that carry a verifiable outcome. When requesting them, specify the families, the floor per family, the outcome evidence each task needs, and the share you will hold out.
Teams that cannot write enough tasks themselves can request records through SourceX, which sources datasets from U.S. businesses for each request instead of holding inventory; a request names scope, volume, language, format, timing and licensing needs. A supplier worth approaching usually has at least two years of steady records, tens of thousands of them in a single system, though rare kinds of work can qualify with smaller sets.
Frequently asked questions
Is 100 tasks enough to compare two AI agents?
Only for large differences. With a per-task difference spread of 0.30, 100 paired tasks can reliably detect a gap of roughly 8 points at the usual settings, not 2 or 3. Run the worksheet with your own pilot spread and decision gap before settling on a number.
Should I run more trials per task or add more tasks?
Use about three trials per task to measure consistency, then spend extra budget on new tasks. Trials reduce noise within a task but cannot cover task types you did not include, and between-task differences usually dominate once the set is small.
What is pass^k and when does it matter?
pass^k estimates the chance an agent succeeds on all of k independent attempts at a task, averaged across tasks. It matters when a deployment cannot tolerate occasional failures, and it usually falls as k rises, so report the k that matches how often the agent will retry real work.
How do I size an evaluation for rare but severe failures?
Ask how low an upper bound you need on the severe failure rate, then find the number of tasks where zero observed failures would put the bound below it. That count is often larger than the one needed to compare average success, so consider a separate targeted set built around known failure triggers.
How should results be reported when the interval is wide?
Report the interval on the difference between agents and call the result inconclusive if it contains both no difference and the decision gap. Do not report two separate scores with overlapping intervals as proof the agents are equivalent.
Sources
- Miller's Adding Error Bars to Evals treats evaluation questions as drawn from a super-population, gives formulas for comparing two models and planning evaluations, and recommends clustered standard errors for related questions and resampling answers to reduce variance. Source
- A 2025 position paper argues that central limit theorem based uncertainty fails on LLM evaluations with fewer than a few hundred datapoints. Source
- tau-bench computes pass^k per task as comb(c, k) divided by comb(n, k), averaged across tasks. Source
- The tau-bench leaderboard reports Pass^1 to Pass^4, with scores falling as k rises, for example 0.604 to 0.383 for one retail configuration. Source
- statsmodels' McNemar test uses the binomial distribution on cases that differ between the paired samples when exact is True. Source
- statsmodels documents that the Clopper-Pearson interval has coverage of at least 1 minus alpha but is conservative, and gives the sample size formula for a desired confidence interval half length. Source
- SWE-bench Verified is a subset of 500 problems that software engineers confirmed are solvable. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.