AI uses for records
Agent evaluation confidence intervals: is a 2-point benchmark lead real?
By SourceX Editorial · Updated
10 sections
Short answer
To quantify uncertainty around a small benchmark lead, run both agents on the same tasks with repeated trials, compute per-task differences, and bootstrap the mean difference by resampling independent task families. In an illustrative 150-task case, a 2.4-point lead carried a 95 percent interval from -3.1 to +8.2 points, so the evaluation did not separate the agents. Report the interval, the unit resampled and untested families.
Evaluation split
Agent evaluation confidence
Key takeaways
- Put the interval on the paired difference between agents, not on each score separately.
- Resample the independent unit, usually a task family, and say which unit you used.
- In the illustrative case, a 2.4-point lead had a 95 percent interval of -3.1 to +8.2 points: not separated.
- Single-trial comparisons inflate both the lead and its uncertainty; fix trials per task in advance.
- An interval says nothing about task families the evaluation never sampled or about grader error.
- Narrowing an interval needs more independent families, not more runs of the same tasks.
How do you quantify uncertainty around a small benchmark score lead?
Quantify it on the difference, not on each score. Run both agents on the same frozen tasks, average each task's result over repeated trials, take the per-task difference, then bootstrap by resampling the independent unit (usually a task family, not a single task) with replacement thousands of times. Report the lead with its 95 percent interval. If the interval includes zero, the honest headline is that the evaluation did not separate the two agents.
That procedure answers the question a buyer actually has. A leaderboard shows two numbers; a decision needs to know whether the gap between them would survive a different draw of tasks and a different set of runs. The worked case below shows how a lead of 2.4 points looks once its uncertainty is attached, and how the answer changes depending on which unit you resample.
Three sources of noise hiding inside an agent's headline score
A headline success rate mixes at least three kinds of variation, and a small lead can come from any of them. Reporting one number hides all three.
At a glance
Three sources of noise hiding inside an agent's headline score
- Task sampling: the evaluation is a sample of the work the agent will face. A statistical treatment of language model evals published by Anthropic frames eval questions as drawn from an unseen super-population and gives formulas for standard errors, for differences between two models, and for planning an evaluation.
- Run-to-run variation: agents are stochastic, so the same task can pass on one trial and fail on the next. The tau-bench leaderboard reports pass^1 through pass^4, and every strategy with reported scores falls as k rises; in the airline domain, the top tool-calling entry drops from 0.460 at pass^1 to 0.225 at pass^4.
- Correlated tasks: tasks built from the same customer, vendor, template or workflow tend to succeed or fail together. scikit-learn's documentation on grouped data puts it plainly: the i.i.d. assumption is broken if the underlying process yields groups of dependent samples, and the eval-statistics paper above recommends clustered standard errors when questions fall into related groups.
The starting state: two agents, 150 illustrative tasks
The numbers in this case are illustrative. They come from a simulation built for this page, not from any real agent or vendor, and are reported so each step can be followed. The setting is an accounts payable exception queue: each task asks the agent to resolve one invoice exception, and success is read from the ERP afterward, not from the agent's final message.
Agent B finished 2.4 points ahead. A results slide would say B is better. The rest of the case tests whether the evaluation supports that sentence.
| Design element | Value (illustrative) | Why it matters |
|---|---|---|
| Task families | 30, one per supplier relationship | Tasks within a family share documents and quirks, so the family is the independent unit |
| Tasks | 150, five per family | Every task run by both agents, so results can be paired |
| Trials per task | 3 per agent, 450 runs each | Separates run-to-run variation from task difficulty |
| Success definition | Exception closed in the ERP with correct posting and approval | A status field or a confident final message does not count |
| Agent A success rate | 57.8 percent of 450 runs | Headline score, before uncertainty |
| Agent B success rate | 60.2 percent of 450 runs | Headline lead of 2.4 points |
The bootstrap interval example, step by step
The asset below is the full procedure with each interval it produced. The point estimate never changes; only the resampling unit does, and with it the honest width of the interval. Each method used 10,000 resamples and a percentile interval.
Paired resampling cut the interval width by about a third compared with treating the two agents as independent samples, because both agents found the same tasks hard. In this data, clustering by family widened the paired interval only slightly, which tells you the per-task differences were not strongly correlated within families here. That is a finding about this evaluation, not a license to skip the step: when differences do cluster, task-level resampling can make an interval look far tighter than it is.
Checklist · 6 items
- Step 1. For each task, average each agent's three trials into a task score between 0 and 1.
- Step 2. Compute the per-task difference, B minus A; the mean over 150 tasks is the 2.4-point lead.
- Step 3. Group the 150 differences into their 30 families.
- Step 4. Draw 30 families with replacement, take the mean difference across their tasks, and repeat 10,000 times.
- Step 5. Read the 2.5th and 97.5th percentiles of those means as the 95 percent interval.
- Step 6. Write the result as the lead with its interval, the unit resampled, the number of families and trials, and whether zero is inside.
0 of 6 ticked. Tick each item as you go.
| Method | What is resampled | 95 percent interval for B minus A | Interval width | Verdict |
|---|---|---|---|---|
| Unpaired bootstrap | Tasks, separately for each agent | -5.6 to +10.7 points | 16.3 points | Not separated |
| Paired task bootstrap | Tasks, same draw for both agents | -3.1 to +8.0 points | 11.1 points | Not separated |
| Paired cluster bootstrap | Task families, same draw for both agents | -3.1 to +8.2 points | 11.3 points | Not separated; report this one |
| Drop-one-family check | Lead recomputed with each family removed | Lead ranged 1.1 to 3.2 points | 2.1 points | No single family drives the lead |
What single trials, pass^3 and family-level results add
One trial per task would have told a different story. Using only each agent's first trial, B led by 6.0 points, and the family-clustered interval ran from -4.0 to +16.0 points. A single run inflates both the lead and the uncertainty, which is why a one-shot comparison is the weakest evidence a vendor can offer.
Reliability tells yet another story. Counting only tasks solved on all three trials, the pass^3 measure tau-bench popularized, A scored 29.3 percent and B 37.3 percent of tasks. Where the business needs the agent to get the same case right every time, that gap may matter more than the average, and it needs its own interval computed the same way.
Family-level results add texture the average hides: B did better in 14 families, worse in 11 and tied in 5. A lead built from many small, mixed wins is a different product decision from one built on a few families where B is far ahead, even when the averages match.
Finally, the share of bootstrap resamples in which B led was about 80 percent. That is a description of the resampling distribution, not a probability that B is better, and it should not appear in a results slide without that caveat.
What the results table should say about a close comparison
Write the comparison so that a reader who never saw the runs can tell what was resampled and what was left out. The rows below are the minimum for any claim that one agent beat another; the right-hand column shows how the illustrative case would fill them in. A results slide that cannot fill every row is not ready to support a vendor or model decision.
| Report field | What to state | Illustrative case entry |
|---|---|---|
| Comparison | Which agents, which version of each and which task set version | Agent A and Agent B on AP exception set version 1 |
| Lead and interval | Point difference with its 95 percent interval and method | +2.4 points, -3.1 to +8.2, paired family-clustered percentile bootstrap |
| Independent unit | What was resampled and how many of them | 30 supplier families, five tasks each |
| Trials | Runs per task per agent, fixed in advance | 3 trials each, 450 runs per agent |
| Reliability view | pass^k or another consistency measure with its own interval | pass^3: 29.3 versus 37.3 percent of tasks, interval to be computed the same way |
| Untested families | Business work with no tasks in the set | Duplicate invoices, foreign-currency exceptions |
| Grader check | How passes were verified against the system of record and the error found | Sample of passes reopened in the ERP |
| Conclusion | Separated, not separated, or separated only in named families | Not separated; collect more families before choosing |
Which supplier families never made it into the 150 tasks?
An interval only describes the population the tasks were drawn from. If the evaluation has no task family for duplicate invoices, foreign-currency exceptions or vendor bank-detail changes, the bootstrap says nothing at all about them, however narrow it gets.
Before computing any interval, set a coverage matrix beside the task list: the families the business actually handles, their share of volume and the cost of a wrong action in each. Mark families with no tasks as untested in the report rather than letting them disappear. A 2-point overall lead can coexist with one agent failing every task in a family that carries most of the financial exposure.
Where severity differs sharply, report severity-weighted results and the unweighted rate side by side, and bootstrap both with the same family draws so the two intervals stay comparable.
Does each pass trace back to the ERP outcome, or only to a status change?
The bootstrap measures sampling noise; it cannot see grading error. If the grader counts a task as passed because the ticket status changed to resolved or because the agent wrote a convincing summary, every resample inherits that bias, and the interval becomes a precise estimate of the wrong thing.
For each task, the evaluation record should link four things: the starting state, the evidence the agent used, the decision it made and the outcome observed afterward in the system of record. In the accounts payable case, that means the exception record, the purchase order and receipt it matched against, the posting it made and the ledger state once approvals cleared. Spot-check a sample of passes against those links before trusting any interval, and report the grader error found alongside the result.
Counterexample: when a bootstrap interval misleads
The bootstrap is a good default, not a universal one. Each case below is a reason to change method or to stop and ask an evaluation statistician, and the business task owner should confirm the family definitions before any of them is computed.
At a glance
Counterexample: when a bootstrap interval misleads
- Few families: with only a handful of independent units, a percentile bootstrap can produce intervals that are too narrow. Get more families or use a method designed for small samples, and say so.
- Scores near 0 or 100 percent: resampling cannot produce values outside what was observed, so an agent that passed every sampled task gets a degenerate interval.
- Many comparisons: comparing many agents, prompts or configurations and reporting the best pair multiplies false leads; adjust for multiplicity or confirm on a fresh set.
- Peeking and stopping: rerunning until the interval excludes zero invalidates it. Fix trials and families before looking.
- Changed tasks: if either agent saw a different task version, the pairing is broken and the paired interval is meaningless.
Getting enough independent task families to settle the question
Narrowing the interval takes more independent families, not more runs of the same ones. Paired bootstrap resampling has a long history for this kind of question; Koehn's 2004 paper on machine translation used bootstrap resampling to judge whether a score difference between systems reflected real quality or chance. In this illustrative case, the clustered standard error of the lead was about 2.9 points, so a rough scaling suggests on the order of 250 comparable families would be needed for the interval's half-width to shrink to 2 points.
Tooling is not the obstacle. SciPy's bootstrap function supports a paired option that resamples the same indices for every array, defaults to the bias-corrected and accelerated method and to 9,999 resamples. Inspect, an open-source evaluation framework, offers a bootstrap standard error metric with 1,000 samples by default and a clustered standard error that reads a cluster key from each sample's metadata. The hard part is finding hundreds of real, independent task families with outcomes you can verify.
Independent families are where SourceX can help an evaluation team: a data request can state how many distinct companies or workflows the set should span, along with scope, volume, language, format, timing and licensing needs, and SourceX then sources records for it from U.S. businesses instead of an inventory. Each supplying business approves scope and terms and signs before its records ship.
What does the plus-or-minus after a leaderboard score actually cover?
Usually one of the three noise sources listed above, and the report must say which. MLE-bench, a set of 75 Kaggle competitions used to test machine learning engineering agents, recommends running at least three seeds and reporting the mean plus or minus one standard error, noting that grading is deterministic while agents can be high-variance. Its aggregation script takes that error from the spread of per-seed scores, so the band measures run-to-run variation on those 75 competitions, not how an agent would fare on a fresh draw of similar ones.
Evaluation libraries usually print the other kind. The EleutherAI evaluation harness computes the standard error of a mean metric as the standard deviation across test items divided by the square root of their count: item sampling within one run. A deck quoting 64.4 plus or minus 1.2 should say which error that is, how many seeds and items lie behind it, and whether the band is one standard error or a 95 percent interval.
We recomputed the leaderboard from the grading reports the repository publishes, and all 28 rows of its main table reproduce to two decimals. Resampling competitions as well as seeds widens every row's 95 percent interval, by 1.6 to about 12 times. The six highest rows:
| Leaderboard row (All, 75 competitions) | Published mean ± SEM | Seed-only 95% interval | Seeds and competitions resampled, 95% |
|---|---|---|---|
| Famou-Agent 2.0, Gemini-3-Pro-Preview | 64.44 ± 1.18 | 62.1 to 66.7 | 53.8 to 74.7 |
| AIBuildAI, Claude-Opus-4.6 | 63.11 ± 0.44 | 62.2 to 64.0 | 52.0 to 73.3 |
| CAIR MARS+, Gemini-3-Pro-Preview | 62.67 ± 0.77 | 61.2 to 64.2 | 52.0 to 73.3 |
| MLEvolve, Gemini-3-Pro-Preview, 12 hours | 61.33 ± 1.33 | 58.7 to 63.9 | 51.1 to 71.6 |
| PiEvolve, Gemini-3-Pro-Preview, 24 hours | 61.33 ± 0.77 | 59.8 to 62.8 | 50.7 to 72.0 |
| Famou-Agent 2.0, Gemini-2.5-Pro | 59.56 ± 0.89 | 57.8 to 61.3 | 48.9 to 70.2 |
Which leaderboard gaps survive once the competitions are resampled?
Very few. Comparing neighbors on the same competition draw, which pairs them the way the page pairs agents on tasks, separates 2 or 3 of the 27 adjacent pairs in the main table, depending on the resampling seed. Non-overlapping seed-only intervals would separate 5, and two or three of those fail once competitions are resampled. Five rows cannot be told apart from the top entry; for the Gemini-2.5-Pro Famou-Agent 2.0 row, the interval's lower end sits at or just below zero.
Check the denominator too. The README marks four main-table rows as computed by padding incomplete seeds with failing scores: in the run groups assigned to them, both PiEvolve rows and the Operand ensemble have no grading report for 22 of the 75 competitions, and Leeroo has none for 35. MLEvolve carries no such note but lacks reports for one competition, and its 61.33 reproduces when that competition is counted as a failure. A padded score is a floor, not an estimate. Even the run-group mapping needed repair: at that commit its stored content is an unresolved merge conflict between two Git LFS pointers, so we rebuilt it from the two versions those pointers name, whose SHA-256 hashes match.
How much of a lead can time or hardware buy, with the agent unchanged?
Enough to reorder a table. With the same agent and model, PiEvolve scored 61.33 given 24 hours and 52.00 given 12: a 9.3-point gain with a paired 95 percent interval of 4.4 to 15.1, both rows padded on the same 22 competitions, and larger than any gap among the six rows above. Entries also ran on hardware from one V100 GPU to one H200. Not every change matters: for GPT-4o on AIDE scaffolding, a second A10 GPU moved the score by -0.6 points (interval -4.1 to 2.8) and a CPU-only setup by +1.8 (-1.9 to 5.5).
Before a benchmark score enters an investment memo, ask for:
- Per-seed and per-task results, so the interval can be recomputed rather than copied.
- Which error the band shows (seeds, items or both), at what confidence and over how many runs.
- Every task in the denominator, with missing or padded ones named.
- The same time limit, hardware, model version and sampling settings for every system compared.
- A paired comparison with the nearest competitor on one task draw, not two separate error bars.
- The benchmark commit, who ran it, and whether anyone outside the company has rerun it.
Where could this recount mislead an investor?
In three ways. Kaggle competitions are not a random sample of anyone's business work, so resampling them describes a population of similar contests, not commercial value. Three seeds give an unstable standard error: AIBuildAI's low-complexity column reads 77.27 plus or minus 0.00 because its three seeds tied. And no interval here detects contamination, grading errors or an agent tuned on these exact competitions.
The checklist applies to any company citing benchmark results while it raises money, SourceX included; SourceX's own investor materials are available on request through its investors page, the right place for a specific diligence question about its evidence.
Methods note: MLE-bench repository at commit 507f92e (April 24, 2026), read October 27, 2026; grading reports fetched from Git LFS and checked against their SHA-256 pointers. Seeds are assembled as the repository's aggregation script does, padded where the README says so and for MLEvolve. A row's score is the share of competitions with any medal. The two-way interval draws competitions with replacement within the low, medium and high splits and the row's seeds with replacement, 10,000 times, and reads the 2.5th and 97.5th percentiles; compared rows share each competition draw. Published figures use seed 20261027; across 30 other seeds, 3 adjacent pairs separated in 19 runs, 2 in 11, and the Gemini-2.5-Pro Famou-Agent 2.0 lower end was zero in 2. Not measured: ability outside these competitions, any business task, any SourceX dataset. Script: mlebench_intervals.py, held for reviewers.
This article is for information only and is not an offer to sell or a solicitation of an offer to buy securities. Any offering by SourceX is made only to verified accredited investors through its offering documents.
Sources for this section
- MLE-bench's README (commit 507f92e, April 2026) describes the benchmark as 75 Kaggle competitions, recommends at least three seeds with scores reported as the mean plus or minus one standard error of the mean, notes that grading is deterministic while agents can be high-variance, and flags leaderboard values computed by padding incomplete seeds with failing scores. Source
- MLE-bench's aggregation script computes the standard error of the mean from per-seed scores and a band of 1.96 standard errors either side of the mean. Source
- MLE-bench's published grading reports and experiment table at commit 507f92e, recounted here: every main-table score reproduced, intervals recomputed with competitions resampled, padded and missing competitions counted, and same-agent budget comparisons measured. Source
- The EleutherAI evaluation harness computes the standard error of a mean metric as the sample standard deviation across items divided by the square root of the item count. Source
Frequently asked questions
What is a confidence interval in agent evaluation?
It is a range for a score or a score difference that reflects how much the result could move with a different draw of tasks and runs. For comparisons, compute it on the paired difference between agents, resampling the independent unit, and report it next to the headline number.
Should I bootstrap tasks or task families?
Resample the unit that is independent. If tasks share a customer, template or workflow, resample those groups. Resampling individual correlated tasks treats them as independent evidence and can make an interval look narrower than the data supports.
How many trials per task does an agent evaluation need?
Enough to separate run-to-run variation from task difficulty; three or more is common for agents. Fix the number before the evaluation starts, use the same number for every agent, and report reliability measures such as pass^k alongside the average.
If the interval includes zero, are the agents equal?
No. It means this evaluation could not tell them apart. The agents may differ by an amount too small to detect with this many families, or differ in specific families that the average blurs. Report it as unresolved and decide whether more data is worth the cost.
Is a bootstrap better than a t-test for benchmark comparisons?
Neither is always better. Standard errors from the central limit theorem work well with many independent units, and clustered versions exist. A bootstrap makes fewer assumptions about the shape of the data. Both fail when units are few or correlated in ways the method ignores.
Sources
- Adding Error Bars to Evals treats eval questions as drawn from an unseen super-population and presents formulas for analyzing eval data, measuring differences between two models and planning an evaluation, including clustered standard errors. Source
- The tau-bench leaderboard reports pass^1 to pass^4, and strategies with reported scores fall as k rises; the top airline tool-calling entry falls from 0.460 to 0.225. Source
- scikit-learn's cross-validation guide states that the i.i.d. assumption is broken when the data contain groups of dependent samples, and provides group-aware splitters. Source
- Koehn (2004) described bootstrap resampling methods to compute the statistical significance of differences in machine translation test results. Source
- scipy.stats.bootstrap defaults to 9,999 resamples, a 0.95 confidence level and the BCa method, and its paired option resamples the same indices for every array. Source
- Inspect provides a bootstrap_stderr metric with 1,000 samples by default and a stderr metric that computes clustered standard errors from a cluster key in sample metadata. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.