AI uses for records
Small benchmark uncertainty statement: the five lines a 60-task score needs
By SourceX Editorial · Updated
9 sections
Short answer
A small benchmark uncertainty statement reports a score as a count over its denominator, a 95 percent interval suited to small samples, the number of independent task clusters behind it, the population the tasks were drawn from and the smallest gap the set can resolve. Applied to 41 of 60 illustrative tasks in 12 families, it turns a quoted 68.3 percent into 68 percent, plausibly 52 to 81.
Evaluation split
Small benchmark uncertainty
Key takeaways
- Report the score as passes over tasks first; the percentage is a derived figure.
- The number of independent clusters, not tasks or trials, sets the width of the interval.
- Below a few hundred tasks, use Wilson, Clopper-Pearson or Bayesian intervals; statsmodels defaults to the normal approximation unless told otherwise.
- Round to the uncertainty: JCGM 101:2008 keeps uncertainty to one or perhaps two significant digits.
- A perfect score on 20 tasks still allows a true failure rate near 14 percent at one-sided 95 percent confidence.
What is a small benchmark uncertainty statement?
It is the short block of text that travels with a benchmark score and says how far the score can be trusted: how many tasks it rests on, how many of them are truly independent, how wide the plausible range is, which population the tasks stand for and which claims are off limits. For a business-task benchmark of a few dozen to a few hundred tasks, that block carries more information than the score.
Small business-task sets are normal, not a defect. Anthropic's engineering guide to agent evaluations says "20-50 simple tasks drawn from real failures is a great start", because early changes to an agent tend to have large, visible effects. The tau2-bench repository fixes 50 frozen tasks per domain for its text and multilingual benchmarks, and splits the 50 airline tasks into 30 for training and 20 for testing. OpenAI's GDPval spans 1,320 tasks but open-sources a gold subset of 220, five for each of its 44 occupations.
Each of those sets can support a useful decision. None of them can support a score quoted to a decimal place with no range beside it. This guide gives a dataset curator or research lead a five-line confidence statement, a resolution ladder that shows how fine a claim a set can carry, and a red-pen table of sentences to strike. Every worked number is an illustrative calculation, not a measurement of any model.
Why does a percentage from 60 tasks claim more than it knows?
Because each task behaves like a coin with an unknown bias, and 60 tosses pin that bias down only loosely. Suppose a model passes 41 of 60 illustrative accounts-payable exception tasks. The point estimate is 68.3 percent, yet a 95 percent Wilson interval on those counts runs from about 56 to 79 percent, a band 23 points wide.
Printing 68.3 percent implies the third digit means something, when even the second barely does. Metrology settled this question long ago. JCGM 101:2008, from the Joint Committee for Guides in Metrology, says it is usually sufficient to report measurement uncertainty to one or perhaps two significant digits, and that the ends of a coverage interval should stop at the same decimal place as the uncertainty. Applied here, the honest report is 68 percent with a range of roughly 56 to 79, before any allowance for related tasks.
NIST's guidance on expressing measurement uncertainty, Technical Note 1297, adds a second habit worth copying: describe how each component of the uncertainty was evaluated, and say how the coverage factor was chosen when it is not the conventional one. A benchmark report that prints a range without its method leaves the reader unable to tell a careful 95 percent interval from a guess.
Count clusters, not tasks or trials
Build the interval on the number of independent units, which in business benchmarks is usually the number of task families, source companies or templates rather than the number of tasks. Tasks written from the same case file, the same customer or the same template tend to pass or fail together, so five of them carry less than five tasks' worth of evidence.
Evan Miller's 2024 paper on adding error bars to evals borrows clustered standard errors from social-science statistics for this situation, and in its table of popular evals the clustered errors can be more than three times the naive ones. In the illustrative example, the 60 tasks come from 12 families of five. With an assumed within-family correlation of 0.15, the design effect is 1 + (5 - 1) x 0.15 = 1.6, the effective sample shrinks to 37.5 tasks, and the interval widens to roughly 52 to 81 percent.
GDPval's gold subset has the same shape at a larger scale: 220 tasks in 44 groups of five. If tasks within an occupation were correlated at 0.2, the subset would hold the information of about 122 independent tasks rather than 220. That is an illustrative calculation, not a figure GDPval reports; the point is that the cluster count belongs in the statement.
Repeated trials are a separate matter. Anthropic's guide defines pass@k as the chance of at least one success in k attempts and pass^k as the chance that all k succeed, and works the example of a 75 percent per-trial rate, where passing three trials in a row has a probability of about 42 percent. Trials sharpen each task's own success rate; they add no new tasks, so they never raise the count that sets the interval.
Which interval method holds up below a few hundred tasks?
Use an interval built for small binomial samples, such as Wilson, Clopper-Pearson or a Bayesian beta interval, and drop the textbook normal approximation. Bowyer, Aitchison and Ivanova argue in a 2025 ICML position paper that methods resting on the central limit theorem suit benchmarks of thousands of examples but give inadequate uncertainty estimates for smaller, specialized ones, and they propose simple frequentist and Bayesian alternatives.
The trap is often a software default. The proportion_confint function in the Python library statsmodels uses the asymptotic normal approximation unless told otherwise, although Wilson, Clopper-Pearson and Jeffreys options sit in the same call. Near the edges the gap is stark: for 19 passes in 20 tasks, the normal approximation runs from about 85 to 105 percent before clipping, while the Wilson interval runs from about 76 to 99 percent.
Perfect and zero scores need a sentence of their own. With no failures in n tasks, the one-sided 95 percent upper bound on the failure rate is 1 - 0.05^(1/n): about 26 percent for 10 tasks, 14 percent for 20 and 5 percent for 60. A clean sweep of a 20-task test split, the size of tau2-bench's airline test set, is still consistent with failing about one task in seven.
Few clusters call for extra care. When a set has only a handful of families, a clustered interval built on a t distribution, or a bootstrap that resamples whole families, comes out wider than the naive one for good reason, and an evaluation statistician should check it before the number leaves the team.
The five-line confidence statement, filled in
Write the statement as five lines, each of which blocks a specific overreach. The table gives what every line must state and an illustrative entry for the 41-of-60 example; keep the structure, replace the entries, and hold the score back until each line is filled.
Read as prose, the filled statement says: 41 of 60 tasks passed, 68 percent; 95 percent Wilson interval 56 to 79 percent on tasks, about 52 to 81 percent allowing for 12 families; drawn from closed exceptions at four companies between January and June 2025; gaps under about 20 points against another model scored on a different 60-task sample are unresolved; set version 1.2 and rubric version 3, not comparable with version 1.0. The figure shows the same pass count earning progressively wider and more honest ranges, line by line:
| Line | What it must state | Illustrative entry | Overreach it blocks |
|---|---|---|---|
| 1. Count | Passes over tasks, trials per task and how trials were combined | 41 of 60 tasks passed; one graded trial per task | A percentage with no denominator |
| 2. Interval | A 95 percent interval, its method and the independent unit | Wilson 56 to 79 percent on tasks; about 52 to 81 percent across 12 families | A narrow range that treats related tasks as independent |
| 3. Frame | Where the tasks came from and the rule that selected them | Closed AP exceptions at four companies, January to June 2025, five drawn at random per family | A claim about all AP work or all companies |
| 4. Resolution | The smallest gap the set can resolve and the claims it cannot support | Gaps under about 20 points against a separate 60-task sample are unresolved | Leaderboard ranks built on noise |
| 5. Version | Benchmark, grader and harness versions, and what they compare with | Set v1.2, rubric v3, harness of September 2026; not comparable with v1.0 | Trend lines drawn across changed tasks or graders |
How fine a claim can the interval carry?
The width of the widest plausible interval decides the finest wording a set can support. The ladder uses the Wilson interval at a 50 percent pass rate, where intervals are widest, and counts independent units, so 220 tasks in 44 correlated families belong on a lower rung than their task count suggests.
The last column is a rough guide for two results scored on separate samples of the same size; paired comparisons on identical tasks usually resolve smaller gaps and need their own interval. The wording column follows the JCGM practice of ending a reported value at the decimal place of its uncertainty:
| Independent units | Widest 95 percent interval | Finest honest wording | Unpaired gap needed (approx.) |
|---|---|---|---|
| 20 | Plus or minus 20 points | Three bands, low, middle or high; never a bare percentage | About 31 points |
| 50 | Plus or minus 13 points | Whole percentages, always beside the interval | About 20 points |
| 100 | Plus or minus 10 points | Whole percentages with the interval | About 14 points |
| 220 | Plus or minus 7 points | Whole percentages; a decimal only next to a stated interval | About 9 points |
| 500 | Plus or minus 4 points | One decimal beside the interval | About 6 points |
| 1,000 | Plus or minus 3 points | One decimal beside the interval | About 4 points |
Strongest objection: could more records make a small benchmark less small?
Only if the new records add independent cases. The leading alternative explanation for a confident-looking result is that a large record count hides repetition: 2,000 tasks generated from one company's ticket templates can behave like a few dozen independent cases, and an interval computed on 2,000 will be far too narrow.
Three mechanisms do the hiding. Repeated cases: near-duplicate tasks share an answer path, so a pass on one predicts a pass on the next. Selection: tasks built from cases that were easy to document, such as closed tickets with clean resolution codes, leave out the messy cases a deployed agent will meet. Unobserved failures: a grader that checks a status field instead of the business outcome can record passes that never happened, and no interval repairs a biased grader.
Version changes are a fourth trap, and benchmark maintainers flag it themselves. The tau2-bench README states that its 2.0 English runs are not input-identical to version 1.0.1 and that 2.0 scores are not comparable with the 1.x leaderboard, citing replaced tasks, changed prompts and a new judge model. A gain measured across such a change says nothing about the model until both versions run on the same tasks.
The check for the objection takes an afternoon: before adding records, count clusters, collapse near-identical tasks and confirm how each task is graded. If the cluster count does not rise, the interval should not shrink.
Red-pen table: sentences a small benchmark cannot carry
Most overreach lives in single sentences of a results summary, so edit at that level. Each row pairs a sentence that the illustrative 60-task evidence does not support with the reason and a replacement it does support:
| Sentence to strike | Why the evidence does not carry it | Sentence the evidence supports |
|---|---|---|
| Model M scores 68.3 percent on AP exceptions. | Three digits on a range 29 points wide | Model M passed 41 of 60 tasks (68 percent; about 52 to 81 percent across 12 families). |
| Model M handles accounts payable exceptions. | Tasks came from four companies' closed cases in one half-year | On closed exceptions from four companies in early 2025, M passed 41 of 60. |
| M beats N, 68 to 63 percent. | A 5-point gap between separate samples sits far inside the resolution | M and N are not distinguishable on this set; a paired run on the same tasks comes next. |
| M never fails on duplicate invoices. | Zero failures in 10 tasks still allows a failure rate near 26 percent | M passed all 10 duplicate-invoice tasks; the set cannot rule out about one failure in four. |
| M improved 6 points since last quarter. | The task set and rubric changed between the two runs | Scores from v1.0 and v1.2 are not comparable; rerun both models on v1.2 before comparing. |
| Results represent mid-market companies. | No frame or selection rule supports that reading | The four companies were chosen by availability; representativeness was not tested. |
Publish, widen or hold: the decision and the next evidence
Publish the score when every line of the statement is filled and the claim you want sits inside the resolution ladder; widen the set when the decision needs a finer resolution than the cluster count allows; hold the claim when the frame or the grader cannot be described. For most small business-task sets, the right move is to publish the count and interval now and plan the next draw around independent families.
Widening means adding clusters. Moving from 12 families to 40, even at five tasks each, lifts a set from the bottom rungs of the ladder toward whole-percentage claims, while 500 more tasks from the same 12 families barely move it. Write the target as a number of families and source companies, with a minimum per family, before collecting anything.
The same terms make a sharper request for outside data. SourceX looks for a buyer's dataset among U.S. businesses after the request arrives rather than selling from inventory, and the request already carries the buyer's scope, volume, language, format, timing and licensing terms. Add the intended use, the number of independent companies and task families wanted, the frame rule and an acceptance test, such as a cap on near-duplicate tasks per family. Candidate records can then be described against those terms, with no promise about availability or about how any model will score.
Frequently asked questions
How many tasks does a business-task benchmark need before its score means anything?
Enough independent clusters to resolve the decision at hand. Anthropic's agent-evaluation guide treats 20 to 50 tasks drawn from real failures as a good start because early changes are large, yet at 50 independent tasks the widest 95 percent interval is still about 13 points either side. Resolving gaps of a few points between models takes hundreds of independent tasks or a paired design.
Should a small eval report a confidence interval or a Bayesian credible interval?
Either can work if it is built for small samples and its method is stated. Wilson and Clopper-Pearson intervals are standard frequentist choices, and a beta posterior with a stated prior is a Bayesian one; a 2025 ICML position paper by Bowyer and colleagues argues for such alternatives over central-limit methods on small benchmarks. Report the method, the unit and the count beside the range.
Do repeated trials per task make a small benchmark bigger?
No. Trials show how consistent a model is on each task, which pass@k and pass^k summarize, but they add no new tasks. The interval on the overall score depends on the number of independent tasks or task clusters, so ten trials of 30 tasks are still 30 tasks.
How many decimal places should a benchmark score have?
As many as its uncertainty supports and no more. JCGM 101:2008 keeps an uncertainty to one or perhaps two significant digits and the reported value to the same decimal place. Below a few hundred independent tasks, that usually means whole percentages with the interval printed beside them.
What does a perfect score on a small benchmark prove?
Less than it seems. With zero failures in 20 tasks, the one-sided 95 percent upper bound on the failure rate is about 14 percent, so the result is consistent with failing roughly one task in seven. Report the count and that upper bound together.
Sources
- Anthropic's guide to agent evaluations says 20-50 simple tasks drawn from real failures is a great start, defines pass@k and pass^k, and works a 75 percent per-trial rate to about 42 percent for three straight passes. Source
- The tau2-bench repository fixes 50 frozen tasks per domain for its text and multilingual benchmarks, splits the airline tasks 30 train and 20 test, and states that 2.0 scores are not comparable with the 1.x leaderboard. Source
- OpenAI's GDPval has 1,320 tasks across 44 occupations from nine industries, with a 220-task gold subset that has five tasks per occupation. Source
- JCGM 101:2008 says measurement uncertainty is usually reported to one or perhaps two significant digits, with coverage interval ends at the same decimal position as the uncertainty (citing GUM 7.2.6). Source
- NIST Technical Note 1297 asks reports to describe how each uncertainty component was evaluated and how the coverage factor was chosen when it is not 2. Source
- Miller (2024) recommends clustered standard errors when eval questions come in related groups and reports clustered errors more than three times the naive ones on some popular evals. Source
- Bowyer, Aitchison and Ivanova (ICML 2025) argue that CLT-based uncertainty methods fail on small, specialized benchmarks and propose frequentist and Bayesian alternatives. Source
- statsmodels' proportion_confint defaults to the asymptotic normal approximation and also offers Wilson, Clopper-Pearson (beta) and Jeffreys intervals. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.