Leadership and readiness
AI benchmark averages hide failure severity: run the wrong-change test
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
An AI benchmark average counts a wrong refund and an unfinished request as the same miss, so split failures by consequence before trusting it. In 1,476 database-checked tau-bench runs, about two in three failed airline runs executed a booking change the task never called for, and the model with the higher published airline average made such a change in more tasks. Ask for the split, repeated-trial exposure and the scoring code.
Evaluation split
AI benchmark average failure
Key takeaways
- A pass rate treats every miss alike; diligence-grade evidence splits failures by what they did to the system of record.
- In tau-bench's published airline runs, 68.0% and 68.5% of failed database-checked runs executed a change the task never called for; in retail, 47.8% and 52.9%.
- Exposure grows with repetition: Claude 3.5 Sonnet's retail tasks with at least one unrequested change rose from 26 to 37 of 77 between four and eight trials.
- The higher published airline average belonged to the agent that left fewer tasks free of unrequested changes, 13 of 46 against 19 of 46.
- Ask which scoring code produced every number: some published answer-task passes in tau-bench cannot all match the expected database.
What does an AI benchmark average leave out?
An average leaves out consequence. A pass rate says how often an agent finished a task correctly and is silent on what the failures did. For an agent that acts inside a booking or order system, a miss that changes nothing usually costs a repeat contact, while a miss that cancels the wrong reservation or exchanges the wrong item moves money and stock that someone has to reverse.
This article keeps three failure classes apart. A missed change leaves the record short of what the task required. An unrequested change writes something to the record that the task never called for. An answer miss leaves the record right but omits information the customer needed. Only the first two touch the system of record, and the second is the one a deployment pays to undo.
For a technical diligence team the question is narrow: before a benchmark score supports a claim that an agent, or the data behind it, is ready for production work, what evidence about its failures should come with the score? The sections below test that question on public run files and turn the result into a worksheet.
Splitting 1,476 tau-bench runs by what each failure changed
We read every published run of tau-bench, the customer-service agent benchmark from Sierra, and sorted each failure by the database writes it executed. The benchmark pairs a language-model agent with a simulated customer in retail and airline tasks and grades a run by comparing the final database with the expected one. Its repository at commit 59a200c ships four historical trajectory files, one per model and domain, and their mean rewards equal the README's Pass^1 figures: 0.604 and 0.692 in retail, 0.420 and 0.460 in airline.
The split uses only database-checked tasks, those with no required answer, where the stored result is the benchmark's own database comparison: 77 of 115 retail tasks and 46 of 50 airline tasks, 1,476 runs in all. For each failed run, the script lists the write calls the environment executed, read from the recorded tool responses, and compares them with the task's expected writes. A failure with at least one executed write that matches no expected write is an unrequested change; any other failure that ended with a wrong database is a missed change.
Ten runs stopped at 62 messages before any check was stored; the two with an unrequested write count as unrequested changes, and the other eight stay in the failure count but in neither class. On passed runs the matching rule flagged 1 of 81 gpt-4o airline passes and none of the other 745, a small false-flag rate measured where the benchmark itself confirmed the database. The published average sits beside the split below; read the last two columns together:
| Run file | Published Pass^1 | Database-checked pass rate | Failed runs with an unrequested change | Tasks with one in trials 0-3 |
|---|---|---|---|---|
| gpt-4o, retail | 0.604 | 55.8% (172 of 308) | 47.8% (65 of 136) | 32 of 77 |
| Claude 3.5 Sonnet (20241022), retail | 0.692 | 66.2% (408 of 616) | 52.9% (110 of 208) | 26 of 77 |
| gpt-4o, airline | 0.420 | 44.0% (81 of 184) | 68.0% (70 of 103) | 27 of 46 |
| Claude 3.5 Sonnet (20241022), airline | 0.460 | 44.8% (165 of 368) | 68.5% (139 of 203) | 33 of 46 |
Why two in three failed airline runs were wrong changes
Airline failures were mostly acts, not omissions. In both airline files about 68% of failed database-checked runs executed at least one write the task did not call for, against roughly half in retail. The writes involved are the ones that move money and seats: the gpt-4o airline file's unrequested writes were 29 flight changes, 28 cancellations, 22 new bookings, 7 baggage updates and 2 travel certificates, and the Claude file's were 51, 45, 40, 29 and 12.
Retail splits more evenly. Its unrequested writes cluster in exchanges of delivered items and item swaps on pending orders, 21 and 23 in the gpt-4o file and 46 and 37 in the Claude file, followed by returns, while about as many failures simply stopped short of a required change. A retailer and an airline reading the same pass rate would face different repair work, and only the split shows which.
The counts describe writes, not dollars. The benchmark records no cost, and a wrong baggage count is cheaper to undo than a cancelled itinerary. Severity weights belong to the business that will run the agent; the class counts are the input those weights need, and a single average throws that input away.
Repeated trials widen the tail, and can flip a ranking
An unrequested change that shows up once in eight attempts is invisible in a one-trial score and close to certain at production volume. tau-bench already measures consistency on the success side: its run.py computes pass^k as the mean, over tasks, of C(c, k) divided by C(n, k), the chance that k trials drawn from a task's n runs all succeed. In the README, the Claude 3.5 Sonnet (20241022) agent falls from 0.692 at Pass^1 to 0.462 at Pass^4 in retail, and gpt-4o from 0.420 to 0.200 in airline.
The same reading applies to harm. The Claude retail file holds eight trials per task. Tasks with at least one unrequested change number 26 of 77 over trials 0-3 and 37 of 77 over all eight, while tasks passed in every trial fall from 38 to 23. A buyer who runs an agent on a task type thousands of times inherits the exposure figure, not the average.
Exposure can also reverse an order. Call a task clean when none of its trials 0-3 made an unrequested change. On published averages the Claude agent leads in airline, 0.460 to 0.420; on clean tasks it trails, 13 of 46 (28%) against 19 of 46 (41%). Paired by task, 7 tasks drew an unrequested change from Claude alone and 1 from gpt-4o alone, with 26 from both and 12 from neither; an exact McNemar test on that split gives p = 0.07, suggestive with 46 tasks rather than decisive. In retail the two measures agree: Claude leads on both, 0.692 and 51 of 77 clean tasks (66%), against 0.604 and 45 of 77 (58%) for gpt-4o, with 12 tasks drawing an unrequested change only from gpt-4o, 6 only from Claude, p = 0.24.
Which scoring code produced the average in the deck?
Ask for the code version behind every run, because an average can mix scoring rules without saying so. In tau-bench, 38 of the 115 retail tasks also require the agent to state a piece of information. A repository commit dated November 10, 2024, titled 'The reward should be zero if the database state is incorrect', made those tasks check the database as well; before it, the reward code checked only the stated answer for them. The four trajectory files were committed on November 22, 2024.
Their stored results for answer tasks keep only the answer check, and some cannot have been scored under the fixed rule. In the gpt-4o retail file, task 2 scores 1.0 in three runs that executed a return and in one run that executed no write at all; the final database can match the expected state in one of those cases or the other, not both. Counting answer tasks whose passes include both a run with no write and a run with a write that always changes stored data flags 4 of 38 in the gpt-4o file and 5 of 38 in the Claude file. Answer-task runs also passed more often, 69.7% and 75.3%, than database-checked runs, 55.8% and 66.2%; the files cannot say how much of that gap is easier tasks and how much is the lighter check.
None of this is concealed: the maintainers fixed the rule, and the README now warns that the repository's tasks are outdated and sends users to tau3-bench. DataComp-LM shows the same hazard in a data benchmark, where the maintainers note that bug fixes in baseline calculations make CORE and EXTENDED scores from before September 2025 not directly comparable with new results, though they expect rank orderings to hold. The diligence step is mechanical: get the raw runs, the scoring code version and a rescore under current code.
Model, prompt, budget or data: separating the causes of a severity gap
A change in model, prompt or budget can move scores with the data untouched, which is why a claim that data drove a gain needs controls. Sclar and colleagues (ICLR 2024) found that formatting changes alone moved LLaMA-2-13B's few-shot accuracy by up to 76 points. Kapoor and colleagues, in AI Agents That Matter (2024), showed that repeated calls with majority voting raise accuracy and that simple baseline agents beat many complex designs on HumanEval at far lower cost, so an accuracy-only leaderboard rewards spending. In tau-bench the customer is itself a language model, and the README names gpt-4o as the default simulator: one more moving part between two scores.
At a glance
Model, prompt, budget or data: separating the causes of a severity gap
The control a data claim needs is an ablation. DataComp-LM has participants train with standardized training code and scale-specific hyperparameters, change only the dataset, and judges each dataset on 53 downstream tasks. Applied to an agent, that means the same model, prompt, tools, inference budget and simulator, run with and without the dataset, with the failure split reported for both arms. Hold these fixed before crediting data with a fall in severe failures:
- Model version and decoding settings, including sampling temperature
- System prompt, policy text and tool schemas, character for character
- Inference budget: calls per task, retries, voting and maximum steps
- User simulator model and strategy, or the pool of human testers
- Task set and split, frozen before either arm runs
- Scoring code version, identical for both arms
A seven-row worksheet for underwriting a benchmark's failures
Use the worksheet to turn a benchmark slide into evidence. Each row names what to request, the test it must pass, the red flag that should stop the memo, and how the public tau-bench files read against it on October 27, 2026.
The worksheet is an editorial tool; its tau-bench column comes from the analysis above, and it rates no company. The replication checklist assumes a company will share raw runs under a confidentiality agreement if it will not publish them:
- Obtain raw run files with per-run tool calls, tool responses and stored scores, not summary tables.
- Confirm that the published average reproduces from those files to three decimals.
- Rerun the failure classifier, and publish its counting rule and its false-flag rate on passed runs.
- Rescore every run under the current scoring code and report any number that moves.
- Report severe-failure exposure per task for the same trial count used for pass^k.
- Compare arms that differ only in the claimed factor, paired by task, with an exact or task-clustered test.
- Archive commit hashes, data files and scripts so a second analyst gets the same numbers.
| Check | What to request | Passes when | Red flag | tau-bench reading |
|---|---|---|---|---|
| Consequence split | Failed runs classed as unrequested change, missed change or answer miss, with the rule | Counts per class and domain, plus the rule's false-flag rate | A pass rate with no failure detail | Airline: 68.0% and 68.5% of failures were unrequested changes |
| Repeated-trial exposure | Tasks with at least one severe failure across k trials | Reported for the same k as pass^k | One trial per task | Claude retail: 26 of 77 tasks at four trials, 37 at eight |
| Scoring-rule audit | Code version for every run and a rescore under current code | One version, or both scores shown | Averages that predate a scoring fix | Answer tasks store only the answer check; 4 and 5 of 38 show conflicting passes |
| Independent units | Task-level results with trials grouped under each task | Uncertainty and paired tests computed over tasks | Trials treated as independent cases | 46 airline tasks: a 7-to-1 split gives p = 0.07 |
| Controls | Model, prompt, budget and simulator for each arm | Only the claimed factor differs | Gains compared across different models or budgets | The files compare two models, so gaps are model gaps |
| Data ablation | The same system run with and without the dataset | Failure split reported for both arms | No without-data baseline | Not applicable: the files hold no data arm |
| Leakage | Task publication dates against training cutoffs, plus held-out tasks | Held-out tasks score in line with public ones | Only long-public tasks | Tasks public since 2024; the README flags them as outdated |
Where the wrong-change reading could overstate or understate risk
Name the ways this test can fail before leaning on it. Each item says what could make the wrong-change reading overstate or understate risk:
- The matching rule can misjudge equivalent writes. It normalizes arguments but cannot prove that two different write sequences reach the same state; its false-flag rate was 1 in 826 passed runs, measured only where the benchmark stored a database check.
- Writes are a proxy for cost. A wrong answer with no write can still be expensive, and some unrequested writes are trivial to reverse, so severity weights must come from the business that bears the cost.
- The customers are simulated and the agents date from 2024. Simulated users behave differently from real ones, and newer agents or the updated tau3-bench tasks may split differently.
- Small task counts limit conclusions. With 46 airline tasks, the ranking flip rests on eight discordant tasks and does not reach conventional significance.
- AI performance statements draw regulatory attention. On March 18, 2024, the SEC announced settled charges against two investment advisers, Delphia (USA) and Global Predictions, over false and misleading statements about their use of AI; the case concerned marketing rather than benchmarks, but it shows that AI claims outrunning the evidence carry legal risk.
- A clean public split does not transfer by itself. Public task mixes differ from any one company's queue, so the split should be rerun on held-out tasks drawn from the buyer's own records.
What to ask a data company about its failures, and where SourceX stands
SourceX sits on the supply side of this question: it arranges licenses for selected business records, handling the work from qualifying a supplier and checking rights through contracting, preparation, delivery and payment. It has published no benchmark results of its own, and a planned study is not evidence of traction.
SourceX is raising from accredited investors under Rule 506(c); its investor materials, including company and commercial metrics, go to investors on request through the investors page rather than onto public pages. A diligence team can put the same questions to SourceX or to any other data company:
Checklist · 6 items
- Which failure classes did the data reduce, and did unrequested changes fall or only missed changes?
- What was held fixed between the arms with and without the data?
- How many trials ran per task, and what is the per-task exposure to severe failures?
- Which scoring code version produced each number in the deck, and does each number survive a rescore?
- Which test tasks came from records the model could not have seen, and how were they chosen?
- Who set the severity weights, and were they fixed before any results came in?
0 of 6 ticked. Tick each item as you go.
Method and limits of the tau-bench recount
Files: sierra-research/tau-bench at commit 59a200c6d575d595120f1cb70fea53cef0632f6b, folder historical_trajectories, files gpt-4o-retail, sonnet-35-new-retail, gpt-4o-airline and sonnet-35-new-airline, with 460, 920, 200 and 400 runs. The Claude files are named sonnet-35-new; their means match the README rows for claude-3-5-sonnet-20241022. Script: tau_write_severity.py, run with python3 -I on October 27, 2026; it parses the JSON files only, runs no repository code and uses no randomness, so no seed applies.
Counting rules as the script applies them: a run passes when its reward is 1.0; database-checked tasks are those with no required outputs; an executed write is an assistant message whose first tool call names one of 13 write tools and whose recorded response does not start with Error; expected writes come from the task's action list; matching is one to one on tool name and normalized arguments, with numbers as floats, item lists sorted where the tool sorts them and item pairs kept together for pending-order swaps; exposure counts use trials 0-3 unless stated. The conflicting-pass check flags an answer task when its passes include a run with no executed write and a run with an executed cancellation, exchange, return, pending-order item swap, booking or reservation cancellation.
Not measured: dollar costs, real customers, current models, the updated tau3-bench tasks and any SourceX data. The worksheet and the checklists are editorial tools written for this article, and nothing here reports a SourceX result.
Frequently asked questions
What is failure severity in an AI benchmark?
Failure severity describes what a failed run did, not just that it failed. For an agent that writes to business systems, the useful split is between missed changes, unrequested changes and answers that omit needed information. Weights can then be applied to each class, but the raw class counts should be reported first so others can apply their own weights.
Does pass^k already capture severe failures?
No. Pass^k measures how consistently a task succeeds across k trials, so it rewards reliability but treats every failure the same. The severe-failure counterpart is the share of tasks with at least one unrequested change across k trials, which in the Claude retail file rose from 26 to 37 of 77 tasks between four and eight trials.
How can an investor tell whether data, not the model, drove a benchmark gain?
Ask for an ablation: the same model, prompt, tools, inference budget and simulator, run with and without the dataset, with results paired by task. DataComp-LM is a public example of the design, standardizing training code and hyperparameters while only the dataset changes. Gains measured across different models or budgets say nothing about the data.
Why might a published benchmark average change after the fact?
Scoring code gets fixed. tau-bench changed its reward in November 2024 so that tasks needing a stated answer also fail when the database is wrong, and DataComp-LM's maintainers say scores from before September 2025 are not directly comparable with new ones. Ask which code version scored each number and request a rescore.
Are tau-bench's historical files a fair basis for judging today's agents?
They are a fair basis for testing a method, not for ranking current products. The files come from 2024 models run against tasks the README now calls outdated, with simulated customers. Rerun the split on current runs and on held-out tasks before drawing conclusions about any specific agent.
Sources
- tau-bench's repository at commit 59a200c holds four historical trajectory files, a README leaderboard with Pass^1 to Pass^4 figures, a warning that its tasks are outdated with a pointer to tau3-bench, gpt-4o as the default user simulator, and run.py's pass^k formula. Source
- A November 10, 2024 tau-bench commit made the reward zero whenever the database state is incorrect, including for tasks that also check a stated answer. Source
- The historical trajectory files were added to tau-bench on November 22, 2024. Source
- Sclar and colleagues (ICLR 2024) report accuracy differences of up to 76 points for LLaMA-2-13B from prompt formatting changes in few-shot settings. Source
- AI Agents That Matter (2024) shows repeated calls with majority voting raise accuracy and that simple baseline agents outperform many complex agents on HumanEval at lower cost, arguing for cost-controlled evaluation. Source
- DataComp-LM trains each curated dataset with standardized training code and scale-specific hyperparameters and evaluates it on 53 downstream tasks; its maintainers note that baseline bug fixes make CORE and EXTENDED scores before September 2025 not directly comparable with new results. Source
- On March 18, 2024 the SEC announced settled charges against investment advisers Delphia (USA) and Global Predictions for false and misleading statements about their use of artificial intelligence. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.