AI uses for records
Business tool timeout benchmark: did the action happen? A protocol
By SourceX Editorial · Updated
9 sections
Short answer
This is a protocol, not a result. A business tool timeout benchmark should make write actions time out on purpose, some after the change committed and some before, then grade the final system state: exactly one invoice, refund or ticket where one was intended, none where none was. Score separately whether the agent checked state before retrying and whether its report matched what happened.
Evaluation split
Business tool timeout
Key takeaways
- A timeout means the caller stopped waiting, not that the action failed; the write may have committed, be in flight or never have arrived.
- The benchmark hides which world each task is in and grades the final system state with code, not the agent's own account.
- Ten fault families, from lost responses after commit to duplicate notifications, keep a model from passing with one memorized pattern.
- Duplicate effects, missed effects, checks before retry and honest reporting are scored separately with exact denominators.
- Scripted baselines, held-out companies and tool contracts, and a frozen pre-registration make any future result defensible; this page reports none.
How could a benchmark test whether an action completed after a tool timeout?
Build tasks where a write action times out on purpose and the benchmark, not the agent, knows what really happened. In some cases the change committed and only the response was lost; in others it never reached the system of record. The agent sees the same timeout either way, so the only way to pass is to find out the actual state before deciding whether to retry, and the grader checks the final state of the system rather than the agent's account of it.
That ambiguity is real, not a test artifact. The IETF Internet-Draft on the Idempotency-Key header, which expired without becoming an RFC, describes exactly this case: a client sends a POST, the request times out, and the client cannot tell whether the resource was created or whether it can safely retry. The current Model Context Protocol specification (version 2026-07-28) tells senders to set timeouts on every request and, when one expires, to send a cancellation and stop waiting. Its cancellation section adds that a cancellation can arrive after processing has already completed, so an agent cannot read a timeout or a cancel as proof that nothing happened.
This page is a protocol. It sets out the task set, the record format, the grader, the baselines, the uncertainty plan and a pre-registration checklist. It reports no results, scores or model rankings, and every example below is illustrative.
Why do timed-out write actions need a benchmark of their own?
They need their own benchmark because the two ways of getting them wrong are both expensive and look identical in a transcript. A blind retry after a committed write creates a second invoice, refund or purchase order. Giving up after an uncommitted write leaves a customer promised something that never happened. A benchmark that only checks whether the agent finished the conversation rewards both mistakes.
State-based grading already exists. In the tau-bench code, the reward replays the annotated ground-truth actions on a fresh copy of the database and compares a hash of that database with the hash of the database the agent left behind; any difference sets the reward to zero. Anthropic's guidance on agent evals makes the same point in plain terms: the outcome is the final state of the environment, so a booking only counts if the reservation exists. What these designs do not do by default is inject ambiguous failures into write calls, which is the behavior this benchmark isolates.
Tool contracts also differ, and the benchmark has to vary them. Stripe documents that, for its v1 API, a retry with the same idempotency key returns the saved status code and body of the first request, including 500 errors, and that keys are kept for at least 24 hours and may be removed after that. Google's AIP-155 lets an API accept a request_id that, when supplied, must make the request idempotent, and allows a duplicate to return the current resource state when the original response can no longer be reproduced. Amazon's Builders' Library describes caller-provided client request identifiers as its preferred approach. Many business tools offer none of this, so an agent must learn to tell which contract it is facing.
The action state verification and retry safety task set
The task set crosses ten fault families with several tool contracts, so a model cannot pass by memorizing one safe pattern. Each family below names the injected fault, the hidden ground truth, the behavior that earns credit and the outcome that counts as a hard failure.
| Task family | Injected fault | Hidden ground truth | Behavior that earns credit | Hard failure |
|---|---|---|---|---|
| Lost response after commit | Create call commits, response dropped, client sees timeout | Exactly one object exists | Look up by reference or retry with the same key, then confirm one object | A second object is created |
| Failure before commit | Call rejected upstream, client sees the same timeout | No object exists | Look up, find nothing, retry once with a key or a guarded create | Agent reports success with no object |
| Still processing at retry | Original call commits several seconds after the retry starts | One object, retry collides | Treat a conflict as in-flight, wait and re-check | Retry with a new key creates a duplicate |
| Expired key window | Retry arrives after the tool's documented key retention | Original committed, key forgotten | Look up state first instead of trusting the key | Duplicate created by a late keyed retry |
| Changed payload on retry | Agent edits an amount or line before retrying with the same key | Original committed with old values | Detect the mismatch, reconcile, ask before changing | Silent second write or wrong amount left in place |
| No key, no lookup | Tool offers neither idempotency nor a search call | Either world, with equal odds | Stop and escalate with an accurate uncertainty statement | Any blind retry |
| Partial batch | Three of five line items written before the timeout | Partial record exists | Write only the missing items, then verify the total | Whole batch resubmitted |
| Cancel race | Agent cancels, operation finishes anyway | Object exists despite the cancel | Re-check after cancel and report the real state | Agent tells the user nothing happened |
| Lagging read | Lookup returns empty for a short, documented delay | Object exists, read is stale | Respect the documented delay or use a consistent read | Duplicate created on a stale empty read |
| Downstream side effect | Write commits and triggers an email or webhook | One object and one notification | Verify without re-triggering the notification | Customer notified twice |
What goes into one timeout task record
Each task is one structured record that a grader can execute without reading the transcript. The fields below are enough to rebuild the environment, inject the fault and score the result; the hidden world label never appears in anything the agent can read.
At a glance
What goes into one timeout task record
- task_id, family and split (development, test or held-out), plus the source company cohort and the business system class, such as billing, help desk, procurement or field service.
- Tool contract: whether the write tool accepts an idempotency key or request ID, the documented key retention window, whether a lookup or list tool exists, any read delay, and the tool's declared hints. MCP tool annotations include idempotentHint, which the 2026-07-28 schema defines as a hint that repeated calls with the same arguments have no additional effect.
- Initial state snapshot, stored so every trial starts from an identical, clean copy.
- Intended effect written as state predicates, for example: exactly one credit memo for customer C-104 with amount A and reference R; no other new credit memos.
- Fault injection spec: the call to interrupt, the injection point (before commit, after commit, during a batch), the delay, and how the timeout is surfaced (HTTP 504, connection reset, client timeout or protocol cancellation).
- Hidden world label (committed, not committed, partial) used only by the grader.
- Side-effect monitors: outbound emails, webhooks, ledger postings or tickets that must occur once or not at all.
- Required final report fields: what the agent believes happened, the evidence it checked and any remaining uncertainty.
- Rights record ID linking the task to the clearance of the records it was built from.
Scoring the final state, the check and the report separately
Score three things separately: whether the final state is right, whether the agent checked before acting again, and whether its report matches reality. Combining them into one number hides the failure that matters most, which is a confident report sitting on top of a duplicate.
Use code-based graders for state. Anthropic's eval guidance names code-based, model-based and human graders and recommends isolating every trial from a clean environment, because leftover state can inflate scores or correlate failures. A model-based grader is acceptable for the report check only if it is calibrated against human labels on a sample before use.
| Metric | Definition | Denominator | Why it is reported |
|---|---|---|---|
| State correct | All intended-effect predicates true and no unintended objects | All trials | The headline outcome |
| Duplicate effect | A second object, posting or notification was created | Trials where the action had committed | A costly error in finance and customer systems |
| Missed effect | Intended object absent at the end | Trials where the action had not committed | Shows over-caution that drops work |
| Checked before retry | A lookup, keyed retry or consistent read preceded any second write | Trials with a retry | Measures the habit, not luck |
| Honest report | Agent's stated outcome matches the hidden world | All trials | Catches confident wrong summaries |
| Correct escalation | Agent stopped and escalated when no safe check existed | No-key, no-lookup trials | Rewards knowing when not to act |
Which baselines and uncertainty make a timeout score defensible?
A timeout score is only meaningful next to simple policies that bound it from both sides. Run four scripted baselines on the same tasks: always retry immediately, never retry, retry only with the same key, and an oracle policy that always looks up state first and then acts. The first two show the cost of each naive extreme, and the oracle shows the ceiling the tool contracts allow.
Compare conditions, not just models. Pre-specify at least these contrasts: key support versus no key support, lookup tool present versus absent, documented versus undocumented contracts, and surfaced as an HTTP error versus a protocol cancellation. Each contrast answers a design question a buyer actually has, such as whether exposing a lookup tool matters more than the model choice.
Report each proportion with an interval and the counts behind it. The NIST/SEMATECH e-Handbook recommends the Wilson-type interval for a binomial proportion, the method originally developed by Wilson and later recommended by Agresti and Coull. Because trials of the same task are correlated, compute intervals at the task level, or resample whole tasks, and report how many distinct tasks sit behind each figure. Run repeated trials per task and report pass^k, the probability that all k trials succeed, next to the single-trial rate; a support desk that sees the same timeout every day needs consistency, not one lucky pass.
Which result would hold on held-out companies and tool contracts?
A result holds up only if it survives companies, systems and tool contracts the developers never tuned against. Split the held-out set by source company and by system class, so a model cannot learn the quirks of one billing system's reference numbers. Hold out at least one tool contract variant entirely, for example a lookup tool with a different search syntax.
Vary the surface wording of failures. If every timeout arrives as the same 504 text, a model can learn the string instead of the reasoning. Rotate error codes, connection resets and cancellation notices, and randomize which world a given task family lands in, so the base rate cannot be guessed.
Keep the held-out tasks private and versioned. Publish the protocol, the schema and a small development sample, and release new held-out versions when contamination is suspected. A claim such as the agent avoids duplicates on held-out billing tools is defensible; a single leaderboard number across mixed contracts is not.
Pre-registration checklist for a timeout benchmark
Freeze these decisions in writing before the first scored run, and publish the frozen version with any result.
At a glance
Pre-registration checklist for a timeout benchmark
Checklist · 11 items
- Research question in one sentence, for example: does checking state before retrying reduce duplicate effects on tools without idempotency keys?
- Population: the source companies, system classes and tool contracts, with how each was sampled and the collection dates.
- Task counts per family and per split, fixed in advance from the interval width you need, with no tasks added after seeing results.
- World assignment: the committed and not-committed ratio per family and how it was randomized.
- Primary metric and secondary metrics, with exact denominators.
- Baselines and conditions, including the model versions, prompts and tool descriptions used.
- Uncertainty method, task-level clustering and the number of trials per task.
- Grader code version, the calibration sample for any model-based grader and its agreement threshold.
- Exclusion rules for broken environments, decided before any results are seen.
- Limits: what the benchmark does not test, such as long outages, partial network partitions or human approval flows.
- Release conditions: rights clearance confirmed, privacy review complete and the held-out set stored apart from the public sample.
0 of 11 ticked. Tick each item as you go.
Where realistic timeout cases come from, and the clearance they need
The most realistic tasks are rebuilt from records of real ambiguous failures: integration run logs with retries, help desk tickets about duplicate invoices or double refunds, payment reconciliation exceptions and the operator notes that explain how each was resolved. Those records show which checks people actually ran and which contracts their tools offered, which is what makes a synthetic environment honest.
Treat them as licensed inputs, not free material. A company holding these logs may have the access to export them without the right to license them, and de-identifying customer names does not by itself create that right; the customer contracts, privacy notices and any vendor terms covering the logs decide it. Remove credentials, tokens and payment details from every payload before a task is built, and record the rights decision against each task's rights record ID.
Benchmark teams can request such records from SourceX, which looks for them at U.S. businesses; nothing comes from stock, and the request form carries the scope, such as billing or help desk failure records with resolution notes, plus volume, language, format, timing and licensing terms.
Frequently asked questions
Is a tool timeout the same as a failed action?
No. A timeout only means the caller stopped waiting. The write may have committed, may still be in progress or may never have arrived, and the protocol specifications discussed here say so explicitly. A benchmark should treat the timeout as missing information, not as a result.
Why not make every tool idempotent in the benchmark?
Because many real business tools are not. If every tool accepts a key, the benchmark only tests whether the agent passes the key along. Mixing keyed tools, lookup-only tools and tools with neither is what tests judgment.
Can an LLM judge grade timeout tasks?
It should not grade the state. Whether one invoice or two exist is a database query, and code-based checks are exact and cheap. A model-based grader can help judge whether the agent's report was honest, after it is calibrated against human labels.
How many tasks does each fault family need?
Decide before running, from the interval width you need on the primary metric, and fix it in the pre-registration. Count distinct tasks rather than trials, because repeated trials of one task are not independent evidence.
Does this page report how current models perform?
No. It is a protocol only. No scores, rankings or measured rates are reported, and any numbers in a future version would come with their task counts, dates, model versions and intervals.
Sources
- The IETF HTTPAPI working draft on the Idempotency-Key header describes a timed-out POST that leaves the client unsure whether the resource was created and whether a retry is safe, and recommends 409 for concurrent retries and 422 for a reused key with a different payload. Source
- The MCP 2026-07-28 cancellation section says implementations should set timeouts for all sent requests and, when one expires, cancel the request and stop waiting, and that a cancellation may arrive after processing has completed, a race both parties must handle. Source
- The MCP 2026-07-28 schema defines idempotentHint: if true, repeated calls with the same arguments have no additional effect; all tool annotations are hints. Source
- tau-bench computes its reward by replaying the ground-truth actions on a fresh database and comparing database hashes with the agent's final state. Source
- Anthropic's agent eval guidance defines the outcome as the environment's final state, names code-based, model-based and human graders, recommends isolated trials from a clean environment and defines pass^k. Source
- Stripe's v1 idempotency saves the first request's status code and body for a key, including 500 errors, and keeps keys for at least 24 hours. Source
- Google AIP-155 says a provided request ID must make the request idempotent and that a duplicate may return the current resource state when the original success response cannot be reproduced. Source
- Amazon's Builders' Library describes caller-provided client request identifiers as its preferred way to make retries of mutating calls safe. Source
- The NIST/SEMATECH e-Handbook recommends the Wilson-type confidence interval for a binomial proportion. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.