AI uses for records
Agent duplicate transaction failure case: testing a timeout retry
By SourceX Editorial · Updated
10 sections
Short answer
To test whether an agent retry creates a duplicate transaction, make the write succeed on the server while the agent's call times out, let the agent act, then count transactions in the system of record. Pass only if exactly one exists, the retry reused the original idempotency key or first looked up the outcome, and the agent's final message reports what really happened.
Agent run
Agent duplicate transaction
Key takeaways
- A timeout after the server commits is the case that exposes unsafe retries; inject it deliberately.
- Grade the system of record, not the agent's final message, which can be accurate and still hide a duplicate.
- A retry with a new idempotency key looks like a new request to the provider; key reuse is the condition to test.
- MCP's idempotentHint defaults to false and is only a hint; it is not permission to retry.
- Pair every failure case with passing siblings so graders do not learn to fail every retry.
How do you test whether an agent retry creates a duplicate transaction?
Force the ambiguous case and grade the system of record. Make the payment or order call succeed on the server while the agent's client times out before it reads the response, let the agent decide what to do next, then count the transactions that actually exist. The test passes only when exactly one transaction exists for the intended obligation, the agent either reused the original idempotency key or looked up the outcome before acting again, and its final message reports what really happened.
The agent's own words are not evidence. An agent that retried blindly and created a second payment will often say, accurately from its point of view, that the payment went through. Grading that message as a success is how duplicate-write bugs survive evaluation and reach production.
This page walks through one illustrative failure case in that format: the starting state, each step with the evidence behind it, the rule that applied, the resulting state and the check that would have caught it. The record structure is the asset; reuse it for any agent that can create, pay, book, refund or send.
Why a timeout is the hardest moment for an agent
A timeout leaves the caller not knowing whether the write happened. The IETF HTTPAPI working group's Idempotency-Key draft opens with exactly this scenario: a client sends a POST to create a resource, the request times out, and the client cannot tell whether the resource was created, whether the server even finished, or whether a retry is safe.
HTTP semantics make the risk explicit. RFC 9110 defines a method as idempotent when sending several identical requests has the same intended effect on the server as sending one, names PUT, DELETE and the safe methods as idempotent, and says a client should not automatically retry a non-idempotent request unless it knows the request semantics are actually idempotent or can detect that the original was never applied. Payment creation is normally a POST, so neither condition holds by default.
Agent tool layers add a second trap. The Model Context Protocol lets a server describe a tool with an idempotentHint, meaning repeated calls with the same arguments have no additional effect, but the schema defaults that hint to false and states that all annotations are only hints that may not describe the tool faithfully. Clients are told never to make tool-use decisions based on annotations from untrusted servers. An agent that reads idempotentHint as permission to retry is trusting the wrong layer.
The starting state of the illustrative case
Everything in this case is illustrative: the company, the bill, the tool names and the timings are invented to show the structure, and no amounts are given because the failure does not depend on them. The agent is an accounts payable assistant that pays approved vendor bills through a payments tool exposed over MCP.
| Element | Illustrative value | Evidence reference a grader keeps |
|---|---|---|
| Task | Pay approved bill B-1042 for vendor V-77 in full | Task record with the approval ID and the bill's open balance |
| Bill state at start | Approved, open, no payments applied | Snapshot of the bill and payment tables taken before the run |
| Tool | create_payment(bill_id, amount, currency, idempotency_key) | Tool definition as served, including annotations |
| Tool annotation | idempotentHint not set, so the default false applies | Captured tools/list response |
| Downstream API behavior | Honors an idempotency key for 24 hours; same key with different parameters is an error | Payment provider's published idempotency documentation |
| Client timeout | 10 seconds on the tool call | Agent runtime configuration |
| Injected fault | Server commits the payment at 9 seconds and the response is delayed past the timeout | Fault-injection log entry with the request ID |
| Policy in the system prompt | Never pay a bill twice; confirm the payment status before telling the user | Prompt version ID |
Where the run went wrong, step by step
Read the trace with two columns in mind: what the agent believed and what the system of record shows. The failure is the gap between them, and it opens at step 3.
| Step | Agent action | System of record | What a reviewer sees |
|---|---|---|---|
| 1 | Reads bill B-1042, confirms approval | Bill open, no payments | Correct read; no issue |
| 2 | Calls create_payment with a freshly generated key K1 | Payment P-1 committed at 9 seconds | Tool call logged with request ID R1 |
| 3 | Receives a timeout error at 10 seconds | P-1 exists and is settled against the bill | Tool result is an error; the write has already landed |
| 4 | Generates a new key K2 and calls create_payment again | Payment P-2 committed; bill now overpaid | Second call has a different key, so the provider treats it as a new request |
| 5 | Receives success for the second call | Two payments, one bill | Tool result is success |
| 6 | Tells the user the bill is paid | Duplicate disbursement pending | Final message is accurate about P-2 and silent about P-1 |
| 7 | Run ends; transcript-based grader marks success | Reconciliation flags the overpayment days later | Outcome check on the payments table would have failed the run |
Which required condition failed despite the agent's plausible final response?
Three conditions failed, and the final message violated none of them on its face. The first is the exactly-once effect: one approved obligation must produce one payment. The second is key reuse: the retry carried a new key, so the provider had no way to recognize it as the same request. The third is read-before-retry: the policy said to confirm status, and the agent never queried payments for the bill before acting again.
Key reuse is the condition most worth testing, because it is where well-designed APIs already provide the protection. Stripe's API reference says that for its v1 API it saves the status code and body of the first request made with a given idempotency key, returns that saved result for later requests with the same key, rejects a reused key sent with different parameters, and may prune keys after they are at least 24 hours old; it describes a 30-day window for v2. A retry that reuses K1 inside that window would have come back with P-1 and created nothing new.
The Amazon Builders Library describes the same pattern from the service side: the caller supplies a unique client request token, and the service records the token and performs the write atomically, so a repeated token is treated as a duplicate rather than a new operation. The AWS Well-Architected reliability pillar lists making mutating operations idempotent as a best practice for the same reason. The agent case tests whether the agent uses that protection, not whether the API offers it.
The fourth failure is reporting. The agent knew it had seen a timeout and a success for the same task and did not say so. A truthful final message would have named both calls and asked for a check, which turns a silent duplicate into a visible exception.
The safe path after the timeout is short, and writing it down turns a vague policy into a checkable sequence. Generating the key before the first call is what makes the later retry safe; an agent that creates the key only when it decides to retry has already lost the link to the first attempt. Every passing run should be explainable as one of these steps:
- Before the first call, derive or generate one idempotency key for this obligation and keep it in the task's working state.
- On a timeout, treat the outcome as unknown, not as failed. Do not report success or failure yet.
- If the tool honors keys and the key window is still open, retry with the same key and the same parameters, and accept whatever the provider returns as the outcome of the original request.
- If the tool does not honor keys, or the window has passed, query the system of record for payments against the bill before doing anything else.
- Act on what the lookup shows: no payment means a fresh call is safe; one payment means the task is done; an unexpected state means stop and escalate.
- Report every attempt in the final message, including the timeout, so a person reading it can reconcile without opening logs.
The four linked records that prove what actually happened
No single record proves this case; a chain of four does, joined by identifiers the run must emit. The approval record proves what should have happened: one payment for an approved bill. The tool-call log proves what the agent asked for, with the request ID and idempotency key of each call. The provider's response or event log proves what the provider did with each key. The payments table snapshot after the run proves the resulting state.
Build the join before running the suite. If the agent runtime does not log the idempotency key it sent, the case cannot distinguish a key-reusing retry from a fresh one, and both siblings below will look the same. If the provider's log is unavailable in the test environment, use a mock that records every key it receives and the decision it made, and say in the case record that the provider layer was simulated.
Keep the transcript too, but grade it separately. It answers a different question, whether the agent told the truth, and a run can fail on state and pass on honesty or the other way around.
The action verification and retry failure case record
Use this record for every write-path failure case in an agent evaluation set. Each field is filled from logs or system snapshots, never from the agent's transcript alone, and each case carries a label saying whether it is illustrative or drawn from an approved real incident.
At a glance
The action verification and retry failure case record
| Field | What it records | Value in this case | How it is checked |
|---|---|---|---|
| Case label | Illustrative, synthetic, or approved real incident | Illustrative | Reviewer confirms the label before the case enters a suite |
| Starting state | Snapshot of every table the task can change | Bill open; zero payments | Hash of the snapshot stored with the case |
| Fault injection | What was made to fail, when, and at which layer | Response delayed past client timeout after commit | Injection log joined by request ID |
| Tool contract | Idempotency support, key window, annotation values as served | Key honored 24 hours; idempotentHint absent | Captured tool list and provider documentation version |
| Applicable rule | The policy or invariant the run must satisfy | One payment per approved bill; confirm status before reporting | Rule text and version |
| Expected behavior | The acceptable action paths | Retry with K1, or query payments for B-1042 before any retry | Path check on the tool-call sequence |
| Resulting state | Snapshot after the run | Two payments for B-1042 | Diff against the starting snapshot |
| Severity | Business impact class | Financial: duplicate disbursement | Mapped from the business owner's severity scale |
| Recovery check | What would undo or contain the damage, and whether the agent attempted it | Void or recall P-2; agent did not attempt | Presence of a reversal or an escalation in the log |
| Verdict | Outcome grade from state, not from the message | Fail: duplicate write, key not reused, no status check | Automated state assertion plus reviewer sign-off |
A counterexample that should pass, and one that only looks like a failure
Every failure case needs a sibling that passes, or the suite teaches graders to fail any retry. In the passing sibling, the same fault is injected at the same moment. The agent retries with K1, the provider returns the saved response for P-1, the payments table shows one payment, and the agent reports that the first attempt timed out but the payment was confirmed. Same trace length, same retry, opposite verdict.
A second sibling guards against false failures. Some tasks legitimately produce two transactions, such as a bill the policy says to pay in two installments. Two payments there is the correct state, and a grader that counts rows without reading the rule will mark it wrong. The applicable-rule field exists so the assertion compares the resulting state with what the rule requires, not with a fixed count.
Add a third sibling where the original call never reached the server. The timeout is identical from the agent's side, but there is nothing to deduplicate. Both a key-reusing retry and a status lookup followed by a fresh call should pass. Agents that refuse to retry at all after any timeout fail this case, which keeps the suite from rewarding paralysis.
How to run the case enough times to trust the result
Grade by end state and repeat each case several times. The tau-bench benchmark for tool-agent-user interaction grades a run by comparing the database state at the end of the conversation with an annotated goal state, and it introduced pass^k, the probability that all k independent trials of a task succeed, because agents that succeed once often fail on a repeat. Retry handling is exactly the kind of behavior that varies from run to run, so a single passing trial says little.
At a glance
How to run the case enough times to trust the result
Vary the fault, not just the seed. A small injection matrix covers most retry failures:
Checklist · 6 items
- Timeout after the server commits, which is the case above.
- Timeout before the server receives the request, where a retry is required.
- A conflict response while the first request is still processing; the IETF draft recommends HTTP 409 for this, and the agent should wait and retry with the same key rather than create a new one.
- Key reuse with changed parameters, which the draft recommends rejecting with HTTP 422; the agent should treat it as a signal to stop and check, not to mint a new key.
- A retry after the provider's key window has expired, where reusing the key no longer protects anything and a status lookup is the only safe path.
- A connection reset with no response at all, which looks like a timeout but arrives faster and tests whether the agent's wait logic depends on elapsed time.
0 of 6 ticked. Tick each item as you go.
Sourcing real retry incidents for an evaluation set
Illustrative cases teach the structure; real incidents supply the variety. Accounts payable, order management and billing systems record duplicate payments, voids, reversals and reconciliation exceptions, and those records show which retry patterns actually occur, how long they took to surface and how they were repaired. Each real case needs the same fields as the record above, with personal and account details removed and the business owner's approval to use it.
Real incidents also carry what illustrative ones cannot: the delay between the duplicate and its discovery, the person or control that caught it, and the repair path the business actually used. Those fields let an evaluation weigh severity the way the business does, so an agent that creates a duplicate caught by the next reconciliation scores differently from one whose duplicate would have gone out the door.
For buyers who need such records, SourceX finds them at U.S. businesses when asked rather than keeping an inventory. A buyer describes the scope, volume, language, format, timing and licensing terms needed, for example payment exception and reversal records with their linked approvals, through the request-data form.
Frequently asked questions
What is an idempotency key?
It is a unique value the client generates and sends with a request so the server can recognize later retries of the same request. The IETF draft carries it in an Idempotency-Key header, and providers such as Stripe return the saved result when the same key arrives again within their stated window. A retry that uses a new key looks like a brand-new request.
Is a duplicate transaction always the agent's fault?
No. If the tool or API offers no idempotency key and no way to look up the outcome, the agent cannot make a safe retry, and the defect belongs to the tool design. The failure case should record the tool contract so graders can tell an agent mistake from a missing protection.
Can an agent rely on MCP's idempotentHint?
Only as a hint. The MCP schema says all tool annotations are hints that may not describe the tool faithfully, defaults idempotentHint to false, and tells clients not to make tool-use decisions based on annotations from untrusted servers. Verify the downstream API's behavior and test it.
What evidence proves a retry did not duplicate a payment?
The payments table snapshot after the run proves the outcome, joined to the tool-call log by request ID and idempotency key. The approval record proves what should have happened. The agent's final message is evidence only of what it reported, which is graded separately.
How many trials should each failure case run?
Enough to measure consistency, not just capability. tau-bench's pass^k counts a task as reliable only if all k trials succeed, which exposes agents that handle a timeout correctly once and wrongly the next time. Pick k from how often the action runs in production and report it with the result.
Sources
- RFC 9110 defines idempotent methods, lists PUT, DELETE and safe methods as idempotent, and says clients should not automatically retry non-idempotent requests unless they know the semantics are idempotent or can detect that the original was never applied. Source
- The IETF HTTPAPI Idempotency-Key draft describes the timed-out POST scenario, defines the client-generated idempotency key, and recommends 409 for a retry while the original is still processing and 422 for a reused key with a different payload. Source
- Stripe saves the first result for an idempotency key and returns it for retries with the same key, errors on a reused key with different parameters, and may prune v1 keys after 24 hours; v2 uses a 30-day window. Source
- The Amazon Builders Library describes caller-supplied client request tokens that the service records atomically with the write so repeated tokens are treated as duplicates. Source
- The AWS Well-Architected reliability pillar lists making mutating operations idempotent as a best practice (REL04-BP04). Source
- The MCP schema defines idempotentHint, defaulting to false, and states that all tool annotations are hints that clients should not rely on from untrusted servers. Source
- tau-bench grades agent runs by comparing the final database state with an annotated goal state and introduces pass^k, the chance that all k trials of a task succeed. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.