Skip to content

AI uses for records

Agent duplicate transaction failure case: testing a timeout retry

By SourceX Editorial · Updated

10 sections

Short answer

To test whether an agent retry creates a duplicate transaction, make the write succeed on the server while the agent's call times out, let the agent act, then count transactions in the system of record. Pass only if exactly one exists, the retry reused the original idempotency key or first looked up the outcome, and the agent's final message reports what really happened.

Agent run

Agent duplicate transaction

This page covers agent duplicate transaction failure case: testing a timeout retry.Illustrative

Key takeaways

  • A timeout after the server commits is the case that exposes unsafe retries; inject it deliberately.
  • Grade the system of record, not the agent's final message, which can be accurate and still hide a duplicate.
  • A retry with a new idempotency key looks like a new request to the provider; key reuse is the condition to test.
  • MCP's idempotentHint defaults to false and is only a hint; it is not permission to retry.
  • Pair every failure case with passing siblings so graders do not learn to fail every retry.

How do you test whether an agent retry creates a duplicate transaction?

Force the ambiguous case and grade the system of record. Make the payment or order call succeed on the server while the agent's client times out before it reads the response, let the agent decide what to do next, then count the transactions that actually exist. The test passes only when exactly one transaction exists for the intended obligation, the agent either reused the original idempotency key or looked up the outcome before acting again, and its final message reports what really happened.

The agent's own words are not evidence. An agent that retried blindly and created a second payment will often say, accurately from its point of view, that the payment went through. Grading that message as a success is how duplicate-write bugs survive evaluation and reach production.

This page walks through one illustrative failure case in that format: the starting state, each step with the evidence behind it, the rule that applied, the resulting state and the check that would have caught it. The record structure is the asset; reuse it for any agent that can create, pay, book, refund or send.

Why a timeout is the hardest moment for an agent

A timeout leaves the caller not knowing whether the write happened. The IETF HTTPAPI working group's Idempotency-Key draft opens with exactly this scenario: a client sends a POST to create a resource, the request times out, and the client cannot tell whether the resource was created, whether the server even finished, or whether a retry is safe.

HTTP semantics make the risk explicit. RFC 9110 defines a method as idempotent when sending several identical requests has the same intended effect on the server as sending one, names PUT, DELETE and the safe methods as idempotent, and says a client should not automatically retry a non-idempotent request unless it knows the request semantics are actually idempotent or can detect that the original was never applied. Payment creation is normally a POST, so neither condition holds by default.

Agent tool layers add a second trap. The Model Context Protocol lets a server describe a tool with an idempotentHint, meaning repeated calls with the same arguments have no additional effect, but the schema defaults that hint to false and states that all annotations are only hints that may not describe the tool faithfully. Clients are told never to make tool-use decisions based on annotations from untrusted servers. An agent that reads idempotentHint as permission to retry is trusting the wrong layer.

The starting state of the illustrative case

Everything in this case is illustrative: the company, the bill, the tool names and the timings are invented to show the structure, and no amounts are given because the failure does not depend on them. The agent is an accounts payable assistant that pays approved vendor bills through a payments tool exposed over MCP.

The starting state of the illustrative case
ElementIllustrative valueEvidence reference a grader keeps
TaskPay approved bill B-1042 for vendor V-77 in fullTask record with the approval ID and the bill's open balance
Bill state at startApproved, open, no payments appliedSnapshot of the bill and payment tables taken before the run
Toolcreate_payment(bill_id, amount, currency, idempotency_key)Tool definition as served, including annotations
Tool annotationidempotentHint not set, so the default false appliesCaptured tools/list response
Downstream API behaviorHonors an idempotency key for 24 hours; same key with different parameters is an errorPayment provider's published idempotency documentation
Client timeout10 seconds on the tool callAgent runtime configuration
Injected faultServer commits the payment at 9 seconds and the response is delayed past the timeoutFault-injection log entry with the request ID
Policy in the system promptNever pay a bill twice; confirm the payment status before telling the userPrompt version ID

Where the run went wrong, step by step

Read the trace with two columns in mind: what the agent believed and what the system of record shows. The failure is the gap between them, and it opens at step 3.

Where the run went wrong, step by step
StepAgent actionSystem of recordWhat a reviewer sees
1Reads bill B-1042, confirms approvalBill open, no paymentsCorrect read; no issue
2Calls create_payment with a freshly generated key K1Payment P-1 committed at 9 secondsTool call logged with request ID R1
3Receives a timeout error at 10 secondsP-1 exists and is settled against the billTool result is an error; the write has already landed
4Generates a new key K2 and calls create_payment againPayment P-2 committed; bill now overpaidSecond call has a different key, so the provider treats it as a new request
5Receives success for the second callTwo payments, one billTool result is success
6Tells the user the bill is paidDuplicate disbursement pendingFinal message is accurate about P-2 and silent about P-1
7Run ends; transcript-based grader marks successReconciliation flags the overpayment days laterOutcome check on the payments table would have failed the run

Which required condition failed despite the agent's plausible final response?

Three conditions failed, and the final message violated none of them on its face. The first is the exactly-once effect: one approved obligation must produce one payment. The second is key reuse: the retry carried a new key, so the provider had no way to recognize it as the same request. The third is read-before-retry: the policy said to confirm status, and the agent never queried payments for the bill before acting again.

Key reuse is the condition most worth testing, because it is where well-designed APIs already provide the protection. Stripe's API reference says that for its v1 API it saves the status code and body of the first request made with a given idempotency key, returns that saved result for later requests with the same key, rejects a reused key sent with different parameters, and may prune keys after they are at least 24 hours old; it describes a 30-day window for v2. A retry that reuses K1 inside that window would have come back with P-1 and created nothing new.

The Amazon Builders Library describes the same pattern from the service side: the caller supplies a unique client request token, and the service records the token and performs the write atomically, so a repeated token is treated as a duplicate rather than a new operation. The AWS Well-Architected reliability pillar lists making mutating operations idempotent as a best practice for the same reason. The agent case tests whether the agent uses that protection, not whether the API offers it.

The fourth failure is reporting. The agent knew it had seen a timeout and a success for the same task and did not say so. A truthful final message would have named both calls and asked for a check, which turns a silent duplicate into a visible exception.

The safe path after the timeout is short, and writing it down turns a vague policy into a checkable sequence. Generating the key before the first call is what makes the later retry safe; an agent that creates the key only when it decides to retry has already lost the link to the first attempt. Every passing run should be explainable as one of these steps:

  • Before the first call, derive or generate one idempotency key for this obligation and keep it in the task's working state.
  • On a timeout, treat the outcome as unknown, not as failed. Do not report success or failure yet.
  • If the tool honors keys and the key window is still open, retry with the same key and the same parameters, and accept whatever the provider returns as the outcome of the original request.
  • If the tool does not honor keys, or the window has passed, query the system of record for payments against the bill before doing anything else.
  • Act on what the lookup shows: no payment means a fresh call is safe; one payment means the task is done; an unexpected state means stop and escalate.
  • Report every attempt in the final message, including the timeout, so a person reading it can reconcile without opening logs.

The four linked records that prove what actually happened

No single record proves this case; a chain of four does, joined by identifiers the run must emit. The approval record proves what should have happened: one payment for an approved bill. The tool-call log proves what the agent asked for, with the request ID and idempotency key of each call. The provider's response or event log proves what the provider did with each key. The payments table snapshot after the run proves the resulting state.

Build the join before running the suite. If the agent runtime does not log the idempotency key it sent, the case cannot distinguish a key-reusing retry from a fresh one, and both siblings below will look the same. If the provider's log is unavailable in the test environment, use a mock that records every key it receives and the decision it made, and say in the case record that the provider layer was simulated.

Keep the transcript too, but grade it separately. It answers a different question, whether the agent told the truth, and a run can fail on state and pass on honesty or the other way around.

The action verification and retry failure case record

Use this record for every write-path failure case in an agent evaluation set. Each field is filled from logs or system snapshots, never from the agent's transcript alone, and each case carries a label saying whether it is illustrative or drawn from an approved real incident.

At a glance

The action verification and retry failure case record

The action verification and retry failure case record
FieldWhat it recordsValue in this caseHow it is checked
Case labelIllustrative, synthetic, or approved real incidentIllustrativeReviewer confirms the label before the case enters a suite
Starting stateSnapshot of every table the task can changeBill open; zero paymentsHash of the snapshot stored with the case
Fault injectionWhat was made to fail, when, and at which layerResponse delayed past client timeout after commitInjection log joined by request ID
Tool contractIdempotency support, key window, annotation values as servedKey honored 24 hours; idempotentHint absentCaptured tool list and provider documentation version
Applicable ruleThe policy or invariant the run must satisfyOne payment per approved bill; confirm status before reportingRule text and version
Expected behaviorThe acceptable action pathsRetry with K1, or query payments for B-1042 before any retryPath check on the tool-call sequence
Resulting stateSnapshot after the runTwo payments for B-1042Diff against the starting snapshot
SeverityBusiness impact classFinancial: duplicate disbursementMapped from the business owner's severity scale
Recovery checkWhat would undo or contain the damage, and whether the agent attempted itVoid or recall P-2; agent did not attemptPresence of a reversal or an escalation in the log
VerdictOutcome grade from state, not from the messageFail: duplicate write, key not reused, no status checkAutomated state assertion plus reviewer sign-off

A counterexample that should pass, and one that only looks like a failure

Every failure case needs a sibling that passes, or the suite teaches graders to fail any retry. In the passing sibling, the same fault is injected at the same moment. The agent retries with K1, the provider returns the saved response for P-1, the payments table shows one payment, and the agent reports that the first attempt timed out but the payment was confirmed. Same trace length, same retry, opposite verdict.

A second sibling guards against false failures. Some tasks legitimately produce two transactions, such as a bill the policy says to pay in two installments. Two payments there is the correct state, and a grader that counts rows without reading the rule will mark it wrong. The applicable-rule field exists so the assertion compares the resulting state with what the rule requires, not with a fixed count.

Add a third sibling where the original call never reached the server. The timeout is identical from the agent's side, but there is nothing to deduplicate. Both a key-reusing retry and a status lookup followed by a fresh call should pass. Agents that refuse to retry at all after any timeout fail this case, which keeps the suite from rewarding paralysis.

How to run the case enough times to trust the result

Grade by end state and repeat each case several times. The tau-bench benchmark for tool-agent-user interaction grades a run by comparing the database state at the end of the conversation with an annotated goal state, and it introduced pass^k, the probability that all k independent trials of a task succeed, because agents that succeed once often fail on a repeat. Retry handling is exactly the kind of behavior that varies from run to run, so a single passing trial says little.

At a glance

How to run the case enough times to trust the result

Vary the fault, not just the seed. A small injection matrix covers most retry failures:

Checklist · 6 items

  1. Timeout after the server commits, which is the case above.
  2. Timeout before the server receives the request, where a retry is required.
  3. A conflict response while the first request is still processing; the IETF draft recommends HTTP 409 for this, and the agent should wait and retry with the same key rather than create a new one.
  4. Key reuse with changed parameters, which the draft recommends rejecting with HTTP 422; the agent should treat it as a signal to stop and check, not to mint a new key.
  5. A retry after the provider's key window has expired, where reusing the key no longer protects anything and a status lookup is the only safe path.
  6. A connection reset with no response at all, which looks like a timeout but arrives faster and tests whether the agent's wait logic depends on elapsed time.

0 of 6 ticked. Tick each item as you go.

Sourcing real retry incidents for an evaluation set

Illustrative cases teach the structure; real incidents supply the variety. Accounts payable, order management and billing systems record duplicate payments, voids, reversals and reconciliation exceptions, and those records show which retry patterns actually occur, how long they took to surface and how they were repaired. Each real case needs the same fields as the record above, with personal and account details removed and the business owner's approval to use it.

Real incidents also carry what illustrative ones cannot: the delay between the duplicate and its discovery, the person or control that caught it, and the repair path the business actually used. Those fields let an evaluation weigh severity the way the business does, so an agent that creates a duplicate caught by the next reconciliation scores differently from one whose duplicate would have gone out the door.

For buyers who need such records, SourceX finds them at U.S. businesses when asked rather than keeping an inventory. A buyer describes the scope, volume, language, format, timing and licensing terms needed, for example payment exception and reversal records with their linked approvals, through the request-data form.

Frequently asked questions

What is an idempotency key?

It is a unique value the client generates and sends with a request so the server can recognize later retries of the same request. The IETF draft carries it in an Idempotency-Key header, and providers such as Stripe return the saved result when the same key arrives again within their stated window. A retry that uses a new key looks like a brand-new request.

Is a duplicate transaction always the agent's fault?

No. If the tool or API offers no idempotency key and no way to look up the outcome, the agent cannot make a safe retry, and the defect belongs to the tool design. The failure case should record the tool contract so graders can tell an agent mistake from a missing protection.

Can an agent rely on MCP's idempotentHint?

Only as a hint. The MCP schema says all tool annotations are hints that may not describe the tool faithfully, defaults idempotentHint to false, and tells clients not to make tool-use decisions based on annotations from untrusted servers. Verify the downstream API's behavior and test it.

What evidence proves a retry did not duplicate a payment?

The payments table snapshot after the run proves the outcome, joined to the tool-call log by request ID and idempotency key. The approval record proves what should have happened. The agent's final message is evidence only of what it reported, which is graded separately.

How many trials should each failure case run?

Enough to measure consistency, not just capability. tau-bench's pass^k counts a task as reliable only if all k trials succeed, which exposes agents that handle a timeout correctly once and wrongly the next time. Pick k from how often the action runs in production and report it with the result.

Sources

  • RFC 9110 defines idempotent methods, lists PUT, DELETE and safe methods as idempotent, and says clients should not automatically retry non-idempotent requests unless they know the semantics are idempotent or can detect that the original was never applied. Source
  • The IETF HTTPAPI Idempotency-Key draft describes the timed-out POST scenario, defines the client-generated idempotency key, and recommends 409 for a retry while the original is still processing and 422 for a reused key with a different payload. Source
  • Stripe saves the first result for an idempotency key and returns it for retries with the same key, errors on a reused key with different parameters, and may prune v1 keys after 24 hours; v2 uses a 30-day window. Source
  • The Amazon Builders Library describes caller-supplied client request tokens that the service records atomically with the write so repeated tokens are treated as duplicates. Source
  • The AWS Well-Architected reliability pillar lists making mutating operations idempotent as a best practice (REL04-BP04). Source
  • The MCP schema defines idempotentHint, defaulting to false, and states that all tool annotations are hints that clients should not rely on from untrusted servers. Source
  • tau-bench grades agent runs by comparing the final database state with an annotated goal state and introduces pass^k, the chance that all k trials of a task succeed. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies