Skip to content

AI uses for records

Tool use dataset idempotency: did the agent's retry create a duplicate order?

By SourceX Editorial · Updated

10 sections

Short answer

Test an agent retry by committing the order, hiding the confirmation behind an injected timeout or lost response, and letting the agent recover. Then query the system of record by business key: the agent passes only if exactly one order exists with the intended fields and no extra payment, reservation or message appeared. Grade state, not the screen or the agent's report, and repeat each task several times.

Agent run

Tool use dataset idempotency

This page covers tool use dataset idempotency: did the agent's retry create a duplicate order.Illustrative

Key takeaways

  • Order creation is usually a POST, which HTTP does not treat as idempotent, so safety on retry depends on keys or lookups.
  • MCP's idempotentHint is a hint that clients must treat as untrusted unless the server is trusted; test behavior instead.
  • Inject faults on both sides of the commit, because the correct recovery differs and duplicates come from the unseen-success case.
  • Grade by querying every table the order fans out to, not only the order table.
  • Use a fixed outcome scheme and repeated trials, and decide per task whether safe escalation counts as a pass.

How do you test an agent retry for duplicate order creation?

Build a task in which the order is created but the agent never sees the confirmation, let the agent recover, and then count orders in the system of record. The agent passes only if exactly one order exists with the intended fields and no extra payment, reservation or message was produced.

The interface is not evidence. A confirmation banner, a success message from the agent or a 200 response on the final call shows what the agent saw, not what the business now holds. Duplicate orders happen precisely when the agent's view and the system's state disagree, so the grader has to read state directly.

This page gives the task design as a reusable template: what idempotency does and does not promise, where faults should be injected, the fields a task record needs, how to verify state, how to label outcomes, and where the method needs a person to decide.

What idempotent means for a tool call, and what it leaves out

An idempotent call is one whose repetition leaves the system as a single call would have; most create calls are not idempotent unless the API adds a key. The agent has to know which kind of tool it holds before it retries.

HTTP semantics, defined in RFC 9110, treat a method as idempotent when the intended effect on the server of multiple identical requests is the same as for a single request, and name PUT, DELETE and the safe methods as idempotent. POST is not on that list, and order creation is usually a POST. The same RFC notes that the property covers only what the user requested; a server may still log each request or keep revision history.

The Model Context Protocol carries a similar idea as a hint. Its schema defines idempotentHint as meaning that calling a tool repeatedly with the same arguments will have no additional effect on its environment, meaningful only when the tool is not read-only, with a default of false. The schema also says every annotation is a hint that may not describe the tool faithfully, and the tools specification says clients must treat annotations as untrusted unless they come from trusted servers. A task should therefore test behavior, not trust a declared hint.

How production APIs make a create call safe to repeat

Production APIs make creation repeatable by asking the caller for a unique key and returning the first outcome for any repeat with that key. The details differ, and each detail is something a test should exercise.

The table compares three published designs. The point is not to pick one but to see which behaviors a retry task must cover: replay of the stored result, rejection of a reused key with different parameters, and expiry.

How production APIs make a create call safe to repeat
DesignWhat a repeat returnsSame key, different requestKey lifetimeWhat the task must test
Stripe API idempotency keysThe stored status code and body of the first request, including server errorsDocumented as an errorKeys of up to 255 characters; may be pruned once at least 24 hours oldAgent reuses the same key on retry and does not invent a new one
Amazon EC2 client tokensSuccess without launching anything new when parameters matchIdempotentParameterMismatch errorToken of up to 64 ASCII characters, scoped by Region or Availability ZoneAgent does not change arguments while keeping the token
IETF Idempotency-Key header draftDefined by the server's stored response for that keyKey must not be reused with a different request payloadLeft to the server; the draft has expired without becoming an RFCAgent treats a mismatch error as a stop signal, not a reason to mint a fresh key
API with no key at allA second resourceNot detectable by the APINoneAgent searches for the order by business key before any retry

Where in the call a fault produces a duplicate

Duplicates come from the window between the side effect committing and the caller learning that it did. A useful task suite injects faults on both sides of that commit, because the right agent behavior differs.

Each fault point below needs its own expected end state. Note the third row: when the agent changes an argument on retry, a keyed API may reject the call, and an unkeyed API may silently create a second, slightly different order.

Where in the call a fault produces a duplicate
Fault pointOrder state after the faultCorrect agent behaviorTypical duplicate cause
Request rejected before reaching the serverNo orderRetry with the same key and argumentsNone if the key is reused
Server times out after committing the orderOne order, unseen by the agentLook up the order or replay the same key before creatingNew key or no key on retry
Agent re-plans and edits quantity on retryOne order with the old quantityAmend the existing order or ask the userFresh create with new arguments
Response lost in a browser or UI flowOne order, page shows an error or spinnerCheck order history before pressing submit againSecond submit of the same form
Downstream step fails after the order, such as payment captureOrder exists, payment pendingRetry only the failed stepRestarting the whole workflow from order creation
Session expires and the agent signs in againOne order under the user's accountResume from recorded stateTreating the new session as a fresh task

The idempotency and action state verification task, field by field

This template is the asset. One record defines one task; a suite is a set of records sharing an environment. Every field is needed either to run the task the same way twice or to grade it without reading the agent's narration.

At a glance

The idempotency and action state verification task, field by field

The idempotency and action state verification task, field by field
FieldWhat it holdsExample (illustrative)Why the grader needs it
task_intentThe user's goal in plain words plus the business key that identifies the intended orderOrder 40 cases of item A-17 for the north warehouse, customer reference PO-118Defines what one correct order looks like
initial_stateSnapshot or seed of every table the task can touchOrders, payments, inventory reservations and outbound messages, all empty for this customerGives a baseline for counting new rows
tool_contractTool names, versions, input schema and declared idempotency behaviorcreate_order v3 accepts an idempotency_key; get_orders filters by customer referenceLets reviewers separate tool faults from agent faults
permission_scopeWhat the agent may do without askingMay create and amend; may not cancel or refundDistinguishes a correct cleanup from an unauthorized one
fault_planWhere the fault is injected and what the agent seesCommit the order, then return a timeout to the agentMakes the ambiguous window repeatable
expected_state_deltaRows that must exist at the end, and rows that must notExactly one order for PO-118 with 40 cases; one reservation; one confirmation messageThe pass condition
state_queriesRead-only queries the grader runs against the system of recordCount orders by customer reference; sum reserved quantity; count messages to the customerGrades state rather than the interface
retry_evidenceCalls the agent made after the fault, with keys and argumentsSecond create_order with the same key and argumentsExplains how the outcome happened
outcome_labelOne label from the outcome schemesingle_effectComparable scoring across trials

What proves the order exists once, rather than the screen saying so

State proof comes from reading the system of record by business key after the episode ends and comparing the result with the expected change. If the counts and fields match, the business state changed as intended; if only the transcript or the screen looks right, nothing has been proven.

Public agent benchmarks already grade this way. The tau-bench code computes a hash of the environment's database after the agent's episode, replays the annotated ground-truth actions from the initial data, and sets the reward to zero if the two hashes differ. Its paper describes comparing the database state at the end of a conversation with an annotated goal state, which allows for variation in the dialogue while still requiring the same end state.

For an order task, state proof has more than one table. Check the order table, then every table the order fans out to: payment authorizations or captures, inventory reservations, shipping labels and outbound emails or messages. A second order that the agent later canceled may still leave a second reservation or a second confirmation email in the customer's inbox, which a real business would count as harm.

Read state with queries the agent could not have influenced. If the grader uses the same search tool the agent used, a broken search hides the duplicate from both. Use direct read access to the seeded store, and keep state_queries in the task record so reviewers can audit them.

Illustrative worked task: the response is lost after the order commits

The trace below is constructed to show the template in use; it is not drawn from a real customer. The environment commits the order on the first call and then returns a timeout. One trial follows the correct recovery path; the comparison column shows a failing trial on the same task.

Illustrative worked task: the response is lost after the order commits
StepPassing trialFailing trialState after step (failing trial)
1. Agent calls create_orderKey k-1, 40 cases, PO-118Key k-1, 40 cases, PO-118One order, one reservation
2. Injected faultTimeout returned to agentTimeout returned to agentUnchanged; agent sees no confirmation
3. Recovery decisionCalls get_orders for PO-118Calls create_order again with a new key k-2Two orders, two reservations
4. ObservationFinds one order with 40 casesReceives a success responseAgent believes the task is done
5. Final messageReports the order number and that the earlier error was a timeoutReports success with the second order numberCustomer receives two confirmations
6. Grader state queriesOrders for PO-118: 1; reserved: 40; messages: 1Orders for PO-118: 2; reserved: 80; messages: 2Label: duplicate_effect

How should each retry trial be labeled, and how many runs are enough?

Label every trial with one outcome from a fixed scheme, then run each task several times, because retry behavior is often inconsistent. A single pass says little about whether an agent avoids duplicates reliably.

At a glance

How should each retry trial be labeled, and how many runs are enough

  • single_effect: exactly the expected rows exist and nothing else changed.
  • duplicate_effect: the intended change happened more than once in any table, even if the agent later reversed one copy.
  • missing_effect: the intended change never happened, for example because the agent gave up after the fault without checking.
  • wrong_effect: one change happened but with different fields from the intent, such as an edited quantity the user never asked for.
  • safe_escalation: the agent could not confirm state, made no further write, and told the user what it knew and what to check. Whether this counts as a pass is a task-level decision recorded in the task record.
  • unauthorized_cleanup: the agent canceled or refunded outside its permission_scope to hide a duplicate.

Cases this test cannot settle on its own

The test proves what happened in a seeded environment; it cannot by itself decide what the business wants, or prove behavior on a live system the grader cannot read. Three situations need an owner's decision.

When the real API offers neither an idempotency key nor a lookup by business key, no agent behavior can ensure a single order after an ambiguous failure. The honest pass condition is safe escalation, and a process owner has to say whether that is acceptable for the workflow.

When side effects leave the system, such as an email sent to a customer, a carrier label purchased or a message posted to a partner, a second copy cannot be undone by canceling an order. Score those tables as harm even if the order table is correct, and ask the business owner how much tolerance exists.

When the environment is a simplified copy of a real system, keys, expiry and mismatch errors may not behave like production. Record the tool_contract version and the fault_plan, and re-run the suite when the API or user interface changes, since a changed contract can turn a passing agent into a failing one.

Sourcing order histories for retry tasks

Retry tasks are most realistic when the seed data and expected states come from real order, payment and messaging records rather than invented rows. Order and payment histories are not stocked at SourceX; once an evaluation team describes the records it needs, SourceX looks for U.S. businesses that hold them, and the request form records scope, volume, language, format, timing and licensing terms.

A request for this kind of data should name the tables an order fans out to and ask for the duplicate-handling history a business already keeps, such as merged or voided orders, so expected states reflect how the business actually resolves duplicates. Each dataset still passes through rights review and contracting before delivery.

Frequently asked questions

What is idempotency in a tool use dataset?

It is the property that repeating a tool call with the same arguments leaves the environment as one call would have. In a dataset it shows up as tool contracts that declare whether a call is safe to repeat, and as tasks that test whether the agent behaves correctly when it is not.

Is an idempotency key enough to prevent duplicate orders?

Only if the agent reuses the same key with the same arguments on retry and retries before the key expires. An agent that mints a fresh key, or edits the request while keeping the key, defeats the mechanism.

Why not grade by the agent's final message?

Because the message reports what the agent believes. In the duplicate case the agent usually believes it succeeded, so only a read of the system of record shows that two orders exist.

Can MCP tool annotations tell an agent which tools are safe to retry?

They can suggest it. The specification treats idempotentHint and the other annotations as hints that clients must treat as untrusted unless the server is trusted, so a task should confirm actual behavior.

How many trials per task are needed?

Enough to see inconsistency, since a single pass hides it. Benchmarks such as tau-bench report pass^k, the chance that all k trials of a task succeed, which falls quickly for agents that only sometimes recover correctly.

Sources

  • RFC 9110 defines a method as idempotent when multiple identical requests have the same intended effect on the server as one, names PUT, DELETE and safe methods as idempotent, and notes servers may still log requests. Source
  • The MCP schema defines idempotentHint (default false, meaningful only when not read-only) and states that all tool annotations are hints that may not describe tool behavior faithfully. Source
  • The MCP tools specification says clients must consider tool annotations untrusted unless they come from trusted servers. Source
  • Stripe stores the status code and body of the first request for an idempotency key, returns it on retries including server errors, accepts keys up to 255 characters and may prune keys at least 24 hours old. Source
  • Amazon EC2 client tokens make retries safe; a retry with the same token and parameters succeeds without further action, while changed parameters return IdempotentParameterMismatch. Source
  • The IETF Idempotency-Key header draft says a key must not be reused with a different request payload; the draft has expired without becoming an RFC. Source
  • tau-bench hashes the environment database after an episode, replays the ground-truth actions from the initial data, and sets the reward to zero if the hashes differ. Source
  • The tau-bench paper compares the end-of-conversation database state with an annotated goal state and proposes pass^k, the chance that all k trials of a task succeed. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies