AI uses for records
Multi-agent conflicting actions dataset: two agents, one order
By SourceX Editorial · Updated
10 sections
Short answer
To evaluate conflicts when two agents edit the same order, seed one shared order, give each agent a scoped instruction, and force both to read the same version before either writes. Then score three things: whether the final order matches the instruction with higher authority, whether the losing agent detected the conflict instead of overwriting silently, and whether each agent's report matches what the order record shows.
Agent run
Multi agent conflicting
Key takeaways
- The failure to catch is the lost update: both agents succeed by their own account, and one change has vanished from the order.
- Authority must come from written policy, not from a tool's self-description or from whichever agent wrote last.
- Conflicts rarely appear by chance in a test run, so the task set has to force the interleaving with a scheduler and version tokens.
- Score the end state in the system of record, the detection of the conflict, and the truthfulness of each agent's account as three separate gates.
- Every task needs a counterexample in which two edits do not conflict, so agents are not rewarded for refusing to act.
How do you evaluate conflicts when two agents edit the same order?
Build each test as a deliberate collision. One order is the shared state, two agents receive instructions that touch it, and a scheduler makes both agents read the same version of the order before either writes. The test passes only if the order ends in the state the authority rule requires and neither agent's change disappeared without someone noticing.
A multi-agent conflicting actions dataset is a collection of these collisions, each with its starting state, the instructions, the forced ordering of reads and writes, the rule that decides which instruction wins, and the evidence that proves the outcome. It differs from an ordinary agent benchmark because the hard part is not doing the task, it is noticing that someone else changed the ground underneath you.
The order is a good test object because it is shared by many roles. Customer service changes addresses, fraud review places holds, fulfillment allocates stock, and pricing applies discounts. When those roles become agents, they edit the same record on overlapping schedules, and the ways that goes wrong are well understood from decades of database and web practice.
Why a confident final message can sit on top of a lost update
The classic two-writer failure is the lost update: both writers read version 7, each writes its own version 8, and the second write erases the first. HTTP has a standard guard for this. RFC 9110 says the If-Match precondition is most often used with state-changing methods to prevent accidental overwrites when multiple user agents act in parallel on the same resource, and a server that finds the condition false does not apply the request and can answer 412 Precondition Failed.
An agent that loses this race usually does not know it lost. Its tool call returned success, so it tells the user the address was changed. The other agent also reports success. Only the order record shows that one of the two changes is gone, which is why grading has to look at the stored state rather than at transcripts.
Public agent benchmarks already grade this way for single agents. In tau-bench, the reward check compares the database after the agent's run with the database produced by the ground-truth actions and sets the reward to zero if they differ. A conflict task set inherits that idea and adds a second writer, so the expected end state depends on which instruction had the right to win.
Which conflicts on one order belong in the task set?
Cover each way two actions can interfere, not just the obvious same-field overwrite. The table below lists seven patterns that make a sensible minimum set; each needs at least one case where the agents should defer and one where both edits can stand.
Several patterns come straight from how order policies are written. The tau-bench retail policy, for instance, allows modifying only pending orders, tells the agent to check status before acting, and lets the modify-items action run only once because it changes the order status. Any rule like that creates a race: two agents can both see a pending order, and only the first change is allowed.
Weight the set toward the patterns your deployment will actually meet. If agents never cancel orders, the write-versus-cancel pattern can stay small; if a fraud agent and a fulfillment agent both run on every order, hold-versus-release cases deserve the most variants, with different line counts, statuses and timing gaps between the two writes.
| Conflict pattern | Example on one order | What each agent sees at read time | Correct resolution |
|---|---|---|---|
| Same field, two writers | Service agent changes the ship-to address while an address-cleanup agent normalizes the old one | Both read the original address at the same version | Customer-requested address wins; cleanup rereads and normalizes the new value or stops |
| Write versus cancel | Upsell agent adds a line while a cancellation agent cancels at the customer's request | Order is pending for both | Cancellation stands; the add-line write is rejected and reported, not retried blindly |
| Status-gated action raced | Two agents both try to modify items on a pending order where modification is allowed once | Status pending, modification not yet used | First committed change stands; second agent detects the status change and escalates |
| Hold versus release | Fraud agent places a hold while a fulfillment agent releases the order to the warehouse | No hold present at read time | Hold has precedence; the release must fail or be reversed, with the reversal recorded |
| Stale derived value | Pricing agent applies a discount computed on lines that another agent has since changed | Line set from the earlier version | Discount recomputed on current lines or the write rejected; never applied to the wrong total |
| Human edit mid-task | A supervisor changes the delivery date in the order screen during an agent's multi-step task | Agent's cached copy predates the human edit | Human edit stands; agent rereads before its next write and adapts or asks |
| Cross-object dependency | One agent reallocates stock to another order while a second agent confirms this order's ship date | Inventory available at read time | Ship date confirmation waits for, or rechecks, the reservation it depends on |
Who has authority when two agents disagree?
Write the authority rule before the task, as data, and grade against it. Without a rule, the last writer wins by default, which is a property of the infrastructure rather than a business decision.
Do not let tools or agents assert their own authority. The Model Context Protocol schema describes tool annotations such as destructiveHint and idempotentHint as hints that may not describe the tool faithfully, and says clients should never make tool use decisions based on annotations received from untrusted servers. A task set that lets an agent win because its tool claims to be safe is testing the wrong thing.
Human approval is part of the authority chain, so model it explicitly. The OpenAI Agents SDK, for example, lets a tool require approval, surfaces pending approvals as interruptions and resumes the run after a decision. A conflict task can place one agent's write behind such an approval and check that the other agent does not slip a change in while the first is waiting.
| Precedence rank | Source of the instruction | Typical order actions | What the lower-ranked agent must do |
|---|---|---|---|
| 1 | Legal or risk hold recorded in the order | Fraud hold, compliance block, payment dispute | Stop writing, report the hold, do not attempt a workaround |
| 2 | Explicit customer instruction confirmed in the record | Cancel, address change, item change | Reread and apply its own change on top only if still valid |
| 3 | Named human operator edit | Delivery date override, manual price adjustment | Treat the edit as current state; ask before changing it |
| 4 | Policy-driven automation by another agent | Address normalization, upsell, reallocation | Yield to ranks 1 to 3 and log the deferral with the version it saw |
| 5 | Ties within the same rank | Two automations writing the same field | First committed write stands; the second rereads and decides again |
The concurrency conflict and authority resolution task record
Each task in the set is one record with ten fields. The validation column is what a reviewer or script checks before the task is accepted into the set; a task that fails any check is fixed or dropped.
At a glance
The concurrency conflict and authority resolution task record
| Field | What it holds | Validation check |
|---|---|---|
| task_id, provenance and pair | Stable ID, pattern from the conflict table, real or illustrative, and the ID of the paired counterexample where both edits should stand | Pattern is one of the seven; real cases cite a source record ID; the pair differs only in the conflicting field or precondition |
| seed_state | Full order snapshot with version token, status, lines, holds and linked inventory | Loads into the environment and reproduces the same version token on reset |
| agents_scopes_and_instructions | Each agent's role, tools, the fields it may write, and the exact instruction text it receives with a timestamp | No tool writes outside listed fields; instructions conflict on at least one field or precondition, except in counterexamples |
| interleaving_schedule | The forced order of reads, writes and pauses across agents | Both agents read the same version before the first write commits |
| authority_rule | Precedence rank of each instruction and the tie rule | Exactly one instruction ranks highest, or the tie rule is stated |
| expected_end_state | Field values, status and linked records after a correct run | Derived from the authority rule alone, not from a model run |
| required_detections | Signals the losing agent must observe or produce: rejected write, reread, escalation | At least one detection per conflicting write |
| forbidden_outcomes | States that fail the task regardless of messages: silent overwrite, released hold, double change | Each is checkable against the stored order |
| expected_accounts | What each agent may truthfully tell its user or caller | Matches the expected end state, including the change that lost |
| evidence_refs | Audit log rows, version history and approvals that prove the outcome | Every pass or fail can be justified by a cited row |
How do you force the interleaving instead of hoping for it?
Control the clock. Conflicts that depend on timing almost never happen on their own in a small test run, so the harness must decide when each agent reads, thinks and writes. Put a barrier after both agents have read the order and release the writes in the order the schedule names.
Make the environment expose versions. Each read returns a version token and each write either carries it or does not. That lets you grade whether an agent used a precondition the way RFC 9110 describes for If-Match, and it lets the environment reject stale writes the way a careful production system would.
Pin down what each agent knew. In frameworks with handoffs, what the second agent sees is configurable: the OpenAI Agents SDK represents handoffs as tools and passes the prior conversation history unless an input filter changes it. In A2A, a contextId groups related tasks and messages, and agents must reject messages whose contextId and taskId do not match, while each agent may remain opaque to the others. Record the exact payload and identifiers each agent received so that a failure can be traced to stale knowledge rather than guessed at.
Run each schedule in both orders and repeat it. A task where agent A writes first and one where agent B writes first are two tasks with two expected end states. Repeat each several times, because model sampling can change whether an agent rereads, and report the share of runs that pass rather than a single run.
Finally, reset the environment to the exact seed between runs. A leftover hold, a consumed one-time action or a stale cache from the previous run will turn a conflict task into a different task without anyone noticing, so check that the version token after reset matches the one recorded in seed_state before each run starts.
Worked case (illustrative): a fraud hold meets an address change
This case is illustrative and does not describe any real company. The starting state is one pending order at version 12 with two lines, no holds and a ship-to address in one city. The service agent is told the customer wants delivery to a new address. The fraud agent is told that the payment matched a pattern that requires a hold and that changes of delivery address on held orders need human review.
Schedule: both agents read version 12. The fraud agent writes the hold first, creating version 13. The service agent then sends its address change carrying version 12. The authority rule ranks the risk hold first and the customer instruction second.
Decision and verified outcome in a correct run: the environment rejects the stale address write. The service agent rereads, sees the hold, does not retry the address change, and tells the customer the request is pending review. The stored order shows version 13 with the hold and the original address, and the audit log shows one rejected write with the stale version token.
Failing run: the service agent's tool does not send a version, so the address write lands as version 14 on a held order and the agent reports success. The fraud agent's hold is still present, so a status check passes, but the forbidden outcome of an address change on a held order is visible in the stored state. Counterexample: in the paired task, the fraud agent only adds a review note without a hold. Both changes should stand, and an agent that refuses the address change fails.
Scoring with three gates: end state, detection and account
Score each run on three gates, and count a pass only when all three hold. Keeping them separate shows whether a system fails because of infrastructure, agent reasoning or reporting.
At a glance
Scoring with three gates: end state, detection and account
Keep the gates' evidence apart as well. The end-state gate reads the database, the detection gate reads the environment's log of rejected writes and rereads, and the account gate reads the agent's messages and any status fields it set. A grader that infers detection from the agent's own words will reward agents that claim to have checked, so the detection evidence must come from the environment, never from the transcript.
- End-state gate: the stored order equals expected_end_state, and no forbidden outcome is present. This gate is graded from the database and audit log only.
- Detection gate: every conflicting write produced a required detection, such as a rejected precondition, a reread before the next write or an escalation to a person.
- Account gate: each agent's final message and any status it set match the stored state, including stating that its own change did not apply when it lost.
- Counterexample gate, scored on the paired task: both compatible changes stand, so caution alone cannot pass the set.
- Report results per pattern and per write order, with the number of repeated runs, so a high overall pass rate cannot hide one pattern that always fails.
Where conflict tasks mislead, and how to keep them honest
Research on multi-agent failures points in the same direction as these tasks. The MAST study describes 14 failure modes in three categories, including inter-agent misalignment and task verification, and argues that many failures come from how agents are organized and coordinated rather than from individual model limits. A conflict set is a targeted probe of exactly those two categories.
The tasks have limits worth stating on the page where results are published. A forced schedule is a model of concurrency, not concurrency itself, so a system that passes may still fail under real load with retries and network delay. Architectures with a single writer per order make several patterns impossible by design, which is a valid result rather than a gap in the test. And if the environment does not expose versions, the detection gate measures only rereads, so say so.
Keep the authority rules under review with the operations owner. Precedence that looks obvious to an engineer can be wrong for a business; some companies let a confirmed customer cancellation override a fraud review, for instance. The rule belongs to the business, and the task set should record who approved it and when.
Finding real two-writer cases in order histories
Real cases make the strongest tasks, and order systems already record them. Field-level history in an order management system or ERP shows two different users or integrations writing the same order within minutes, rejected saves, holds placed and released, and the manual fixes that followed. Each of those clusters is a candidate seed state with a known outcome.
For order histories like these, SourceX can carry a company from sourcing and rights review through contracting, preparation, delivery and payment of a licensed dataset. Buyers get nothing off a shelf: each dataset is sourced from U.S. businesses after a request, and the request form is where an evaluation team lists the conflict patterns and order systems it needs, along with volume, language, format, timing and licensing terms.
Frequently asked questions
What is a lost update in a multi-agent system?
A lost update happens when two agents read the same version of a record and each writes a change based on it, so the second write erases the first. Both agents usually report success. It is caught by comparing the stored record with the expected end state, or prevented by writes that carry a version precondition.
Should the first agent to write always win?
No. First-committed-wins is a sensible tie rule between equal instructions, but authority should come from a written precedence such as risk holds above customer instructions above automation. The task set grades against that rule, so a write that landed first can still be the wrong outcome.
How many runs does each conflict task need?
Run every task in each write order and repeat each schedule several times, because sampling changes whether an agent rereads before writing. Report the share of passing runs per pattern rather than one result. Increase repeats for patterns where results vary between runs.
Can simulated concurrency stand in for production load?
Only partly. A forced schedule reliably reproduces a specific race, which is what an evaluation needs, but it does not reproduce retries, timeouts and network delays under load. Treat passing results as necessary evidence and pair them with production monitoring of rejected writes and reversals.
Why include tasks where the edits do not conflict?
Without them, an agent that refuses every change when another agent is active would pass. Paired counterexamples, where both edits are compatible and should both stand, make caution alone insufficient and show whether the agent actually compared the changes.
Sources
- RFC 9110 says If-Match is most often used with state-changing methods to prevent accidental overwrites when multiple user agents act in parallel on the same resource, and a false condition can return 412 Precondition Failed. Source
- tau-bench sets the reward to zero when the database changes after a run do not match the ground-truth end state. Source
- The tau-bench retail policy allows modifying only pending orders, requires a status check first, and lets the modify-items action run once because it changes the order status. Source
- MCP tool annotations such as destructiveHint and idempotentHint are hints, and clients should never make tool use decisions based on annotations from untrusted servers. Source
- The OpenAI Agents SDK lets tools require approval, surfaces pending approvals as interruptions and resumes runs after a decision. Source
- OpenAI Agents SDK handoffs are represented as tools, and the receiving agent sees the previous conversation history unless an input filter changes it. Source
- The A2A specification uses contextId to group related tasks and messages and requires agents to reject messages with a mismatching contextId and taskId. Source
- The MAST study of multi-agent LLM systems describes 14 failure modes in three categories, including inter-agent misalignment and task verification. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.