AI uses for records
RL environment exception coverage: when the customer changes their mind
By SourceX Editorial · Updated
10 sections
Short answer
Test it by treating each change of mind as a scheduled branch with a fixed trigger point, then checking that the environment's state, rules and reward still agree. Map every branch by when the change arrives relative to irreversible writes, give each one an expected end state built from the customer's final intent, and confirm that resets are deterministic so any failure can be replayed.
Grader rubric
RL environment exception
Key takeaways
- A change of mind is a branch with a trigger point; write both into the task, not just the new request.
- The reward must follow the customer's final confirmed intent, or the environment punishes correct agents.
- Locks belong in the tools as well as the policy, so a forbidden late change can actually be detected.
- Deterministic resets and replay logs come before any agent score is trusted.
- Generated sequences of customer moves find branches that hand-written maps miss.
How do you test an RL environment when a customer changes their mind?
You turn the change of mind into a set of explicit branches and test the environment against each one before any agent is trained on it. A branch fixes three things: what the customer first asks for, what they switch to, and the exact point in the workflow where they switch. For every branch the environment must reach a known starting state on reset, accept only the actions the business allows at that point, and grade the episode against the customer's final intent rather than the first request.
Most RL gyms built from business workflows are validated on the happy path. A task is written, a reference solution is replayed, the reward fires, and the environment ships. The first time a simulated customer says actually, make it the blue one after the order was already modified, three things can go wrong at once: the tool lets the agent do something the business forbids, the reward still expects the original request, or the reset leaves state from the previous attempt.
This page gives an exception coverage map for mind changes and the acceptance checks that make it trustworthy. It is about testing the environment itself; choosing which records can become an environment is a separate question.
Why do mind changes break environments that pass happy-path tests?
They break the assumption that the task's goal is fixed at the start. In tau-bench, the reward function reloads the data, replays the task's reference actions to produce a ground-truth state, hashes it and compares that hash with the state the agent left. That works only if the reference actions encode the customer's final intent. If a task author writes the gold actions for the first request and the simulated user later switches, the environment will punish a correct agent.
Mind changes are not an edge case invented for testing. The tau-bench retail test tasks include user instructions such as changing their mind when the agent asks for confirmation, or first naming one payment method and switching to another if the agent asks them to confirm it. Those instructions are conditional on the agent's behavior, which means the same task can follow different branches on different runs.
Business rules then decide which switches are still possible. The tau2-bench retail policy allows item modification on a pending order only once; after it, the order status becomes pending (items modified) and the agent can no longer modify or cancel the order. A late change of mind on that order has exactly one correct outcome: the agent explains it cannot make the change, or transfers when the request is outside its scope.
When the change arrives matters more than what changes
Classify every branch by its trigger point, because the trigger point decides which actions remain legal. The same request, swap the color, is trivial before any write, routine at the confirmation prompt and forbidden after a one-shot modification. Write the trigger point into the task, not just the new request.
| Trigger point | What is still possible | Typical correct agent behavior | Environment risk to test |
|---|---|---|---|
| Before any lookup or write | Everything the policy allows for the new request | Drop the first request and serve the new one | Reward still keyed to the first request |
| At the confirmation prompt | Any change, because nothing has been written yet | Restate the revised action details and ask again | Tool accepts a write without a fresh confirmation |
| After a reversible write | Undo or amend, if a tool and rule allow it | Amend through the permitted tool and confirm again | Undo tool missing, or leaves residue in state |
| After a one-shot or locking write | Only what the policy permits on the locked record | Explain the limit, refuse or transfer | Tool still allows the forbidden second change |
| After hand-off to a third party | Usually a new process such as a return or exchange | Switch to the follow-on workflow the policy names | Environment lacks the follow-on state transition |
| Oscillation: back to the original | Depends on what was written in between | Track the latest confirmed intent, not the first or second | Reward or state double-counts the reverted change |
The exception coverage map with branching scenarios
The map is one row per branch for each base task. A row is complete when a reviewer can read it and predict the final state without running anything. Below is an illustrative map for one retail base task: a customer with a pending two-item order who wants to change one item's option. Copy the columns; replace the rows with your own workflow's states and rules.
Build the rows in a fixed order. Start from the base task and list every state transition its tools can make. For each transition, ask what the customer could change before it, at it and after it, and which rule governs each case. Drop combinations the policy makes identical, keep the ones where the correct outcome differs, and add one oscillation row for any field a customer can set twice.
Then run the scripted solution for every row before a single agent sees the environment. In this illustration, suppose the scripted solution for B3 fails because the modify tool happily accepts a second call: the environment, not the agent, is broken, and the fix is to make the tool enforce the lock or to add a grader rule that flags the second write. Only when every row's scripted solution reaches its expected end state does the map become an acceptance test.
| Branch | Change and trigger point | Expected end state | Oracle | Rule source |
|---|---|---|---|---|
| B0 base | No change; modify item 1 option | Order pending (items modified), item 1 new option | State hash equals replay of final-intent actions | Modify items rule |
| B1 add item | At confirmation, also modify item 2 | One modification call covering both items | Exactly one modify call; both items changed | Modify only once; collect all items first |
| B2 cancel instead | Before confirmation, cancel the order | Order canceled, refund recorded | Status canceled; no modify call in trace | Cancel only if pending |
| B3 late add | After modification, asks to change item 2 | Unchanged from B0 | No second write; refusal or transfer message | Locked after items modified |
| B4 late cancel | After modification, asks to cancel | Unchanged from B0 | No cancel write; reason stated | Cannot cancel after items modified |
| B5 payment switch | At payment step, names one method, then another | Order uses the second method | Payment field equals final choice | Payment method rules |
| B6 revert | At confirmation, returns to original option | Order unchanged | State hash equals initial state | Confirmation before any write |
| B7 ambiguous | Says maybe the other one and stops | Unchanged until intent is clear | No write without explicit yes | Explicit confirmation rule |
Acceptance checks the environment must pass first
A coverage map is only as good as the environment under it. Before running any agent, an RL environment engineer should check the following and record the evidence in the task file. Gymnasium's reset documentation is a useful anchor: a seed passed to reset initializes the random generator, while a later reset with no seed does not re-seed it, and its environment checker tests that two resets with the same seed produce equal observations.
- State schema: every field a branch reads or writes is named, typed and present at reset, including status values such as locked or modified.
- Deterministic reset: the same seed gives the same initial records and the same simulated-user behavior, or the randomness is documented and logged per episode.
- Solvability per branch: a scripted solution reaches each row's expected end state, and for refusal branches the scripted solution makes no write.
- Allowed actions per state: tools reject what the policy forbids in that state, or the grader flags it; never rely on the agent alone to respect a lock.
- Completion invariants: terminated is reached only by a real end condition; time or turn limits produce a truncated episode, which Gymnasium keeps separate from termination.
- Replay log: each episode stores seed, initial snapshot, user instruction, tool calls and final state so a second engineer can reproduce a failure.
Writing simulated customers who change their minds on cue
Make the change conditional on an observable event in the conversation, not on a turn count. The tau-bench tasks show the pattern: if the agent asks for confirmation, the user changes their mind. Keyed to an event, the branch fires at the same workflow point on every run even when the agent takes a different number of turns to get there.
Keep the simulator honest about what it knows. The tau2-bench simulation guidelines tell the simulated user never to invent information missing from the scenario, to disclose information progressively and to end with a stop, transfer or out-of-scope token. Each branch should say which token ends it; a late-cancel branch that should end in transfer is mis-graded if the simulator stops instead.
Some changes of mind come with an action rather than a message. The tau2-bench paper describes a dual-control setting in which the simulated user has tools and can change the shared environment, not only talk to the agent. If your workflow lets customers edit a cart or cancel from a portal, include branches where the user acts and the agent has to notice.
Finding the branches nobody wrote down
Hand-written maps cover the changes people expect. To find the rest, drive the environment with generated sequences of user moves and check invariants after each step. Hypothesis's rule-based state machines are built for this: you declare rules (here, customer moves such as swap, add, cancel or revert), preconditions that filter rules to states where they apply, and invariants that run after every step.
When an invariant fails, Hypothesis prints a short program of steps that reproduces it, which becomes a new row in the coverage map. Typical invariants for this map are that a locked order never changes, that every write is preceded by a confirmation event and that the reward's target always matches the latest confirmed intent. This tests the environment, not an agent, so it can run in continuous integration whenever tools or rules change.
Five settings that quietly flip an acceptance result
Five assumptions flip results most often. First, what counts as confirmation: if the grader treats any affirmative word as a yes, branch B7 passes for the wrong reason. Second, the hash scope: if the state hash includes timestamps or generated identifiers, correct runs on different branches fail. Third, simulator model drift: a new simulator model may ignore a conditional instruction, so the branch never fires. Fourth, clock behavior: if an order can ship during the episode, late-change branches move to a different trigger point. Fifth, one-shot enforcement: if the lock lives only in the policy text and not in the tool, the environment cannot distinguish an agent that respected it from one that was never tempted.
Record each assumption as a versioned setting in the task file. When any of them changes, rerun the scripted solutions for every branch before trusting new agent scores.
Limits, and when to bring in the process owner
A coverage map built from published policies covers the rules someone wrote down. Real operations also carry discretionary practice: a supervisor who will reopen a locked order for a valued account, or a cancellation window that depends on the warehouse. When the correct outcome of a branch depends on that kind of judgment, stop and get the business process owner to state the rule, or mark the branch out of scope rather than guessing.
The map also cannot show how often each branch happens. Weighting branches in training needs frequencies from real order amendment and cancellation histories; without them, label the mix as designed, not representative.
Real amendment histories as the source of branches
Order systems, booking tools and service desks keep the evidence of customers changing their minds: amendment logs, cancellations after confirmation, reopened tickets and exchanges that followed a modification. Those records give both the branch types and their frequencies.
Teams can commission amendment and cancellation histories through SourceX's request form, giving scope, volume, format, timing and licensing needs; SourceX then searches U.S. businesses for matching records, as it does not keep datasets in stock.
Frequently asked questions
What is exception coverage in an RL environment?
It is the share of realistic deviations from the happy path that the environment can represent, enforce and grade correctly. For customer workflows the most common deviations are changes of mind, missing information and requests the policy does not allow.
Should the reward follow the first request or the final one?
The final confirmed intent, as long as the policy still allows it at that point. When the policy forbids the late change, the correct end state is usually the state before the request, plus a refusal or transfer.
What should the exception coverage map include?
One row per branch with the change, the trigger point, the expected end state, the oracle used to check it and the rule that justifies it. Each row should be predictable without running the environment.
Why trigger the change on an event instead of a turn number?
Agents take different numbers of turns to reach the same point in a workflow. Tying the change to an event such as the confirmation prompt makes the branch fire at the same workflow state on every run.
Can generated tests replace a hand-written map?
No. Stateful property-based testing finds sequences people missed, but someone who knows the business must still decide what the correct outcome is for each new branch it finds.
Sources
- tau-bench computes reward by reloading data, replaying the task's reference actions, hashing the resulting state and comparing it with the hash of the state the agent produced. Source
- tau-bench retail test tasks include user instructions to change their mind when the agent asks for confirmation, and to switch payment method if the agent asks them to confirm it. Source
- The tau2-bench retail policy allows item modification only once per pending order, after which the status becomes pending (items modified) and the agent cannot modify or cancel the order; updates need explicit confirmation. Source
- The tau2-bench user simulation guidelines tell the simulated user not to invent information, to disclose information progressively and to end with stop, transfer or out-of-scope tokens. Source
- The tau2-bench paper describes a dual-control setting in which the simulated user has its own tools and can act on the shared environment. Source
- Gymnasium's Env.reset initializes the random generator from a passed seed and does not reset it when called later without a seed; step separates terminated from truncated. Source
- Gymnasium's environment checker tests that resetting with the same seed produces equivalent observations and random generator state. Source
- Hypothesis rule-based state machines combine rules, preconditions and invariants that run after every step, and print a short reproducing program when a check fails. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.