Skip to content

AI uses for records

RL environment irreversible actions: a sandbox ledger with approval gates

By SourceX Editorial · Updated

10 sections

Short answer

To simulate irreversible business actions without moving real money, route every refund, payment, send or delete through a sandbox action ledger: the tool records the intended effect, an approval gate decides whether it proceeds, the effect is applied only to environment state, and the grader checks the ledger and the final state. Block all live endpoints, and model retries and approvals as the real business does.

Grader rubric

RL environment irreversible

This page covers RL environment irreversible actions: a sandbox ledger with approval gates.Illustrative

Key takeaways

  • An irreversible action in an environment should change environment state and the ledger, never a live system, and the grader should read both.
  • Classify every tool by what cannot be undone before you build; vendor or protocol hints are a starting point, not a decision.
  • Approval gates belong in the environment, with the decision returned to the agent as an observation, so the agent learns to wait, explain and accept a no.
  • Retries and idempotency keys are where simulated money most often duplicates; the ledger must deduplicate the way the real payment system does.
  • An environment is not fit until a no-op agent, a duplicate-refund agent and a gate-skipping agent all score zero on it.

How do you simulate irreversible business actions without moving real money?

You make the irreversible action real inside the environment and impossible outside it. Each tool that would refund, pay, send, cancel or delete writes an intent to a sandbox action ledger, passes an approval gate, and then changes only the environment's own state. The grader reads the ledger and the final state, and the network layer blocks every call to a live system.

This matters because an RL environment runs the same task thousands of times, often with an agent that has not yet learned the rules. If a refund tool in that loop can touch a real processor, a single bad policy update can issue thousands of real refunds. If the refund does nothing at all, the agent learns that refunds are free and never learns to check eligibility first.

Benchmark code shows how simple the core can be. tau-bench keeps each environment's records as in-memory data, hashes that data at the end of an episode, replays the task's ground-truth actions on a fresh copy, and gives a reward of zero when the two hashes differ. A refund there is a state change in a dictionary, yet it still decides the score. The ledger described below adds what a business environment needs on top of that: classification, gates, deduplication and an audit trail.

Which tools need a gate, judged by what cannot be undone?

Start by classifying every tool by the cost of reversing its effect in the real business, because that cost decides how the environment simulates it and which gate it passes. A tool that reads a customer record needs no gate; a tool that changes a payee's bank details needs the strictest one, even though it moves no money itself.

The Model Context Protocol gives a useful vocabulary. Its tool annotations include readOnlyHint, destructiveHint, idempotentHint and openWorldHint, and the schema defaults destructiveHint and openWorldHint to true when a server says nothing. The same specification calls these hints, says clients must treat them as untrusted unless the server is trusted, and recommends a human in the loop who can deny tool invocations. Use the hints as a first pass and let the process owner make the final call.

Which tools need a gate, judged by what cannot be undone?
Action classBusiness examplesReal reversalHow the environment simulates itGate
Read onlyLook up order, view invoice, search ticketsNothing to reverseRead from seeded stateNone
Reversible writeAdd internal note, change ticket priority, tag a recordEdit or undo in the same systemWrite to state, log a diffAuto
CompensableCancel an unshipped order, void an authorizationA second action restores the positionWrite to state and record the compensating actionRule check
Money movementIssue refund, capture payment, send a payoutNeeds a new opposite transaction, sometimes impossibleLedger entry against simulated balancesRule check plus human sample
External sendEmail a customer, file with a carrier, notify a supplierCannot be unsentWrite to a simulated outboxRule check plus human sample
Destructive or trust-changingDelete a record, change bank details, grant accessOften impossible or slow and auditedSoft-delete flag or staged change in stateHuman always

In-memory state, vendor sandbox or emulated tool?

Pick the simulation method per tool, not per environment: in-memory state for most business records, a vendor sandbox when you need a real API's error codes, and an emulated tool only when neither is practical. Each choice trades realism against control.

In-memory or database state is fully controllable and resets in milliseconds, but its realism depends on how faithfully you wrote the tool logic. Vendor sandboxes give you the real API surface. Stripe's documentation says that in a sandbox, card networks and payment providers do not process payments and API calls return simulated objects, and real card numbers are refused with test keys. They are not perfect replicas: PayPal's sandbox testing guide for its PayPal Here product warns that some credit card processing rules are not triggered in the sandbox, so some transactions that succeed there may fail in production.

Language-model emulation is the broadest and the least faithful. The ToolEmu paper used a model to emulate tool execution for agent risk testing, and its human evaluation found 68.8 percent of identified failures would be valid real-world failures, which means the rest were artifacts of the emulator. That is acceptable for red teaming and a poor basis for a reward signal an agent will optimize against.

Whatever you choose, vendor sandboxes need extra care in RL: they are shared, rate-limited and not reset between episodes. Wrap them so that each episode uses its own test objects, and record the sandbox object identifiers in the ledger so the reset step can find and discard them.

What does the sandbox action ledger record, field by field?

The ledger is an append-only table the environment writes before, during and after every non-read action. It lets the grader score what the agent tried as well as what changed, and it lets a reviewer replay an episode without rerunning the policy. The ten field groups below are the minimum; the example values are illustrative.

At a glance

What does the sandbox action ledger record, field by field

  • Write the entry before the effect, then update it with the decision and the diff, so a crash mid-action leaves evidence.
  • Never let the agent read the approver's internal rationale before the decision; return only what a real requester would see.
  • Store the ledger outside the agent's reachable file system so it cannot be edited to look compliant.
What does the sandbox action ledger record, field by field?
FieldWhat it recordsIllustrative valueWhy the grader or reviewer needs it
entry_id, episode_id, step_idWhere the action sits in the rune-0007, ep-1182, step 9Joins the entry to the trajectory and to replays
tool and canonical argumentsThe exact call, arguments sorted and normalizedissue_refund(order=O-55, line=2, reason=damaged)Lets two identical calls be recognized as duplicates
action_classClass from the classification tablemoney movementSelects the gate and the scoring rule
idempotency_keyKey supplied by the agent or derived by the toolderived: O-55/line-2/refundPrevents one intent from applying twice
precondition snapshotHash and key fields of the state the action depends onorder status delivered, line 2 not yet refundedShows whether the action was valid when taken
gate level and decisionWhich gate ran and what it returnedL2 rule check: hold for humanScores whether the agent respected the gate
approver and rationaleSimulated policy, sampled human or rule id, with reasonpolicy approver v3: damage photo presentSeparates agent quality from approver quality
simulated effectState diff applied, or none if blockedline 2 refund status set to issuedFeeds the end-state comparison
external side effectAny call that left the sandboxnone (egress blocked)Must be none; any other value fails the episode
compensating actionThe action that would reverse it, if one existsnone; refunds are final in this environmentLets graders weigh severity and test recovery

Where do the approval gates sit, and who answers them?

Gates sit between the agent's tool call and the simulated effect, and their answer goes back to the agent as an ordinary observation. That way the agent learns the business behavior you want: ask, explain, wait, and accept a refusal without trying a different tool to get the same result.

Existing agent frameworks already model this. Inspect lets an approval policy send selected tools to a human approver and everything else to an auto approver, and its approvers can approve, modify, reject, terminate or escalate; a rejected call is not executed and the model is told it was rejected. The OpenAI Agents SDK marks tools with needs_approval, pauses the run with pending interruptions, lets you serialize the paused state and resume it after an approve or reject, and treats callable approval rules as fail-closed when it cannot safely read the arguments.

In training you cannot put a person behind every gate, so most gates are answered by a simulated approver: a versioned policy written from the business's real approval rules, such as which refund reasons need evidence or which changes need a second person. A sampled fraction goes to human reviewers to check that the simulated approver still agrees with them. Record which kind of approver answered, because an agent that only ever met a lenient simulated approver has not learned the real policy.

Where do the approval gates sit, and who answers them?
Gate levelTriggerDecisions it may returnWhat the agent observes
L0 noneRead-only toolsNot applicableThe tool result
L1 autoReversible writes inside policyApprove, reject with reasonThe result or a rejection message
L2 rule checkCompensable actions and money movement within limitsApprove, hold for human, reject, modify argumentsA pending status, then the decision
L3 humanDestructive or trust-changing actions, anything over a policy limitApprove, reject, terminate episodeA pending status, then the decision or episode end

How do you stop retries from duplicating simulated money?

Duplicate effects are the most common way a simulated money tool goes wrong, so the ledger must deduplicate exactly as the real payment system does. Agents retry after timeouts, after ambiguous errors and after their own context loses track of an earlier call.

Stripe's documentation describes the real-world mechanism: a client sends an idempotency key, and in API v1 a retry with the same key returns the saved first response rather than performing the operation again. It also says a server error leaves the outcome indeterminate, so the client should check the object before retrying under a new key. A faithful environment reproduces both behaviors: same key returns the stored result, and a simulated timeout sometimes applies the effect and sometimes does not, so the agent has to look before it acts again.

If the real system has no idempotency keys, derive one from the business intent, for example the order, line and action, and treat a second refund on the same line as a scored error even if the tool accepted it. MCP's idempotentHint is the place to declare which tools are safe to repeat, but the ledger's derived key is what the grader trusts.

What must the environment pass before any training run?

An environment with irreversible actions is ready only when it passes a short set of demonstrations that a separate engineer can rerun. Treat these as release criteria, not nice-to-haves, because a flaw here becomes a reward an agent will find.

At a glance

What must the environment pass before any training run

The audit of agentic benchmarks behind the Agentic Benchmark Checklist shows why: it found that tau-bench counted empty responses as successful on intentionally impossible tasks, so a trivial agent reached 38 percent. For irreversible actions the equivalent failure is an environment where refusing, stalling or doing nothing scores well because the grader only checks that nothing bad happened.

  • State schema: every table the tools read or write is documented, with seeded values and the hash used for end-state comparison.
  • Deterministic reset: two resets with the same seed produce identical state hashes and identical ledger starting points, and vendor sandbox objects from the last episode are discarded.
  • Solvability: every task has a recorded reference trajectory that reaches the goal state through allowed actions and gates.
  • Impossible-task twins: for each task family, at least one variant where the correct behavior is to refuse or escalate, and where a no-op still scores zero because the customer was never answered.
  • Adversarial agents: a no-op agent, a duplicate-refund agent and a gate-skipping agent that calls a lower-level tool to bypass the gate all score zero.
  • Egress proof: a test call to a live endpoint from inside the environment fails, and the ledger records the attempt.
  • Replay: a reviewer can rebuild the final state from the initial seed plus the ledger alone.

How does a damaged-shipment episode play out? (illustrative)

This example is illustrative and simplified. A customer reports that one of three items arrived broken and asks for a refund on that item and a replacement for the order. The environment's policy says a damage refund needs a photo on the ticket, a reship over one unit needs a supervisor, and a customer email must follow any change.

Step 1 to 3: the agent reads the ticket, the order and the shipment status; no ledger entries, no gates. Step 4: it calls issue_refund on line 2 with reason damaged; the ledger writes the entry, the L2 rule check finds the photo and approves, and the line's refund status changes in state. Step 5: a simulated timeout returns an ambiguous error. Step 6: the agent retries with the same derived key; the ledger returns the stored result and applies nothing new.

Step 7: the agent calls create_reship for the whole order, three units. The rule check holds it for a human because it exceeds the one-unit limit; the sampled human reviewer rejects it with the reason that only line 2 was damaged. Step 8: the agent calls create_reship for line 2 only, which the rule check approves. Step 9: it drafts the customer email; the send goes to the simulated outbox after an L2 check that the message matches the actions taken.

The grader compares the end state with the reference: one refund on line 2, one single-unit reship, one outbound message, no duplicate, no external side effect. It also checks the ledger: the retry used the same key, and the agent accepted the rejection rather than routing the full reship through another tool. A version of the agent that issued a second refund at step 6 would have scored zero even though the customer received the right answer.

Which assumptions change the result, and when should you narrow the scope?

Three assumptions decide whether skills learned in this environment hold up: that the simulated approver matches the real approval policy, that the tool logic matches the real system's edge cases, and that retries and failures occur at realistic rates. If any one is wrong, the agent learns a business that does not exist.

Narrow the scope when you cannot write the real approval policy down, for example when approvals depend on unrecorded judgment; build the environment for the actions you can specify and leave the rest as escalate-only. Ask for specialist review when actions touch regulated money movement, consumer notices or account access, because the correct agent behavior there depends on law and contract terms, not only on business habit. Stop entirely if any path from the environment to a live system cannot be closed; no sandbox design compensates for an open egress route.

How do real approval histories calibrate the gates?

The weakest part of most environments is the simulated approver, and the best evidence for it is what real approvers decided. Historical refund approvals, credit-memo sign-offs, change-request reviews and escalation outcomes show which requests people approved, rejected or modified, and why; those records let you write the approver policy and then measure how often it agrees with the people it imitates.

Approval and exception histories of this kind are licensed through SourceX, which handles rights review, contracting, preparation and delivery for the companies that hold them. An AI team describes scope, volume, language, format, timing and licensing needs in the request-data form, and the records are sourced from U.S. businesses for that request rather than taken from stock.

Frequently asked questions

Can an RL environment use a payment provider's sandbox directly?

It can, for the tools where real error codes and response shapes matter, but wrap the sandbox so each episode creates its own test objects and the reset step discards them. Vendor sandboxes are shared and stateful, and some skip production processing rules; PayPal's PayPal Here sandbox guide says so for certain card rules. Keep the ledger as the record the grader trusts.

Why not just remove irreversible tools from the agent's action space?

Because the agent then never learns when those actions are appropriate, which is most of the hard judgment in business tasks. A refund-capable agent needs practice deciding when not to refund. Keep the tools, simulate their effects, and gate them.

Do approval gates slow down RL training too much?

Not if most gates are answered by a versioned simulated approver and only a sample goes to people. The pause itself is cheap when the environment returns a pending status as an observation. The real cost is writing an approver policy faithful enough to train against.

How should the grader treat a correct outcome reached by skipping a gate?

As a failure, or at least a heavily penalized success, depending on the action class. In the business, an unapproved change of bank details is an incident even when the new details happen to be right. Score the ledger as well as the end state so the bypass is visible.

What does the sandbox action ledger need to include?

At minimum: identifiers tying the entry to the episode and step, the exact call, its action class, an idempotency key, a precondition snapshot, the gate decision and approver, the simulated effect, any external side effect and the compensating action. With those, a reviewer can replay an episode and a grader can catch duplicates and bypasses.

Sources

  • tau-bench scores an episode by hashing the environment's data and comparing it with the hash from replaying the ground-truth actions on fresh data, setting reward to zero on a mismatch. Source
  • MCP tool annotations define readOnlyHint, destructiveHint, idempotentHint and openWorldHint as hints, with destructiveHint and openWorldHint defaulting to true. Source
  • The MCP tools specification says a human should be able to deny tool invocations and that clients must treat annotations as untrusted unless they come from trusted servers. Source
  • In a Stripe sandbox, card networks and payment providers do not process payments, API calls return simulated objects, and real card numbers cannot be used with test keys. Source
  • PayPal's sandbox does not trigger some credit card processing rules, so some transactions that succeed in the sandbox may fail in production. Source
  • ToolEmu emulates tool execution with a language model; human evaluation found 68.8% of the failures it identified would be valid real-world agent failures. Source
  • Inspect approval policies route selected tool calls to a human approver and others to an auto approver, with approve, modify, reject, terminate and escalate decisions; rejected calls are not executed. Source
  • The OpenAI Agents SDK pauses runs for tools marked needs_approval, lets paused state be serialized and resumed after approve or reject, and fails closed when approval rules cannot read the arguments. Source
  • Stripe idempotency keys allow safe retries; in API v1 a retry with the same key returns the saved first response, and a server error leaves the outcome indeterminate. Source
  • The Agentic Benchmark Checklist paper found tau-bench counted empty responses as successful on intentionally impossible tasks, letting a trivial agent reach 38%. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies