AI data market
Enterprise RL environment acceptance tests: what to run before you buy
By SourceX Editorial · Updated
9 sections
Short answer
An enterprise RL environment should pass twelve acceptance tests before purchase: a state schema that covers what the grader reads, seeded resets that reproduce the initial state, reproducible or declared randomness, isolated episodes, a documented action surface, labeled episode endings, a reference solution that passes, alternative paths that pass, a do-nothing agent that fails, shortcut probes that fail, untouched-record invariants and correctly graded infeasible tasks.
Grader rubric
Enterprise RL environment
Key takeaways
- Accept an RL environment only after twelve scripted tests: six on the environment and six on its tasks and graders.
- Seeded resets must reproduce the full state, not just the first observation, and every random field must be declared.
- A do-nothing agent should pass only on tasks labeled as correct-to-refuse; anything else means the grader checks too little.
- Write shortcut probes from the grader code: the cheapest state change that makes each check true must fail.
- A passing matrix applies to one build, grader version and seed-data version; rerun it after any change.
Which acceptance tests should an enterprise RL environment pass before purchase?
Run twelve tests in two groups before you accept an enterprise RL environment: six that check the environment itself (state, reset, randomness, isolation, actions, endings) and six that check the tasks and graders (solvability, alternative paths, null agents, shortcuts, collateral damage, infeasible tasks). An environment that fails any of the first six cannot produce trustworthy training signal, and one that fails any of the second six rewards the wrong behavior.
The tests are deliberately boring. None of them measures how realistic the business workflow feels or how hard the tasks are; those are judgment calls for your domain reviewers. Acceptance tests answer a narrower question that comes first: does the environment behave the same way every time, and does its reward go up only when the agent actually completes the task.
Public agent benchmarks show why both groups matter. The SWE-bench maintainers record, for every task, the tests a correct patch must turn from failing to passing and the tests that must keep passing. OpenAI later had professional developers screen SWE-bench samples because some unit tests were too narrow and some issue descriptions too vague, which is the kind of defect a grader test catches before training rather than after.
The matrix below is written for a buyer: an AI lab, an agent team or an enterprise evaluating a vendor-built environment for a workflow such as accounts payable exceptions, support refunds or order changes. Each test has a run procedure, a pass example and a fail example, so a reviewer can mark it without interpretation.
- Environment group: state schema coverage, seeded reset, transition reproducibility, episode isolation, action surface, episode ending labels.
- Task and grader group: reference solvability, alternative-path acceptance, null-agent baseline, shortcut probes, collateral invariants, infeasible-task grading.
What has to be defined before any test can run?
Four things must be written down before testing starts: the initial state, the allowed actions, the reset behavior and the observable success criteria. If the seller cannot hand you these as files, the environment is not ready for acceptance, and running tests against undocumented behavior only measures your guesses.
Mature open environments show what written definitions look like. Gymnasium documents that reset with an integer seed reinitializes the environment's random number generator, and that step returns separate terminated and truncated flags. OSWorld stores each task as a record with a snapshot id for the starting machine, a config list of setup steps that produce the initial state, and an evaluator that holds the checking logic. tau2-bench documents exactly which reward components gate the final score and how the target end state is derived.
For an enterprise environment built on a business system such as NetSuite, Zendesk or ServiceTitan clones, ask for the same artifacts in plain files. You will use every one of them in the matrix.
- State schema: every table, object and field in the environment, with types, and a flag on each field the grader reads.
- Initial state recipe: the seed data, fixtures and setup steps that build the starting state for each task, versioned.
- Action catalog: every tool or API the agent may call, its arguments, its side effects and whether it is read-only.
- Reset contract: what reset restores, what it does not, and how the seed controls randomness.
- Success criteria: for each task, the end-state checks and any required communication, written so a script can evaluate them.
- Reference solutions: at least one recorded trajectory per task that the seller claims passes.
- Replay log format: the per-step record of observation, action, result and state hash that you will audit.
Acceptance matrix, part 1: state, reset and action tests
Run the environment tests first because every later result depends on them. If a seeded reset does not reproduce the initial state, a passing reference solution on one run says nothing about the next run.
Gymnasium ships a checker that performs two of these tests automatically for its own API: it resets twice with the same seed and asserts the observations match, then steps with the same seed and action and asserts observations, rewards, termination flags and info match. Use the same logic on an enterprise environment, comparing a hash of the full state rather than only the observation the agent sees.
Test 1 is the one buyers most often skip. If the grader reads a field the schema does not document, the agent is being scored on behavior no one can describe, and any fix the seller makes later changes the task silently. Tests 5 and 6 protect the training loop: an undocumented action is a path around the task, and an unlabeled timeout teaches a policy that running out of time is the same as failing.
| Test | How to run it | Pass example | Fail example |
|---|---|---|---|
| 1. State schema covers what the grader reads | Instrument the grader and list every field it reads; diff against the documented schema | Grader reads invoice.status and invoice.approved_by; both are in the schema with types | Grader reads an internal flag that is missing from the schema, so no one can tell what the agent must change |
| 2. Seeded reset reproduces the initial state | Reset twice with the same seed per task; compare full state hashes and first observations | Both resets yield the same state hash and the same open ticket queue | Second reset reorders the queue or keeps a vendor record created in the previous episode |
| 3. Transitions are reproducible or randomness is declared | Replay one action sequence from the same seed several times; compare state hash after every step | Identical hashes at every step, or a documented random field (for example a generated confirmation number) excluded from the hash | Refund amount differs between replays with no declared source of randomness |
| 4. Episodes are isolated | Run task A, reset, run task B; search B's state for any record or side effect from A | No record, email, log entry or cache from task A is visible in task B | An email sent in task A appears in task B's outbox, so B's grader can pass without any agent action |
| 5. Action surface matches the catalog | Call every documented tool once; attempt undocumented actions such as direct database writes or reading grader files | Documented tools work; undocumented actions are refused and logged | The agent can open the evaluator directory or write to the database without going through a tool |
| 6. Episode endings are labeled | Force a time limit and a true terminal state; inspect the end-of-episode record | Time-limit endings are marked truncated and goal or failure endings are marked terminated | Every ending is recorded the same way, so training cannot tell a timeout from a failed task |
Acceptance matrix, part 2: task and grader tests
The task tests check that the reward is earned. Three of them use simple scripted agents rather than models: a reference agent that replays the seller's solution, a null agent that does nothing or refuses, and a shortcut agent that tries to satisfy the grader without doing the work.
tau2-bench documents a case worth copying into your test plan. In one airline task the correct behavior is to refuse a cancellation, so an agent that only refuses politely receives full reward, by design. That is correct for that task and wrong for most others, which is why the null-agent test checks that do-nothing passes only on tasks explicitly labeled as no-action-correct.
Tests 8 and 11 pull in opposite directions, and both are needed. Outcome-based grading accepts any trajectory that reaches an equivalent end state, which tau2-bench adopts deliberately so correct but different solutions are not penalized. That same tolerance lets a destructive trajectory pass unless the grader also checks the records that must not change, the role SWE-bench gives to its tests that must keep passing after a patch.
| Test | How to run it | Pass example | Fail example |
|---|---|---|---|
| 7. Reference solution passes on a fresh environment | Reset, replay the seller's recorded trajectory, run the grader | Every reference trajectory scores as a pass on a clean reset | A reference trajectory passes only after a manual fix or when run in a specific order |
| 8. Alternative valid path passes | Write a second correct trajectory with different but valid steps, such as looking up the vendor before the invoice | The grader accepts any trajectory that produces an equivalent end state | The grader requires the exact call sequence of the reference solution |
| 9. Null agent fails except on labeled tasks | Run an agent that takes no actions, or only refuses, on every task | Do-nothing passes only on tasks the seller labeled as correct-to-refuse | Do-nothing passes on tasks that require an approval, refund or record update |
| 10. Shortcut probes fail | Try known shortcuts: set the status field directly, delete or edit test files, write the expected string without doing the work | Each shortcut is either blocked by the action surface or scored as a fail | Setting a ticket to resolved passes even though the customer's refund was never issued |
| 11. Collateral invariants hold | List records that must not change for each task and check them after every run | Unrelated vendors, invoices and users are unchanged after a passing run | A run that also deletes an unrelated purchase order still scores as a pass |
| 12. Infeasible tasks are graded correctly | Include tasks that cannot be completed with the given tools or permissions; check the expected answer | The correct outcome is reporting that the task cannot be done, and the grader rewards that | The grader only rewards completion, so infeasible tasks teach the agent to fake progress |
How do you run the acceptance matrix and keep the evidence?
Run the matrix as a script, not a review meeting, and keep a replay log for every run so a second person can verify each verdict. The log is the acceptance evidence; a spreadsheet of pass and fail marks without logs is an opinion.
At a glance
How do you run the acceptance matrix and keep the evidence
Shortcut probes deserve their own budget. DeepMind describes specification gaming as behavior that satisfies the literal specification of an objective without achieving the intended outcome, and an RL environment is where that happens at scale because a policy is optimized directly against the grader. Write probes from the grader code itself: for each check, ask what the cheapest state change is that makes it return true.
- Pin versions: record the environment build, the seed data version, the grader version and the dependency lockfile in the run header.
- Run the environment tests (1 to 6) on every task, not a sample, because they are cheap and a single leaking task contaminates training.
- Run the scripted agents (reference, alternative path, null, shortcut) on every task.
- Log per step: observation, action and arguments, tool result, state hash, reward components and the end-of-episode label.
- Store failing runs with the full state diff so the seller can reproduce them from the seed and action list.
- Re-run the full matrix after every environment, grader or seed-data change; a passing matrix applies only to the version it ran on.
- Have a business process owner review a sample of passing end states to confirm they match how the workflow is actually closed out.
Which environment assumptions change the result of an acceptance test?
The verdict on several tests depends on assumptions the seller should state in writing. If an assumption changes, rerun the affected tests instead of carrying the old verdict forward.
The most common silent change is in the grader's comparison rule. When equivalence is computed over a hash of stored state, any formatting difference in stored values, such as a number written with or without a decimal or a timestamp in a different time zone, can make two correct trajectories disagree on the end state.
| Assumption | Tests it affects | What changes the decision | What to ask for |
|---|---|---|---|
| How end-state equivalence is computed (hash, field checks, assertions) | 2, 3, 8, 11 | Formatting differences such as number types or timestamps break equivalence for correct runs | The comparison function and the list of fields excluded from it |
| Which fields are allowed to be random | 3 | Undeclared randomness makes reproducibility impossible to verify | A list of random fields and the seed that controls each |
| Whether external services are live or mocked | 2, 4, 7 | Live email, payment or web services make resets incomplete and results drift | A statement of every external call and its mock |
| Simulated user behavior | 7, 8, 9 | A model-driven user can vary between runs and change task difficulty | The simulator model, its prompt version and its sampling settings |
| Time and calendar handling | 2, 3, 6 | Due dates computed from the wall clock change between runs | A frozen clock or a documented time source |
| Permission model for the agent's account | 5, 10, 12 | Broader permissions than production let shortcuts succeed | The role and permission set the agent runs under |
How should uncertainty and repeated trials be shown?
Report every grader-dependent result per task and across several trials, because one run of a stochastic agent says little. tau2-bench reports pass^k, the probability that all k independent trials of a task succeed, computed from n trials with c successes as the number of ways to choose k from c divided by the number of ways to choose k from n.
For acceptance, uncertainty matters most in the scripted-agent tests. A reference solution should pass on every trial; if it passes on some trials and not others, the environment or simulator is nondeterministic and test 3 should have caught it. For model-driven probes, report the number of tasks, the number of trials per task and the count of passes, and do not summarize a small suite as a single percentage.
No acceptance numbers are reported on this page. The matrix defines what to measure; the thresholds are a contract decision between buyer and seller, written into the statement of work before testing begins.
A practical way to write the thresholds is per test rather than as one score. Environment tests 1 to 6 and scripted-agent tests 7, 9 and 10 are usually all-or-nothing per task: a task either resets cleanly or it does not. Tests that involve a model or a simulated user need an agreed trial count and an agreed rule for what happens when a task passes on some trials and fails on others, such as quarantining the task until the seller explains the variance.
Worked scenario: an accounts payable exception environment (illustrative)
This scenario is illustrative and does not describe a real vendor or dataset. A seller offers an environment cloned from an accounts payable workflow: an agent receives an invoice that fails a three-way match against a purchase order and a goods receipt, and must either correct a coding error, request a credit memo or route the invoice to a named approver.
Test 2 fails on the first run: a second seeded reset keeps a credit memo request created in the previous episode. The seller fixes the reset to truncate the requests table. Test 9 then fails: a null agent passes 40 of an illustrative 120 tasks because those tasks only check that the invoice is not paid, and doing nothing leaves it unpaid. The seller adds a required end state (routed, corrected or credit requested) to each affected task and labels the few where holding the invoice is genuinely correct.
Test 10 finds one shortcut: the agent can set the invoice status to approved through a generic update tool without any approver. The seller removes status from that tool's writable fields. After the fixes, all twelve tests pass on the current build, and the buyer records the build hash, grader version and seed data version in the acceptance certificate.
Where do licensed business records fit in an RL environment purchase?
Realistic enterprise environments are usually built from real workflow records: closed tickets, invoices, approvals and their outcomes, which define both the initial states and what a correct end state looks like. Before buying, confirm that the seller can show the right to use those records for training and environment building, separately from any privacy review of them; de-identification alone does not create a right to license.
Teams that need workflow records to seed an environment can ask SourceX to find them: it keeps no inventory and instead sources records from U.S. businesses against each request. The request-data form asks for scope, volume, language, format, timing and licensing needs, which is the place to state that the records will be used for environment building. No records move until the supplying company has approved scope and terms and signed.
Frequently asked questions
What is an acceptance test for an RL environment?
It is a scripted check, run before purchase or before training, that confirms the environment resets and transitions reproducibly and that its reward goes up only when the task is actually completed. It differs from a benchmark score, which measures an agent; an acceptance test measures the environment and its grader.
How many tasks should an acceptance run cover?
Run the environment tests and scripted-agent tests on every task, because they are cheap and one leaking or trivially passable task can contaminate training. Sampling is reasonable only for slow model-driven probes, and the sample size and selection rule should be written down.
Why test with an agent that does nothing?
A do-nothing or refuse-only agent reveals tasks whose grader checks only that something bad did not happen. In tau2-bench, polite refusal earns full reward on a task where refusing is correct, so the test compares null-agent passes against tasks labeled as correct-to-refuse.
What is the difference between terminated and truncated?
In the Gymnasium API, terminated means the agent reached a terminal state of the task, such as the goal or a failure, while truncated means a condition outside the task ended the episode, typically a time limit. Training code treats the two differently, so the environment must label them.
Does a passing acceptance matrix mean the environment is realistic?
No. The matrix shows that the environment is reproducible and that its reward cannot be gamed by the probes you ran. Realism, task difficulty and coverage of business exceptions need a separate review by a process owner who knows how the workflow is closed out.
Can an environment built from business records be licensed for RL training?
Only if the record owner grants rights that cover training and environment building. Access to the records and a privacy review are separate questions from licensing rights, and counsel should review the specific terms.
Sources
- Gymnasium documents that reset with an integer seed reinitializes the environment's random number generator, and that step returns separate terminated and truncated flags, where truncated covers conditions outside the task such as a time limit. Source
- Gymnasium's environment checker resets twice with the same seed and asserts the observations match, and steps with the same seed and action and asserts observations, rewards, termination and info match. Source
- tau2-bench scores tasks on outcomes: a reference trajectory is replayed on a fresh environment to derive the target database end state, any trajectory reaching an equivalent end state passes, and a polite refusal earns full reward on a task where refusing is correct. Source
- tau2-bench computes pass^k from n trials with c successes as comb(c, k) divided by comb(n, k), citing the tau-bench paper. Source
- OSWorld stores each task with a snapshot id for the starting environment, a config of setup steps that build the initial state and an evaluator that contains the evaluation logic. Source
- OSWorld maintains a separate list of infeasible tasks across applications such as GIMP, VS Code and Chrome. Source
- WebArena instructs users to reset the environment to its initial state after evaluating its 812 examples by stopping, removing and restarting the site containers. Source
- Each SWE-bench task records FAIL_TO_PASS tests that a correct patch must make pass and PASS_TO_PASS tests that must keep passing. Source
- OpenAI ran a human annotation campaign with professional software developers to screen SWE-bench samples for unit tests that were too narrow and issue descriptions that were underspecified. Source
- DeepMind defines specification gaming as behavior that satisfies the literal specification of an objective without achieving the intended outcome. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.