AI uses for records
RL environment vendor pilot: a two-week plan with acceptance and stop criteria
By SourceX Editorial · Updated
10 sections
Short answer
A two-week RL environment pilot should start from a signed brief that fixes one workflow, a task sample, the initial state, allowed actions, reset behavior and observable success criteria. Week one proves seeded resets, task solvability and grading against end states; week two runs a trivial baseline and repeated agent trials. Written acceptance and stop criteria turn the evidence into a go, narrow or stop decision.
Grader rubric
RL environment vendor pilot
Key takeaways
- Sign a brief before day one that fixes the workflow, task sample, initial state, allowed actions, reset behavior and success criteria.
- Use week one to prove the environment: seeded resets, a reference solution for every task, and a grader audit with do-nothing runs.
- Use week two for agents: a trivial baseline, repeated trials reported as pass^k, edge tasks and an independent replay.
- Write stop criteria in advance so ending a weak pilot is a rule, not a negotiation.
- A two-week pilot shows whether one workflow is built and graded correctly; it does not show coverage or training gains.
What should a two-week enterprise RL environment evaluation pilot include?
It should include a signed brief, a fixed sample of tasks from one business workflow, and four definitions agreed before any agent runs: the initial state, the allowed actions, the reset behavior and the observable success criteria. Across ten working days the buyer reproduces resets, confirms each task is solvable, runs repeated trials, checks the grader against the end state and stops early if any stop criterion trips. The output is an evidence package and a go, narrow or stop decision.
A vendor pilot often goes wrong in a predictable way. The buyer receives a demo, a dashboard of agent scores and a promise that the environment resembles their operations, then has no way to tell whether a low score means a weak agent, an unsolvable task or a grader that reads the wrong thing. Two weeks is enough to settle those questions for one workflow if the work is planned around evidence the buyer can reproduce without the vendor in the room.
This page is the plan. It gives the brief template, the day-by-day schedule, the acceptance and stop criteria, the assumptions that change results, a worked example and the limits of what a short pilot can show. Each check could fill its own guide; here the focus is how to sequence them so the pilot ends in a decision rather than a pile of scores.
The pilot brief: what both sides sign before day one
Write the brief before day one and have the vendor countersign it, because every acceptance criterion depends on definitions that are easy to change mid-pilot. The template below is the original asset of this page, version 1.0, dated 2026-10-18. Fill every row; a blank row is itself a stop signal.
| Brief section | What to fill in | Owner | Illustrative entry |
|---|---|---|---|
| Workflow and task sample | One workflow, the task list, and how tasks were sampled | Business process owner | Supplier invoice exceptions; 30 tasks across 5 exception types |
| Initial state | What exists at reset: records, users, open items, clock time | RL environment engineer | Seeded ledger snapshot with 12 open exceptions, clock fixed to a business day |
| Allowed actions | Tools, arguments and permissions the agent has | RL environment engineer | Read invoice, read PO, place hold, request credit memo, message supplier |
| Reset behavior | How state returns to the start, and what randomness remains | Vendor | Seeded reset; supplier reply wording varies within a documented set |
| Success criteria | Observable end-state checks per task, plus required messages | Business process owner | Hold placed and reason code set; supplier told the reason |
| Acceptance criteria | Thresholds and checks that make the pilot pass | Buyer evaluation lead | See the acceptance list on this page |
| Stop criteria | Conditions that end the pilot early | Buyer evaluation lead | See the stop list on this page |
| Evidence package | Files the vendor hands over and the buyer keeps | Vendor | Environment build, seeds, task files, grader code, replay logs |
| Data and access | What data is used, who may see it, where it runs | Security and legal | Synthetic or licensed records only; buyer-controlled sandbox |
Ten working days, with a gate at the end of each week
Spend the first week proving the environment behaves as declared and the second week measuring agents in it. The order matters: agent scores from an environment that has not passed week one cannot be interpreted.
| Days | Work | Evidence produced | Gate |
|---|---|---|---|
| 1 | Install in the buyer's sandbox; walk the brief line by line with the vendor | Build identifier, signed brief, open-questions list | Environment runs without vendor hands on keyboard |
| 2 | Reset tests: same seed twice, then different seeds | Paired state dumps and their comparison | Same seed gives the same initial state, or randomness is documented |
| 3 | Solvability: replay a reference solution for every task on a fresh reset | Replay logs and end states per task | Every task reaches its success state at least once |
| 4 | Grader audit: feed known-good, known-bad and do-nothing runs to the grader | Grader verdicts against expected verdicts | No do-nothing or wrong run passes |
| 5 | Week-one review | Short written findings | Go to week two, narrow scope, or stop |
| 6-7 | Baseline agents: a trivial agent and the buyer's current agent, repeated trials | Per-task results over several trials | Trivial agent score recorded and explained |
| 8 | Exceptions and edge tasks from the business owner's list | Results and transcripts for edge tasks | Edge tasks are graded on end state, not wording |
| 9 | Independent replay: a second engineer reruns a sample from the evidence package | Matching or explained-different results | Results reproduce without the vendor |
| 10 | Decision memo | Go, narrow or stop, with reasons and open risks | Signed by the process owner and environment engineer |
Acceptance criteria that decide whether the environment passes
Accept the environment only if every item below holds, each with a file in the evidence package that a person who was not in the pilot can check. The items follow how mature open environments define these behaviors, so a vendor that has built carefully should find them familiar.
- Seeded reset reproduces the initial state. Gymnasium's reset takes a seed that initializes the environment's random generator, and its environment checker flags an environment as non-deterministic when the same seed produces different observations. Ask for the same property, or for written documentation of what varies.
- Episode endings are reported honestly. Gymnasium separates terminated, meaning the task reached a terminal state, from truncated, meaning a limit outside the task such as a time limit ended it. A pilot should see both and never count a timeout as a completed task.
- Every task is solvable on a fresh reset. A reference solution replayed from the initial state reaches the success state; tau2-bench, for example, replays a reference action list on a fresh environment to derive the target database end state.
- The grader reads the end state. tau2-bench compares the predicted environment's database end state to the target by hash and gates the reward on required communication as well, so any trajectory that reaches an equivalent end state passes. Prefer that over a grader that judges the transcript alone.
- A trivial agent does not score well. Run a do-nothing or minimal agent; if it passes a meaningful share of tasks, the tasks or grader are flawed. The Agentic Benchmark Checklist paper reports this kind of flaw: a published agent benchmark that counted empty responses as successful.
- Results hold over repeated trials. Report pass^k, the chance that all of k independent trials succeed, which tau2-bench computes from the trial count and success count, alongside the single-trial rate.
- The replay log is complete enough for a second engineer to rerun a sample and get the same verdicts.
Stop criteria: reasons to end the pilot before day ten
Stop early when continuing would only produce numbers no one can interpret. Each criterion below is cheap to observe and should be in the signed brief, so stopping is a rule being applied rather than a negotiation.
- The environment cannot be run in the buyer's sandbox without vendor staff operating it by day two.
- The same seed gives different initial states and the vendor cannot document what varies or why.
- Any task in the sample cannot be solved from a fresh reset by its own reference solution, and more than a small number are affected.
- A do-nothing run passes the grader on any task.
- The grader or task files are withheld, so verdicts cannot be checked.
- Production credentials, live customer data or real external side effects appear anywhere in the environment.
- The business process owner judges that the success criteria do not match how the work is actually closed in their operation.
Assumptions to pin before comparing two vendors' results
Results from two vendors, or from one vendor in two weeks, are comparable only when these assumptions are pinned and written into the evidence package. Change any one of them and an agent's score can move without the agent changing at all.
| Assumption | Why it moves results | How to pin it |
|---|---|---|
| Seeds and randomness | Different initial states make tasks easier or harder | Fixed seed list per trial, recorded per episode |
| Initial snapshot | OSWorld ties each task to a snapshot and setup config that define the initial state | Snapshot identifier and setup script version per task |
| Reward basis | Adding or removing a check, such as required messages, changes what counts as success | Grader version and the list of checks that gate the reward |
| Clock and dates | Due dates, aging buckets and deadlines depend on the current date | Frozen clock or recorded start time |
| Shared state between runs | Leftover changes from earlier runs alter later tasks; WebArena tells users to reset its sites after evaluating its examples | Full reset between trials, verified by a state dump |
| User simulator | A simulated requester that phrases things differently changes difficulty | Simulator model, prompt and version recorded |
Worked example (illustrative): an accounts payable exception environment
This example is constructed for illustration; the counts are illustrative and do not describe any real vendor or buyer. An AI team pilots a vendor's accounts payable exception environment built around three-way matching of purchase orders, receipts and supplier invoices.
Week one: the environment installs on day one. On day two, ten paired resets with the same seed match on every record except supplier reply wording, which the vendor documents as drawn from a fixed set of templates. On day three, 28 of 30 tasks reach their success state from the reference solution; two fail because the reference solution relies on a credit memo that the seeded state never creates. The vendor fixes the seed data and both pass on rerun. On day four, a do-nothing run fails every task, but a run that places a hold on every invoice passes 4 tasks, because the grader checked for a hold and not for the reason code. The grader is corrected and those 4 tasks are rerun.
Week two: across 5 trials per task, the buyer's agent shows a much lower pass^5 than its single-trial rate on price-variance exceptions, which is the useful finding of the pilot. A second engineer reruns 6 tasks from the evidence package on day nine and gets matching verdicts. The decision memo recommends going ahead, narrowed to the three exception types where the success criteria matched the process owner's practice, with the other two returned to the vendor for redesign.
When documented randomness is acceptable instead of a deterministic reset
A deterministic reset is the default expectation, but some realistic environments need randomness: a simulated supplier that replies differently, network delays, or a user simulator built on a language model. The exception is acceptable when the vendor documents every random component, exposes its seed or its fixed set of possible values, and shows that the success check does not depend on which value was drawn.
In that case, replace the same-state test with a same-distribution test. Run enough trials to see the spread, report pass^k rather than a single rate, and record the drawn values per episode so a disputed result can be replayed exactly. If the randomness cannot be recorded, the environment fails the replay criterion, however realistic it is.
The exception does not cover hidden randomness in the grader. A grader that gives different verdicts for the same end state, for instance a language-model judge without fixed settings, is a stop criterion, not a documented-randomness case.
What two weeks cannot tell you, and who signs the decision
Two weeks shows whether an environment behaves as declared and whether a handful of tasks are graded fairly. It cannot show coverage of a whole workflow, long-run stability across vendor releases, or whether training in the environment improves an agent on real work. Say so in the decision memo, and list what a longer engagement would need to measure.
The pilot also inherits the limits of its sample. Thirty tasks from five exception types say little about the sixth type nobody sampled. The process owner should state which real cases the sample leaves out, and the memo should carry that list forward.
Two people sign the decision: an RL environment engineer, who vouches for reset, solvability, grading and replay, and the business process owner, who vouches that the success criteria match how the work is really closed. Request specialist review before signing when the environment touches regulated records, models money movement or approvals with real-world effect, or when the two signers disagree about what success means.
Grounding pilot tasks in real workflow records
The weakest part of many enterprise environments is the seed state: invented invoices and tickets that are too clean to contain the exceptions agents struggle with. A pilot built on realistic histories tests the environment against the cases that matter.
SourceX holds no pilot environments or record sets in stock. When an AI team asks, it sources workflow records from U.S. businesses, and the request-data form captures the workflow, record types, volume, format, timing and licensing terms a pilot's seed state needs. The supplying business approves the scope and terms and signs the agreement before any invoice, ticket or log reaches the pilot.
Frequently asked questions
How long should an RL environment vendor pilot run?
Two weeks is enough for one workflow if the brief is signed before day one: one week to prove reset, solvability and grading, and one week to measure agents. Broader coverage or training effects need a longer engagement with its own success criteria.
How many tasks should a two-week pilot use?
Enough to cover each exception type the business owner cares about, with several trials per task, while leaving time for reference replays and a grader audit. A few dozen well-specified tasks usually teach more than hundreds that nobody can audit inside the window.
What should the vendor hand over at the end of the pilot?
The environment build, the seeds, the task and initial-state files, the grader code and version, and complete replay logs. Without these a second engineer cannot reproduce the verdicts, and the pilot result rests on trust.
What is pass^k and why does it matter for a pilot?
pass^k is the probability that all k independent trials of a task succeed. It exposes agents and environments that succeed sometimes but not reliably, which a single-trial success rate hides.
Can a pilot use real company data?
Only data the buyer has the right to use for this purpose, in a sandbox the buyer controls, with no production credentials or real side effects. Synthetic or properly licensed records are the usual choice, and security and legal should approve the data row of the brief.
Sources
- Gymnasium's reset accepts a seed that initializes the environment's random number generator, and step distinguishes terminated (a terminal state of the task) from truncated (a limit outside the task, typically a time limit). Source
- Gymnasium's environment checker reports an environment as non-deterministic when resetting with the same seed produces observations that are not equivalent. Source
- tau2-bench replays a reference action list on a fresh environment to derive the target database end state, compares end states by hash, and by default gates reward on both the database state and required communication. Source
- tau2-bench computes the pass^k metric from the number of trials and the number of successful trials. Source
- OSWorld task examples specify an environment snapshot, setup configuration for the initial state, and an evaluator for each task. Source
- WebArena's environment instructions tell users to reset the sites to their initial state after evaluating the benchmark's examples. Source
- The Agentic Benchmark Checklist paper frames rigorous agent evaluation as task validity plus outcome validity and reports that one published agent benchmark counted empty responses as successful. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.