Skip to content

AI uses for records

RL environment vendor pilot: a two-week plan with acceptance and stop criteria

By SourceX Editorial · Updated

10 sections

Short answer

A two-week RL environment pilot should start from a signed brief that fixes one workflow, a task sample, the initial state, allowed actions, reset behavior and observable success criteria. Week one proves seeded resets, task solvability and grading against end states; week two runs a trivial baseline and repeated agent trials. Written acceptance and stop criteria turn the evidence into a go, narrow or stop decision.

Grader rubric

RL environment vendor pilot

This page covers RL environment vendor pilot: a two-week plan with acceptance and stop criteria.Illustrative

Key takeaways

  • Sign a brief before day one that fixes the workflow, task sample, initial state, allowed actions, reset behavior and success criteria.
  • Use week one to prove the environment: seeded resets, a reference solution for every task, and a grader audit with do-nothing runs.
  • Use week two for agents: a trivial baseline, repeated trials reported as pass^k, edge tasks and an independent replay.
  • Write stop criteria in advance so ending a weak pilot is a rule, not a negotiation.
  • A two-week pilot shows whether one workflow is built and graded correctly; it does not show coverage or training gains.

What should a two-week enterprise RL environment evaluation pilot include?

It should include a signed brief, a fixed sample of tasks from one business workflow, and four definitions agreed before any agent runs: the initial state, the allowed actions, the reset behavior and the observable success criteria. Across ten working days the buyer reproduces resets, confirms each task is solvable, runs repeated trials, checks the grader against the end state and stops early if any stop criterion trips. The output is an evidence package and a go, narrow or stop decision.

A vendor pilot often goes wrong in a predictable way. The buyer receives a demo, a dashboard of agent scores and a promise that the environment resembles their operations, then has no way to tell whether a low score means a weak agent, an unsolvable task or a grader that reads the wrong thing. Two weeks is enough to settle those questions for one workflow if the work is planned around evidence the buyer can reproduce without the vendor in the room.

This page is the plan. It gives the brief template, the day-by-day schedule, the acceptance and stop criteria, the assumptions that change results, a worked example and the limits of what a short pilot can show. Each check could fill its own guide; here the focus is how to sequence them so the pilot ends in a decision rather than a pile of scores.

The pilot brief: what both sides sign before day one

Write the brief before day one and have the vendor countersign it, because every acceptance criterion depends on definitions that are easy to change mid-pilot. The template below is the original asset of this page, version 1.0, dated 2026-10-18. Fill every row; a blank row is itself a stop signal.

The pilot brief: what both sides sign before day one
Brief sectionWhat to fill inOwnerIllustrative entry
Workflow and task sampleOne workflow, the task list, and how tasks were sampledBusiness process ownerSupplier invoice exceptions; 30 tasks across 5 exception types
Initial stateWhat exists at reset: records, users, open items, clock timeRL environment engineerSeeded ledger snapshot with 12 open exceptions, clock fixed to a business day
Allowed actionsTools, arguments and permissions the agent hasRL environment engineerRead invoice, read PO, place hold, request credit memo, message supplier
Reset behaviorHow state returns to the start, and what randomness remainsVendorSeeded reset; supplier reply wording varies within a documented set
Success criteriaObservable end-state checks per task, plus required messagesBusiness process ownerHold placed and reason code set; supplier told the reason
Acceptance criteriaThresholds and checks that make the pilot passBuyer evaluation leadSee the acceptance list on this page
Stop criteriaConditions that end the pilot earlyBuyer evaluation leadSee the stop list on this page
Evidence packageFiles the vendor hands over and the buyer keepsVendorEnvironment build, seeds, task files, grader code, replay logs
Data and accessWhat data is used, who may see it, where it runsSecurity and legalSynthetic or licensed records only; buyer-controlled sandbox

Ten working days, with a gate at the end of each week

Spend the first week proving the environment behaves as declared and the second week measuring agents in it. The order matters: agent scores from an environment that has not passed week one cannot be interpreted.

Ten working days, with a gate at the end of each week
DaysWorkEvidence producedGate
1Install in the buyer's sandbox; walk the brief line by line with the vendorBuild identifier, signed brief, open-questions listEnvironment runs without vendor hands on keyboard
2Reset tests: same seed twice, then different seedsPaired state dumps and their comparisonSame seed gives the same initial state, or randomness is documented
3Solvability: replay a reference solution for every task on a fresh resetReplay logs and end states per taskEvery task reaches its success state at least once
4Grader audit: feed known-good, known-bad and do-nothing runs to the graderGrader verdicts against expected verdictsNo do-nothing or wrong run passes
5Week-one reviewShort written findingsGo to week two, narrow scope, or stop
6-7Baseline agents: a trivial agent and the buyer's current agent, repeated trialsPer-task results over several trialsTrivial agent score recorded and explained
8Exceptions and edge tasks from the business owner's listResults and transcripts for edge tasksEdge tasks are graded on end state, not wording
9Independent replay: a second engineer reruns a sample from the evidence packageMatching or explained-different resultsResults reproduce without the vendor
10Decision memoGo, narrow or stop, with reasons and open risksSigned by the process owner and environment engineer

Acceptance criteria that decide whether the environment passes

Accept the environment only if every item below holds, each with a file in the evidence package that a person who was not in the pilot can check. The items follow how mature open environments define these behaviors, so a vendor that has built carefully should find them familiar.

  • Seeded reset reproduces the initial state. Gymnasium's reset takes a seed that initializes the environment's random generator, and its environment checker flags an environment as non-deterministic when the same seed produces different observations. Ask for the same property, or for written documentation of what varies.
  • Episode endings are reported honestly. Gymnasium separates terminated, meaning the task reached a terminal state, from truncated, meaning a limit outside the task such as a time limit ended it. A pilot should see both and never count a timeout as a completed task.
  • Every task is solvable on a fresh reset. A reference solution replayed from the initial state reaches the success state; tau2-bench, for example, replays a reference action list on a fresh environment to derive the target database end state.
  • The grader reads the end state. tau2-bench compares the predicted environment's database end state to the target by hash and gates the reward on required communication as well, so any trajectory that reaches an equivalent end state passes. Prefer that over a grader that judges the transcript alone.
  • A trivial agent does not score well. Run a do-nothing or minimal agent; if it passes a meaningful share of tasks, the tasks or grader are flawed. The Agentic Benchmark Checklist paper reports this kind of flaw: a published agent benchmark that counted empty responses as successful.
  • Results hold over repeated trials. Report pass^k, the chance that all of k independent trials succeed, which tau2-bench computes from the trial count and success count, alongside the single-trial rate.
  • The replay log is complete enough for a second engineer to rerun a sample and get the same verdicts.

Stop criteria: reasons to end the pilot before day ten

Stop early when continuing would only produce numbers no one can interpret. Each criterion below is cheap to observe and should be in the signed brief, so stopping is a rule being applied rather than a negotiation.

  • The environment cannot be run in the buyer's sandbox without vendor staff operating it by day two.
  • The same seed gives different initial states and the vendor cannot document what varies or why.
  • Any task in the sample cannot be solved from a fresh reset by its own reference solution, and more than a small number are affected.
  • A do-nothing run passes the grader on any task.
  • The grader or task files are withheld, so verdicts cannot be checked.
  • Production credentials, live customer data or real external side effects appear anywhere in the environment.
  • The business process owner judges that the success criteria do not match how the work is actually closed in their operation.

Assumptions to pin before comparing two vendors' results

Results from two vendors, or from one vendor in two weeks, are comparable only when these assumptions are pinned and written into the evidence package. Change any one of them and an agent's score can move without the agent changing at all.

Assumptions to pin before comparing two vendors' results
AssumptionWhy it moves resultsHow to pin it
Seeds and randomnessDifferent initial states make tasks easier or harderFixed seed list per trial, recorded per episode
Initial snapshotOSWorld ties each task to a snapshot and setup config that define the initial stateSnapshot identifier and setup script version per task
Reward basisAdding or removing a check, such as required messages, changes what counts as successGrader version and the list of checks that gate the reward
Clock and datesDue dates, aging buckets and deadlines depend on the current dateFrozen clock or recorded start time
Shared state between runsLeftover changes from earlier runs alter later tasks; WebArena tells users to reset its sites after evaluating its examplesFull reset between trials, verified by a state dump
User simulatorA simulated requester that phrases things differently changes difficultySimulator model, prompt and version recorded

Worked example (illustrative): an accounts payable exception environment

This example is constructed for illustration; the counts are illustrative and do not describe any real vendor or buyer. An AI team pilots a vendor's accounts payable exception environment built around three-way matching of purchase orders, receipts and supplier invoices.

Week one: the environment installs on day one. On day two, ten paired resets with the same seed match on every record except supplier reply wording, which the vendor documents as drawn from a fixed set of templates. On day three, 28 of 30 tasks reach their success state from the reference solution; two fail because the reference solution relies on a credit memo that the seeded state never creates. The vendor fixes the seed data and both pass on rerun. On day four, a do-nothing run fails every task, but a run that places a hold on every invoice passes 4 tasks, because the grader checked for a hold and not for the reason code. The grader is corrected and those 4 tasks are rerun.

Week two: across 5 trials per task, the buyer's agent shows a much lower pass^5 than its single-trial rate on price-variance exceptions, which is the useful finding of the pilot. A second engineer reruns 6 tasks from the evidence package on day nine and gets matching verdicts. The decision memo recommends going ahead, narrowed to the three exception types where the success criteria matched the process owner's practice, with the other two returned to the vendor for redesign.

When documented randomness is acceptable instead of a deterministic reset

A deterministic reset is the default expectation, but some realistic environments need randomness: a simulated supplier that replies differently, network delays, or a user simulator built on a language model. The exception is acceptable when the vendor documents every random component, exposes its seed or its fixed set of possible values, and shows that the success check does not depend on which value was drawn.

In that case, replace the same-state test with a same-distribution test. Run enough trials to see the spread, report pass^k rather than a single rate, and record the drawn values per episode so a disputed result can be replayed exactly. If the randomness cannot be recorded, the environment fails the replay criterion, however realistic it is.

The exception does not cover hidden randomness in the grader. A grader that gives different verdicts for the same end state, for instance a language-model judge without fixed settings, is a stop criterion, not a documented-randomness case.

What two weeks cannot tell you, and who signs the decision

Two weeks shows whether an environment behaves as declared and whether a handful of tasks are graded fairly. It cannot show coverage of a whole workflow, long-run stability across vendor releases, or whether training in the environment improves an agent on real work. Say so in the decision memo, and list what a longer engagement would need to measure.

The pilot also inherits the limits of its sample. Thirty tasks from five exception types say little about the sixth type nobody sampled. The process owner should state which real cases the sample leaves out, and the memo should carry that list forward.

Two people sign the decision: an RL environment engineer, who vouches for reset, solvability, grading and replay, and the business process owner, who vouches that the success criteria match how the work is really closed. Request specialist review before signing when the environment touches regulated records, models money movement or approvals with real-world effect, or when the two signers disagree about what success means.

Grounding pilot tasks in real workflow records

The weakest part of many enterprise environments is the seed state: invented invoices and tickets that are too clean to contain the exceptions agents struggle with. A pilot built on realistic histories tests the environment against the cases that matter.

SourceX holds no pilot environments or record sets in stock. When an AI team asks, it sources workflow records from U.S. businesses, and the request-data form captures the workflow, record types, volume, format, timing and licensing terms a pilot's seed state needs. The supplying business approves the scope and terms and signs the agreement before any invoice, ticket or log reaches the pilot.

Frequently asked questions

How long should an RL environment vendor pilot run?

Two weeks is enough for one workflow if the brief is signed before day one: one week to prove reset, solvability and grading, and one week to measure agents. Broader coverage or training effects need a longer engagement with its own success criteria.

How many tasks should a two-week pilot use?

Enough to cover each exception type the business owner cares about, with several trials per task, while leaving time for reference replays and a grader audit. A few dozen well-specified tasks usually teach more than hundreds that nobody can audit inside the window.

What should the vendor hand over at the end of the pilot?

The environment build, the seeds, the task and initial-state files, the grader code and version, and complete replay logs. Without these a second engineer cannot reproduce the verdicts, and the pilot result rests on trust.

What is pass^k and why does it matter for a pilot?

pass^k is the probability that all k independent trials of a task succeed. It exposes agents and environments that succeed sometimes but not reliably, which a single-trial success rate hides.

Can a pilot use real company data?

Only data the buyer has the right to use for this purpose, in a sandbox the buyer controls, with no production credentials or real side effects. Synthetic or properly licensed records are the usual choice, and security and legal should approve the data row of the brief.

Sources

  • Gymnasium's reset accepts a seed that initializes the environment's random number generator, and step distinguishes terminated (a terminal state of the task) from truncated (a limit outside the task, typically a time limit). Source
  • Gymnasium's environment checker reports an environment as non-deterministic when resetting with the same seed produces observations that are not equivalent. Source
  • tau2-bench replays a reference action list on a fresh environment to derive the target database end state, compares end states by hash, and by default gates reward on both the database state and required communication. Source
  • tau2-bench computes the pass^k metric from the number of trials and the number of successful trials. Source
  • OSWorld task examples specify an environment snapshot, setup configuration for the initial state, and an evaluator for each task. Source
  • WebArena's environment instructions tell users to reset the sites to their initial state after evaluating the benchmark's examples. Source
  • The Agentic Benchmark Checklist paper frames rigorous agent evaluation as task validity plus outcome validity and reports that one published agent benchmark counted empty responses as successful. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies