Skip to content

AI uses for records

RL environment reset reproducibility: can you replay yesterday's failure?

By SourceX Editorial · Updated

10 sections

Short answer

To test whether an environment can reproduce a prior failure, reset it from a written specification that pins every input to the starting state, replay the recorded actions, and compare state and verdict at each step against the original log. Repeat the replay several times. If every attempt reaches the same failure at the same step, the reset is reproducible; if not, the log shows where and why it drifts.

Grader rubric

RL environment reset

This page covers RL environment reset reproducibility: can you replay yesterday's failure.Illustrative

Key takeaways

  • A failure you cannot replay cannot be debugged, graded consistently or used as a regression test, so replay is the first acceptance check for any training environment.
  • A reset has to restore far more than a seed: data store, files, clock, tool and grader versions, simulated users and any stubbed external service.
  • Replaying recorded actions tests the environment; rerunning the agent tests the agent. Run both and never mix their numbers.
  • Classify every replay as reproduced, diverged at a named step, or environment fault before counting anything.
  • Where exact replay is impossible, write the tolerance and the stochastic parts into the specification instead of hiding them.

How do you test whether an environment can reproduce a prior failure?

Restore the exact starting state the failed episode began from, feed the environment the same actions in the same order, and check that each observation, state change and the final verdict match the original record. If they match on repeated attempts, the environment reproduces the failure. If they do not, the first step where they differ tells you which part of the reset is incomplete.

This is a narrower test than asking whether an environment is realistic or well designed. It asks one operational question: when an agent fails on Tuesday, can an engineer bring back Tuesday on Wednesday? Without that, a failure cannot be debugged, a fix cannot be confirmed, and the same episode can be graded differently on two days.

The test needs two artifacts the environment owner should already have. One is a reset specification that says what the starting state is made of and how each part is pinned. The other is a reproducibility log that records every replay attempt and its outcome. Both are laid out below as templates you can copy.

What must a reset restore before a replay means anything?

A reset must restore every input that can change what the agent observes or what the grader concludes, not only the random seed. In a simulator that may be little more than a seed; in an environment built from business systems it is usually a database snapshot, a file tree, a clock, a set of tool versions, a simulated user and a grader.

Seeding itself has rules that are easy to get wrong. Gymnasium's reset documentation says that if an environment already has a random number generator and reset is called with seed set to None, the generator is not reset; passing an integer resets it. The library's guidance is to pass a seed right after initialization and then leave it alone. A replay harness that calls reset without a seed and expects the same start state is relying on an assumption the API does not make.

State outside the random generator needs the same treatment. In the tau-bench reference code, each reset reloads the domain data from its source and clears the action list, and the reward is computed by replaying the gold actions on freshly loaded data and comparing a hash of the final data. That pattern, reload from a fixed snapshot and compare a state hash, is the one to copy for record-based environments: if the snapshot is not reloaded, the second episode starts from the first episode's leftovers.

Make the reset prove itself. At the end of every reset, before the first action, compute a hash over the loaded data, the file tree and the key settings, and write it into the episode record. A replay that starts from a different hash has already failed, and the comparison takes seconds instead of the hours it can take to notice a drifted state halfway through a long episode.

The reset specification: ten fields to fill in before acceptance

Ask the environment owner to complete one specification per environment version. Each field names a part of the starting state, how it is pinned, and how a replay harness can check it. A blank field is a finding in its own right: it marks an input that can change between runs without anyone noticing.

  • The code and runtime row follows Docker's own distinction: a tag can be moved to different content, while pulling by digest fetches the same image each time.
  • The episode end row follows Gymnasium's split between terminated, meaning the task reached a terminal state, and truncated, meaning a limit outside the task cut it off. Record which one ended the original episode.
  • The declared randomness row is what keeps the specification honest when an environment uses a language-model user or other sampling that cannot be fully pinned.
The reset specification: ten fields to fill in before acceptance
FieldWhat to recordHow to check it on replayWhat breaks if it is missing
Code and runtimeEnvironment version and source commit, plus the container image pinned by digest, not by tagCompare the running version and image digest to the logged onesFixes are compared across different code, or a retagged image silently changes libraries
Data snapshotSnapshot identifier and content hash of every table or file loaded at resetHash the loaded state before the first actionEpisodes start from the previous episode's leftovers
Seeds and process settingsSeed passed at reset, seeds for any library or simulator that draws randomness, hash seed, locale and time zoneLog the values actually used, not only the configured onesStarting state, stochastic transitions and the ordering of sets and dates shift
ClockFrozen or offset clock value at resetCompare the first timestamp the agent seesDue dates, aging rules and expiry checks flip
External servicesStub or recorded response set for every outside call, with its versionFail the replay on any unrecorded outbound callLive services return different answers
Simulated userUser simulator model, prompt version, sampling settings and seedCompare the first simulated messageThe conversation branches before the agent acts
Tool setTool names, schemas and implementation versions available to the agentDiff the tool list at reset against the logActions that were valid become invalid or vice versa
Episode end rulesStep limit, time limit and the difference between task completion and cut-offCheck the end reason recorded on replayA timeout is graded as a task failure or the reverse
Grader versionGrader code version and reference outcome versionRe-grade the original episode with the pinned graderThe same final state gets a different verdict
Declared randomnessAny part intentionally left stochastic and its expected distributionCompare outcome frequencies, not single outcomesNormal variation is mistaken for a broken reset

The reproducibility log: one row per replay attempt

The log turns a vague claim that an environment is reproducible into a record a reviewer can audit. Keep one row per replay attempt and never overwrite an earlier row. The last column is a classification, chosen from a short fixed list, so attempts can be counted without reading free text.

  • Reproduced: start state, every step and the final verdict match the original.
  • Diverged at step N: the start state matched but an observation, state change or verdict differed at step N.
  • Environment fault: the replay could not run as intended, such as a missing service, a crashed container or a grader that produced no output.
  • Declared stochastic: the divergence falls inside randomness the specification declares, and outcome frequencies are compared instead.
The reproducibility log: one row per replay attempt
Log fieldContentExample entry (illustrative)
Replay ID and timeUnique attempt ID and wall-clock time of the replay runR-0412, run on the day after the original
Source episodeID of the original failed episode and its stored trajectoryEpisode E-1187, 9 steps
Specification versionVersion of the reset specification used for this attemptReset spec v3
Start-state checkWhether the loaded state hash matched the logged hash before step 1Matched
Step comparisonFirst step where observation or state differs, or noneDiffers at step 6
Final verdictGrader verdict on the replay compared with the original verdictOriginal fail, replay pass
ClassificationReproduced, diverged at step N, environment fault, or declared stochasticDiverged at step 6
Cause and ownerSuspected cause of any divergence and who must fix itUnpinned clock, environment engineer

Which environment assumptions change the result of the acceptance test?

The acceptance result depends most on assumptions the environment owner did not write down: that transitions are deterministic, that the software stack behaves the same everywhere, and that nothing outside the container answers differently today. Test each one directly rather than trusting it.

Determinism of transitions can be checked mechanically. Gymnasium's environment checker includes tests that seeded resets produce the same observations and that one step from a seeded reset with the same action produces the same observation, reward and termination flags. A business-system environment deserves the same check: reset twice from the same specification, take the same first action, and compare the resulting state hash.

The software stack is a frequent source of drift. PyTorch's reproducibility notes say fully reproducible results are not promised across releases, individual commits or platforms, and may differ between CPU and GPU runs even with identical seeds. Python randomizes string hashing by default, so anything that depends on set or dictionary ordering of strings can change between processes unless PYTHONHASHSEED is fixed. Neither shows up in a seed field.

The last assumption is isolation. Environments that call live services, read the current date or use a language-model user simulator carry inputs that change on their own. Either pin them in the specification or declare them as randomness and judge the replay by outcome frequency.

Replaying the actions or rerunning the agent: two different tests

Replaying the recorded actions tests whether the environment is reproducible; rerunning the agent from the same reset tests whether the agent is consistent. Both matter, but they answer different questions and their numbers should never be pooled.

Action replay holds the agent fixed by removing it. SWE-agent, for example, stores each run as a trajectory file of thought, action and observation turns, writes the configuration needed to repeat the run, and offers a replay command that executes the stored actions again in the environment. If an action replay diverges, the environment or its reset is at fault, because the actions were identical.

A policy rerun lets the agent choose again, so differences can come from model sampling as well as from the environment. The tau-bench paper measures this with pass^k, the chance that all k independent trials of a task succeed, and notes that its tasks are written so that a user instruction leads to a single correct database outcome. Seed variance is also large on its own: Henderson and colleagues showed that runs differing only by random seed can produce statistically different result distributions in deep RL. Run action replay first; only when it passes does a policy rerun tell you about the agent.

Illustrative replay: the refund task that failed on Tuesday

This worked example is illustrative; the environment, counts and causes are invented to show the method. An environment built from a support desk and billing system asks the agent to issue a refund if an invoice is within its refund window. On Tuesday the agent refused a refund the reference outcome says was due. The team logs it as episode E-1187 and tries to replay it on Wednesday.

First pass: 10 action replays, all with a matching start-state hash, all diverging at step 6, where the billing tool reports the invoice as outside its window. The log points to the clock. The refund window is computed from the current date, and the reset specification left the clock field blank, so every replay saw a later date than the original episode did.

Second pass: the owner freezes the clock at the original episode's start time and adds it to the specification. All 10 replays now reproduce the original observations and the original failing verdict, which means the failure is real agent behavior on a stable state, not an artifact. Only now is the episode fit to become a regression test.

The exception: a second failure, E-1202, involves a simulated customer who changes their mind. With the simulator's sampling left on, 3 of 10 replays reach the same failure, illustratively. The specification declares the simulator as stochastic, so the team either pins the simulator's seed and model version for regression use or records the outcome frequency and treats single replays as samples.

Separating environment faults from agent failures before you count

Classify each replay before counting anything, because an environment fault counted as an agent failure inflates failure rates and hides a broken reset. A missing service, a container that failed to start or a grader that produced no output is not evidence about the agent.

Mature harnesses already make this distinction. The SWE-bench grading code flags a likely environment fault when a run produces no parseable test output and keeps the instance unresolved either way, so the fault is visible for triage without being scored as a success. Copy both halves of that rule: mark the fault explicitly, and never let a run that did not execute count as a pass.

A practical acceptance threshold follows from the log. For declared deterministic parts, every replay in a batch should be classified as reproduced; one unexplained divergence is a defect to fix, not noise to average away. For declared stochastic parts, agree in advance how many replays will be run and what outcome frequency range is acceptable.

Once a failure is classified as reproduced, keep it. Store the source episode, its reset specification version and its replay log together as a regression case, and replay the whole set whenever the environment, grader or tool set changes. A case that stops reproducing after a change is either a fix to confirm or a new source of drift to explain.

When exact replay is out of reach, and who signs off

Some environments cannot replay exactly: GPU-dependent models, third-party services that cannot be recorded, or simulators whose sampling cannot be fixed. In those cases the specification should say which parts are approximate, what tolerance applies to each and how replays are judged, and the acceptance decision should be made on that basis rather than on a claim of full determinism.

Bring in the RL environment engineer for anything in the runtime, seed and tool rows, and the business process owner for the data snapshot, clock and grader rows. The process owner is the person who can say whether a frozen date or a stubbed payment response still represents how the work is actually done. If neither can explain a divergence after two replay batches, stop using the affected episodes for training or evaluation until the cause is found.

Environments built from licensed business records

When an environment's starting state comes from real business records, reproducibility also depends on the data delivery: a snapshot that can be reloaded identically, with timestamps, identifiers and reference outcomes kept consistent through de-identification. Settle those requirements before records are prepared, not after an environment fails its replay test.

Environment builders can ask SourceX for licensed datasets drawn from U.S. businesses' operational records. Nothing is held in stock, so each request states the snapshot, field and outcome requirements an environment needs, alongside volume, format, timing and licensing terms.

Frequently asked questions

What is a reset specification for an RL environment?

It is a written list of every input that defines an environment's starting state and how each is pinned, such as the code version, container image, data snapshot, seeds, clock, simulated user, tools and grader. It lets anyone restore the same start state later. A blank field marks an input that can change between runs.

Is setting a random seed enough to make an environment reproducible?

No. A seed controls one source of randomness, while data, clocks, library versions, hash ordering, external services and graders can all change independently. In Gymnasium, calling reset with no seed also leaves an existing generator unreset, so seeding has to be done deliberately.

What should the reset specification and reproducibility log include?

The specification should cover version, runtime image, data snapshot, seeds, process settings, clock, external services, simulated user, tools, end rules, grader and declared randomness. The log should record the replay ID, source episode, specification version, start-state check, first differing step, final verdict, classification and owner of any fix.

How many replays are needed to call a failure reproducible?

There is no universal number. For parts declared deterministic, every replay in an agreed batch should match, and a single unexplained divergence is a defect. For stochastic parts, agree the number of replays and the acceptable outcome frequency before running them.

What is the difference between replaying actions and rerunning the agent?

Action replay feeds the environment the exact recorded actions, so any divergence comes from the environment or its reset. Rerunning the agent lets it choose again, so differences also reflect model sampling. Test the environment with action replay first, then measure agent consistency.

Sources

  • Gymnasium's reset does not reset an existing random generator when seed is None, resets it for an integer seed, and distinguishes terminated from truncated episodes. Source
  • Gymnasium's environment checker tests that seeded resets and a step after a seeded reset are deterministic for the same seed and action. Source
  • PyTorch does not promise fully reproducible results across releases, commits or platforms, or between CPU and GPU, even with identical seeds. Source
  • Python enables hash randomization by default, and PYTHONHASHSEED sets a fixed hash seed. Source
  • Pulling a Docker image by digest fetches the same image every time, while tags can be changed. Source
  • The tau-bench environment reloads its data at each reset and grades by comparing a hash of the final data with the result of replaying the gold actions. Source
  • The tau-bench paper defines pass^k as the chance that all k independent trials succeed and designs tasks to lead to a unique database outcome. Source
  • Henderson and colleagues show that runs differing only in random seed can produce statistically different result distributions in deep RL. Source
  • SWE-agent stores runs as thought, action and observation trajectories with a configuration file to repeat the run, and can replay stored actions. Source
  • SWE-bench's grader flags likely environment faults when no test output can be parsed and keeps such instances unresolved. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies