AI uses for records
Agent trajectory replay benchmark: can another team rerun your data?
By SourceX Editorial · Updated
9 sections
Short answer
For another team to replay a trajectory dataset, each episode needs ordered steps with case and step IDs, actor, time, the observation as seen, the action as issued, the tool response as returned under a matching call ID, linked artifacts and a verified outcome. The package also needs the environment build, starting-state snapshot, seeds and a success rule a second team can compute. A replay challenge tests all of it.
Agent run
Agent trajectory replay
Key takeaways
- State whether the dataset supports log replay, re-execution, reproduction or replication; each needs different contents.
- Ship the environment build, starting-state snapshot, seeds and a computable success rule, not just the step log.
- Pair every tool call and response with a shared ID and keep responses exactly as returned, errors included.
- Let a team that did not build the package draw the sample, log every question it asks and report the highest level reached.
- Label episodes that cannot be re-executed, with the cause, instead of dropping them.
What must a trajectory dataset include for another team to replay it?
It must include every step in order with stable case and step IDs, the actor and time of each step, the observation exactly as the agent saw it, the action exactly as issued, the tool response exactly as returned, links to any files the step produced, and a verified outcome. It also needs what the steps cannot carry: a way to rebuild the environment, the starting state, the seeds and the rule that decides success.
Offline reinforcement learning datasets show the environment half of that list in a standard form. Minari's dataset standard stores, alongside the episodes, a JSON copy of the Gymnasium environment spec, a link to the code that generated the data, pip-style requirements to load the environment and the Minari version; each episode's reset seed and options sit in the episode metadata. Its loader can then recover the environment the data came from, and a dataset can declare a separate evaluation environment when collection and evaluation differ.
Business agent trajectories rarely arrive that complete. A ticket history or tool log usually has the actions and some outcomes but not the starting state, the exact responses or the grading rule. A replay benchmark is the test that exposes which of those pieces are missing before anyone trains or scores on the data.
Log replay, re-execution or reproduction: which claim is the dataset making?
State the level up front, because each level needs different contents and proves something different. A dataset that only supports log replay should not be described as reproducible.
ACM's artifact badging policy is a useful reference for the wording. In its current terms, reproducibility means a different team obtaining the same result with the same experimental setup, typically the authors' artifacts, and replicability means a different team obtaining it with a different setup. For trajectory data, that maps onto a ladder from simply re-reading the record to rebuilding the result independently.
| Level | What the second team does | What it shows | What the package must ship |
|---|---|---|---|
| Log replay | Steps through recorded observations and actions without running anything | The record is complete and internally consistent | Ordered steps, IDs, observations, actions, responses, outcome |
| Re-execution | Restores the starting state and feeds the recorded actions into the environment | The environment produces the recorded observations and outcome | Environment build, starting state, seeds, tool versions |
| Reproduction | Re-runs the episode or the scoring with the provided setup and gets the same outcome | The result does not depend on the original team or machine | All of the above plus the grader and the run instructions |
| Replication | Builds its own environment or grader from the documentation and compares | The result survives a different implementation | Task specs and success rules written precisely enough to reimplement |
What should the reproducible replay challenge and reporting template include?
The challenge hands the package to a team that did not build it, gives them nothing else, and asks them to reach a stated replay level on a sample of episodes within a fixed effort budget. Every question they have to ask the original team is logged as a defect in the package.
Run it in stages so a failure points at a specific gap. The table below is the protocol; the reporting template in a later section records the result.
Independence is what gives the result weight. The challenge team should sit outside the group that collected and cleaned the data, draw its own random sample after receiving the package, and work only from what ships. If the builders help informally, every answer they give is a fact the next buyer will not have, so it belongs in the question log and, after the challenge, in the documentation. Agree the effort budget before starting as well; a package that replays only after weeks of reverse engineering has not passed, even if the final numbers match.
Repeat the challenge whenever the package changes in a way that could move a result: a new records extract, a new environment build, a changed grader or a different redaction rule. Keep each completed report with the version it tested, so a buyer can see which replay results carried forward and which had to be earned again.
| Stage | Challenge team input | Pass rule | Record |
|---|---|---|---|
| 1. Inventory | Package manifest and documentation only | Every file the manifest lists is present and checksums match | Missing or mismatched files |
| 2. Schema check | Episode and step files | Every step has IDs, actor, time, observation, action and, for tool calls, a matched response | Steps failing each check, by field |
| 3. Log replay | Random sample of episodes drawn by the challenge team | Each sampled episode reads end to end and reaches the recorded outcome | Episodes with gaps or contradictions |
| 4. Environment rebuild | Build instructions, images, requirements, starting state | Environment starts from documented steps without help | Questions asked and time taken |
| 5. Re-execution | Recorded actions and seeds for the sample | Recorded observations and outcome recur, or each difference is explained | Divergent steps with first point of divergence |
| 6. Outcome check | Grader or success rule | Independent scoring gives the recorded verdict | Verdict mismatches |
| 7. Sign-off | All stage records | Highest level reached is stated with its exceptions | Completed replay report |
Pairing every tool call with the response it actually got
Give every tool call an identifier and store the response under the same identifier, so a checker can pair them without guessing from order or timing. Agents often issue several calls before reading any result, and order alone does not settle which response answered which call.
OpenTelemetry's generative AI semantic conventions give a usable vocabulary. They define an execute_tool operation and attributes for the tool name, a tool call identifier supplied by the model or provider, the arguments passed and the result returned, all currently marked as in development. The conventions warn that tool arguments and results may contain sensitive information, which is the reminder a trajectory dataset needs before it ships raw tool payloads.
Keep the response as it was returned, including errors, timeouts and partial results. A cleaned or summarized response makes log replay look tidy and makes re-execution impossible to check, because the challenge team has nothing exact to compare against.
Can a second team reconstruct the same task state and outcome?
Only if the package fixes the starting state and defines the outcome as something a second team can compute. With those two pieces, a sampled episode can be re-executed and scored independently; without them, the best available level is log replay.
Software-engineering benchmarks show what a fixed starting state looks like. Each SWE-bench task instance names the repository, the base commit, the package version, a test patch and the tests expected to go from failing to passing and to stay passing, and its harness runs evaluation in Docker containers built in base, environment and instance layers for reproducibility. A business trajectory needs the equivalent: a dated snapshot of the records the agent could see, the configuration in force and the tests that define success.
The outcome rule matters as much. In tau-bench, the success check reloads the original data, applies the task's ground-truth actions, hashes the resulting database and compares that with a hash of the database the agent left. A second team can recompute that verdict from the package alone. A label such as "resolved" copied from a ticket status cannot be recomputed and should be shipped as a recorded outcome, clearly separated from a verified one.
Shared state is the last trap. WebArena's instructions say that once its 812 examples have been evaluated, the environment should be reset to its initial state by removing and re-creating its site containers. A replay that runs episodes against state left over from earlier episodes will diverge for reasons that have nothing to do with the data.
The replay report: what the challenge team writes down
The challenge team fills one report per package, so different suppliers and versions can be compared on the same form. Copy the fields below into a document or issue template and keep the completed report with the dataset version it describes.
At a glance
The replay report: what the challenge team writes down
| Report field | What to write | Illustrative entry |
|---|---|---|
| Package and version | Dataset name, version, manifest checksum | service-desk-trajectories v2, manifest checksum recorded |
| Challenge team | Who ran it and their independence from the builders | Two engineers from a separate evaluation group |
| Sample | How episodes were drawn and how many | Random draw of 50 episodes across all task types |
| Highest level reached | Log replay, re-execution, reproduction or replication | Re-execution for 46 of 50; log replay only for 4 |
| Divergences | Episode ID, first divergent step, cause | Four episodes diverge at a call to an external rate service |
| Outcome agreement | Recorded verdicts the independent check confirmed or contradicted | All 50 confirmed under the stated rule; 2 flagged as reopened after the window |
| Questions asked | Every question put to the builders and the answer | Three, all about seed data loading |
| Effort | Time to rebuild and to run the sample | Rebuild one day, sample two days |
| Defects and fixes | Package changes required before the next version | Ship external responses as recorded fixtures |
Illustrative: a replay challenge on a service-desk trajectory set
This scenario is illustrative; the counts are invented and describe no real dataset. A supplier packages trajectories from an IT service desk: each episode is a ticket handled with a directory lookup tool, a knowledge base search and a ticketing system, with outcomes taken from whether the ticket reopened within a fixed window.
The challenge team passes inventory and schema checks after one fix, a missing tool call ID on a small set of steps. Log replay works for all 50 sampled episodes. Re-execution against the rebuilt sandbox matches for 46. The other four call a currency or rate service outside the sandbox whose answers have changed since recording, so the environment cannot reproduce the observations.
The outcome check exposes the subtler problem: two episodes marked successful reopened after the window closed, so the recorded verdict is correct under the stated rule but contradicts what a reader would expect. The report keeps both facts, the team flags the window as a documented assumption, and the supplier adds a field with the reopen date so buyers can apply a different window if they choose.
Episodes that cannot be re-executed, and how to mark them
Some episodes can never be re-executed exactly, and the honest response is to label them rather than drop them. The common causes are live external services, simulated users driven by a language model, time-dependent logic and redaction.
At a glance
Episodes that cannot be re-executed, and how to mark them
tau-bench is a clear example of the simulated-user case: its user is simulated by a language model, with the model and strategy set by flag. Feeding a recorded agent action into a fresh simulator can produce a different user reply on the next turn, so the episode drifts. Record the simulator's turns as observations and replay them as fixtures, or declare the episode log-replay only.
Redaction needs the same discipline. If personal data in observations is masked, the starting state must be masked the same way, or re-execution will produce unmasked observations that fail to match the record and, worse, expose the data the masking was meant to remove.
Checklist · 4 items
- replay_level: the highest level the episode supports
- nondeterminism_source: external_service, simulated_user, clock, random_seed, none
- fixture_available: whether recorded responses can stand in for the live source
- redaction_consistent: whether the starting state and observations were masked with the same rules
0 of 4 ticked. Tick each item as you go.
Requesting replay-ready trajectories from a data supplier
Ask for the replay level you need in the request itself, and make the replay report part of acceptance. A supplier who knows the challenge is coming will ship starting-state snapshots, tool call IDs and raw responses instead of a cleaned transcript.
SourceX does not keep trajectory data on the shelf; for each buyer it locates U.S. businesses whose records match the request and manages rights review, contracting, preparation and delivery with those owners. Put the replay level, the snapshot dates and whether the license must cover re-execution and benchmark use into that request, alongside scope, volume, language, format and timing.
Frequently asked questions
What is an agent trajectory replay benchmark?
It is a test of whether a trajectory dataset can be replayed by a team that did not build it. The team receives only the package, rebuilds the environment where possible, replays or re-executes a sample of episodes and reports the highest replay level reached, with every divergence and question recorded.
Is log replay enough for training data?
For supervised fine-tuning on recorded actions, a complete and consistent log is often enough. For evaluation, reinforcement learning or any claim that outcomes are verified, the dataset needs to support re-execution, because only then can a second team confirm that the recorded actions lead to the recorded result.
Why do tool call IDs matter so much?
Agents can issue several tool calls before reading any results, so matching responses to calls by position or time is unreliable. A shared identifier on the call and its response lets a checker pair them exactly and spot missing or duplicated responses.
How large should the replay sample be?
Large enough to include every task type and tool in the package, drawn at random by the challenge team rather than chosen by the supplier. Fix the size before the challenge starts and write it in the report so later versions are tested the same way.
What if the environment cannot be shared with the buyer?
Ship recorded tool responses as fixtures and the starting-state snapshot the agent saw, so the buyer can at least log-replay and re-score. State the limit in the report; a package that supports log replay only should not be described as reproducible.
Does a replay benchmark need the original agent model?
No. Re-execution feeds the recorded actions into the environment, so the model that produced them is not needed. It is useful to record the model and its version as metadata, but the benchmark tests the data and the environment, not the agent.
Sources
- Minari's dataset standard stores the Gymnasium environment spec, a code permalink, pip-style requirements and the Minari version as dataset metadata, with each episode's reset seed and options in episode metadata. Source
- Minari can recover the environment used to create a dataset, and a dataset can declare a separate evaluation environment. Source
- Each SWE-bench task instance lists the repository, base commit, version, test patch and the FAIL_TO_PASS and PASS_TO_PASS tests. Source
- The SWE-bench harness runs evaluation in Docker containers built from base, environment and instance images to create reproducible environments. Source
- tau-bench's success check reloads the data, applies the ground-truth actions and compares the resulting database hash with the hash of the agent's final database. Source
- tau-bench simulates the user with a language model whose model and strategy are set by command-line flags. Source
- OpenTelemetry's GenAI semantic conventions define an execute_tool operation and tool name, call ID, arguments and result attributes, marked as in development, and warn that arguments and results may contain sensitive information. Source
- WebArena's instructions reset the environment after evaluating its 812 examples by removing and re-creating the site containers. Source
- ACM's current artifact badging terms define reproducibility as a different team with the same experimental setup and replicability as a different team with a different setup. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.