Skip to content

AI uses for records

Agent trajectory replay benchmark: can another team rerun your data?

By SourceX Editorial · Updated

9 sections

Short answer

For another team to replay a trajectory dataset, each episode needs ordered steps with case and step IDs, actor, time, the observation as seen, the action as issued, the tool response as returned under a matching call ID, linked artifacts and a verified outcome. The package also needs the environment build, starting-state snapshot, seeds and a success rule a second team can compute. A replay challenge tests all of it.

Agent run

Agent trajectory replay

This page covers agent trajectory replay benchmark: can another team rerun your data.Illustrative

Key takeaways

  • State whether the dataset supports log replay, re-execution, reproduction or replication; each needs different contents.
  • Ship the environment build, starting-state snapshot, seeds and a computable success rule, not just the step log.
  • Pair every tool call and response with a shared ID and keep responses exactly as returned, errors included.
  • Let a team that did not build the package draw the sample, log every question it asks and report the highest level reached.
  • Label episodes that cannot be re-executed, with the cause, instead of dropping them.

What must a trajectory dataset include for another team to replay it?

It must include every step in order with stable case and step IDs, the actor and time of each step, the observation exactly as the agent saw it, the action exactly as issued, the tool response exactly as returned, links to any files the step produced, and a verified outcome. It also needs what the steps cannot carry: a way to rebuild the environment, the starting state, the seeds and the rule that decides success.

Offline reinforcement learning datasets show the environment half of that list in a standard form. Minari's dataset standard stores, alongside the episodes, a JSON copy of the Gymnasium environment spec, a link to the code that generated the data, pip-style requirements to load the environment and the Minari version; each episode's reset seed and options sit in the episode metadata. Its loader can then recover the environment the data came from, and a dataset can declare a separate evaluation environment when collection and evaluation differ.

Business agent trajectories rarely arrive that complete. A ticket history or tool log usually has the actions and some outcomes but not the starting state, the exact responses or the grading rule. A replay benchmark is the test that exposes which of those pieces are missing before anyone trains or scores on the data.

Log replay, re-execution or reproduction: which claim is the dataset making?

State the level up front, because each level needs different contents and proves something different. A dataset that only supports log replay should not be described as reproducible.

ACM's artifact badging policy is a useful reference for the wording. In its current terms, reproducibility means a different team obtaining the same result with the same experimental setup, typically the authors' artifacts, and replicability means a different team obtaining it with a different setup. For trajectory data, that maps onto a ladder from simply re-reading the record to rebuilding the result independently.

Log replay, re-execution or reproduction: which claim is the dataset making?
LevelWhat the second team doesWhat it showsWhat the package must ship
Log replaySteps through recorded observations and actions without running anythingThe record is complete and internally consistentOrdered steps, IDs, observations, actions, responses, outcome
Re-executionRestores the starting state and feeds the recorded actions into the environmentThe environment produces the recorded observations and outcomeEnvironment build, starting state, seeds, tool versions
ReproductionRe-runs the episode or the scoring with the provided setup and gets the same outcomeThe result does not depend on the original team or machineAll of the above plus the grader and the run instructions
ReplicationBuilds its own environment or grader from the documentation and comparesThe result survives a different implementationTask specs and success rules written precisely enough to reimplement

What should the reproducible replay challenge and reporting template include?

The challenge hands the package to a team that did not build it, gives them nothing else, and asks them to reach a stated replay level on a sample of episodes within a fixed effort budget. Every question they have to ask the original team is logged as a defect in the package.

Run it in stages so a failure points at a specific gap. The table below is the protocol; the reporting template in a later section records the result.

Independence is what gives the result weight. The challenge team should sit outside the group that collected and cleaned the data, draw its own random sample after receiving the package, and work only from what ships. If the builders help informally, every answer they give is a fact the next buyer will not have, so it belongs in the question log and, after the challenge, in the documentation. Agree the effort budget before starting as well; a package that replays only after weeks of reverse engineering has not passed, even if the final numbers match.

Repeat the challenge whenever the package changes in a way that could move a result: a new records extract, a new environment build, a changed grader or a different redaction rule. Keep each completed report with the version it tested, so a buyer can see which replay results carried forward and which had to be earned again.

What should the reproducible replay challenge and reporting template include?
StageChallenge team inputPass ruleRecord
1. InventoryPackage manifest and documentation onlyEvery file the manifest lists is present and checksums matchMissing or mismatched files
2. Schema checkEpisode and step filesEvery step has IDs, actor, time, observation, action and, for tool calls, a matched responseSteps failing each check, by field
3. Log replayRandom sample of episodes drawn by the challenge teamEach sampled episode reads end to end and reaches the recorded outcomeEpisodes with gaps or contradictions
4. Environment rebuildBuild instructions, images, requirements, starting stateEnvironment starts from documented steps without helpQuestions asked and time taken
5. Re-executionRecorded actions and seeds for the sampleRecorded observations and outcome recur, or each difference is explainedDivergent steps with first point of divergence
6. Outcome checkGrader or success ruleIndependent scoring gives the recorded verdictVerdict mismatches
7. Sign-offAll stage recordsHighest level reached is stated with its exceptionsCompleted replay report

Pairing every tool call with the response it actually got

Give every tool call an identifier and store the response under the same identifier, so a checker can pair them without guessing from order or timing. Agents often issue several calls before reading any result, and order alone does not settle which response answered which call.

OpenTelemetry's generative AI semantic conventions give a usable vocabulary. They define an execute_tool operation and attributes for the tool name, a tool call identifier supplied by the model or provider, the arguments passed and the result returned, all currently marked as in development. The conventions warn that tool arguments and results may contain sensitive information, which is the reminder a trajectory dataset needs before it ships raw tool payloads.

Keep the response as it was returned, including errors, timeouts and partial results. A cleaned or summarized response makes log replay look tidy and makes re-execution impossible to check, because the challenge team has nothing exact to compare against.

Can a second team reconstruct the same task state and outcome?

Only if the package fixes the starting state and defines the outcome as something a second team can compute. With those two pieces, a sampled episode can be re-executed and scored independently; without them, the best available level is log replay.

Software-engineering benchmarks show what a fixed starting state looks like. Each SWE-bench task instance names the repository, the base commit, the package version, a test patch and the tests expected to go from failing to passing and to stay passing, and its harness runs evaluation in Docker containers built in base, environment and instance layers for reproducibility. A business trajectory needs the equivalent: a dated snapshot of the records the agent could see, the configuration in force and the tests that define success.

The outcome rule matters as much. In tau-bench, the success check reloads the original data, applies the task's ground-truth actions, hashes the resulting database and compares that with a hash of the database the agent left. A second team can recompute that verdict from the package alone. A label such as "resolved" copied from a ticket status cannot be recomputed and should be shipped as a recorded outcome, clearly separated from a verified one.

Shared state is the last trap. WebArena's instructions say that once its 812 examples have been evaluated, the environment should be reset to its initial state by removing and re-creating its site containers. A replay that runs episodes against state left over from earlier episodes will diverge for reasons that have nothing to do with the data.

The replay report: what the challenge team writes down

The challenge team fills one report per package, so different suppliers and versions can be compared on the same form. Copy the fields below into a document or issue template and keep the completed report with the dataset version it describes.

At a glance

The replay report: what the challenge team writes down

The replay report: what the challenge team writes down
Report fieldWhat to writeIllustrative entry
Package and versionDataset name, version, manifest checksumservice-desk-trajectories v2, manifest checksum recorded
Challenge teamWho ran it and their independence from the buildersTwo engineers from a separate evaluation group
SampleHow episodes were drawn and how manyRandom draw of 50 episodes across all task types
Highest level reachedLog replay, re-execution, reproduction or replicationRe-execution for 46 of 50; log replay only for 4
DivergencesEpisode ID, first divergent step, causeFour episodes diverge at a call to an external rate service
Outcome agreementRecorded verdicts the independent check confirmed or contradictedAll 50 confirmed under the stated rule; 2 flagged as reopened after the window
Questions askedEvery question put to the builders and the answerThree, all about seed data loading
EffortTime to rebuild and to run the sampleRebuild one day, sample two days
Defects and fixesPackage changes required before the next versionShip external responses as recorded fixtures

Illustrative: a replay challenge on a service-desk trajectory set

This scenario is illustrative; the counts are invented and describe no real dataset. A supplier packages trajectories from an IT service desk: each episode is a ticket handled with a directory lookup tool, a knowledge base search and a ticketing system, with outcomes taken from whether the ticket reopened within a fixed window.

The challenge team passes inventory and schema checks after one fix, a missing tool call ID on a small set of steps. Log replay works for all 50 sampled episodes. Re-execution against the rebuilt sandbox matches for 46. The other four call a currency or rate service outside the sandbox whose answers have changed since recording, so the environment cannot reproduce the observations.

The outcome check exposes the subtler problem: two episodes marked successful reopened after the window closed, so the recorded verdict is correct under the stated rule but contradicts what a reader would expect. The report keeps both facts, the team flags the window as a documented assumption, and the supplier adds a field with the reopen date so buyers can apply a different window if they choose.

Episodes that cannot be re-executed, and how to mark them

Some episodes can never be re-executed exactly, and the honest response is to label them rather than drop them. The common causes are live external services, simulated users driven by a language model, time-dependent logic and redaction.

At a glance

Episodes that cannot be re-executed, and how to mark them

tau-bench is a clear example of the simulated-user case: its user is simulated by a language model, with the model and strategy set by flag. Feeding a recorded agent action into a fresh simulator can produce a different user reply on the next turn, so the episode drifts. Record the simulator's turns as observations and replay them as fixtures, or declare the episode log-replay only.

Redaction needs the same discipline. If personal data in observations is masked, the starting state must be masked the same way, or re-execution will produce unmasked observations that fail to match the record and, worse, expose the data the masking was meant to remove.

Checklist · 4 items

  1. replay_level: the highest level the episode supports
  2. nondeterminism_source: external_service, simulated_user, clock, random_seed, none
  3. fixture_available: whether recorded responses can stand in for the live source
  4. redaction_consistent: whether the starting state and observations were masked with the same rules

0 of 4 ticked. Tick each item as you go.

Requesting replay-ready trajectories from a data supplier

Ask for the replay level you need in the request itself, and make the replay report part of acceptance. A supplier who knows the challenge is coming will ship starting-state snapshots, tool call IDs and raw responses instead of a cleaned transcript.

SourceX does not keep trajectory data on the shelf; for each buyer it locates U.S. businesses whose records match the request and manages rights review, contracting, preparation and delivery with those owners. Put the replay level, the snapshot dates and whether the license must cover re-execution and benchmark use into that request, alongside scope, volume, language, format and timing.

Frequently asked questions

What is an agent trajectory replay benchmark?

It is a test of whether a trajectory dataset can be replayed by a team that did not build it. The team receives only the package, rebuilds the environment where possible, replays or re-executes a sample of episodes and reports the highest replay level reached, with every divergence and question recorded.

Is log replay enough for training data?

For supervised fine-tuning on recorded actions, a complete and consistent log is often enough. For evaluation, reinforcement learning or any claim that outcomes are verified, the dataset needs to support re-execution, because only then can a second team confirm that the recorded actions lead to the recorded result.

Why do tool call IDs matter so much?

Agents can issue several tool calls before reading any results, so matching responses to calls by position or time is unreliable. A shared identifier on the call and its response lets a checker pair them exactly and spot missing or duplicated responses.

How large should the replay sample be?

Large enough to include every task type and tool in the package, drawn at random by the challenge team rather than chosen by the supplier. Fix the size before the challenge starts and write it in the report so later versions are tested the same way.

What if the environment cannot be shared with the buyer?

Ship recorded tool responses as fixtures and the starting-state snapshot the agent saw, so the buyer can at least log-replay and re-score. State the limit in the report; a package that supports log replay only should not be described as reproducible.

Does a replay benchmark need the original agent model?

No. Re-execution feeds the recorded actions into the environment, so the model that produced them is not needed. It is useful to record the model and its version as metadata, but the benchmark tests the data and the environment, not the agent.

Sources

  • Minari's dataset standard stores the Gymnasium environment spec, a code permalink, pip-style requirements and the Minari version as dataset metadata, with each episode's reset seed and options in episode metadata. Source
  • Minari can recover the environment used to create a dataset, and a dataset can declare a separate evaluation environment. Source
  • Each SWE-bench task instance lists the repository, base commit, version, test patch and the FAIL_TO_PASS and PASS_TO_PASS tests. Source
  • The SWE-bench harness runs evaluation in Docker containers built from base, environment and instance images to create reproducible environments. Source
  • tau-bench's success check reloads the data, applies the ground-truth actions and compares the resulting database hash with the hash of the agent's final database. Source
  • tau-bench simulates the user with a language model whose model and strategy are set by command-line flags. Source
  • OpenTelemetry's GenAI semantic conventions define an execute_tool operation and tool name, call ID, arguments and result attributes, marked as in development, and warn that arguments and results may contain sensitive information. Source
  • WebArena's instructions reset the environment after evaluating its 812 examples by removing and re-creating the site containers. Source
  • ACM's current artifact badging terms define reproducibility as a different team with the same experimental setup and replicability as a different team with a different setup. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies