Systems and records
Agent trajectory dataset schema: a JSONL format buyers can validate
By SourceX Editorial · Updated
10 sections
Short answer
An agent trajectory dataset schema is independently verifiable when every episode has a versioned header, every step is a separate observation, action or result record, every tool result points back to the call that caused it, and every outcome names how success was checked. Ship it as JSON Lines with a validator, one valid and one failing example, and replay instructions a second team can follow.
Agent run
Agent trajectory dataset
Key takeaways
- Keep observations, actions and results in separate records so every error can be attributed to the agent or the environment.
- Tie records together with an episode id, a gap-free step counter and a call id shared by each tool call and its result.
- Pin the schema version, environment snapshot and tool versions in an episode header, or the episode cannot be replayed.
- Treat success as verified only when a state check, test or named reviewer confirmed it.
- Ship a validator with one valid and one failing example, and have the supplier run it before delivery.
What makes an agent trajectory dataset schema verifiable by someone else?
A buyer can verify a trajectory only if a second team, with no access to the people who produced it, can check that each step is complete, that each tool result belongs to the call that caused it, and that the final outcome was confirmed against the system's real state. Field names matter less than four properties: separation, linkage, versioning and a checkable outcome.
Separation means what the agent saw, what it did and what the environment returned live in different records. Linkage means stable identifiers tie them together: an episode id on every line, a step counter without gaps and a call id shared by each tool call and its result. Versioning means the header pins the schema, the environment snapshot and the version of every tool. A checkable outcome means the final record says how success was verified, by whom, and against which expected state.
The rest of this page turns those properties into a concrete JSON Lines format for tool-using business agents: record types and required fields, an illustrative valid episode, a failing episode with the exact errors a validator reports, the checks themselves and what replay needs. It is written for API and tool-call agents; screen-based computer-use agents need additional observation fields for pixels and coordinates.
Why observations, actions and results belong in separate records
Splitting the three lets a reviewer attribute every error. If the model misread a tool response, the fault is in the action after a correct result; if the tool returned something wrong, the fault is in the result. When all three sit in one blob per turn, nobody can tell which happened, and a training pipeline cannot mask the environment's text out of the loss.
Current tool-calling protocols already draw this line. The Model Context Protocol specification separates protocol errors, returned as JSON-RPC errors for problems such as an unknown tool, from tool execution errors, which are returned inside the tool result with isError set to true so the model can self-correct. Anthropic's tool-use documentation has the result reference the original call by tool_use_id and lets a failed call carry an is_error flag. OpenTelemetry's GenAI conventions define a tool call identifier attribute, gen_ai.tool.call.id, set to the identifier received from the model or provider.
A dataset schema should keep those distinctions rather than flatten them. Record tool failures as results with is_error true, keep protocol failures as their own error results with a code, and never drop an errored step, because recovery after a failure is often the most valuable part of the episode.
Which record types and required fields does the schema need?
Five record types cover a tool-using episode. Every line carries type and episode_id; every in-episode line also carries step and ts. The table lists what each type must contain to pass validation.
| Record type | Required fields | Purpose | Links to |
|---|---|---|---|
| episode | schema_version, episode_id, task_id, instruction, env with snapshot_id and tool versions, started_at | Pins what task was attempted, in which environment, under which versions | Every other line by episode_id |
| observation | episode_id, step, source, content, ts | What the agent received from a user, document or system event, before acting | The action that follows it by step order |
| action | episode_id, step, kind, ts; for tool calls also call_id, tool, args | What the agent did: a tool call, a message or a stop | Its result by call_id |
| result | episode_id, step, call_id, is_error, content, ts | What the environment returned, success or failure | Its action by call_id |
| outcome | episode_id, final_step, success, verified_by, labeler | How the episode ended and how that was confirmed | Expected state or check list |
Field rules that make linkage and replay checkable
Field rules turn the record types into something a validator can enforce. Keep them short and mechanical; any rule that needs judgment belongs in the outcome review, not the schema.
De-identification must preserve these links. If an order number or customer id appears in an observation, a tool argument and a result, replace it with the same pseudonym everywhere in the episode, or the call pairing and the expected-state comparison will both break. Ask the supplier to apply one consistent mapping per episode and to state in the manifest which fields were transformed.
| Field | Rule | Why it matters |
|---|---|---|
| schema_version | Present in the header; changes whenever a field is added, removed or redefined | Lets a buyer run the right validator on old and new deliveries |
| episode_id | Identical on every line of the episode and unique across the dataset | Prevents lines from two episodes being merged by sort order |
| step | Integer starting at 1 with no gaps or repeats inside an episode | A gap means a dropped step; a repeat means a merge error |
| call_id | Shared by exactly one tool-call action and one result | Proves which call produced which response, including failures |
| ts | UTC timestamp with offset; a result never earlier than its call | Catches reordered or reconstructed logs |
| env.tools | Name and version of every tool the agent could call | A result from an undeclared or different version cannot be replayed |
| is_error | Boolean on every result; error text kept in content | Keeps failures and recoveries in the data instead of filtering them out |
| verified_by | One of state_check, test, human_review, with the reference it used | Separates a confirmed outcome from an agent's own claim of success |
An illustrative valid episode in JSON Lines
The episode below is invented for illustration and is not customer data. A support agent receives a return request, looks up the order, opens a return and tells the customer. Each line is one JSON object; the outcome line points to an expected-state file a second team can compare against.
- {"type": "episode", "schema_version": "1.0", "episode_id": "ep-0001", "task_id": "returns-017", "instruction": "Customer asks to return order 5512 as damaged; decide eligibility and act.", "env": {"snapshot_id": "snap-0302a", "tools": {"get_order": "2.1", "create_return": "1.4"}}, "policy_ref": "returns-policy-v7", "started_at": "2026-03-02T14:05:11Z"}
- {"type": "observation", "episode_id": "ep-0001", "step": 1, "source": "user", "content": "Order 5512 arrived damaged. I want to return it.", "ts": "2026-03-02T14:05:11Z"}
- {"type": "action", "episode_id": "ep-0001", "step": 2, "kind": "tool_call", "call_id": "c1", "tool": "get_order", "args": {"order_id": "5512"}, "ts": "2026-03-02T14:05:14Z"}
- {"type": "result", "episode_id": "ep-0001", "step": 3, "call_id": "c1", "is_error": false, "content": {"order_id": "5512", "status": "delivered", "delivered_on": "2026-02-27"}, "ts": "2026-03-02T14:05:15Z"}
- {"type": "action", "episode_id": "ep-0001", "step": 4, "kind": "tool_call", "call_id": "c2", "tool": "create_return", "args": {"order_id": "5512", "reason": "damaged"}, "ts": "2026-03-02T14:05:19Z"}
- {"type": "result", "episode_id": "ep-0001", "step": 5, "call_id": "c2", "is_error": false, "content": {"return_id": "r-88", "status": "open"}, "ts": "2026-03-02T14:05:20Z"}
- {"type": "action", "episode_id": "ep-0001", "step": 6, "kind": "message", "content": "Your return r-88 is open and a label is on its way.", "ts": "2026-03-02T14:05:22Z"}
- {"type": "outcome", "episode_id": "ep-0001", "final_step": 6, "success": true, "verified_by": "state_check", "expected_state_ref": "gold/ep-0001.json", "checks": ["return r-88 exists for order 5512", "reason is damaged"], "labeler": "reviewer-2"}
A failing episode and what the validator reports
This second illustrative episode looks plausible at a glance, which is why it needs a validator. It has no schema version, skips step 1, calls a tool the header never declared, returns a result tied to a call that does not exist, and claims success without saying how it was checked.
- {"type": "episode", "episode_id": "ep-0002", "task_id": "returns-018", "instruction": "Customer asks to return order 6120.", "env": {"snapshot_id": "snap-0302a", "tools": {"get_order": "2.1"}}, "started_at": "2026-03-02T15:00:02Z"}
- {"type": "action", "episode_id": "ep-0002", "step": 2, "kind": "tool_call", "call_id": "c1", "tool": "create_return", "args": {"order_id": "6120"}, "ts": "2026-03-02T15:00:06Z"}
- {"type": "result", "episode_id": "ep-0002", "step": 3, "call_id": "c9", "content": "Return created", "ts": "2026-03-02T15:00:05Z"}
- {"type": "outcome", "episode_id": "ep-0002", "final_step": 3, "success": true}
- Validator output, line 1: episode missing schema_version
- Validator output, line 2: tool create_return not declared in header
- Validator output, line 3: result missing is_error
- Validator output, line 3: result call_id c9 has no earlier action
- Validator output, line 4: outcome missing verified_by and labeler
- Validator output, episode level: call c1 has no result; steps do not start at 1; step numbers have gaps
The checks a buyer's validator should run
Run these checks on every delivered file before any human review. Errors reject the episode; warnings go to a reviewer. A reference validator is a short script, and the supplier should run the same one before shipping so both sides see the same report.
At a glance
The checks a buyer's validator should run
Structural checks prove an episode is complete, not that it is correct. After an episode passes, sample a share of outcomes for a domain expert to re-verify against the expected state, weighted toward episodes with tool errors, long step counts or unusual tools, and record the reviewer's verdict in a separate audit file so the original outcome label is never overwritten.
| Check | Rule | Example failure | Severity |
|---|---|---|---|
| Parse | Each line is valid UTF-8 JSON; no blank lines | Truncated line from a log export | Error |
| Header first | First line is an episode header with schema_version | Header missing or placed last | Error |
| Required fields | Each record type has its required fields | Result without is_error | Error |
| Step sequence | Steps start at 1, no gaps, no repeats | Step 1 missing | Error |
| Call pairing | Every tool call has exactly one result with the same call_id, and the reverse | Result for call c9 with no call | Error |
| Declared tools | Every called tool appears in env.tools with a version | create_return absent from the header | Error |
| Time order | Result timestamp not earlier than its call | Result stamped before the call it answers | Error |
| Outcome evidence | success has verified_by and a reference | success true with nothing else | Error |
| Expected-state file | The referenced file exists and matches the schema of the system state | gold file missing from the delivery | Warning until resolved |
| Privacy scan | No direct identifiers in content and args beyond what the license allows | Customer email address in a result | Error |
Can a second team reconstruct the same task state and outcome?
Only if the delivery includes what replay needs: the starting state, the tool versions, the recorded tool responses and a mechanical success test. Without the starting state a second team can read the trajectory but cannot rerun it; without a success test they can rerun it but cannot score it.
Public agent benchmarks show two workable ways to make outcomes checkable. In tau-bench's reward code, the environment replays the annotated ground-truth actions against freshly loaded data and compares a hash of the resulting database with the hash produced by the agent's run, and it also checks that required outputs appear in the agent's responses. SWE-bench records, for each task, the base commit, the gold patch, a test patch and two test lists, FAIL_TO_PASS and PASS_TO_PASS, and it runs evaluation in Docker for reproducibility.
For business-record trajectories the same pattern translates directly: store an expected-state file or a list of state assertions per episode, the snapshot id the episode started from, and the recorded responses so the episode can run in replay mode when the live system is no longer available. Write replay instructions as numbered steps: load snapshot, install declared tool versions, feed recorded results in order, compare final state with the expected-state file.
Test the instructions before accepting the delivery. Ask someone who did not build the dataset to replay a small random sample of episodes from the written steps alone, and count how many reach the recorded final state. Episodes that diverge point to a missing snapshot, an undeclared tool version or a nondeterministic tool, and each of those is cheaper to fix with the supplier before the full dataset ships than after.
How should the files be packaged and versioned?
Use JSON Lines with one episode per file or many episodes per file, never one giant JSON array. The JSON Lines format has three rules: UTF-8 encoding without a byte order mark, each line a valid JSON value with no blank lines, and a newline as the line terminator.
Add a manifest beside the data listing every file with its record count, episode count, schema_version and a SHA-256 hash, plus the expected-state files and the validator version used. When the schema changes, bump schema_version and keep the old validator available, so episodes delivered under version 1.0 can still be checked after version 1.1 ships.
Applying the schema to trajectories built from business records
Many useful trajectories are reconstructed from operational histories, such as ticket events, order changes and approval logs, rather than recorded live. Reconstruction is where gaps, merged steps and unverifiable outcomes creep in, so put the schema and validator in the request before any records are converted, and ask for the failing-episode report alongside the data.
Trajectory data reconstructed this way can be requested from SourceX, which builds each delivery from the records of U.S. businesses found for that request rather than from inventory. Attach the schema and validator to the request-data form alongside scope, volume, format, timing and licensing needs, so conversion is held to them from the first episode.
Frequently asked questions
Why JSON Lines instead of one JSON file per dataset?
JSON Lines puts one record per line, so files can be streamed, split, concatenated and validated line by line without loading everything into memory. A single malformed line is easy to locate and reject, while a malformed array can break the whole file.
Should failed tool calls be kept in the dataset?
Yes. Record them as results with is_error set to true and the error text in content. Recovery after a failure is often what a model most needs to learn, and filtering failures out makes the data look cleaner than the work it represents.
Where does the agent's reasoning go?
If you have it, put it in a separate optional field or record linked by step, and label who or what produced it. Keep it out of the observation, action and result records so validation and training masks do not depend on it.
What counts as a verified outcome?
An outcome is verified when a mechanical check or a named reviewer confirmed it against the system's state, such as a comparison with an expected-state file or a passing test. An agent's own message saying the task is done is not verification.
Can trajectories reconstructed from ticket histories pass this validator?
They can, if the conversion keeps every event as its own step, pairs each system action with its recorded response and stores the end state. Reconstructed episodes often fail the step-sequence and call-pairing checks first, which is exactly what the validator is for.
Sources
- The Model Context Protocol tools specification (version 2026-07-28) distinguishes protocol errors returned as JSON-RPC errors from tool execution errors reported in tool results with isError set to true. Source
- Anthropic's tool-use documentation has a tool_result block reference the original call through tool_use_id and allows an is_error flag for failed calls. Source
- OpenTelemetry's GenAI semantic conventions define gen_ai.tool.call.id as the tool call identifier supplied by the model or provider. Source
- tau-bench's reward function replays ground-truth actions on freshly loaded data, compares database hashes with the agent's run and checks required outputs in the agent's responses. Source
- SWE-bench task instances include base_commit, patch, test_patch, FAIL_TO_PASS and PASS_TO_PASS fields. Source
- The SWE-bench README states that SWE-bench uses Docker for reproducible evaluations. Source
- The JSON Lines format requires UTF-8 encoding without a byte order mark, one valid JSON value per line with no blank lines, and a newline line terminator. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.