Skip to content

Systems and records

Agent trajectory dataset schema: a JSONL format buyers can validate

By SourceX Editorial · Updated

10 sections

Short answer

An agent trajectory dataset schema is independently verifiable when every episode has a versioned header, every step is a separate observation, action or result record, every tool result points back to the call that caused it, and every outcome names how success was checked. Ship it as JSON Lines with a validator, one valid and one failing example, and replay instructions a second team can follow.

Agent run

Agent trajectory dataset

This page covers agent trajectory dataset schema: a JSONL format buyers can validate.Illustrative

Key takeaways

  • Keep observations, actions and results in separate records so every error can be attributed to the agent or the environment.
  • Tie records together with an episode id, a gap-free step counter and a call id shared by each tool call and its result.
  • Pin the schema version, environment snapshot and tool versions in an episode header, or the episode cannot be replayed.
  • Treat success as verified only when a state check, test or named reviewer confirmed it.
  • Ship a validator with one valid and one failing example, and have the supplier run it before delivery.

What makes an agent trajectory dataset schema verifiable by someone else?

A buyer can verify a trajectory only if a second team, with no access to the people who produced it, can check that each step is complete, that each tool result belongs to the call that caused it, and that the final outcome was confirmed against the system's real state. Field names matter less than four properties: separation, linkage, versioning and a checkable outcome.

Separation means what the agent saw, what it did and what the environment returned live in different records. Linkage means stable identifiers tie them together: an episode id on every line, a step counter without gaps and a call id shared by each tool call and its result. Versioning means the header pins the schema, the environment snapshot and the version of every tool. A checkable outcome means the final record says how success was verified, by whom, and against which expected state.

The rest of this page turns those properties into a concrete JSON Lines format for tool-using business agents: record types and required fields, an illustrative valid episode, a failing episode with the exact errors a validator reports, the checks themselves and what replay needs. It is written for API and tool-call agents; screen-based computer-use agents need additional observation fields for pixels and coordinates.

Why observations, actions and results belong in separate records

Splitting the three lets a reviewer attribute every error. If the model misread a tool response, the fault is in the action after a correct result; if the tool returned something wrong, the fault is in the result. When all three sit in one blob per turn, nobody can tell which happened, and a training pipeline cannot mask the environment's text out of the loss.

Current tool-calling protocols already draw this line. The Model Context Protocol specification separates protocol errors, returned as JSON-RPC errors for problems such as an unknown tool, from tool execution errors, which are returned inside the tool result with isError set to true so the model can self-correct. Anthropic's tool-use documentation has the result reference the original call by tool_use_id and lets a failed call carry an is_error flag. OpenTelemetry's GenAI conventions define a tool call identifier attribute, gen_ai.tool.call.id, set to the identifier received from the model or provider.

A dataset schema should keep those distinctions rather than flatten them. Record tool failures as results with is_error true, keep protocol failures as their own error results with a code, and never drop an errored step, because recovery after a failure is often the most valuable part of the episode.

Which record types and required fields does the schema need?

Five record types cover a tool-using episode. Every line carries type and episode_id; every in-episode line also carries step and ts. The table lists what each type must contain to pass validation.

Which record types and required fields does the schema need?
Record typeRequired fieldsPurposeLinks to
episodeschema_version, episode_id, task_id, instruction, env with snapshot_id and tool versions, started_atPins what task was attempted, in which environment, under which versionsEvery other line by episode_id
observationepisode_id, step, source, content, tsWhat the agent received from a user, document or system event, before actingThe action that follows it by step order
actionepisode_id, step, kind, ts; for tool calls also call_id, tool, argsWhat the agent did: a tool call, a message or a stopIts result by call_id
resultepisode_id, step, call_id, is_error, content, tsWhat the environment returned, success or failureIts action by call_id
outcomeepisode_id, final_step, success, verified_by, labelerHow the episode ended and how that was confirmedExpected state or check list

Field rules that make linkage and replay checkable

Field rules turn the record types into something a validator can enforce. Keep them short and mechanical; any rule that needs judgment belongs in the outcome review, not the schema.

De-identification must preserve these links. If an order number or customer id appears in an observation, a tool argument and a result, replace it with the same pseudonym everywhere in the episode, or the call pairing and the expected-state comparison will both break. Ask the supplier to apply one consistent mapping per episode and to state in the manifest which fields were transformed.

Field rules that make linkage and replay checkable
FieldRuleWhy it matters
schema_versionPresent in the header; changes whenever a field is added, removed or redefinedLets a buyer run the right validator on old and new deliveries
episode_idIdentical on every line of the episode and unique across the datasetPrevents lines from two episodes being merged by sort order
stepInteger starting at 1 with no gaps or repeats inside an episodeA gap means a dropped step; a repeat means a merge error
call_idShared by exactly one tool-call action and one resultProves which call produced which response, including failures
tsUTC timestamp with offset; a result never earlier than its callCatches reordered or reconstructed logs
env.toolsName and version of every tool the agent could callA result from an undeclared or different version cannot be replayed
is_errorBoolean on every result; error text kept in contentKeeps failures and recoveries in the data instead of filtering them out
verified_byOne of state_check, test, human_review, with the reference it usedSeparates a confirmed outcome from an agent's own claim of success

An illustrative valid episode in JSON Lines

The episode below is invented for illustration and is not customer data. A support agent receives a return request, looks up the order, opens a return and tells the customer. Each line is one JSON object; the outcome line points to an expected-state file a second team can compare against.

  • {"type": "episode", "schema_version": "1.0", "episode_id": "ep-0001", "task_id": "returns-017", "instruction": "Customer asks to return order 5512 as damaged; decide eligibility and act.", "env": {"snapshot_id": "snap-0302a", "tools": {"get_order": "2.1", "create_return": "1.4"}}, "policy_ref": "returns-policy-v7", "started_at": "2026-03-02T14:05:11Z"}
  • {"type": "observation", "episode_id": "ep-0001", "step": 1, "source": "user", "content": "Order 5512 arrived damaged. I want to return it.", "ts": "2026-03-02T14:05:11Z"}
  • {"type": "action", "episode_id": "ep-0001", "step": 2, "kind": "tool_call", "call_id": "c1", "tool": "get_order", "args": {"order_id": "5512"}, "ts": "2026-03-02T14:05:14Z"}
  • {"type": "result", "episode_id": "ep-0001", "step": 3, "call_id": "c1", "is_error": false, "content": {"order_id": "5512", "status": "delivered", "delivered_on": "2026-02-27"}, "ts": "2026-03-02T14:05:15Z"}
  • {"type": "action", "episode_id": "ep-0001", "step": 4, "kind": "tool_call", "call_id": "c2", "tool": "create_return", "args": {"order_id": "5512", "reason": "damaged"}, "ts": "2026-03-02T14:05:19Z"}
  • {"type": "result", "episode_id": "ep-0001", "step": 5, "call_id": "c2", "is_error": false, "content": {"return_id": "r-88", "status": "open"}, "ts": "2026-03-02T14:05:20Z"}
  • {"type": "action", "episode_id": "ep-0001", "step": 6, "kind": "message", "content": "Your return r-88 is open and a label is on its way.", "ts": "2026-03-02T14:05:22Z"}
  • {"type": "outcome", "episode_id": "ep-0001", "final_step": 6, "success": true, "verified_by": "state_check", "expected_state_ref": "gold/ep-0001.json", "checks": ["return r-88 exists for order 5512", "reason is damaged"], "labeler": "reviewer-2"}

A failing episode and what the validator reports

This second illustrative episode looks plausible at a glance, which is why it needs a validator. It has no schema version, skips step 1, calls a tool the header never declared, returns a result tied to a call that does not exist, and claims success without saying how it was checked.

  • {"type": "episode", "episode_id": "ep-0002", "task_id": "returns-018", "instruction": "Customer asks to return order 6120.", "env": {"snapshot_id": "snap-0302a", "tools": {"get_order": "2.1"}}, "started_at": "2026-03-02T15:00:02Z"}
  • {"type": "action", "episode_id": "ep-0002", "step": 2, "kind": "tool_call", "call_id": "c1", "tool": "create_return", "args": {"order_id": "6120"}, "ts": "2026-03-02T15:00:06Z"}
  • {"type": "result", "episode_id": "ep-0002", "step": 3, "call_id": "c9", "content": "Return created", "ts": "2026-03-02T15:00:05Z"}
  • {"type": "outcome", "episode_id": "ep-0002", "final_step": 3, "success": true}
  • Validator output, line 1: episode missing schema_version
  • Validator output, line 2: tool create_return not declared in header
  • Validator output, line 3: result missing is_error
  • Validator output, line 3: result call_id c9 has no earlier action
  • Validator output, line 4: outcome missing verified_by and labeler
  • Validator output, episode level: call c1 has no result; steps do not start at 1; step numbers have gaps

The checks a buyer's validator should run

Run these checks on every delivered file before any human review. Errors reject the episode; warnings go to a reviewer. A reference validator is a short script, and the supplier should run the same one before shipping so both sides see the same report.

At a glance

The checks a buyer's validator should run

Structural checks prove an episode is complete, not that it is correct. After an episode passes, sample a share of outcomes for a domain expert to re-verify against the expected state, weighted toward episodes with tool errors, long step counts or unusual tools, and record the reviewer's verdict in a separate audit file so the original outcome label is never overwritten.

The checks a buyer's validator should run
CheckRuleExample failureSeverity
ParseEach line is valid UTF-8 JSON; no blank linesTruncated line from a log exportError
Header firstFirst line is an episode header with schema_versionHeader missing or placed lastError
Required fieldsEach record type has its required fieldsResult without is_errorError
Step sequenceSteps start at 1, no gaps, no repeatsStep 1 missingError
Call pairingEvery tool call has exactly one result with the same call_id, and the reverseResult for call c9 with no callError
Declared toolsEvery called tool appears in env.tools with a versioncreate_return absent from the headerError
Time orderResult timestamp not earlier than its callResult stamped before the call it answersError
Outcome evidencesuccess has verified_by and a referencesuccess true with nothing elseError
Expected-state fileThe referenced file exists and matches the schema of the system stategold file missing from the deliveryWarning until resolved
Privacy scanNo direct identifiers in content and args beyond what the license allowsCustomer email address in a resultError

Can a second team reconstruct the same task state and outcome?

Only if the delivery includes what replay needs: the starting state, the tool versions, the recorded tool responses and a mechanical success test. Without the starting state a second team can read the trajectory but cannot rerun it; without a success test they can rerun it but cannot score it.

Public agent benchmarks show two workable ways to make outcomes checkable. In tau-bench's reward code, the environment replays the annotated ground-truth actions against freshly loaded data and compares a hash of the resulting database with the hash produced by the agent's run, and it also checks that required outputs appear in the agent's responses. SWE-bench records, for each task, the base commit, the gold patch, a test patch and two test lists, FAIL_TO_PASS and PASS_TO_PASS, and it runs evaluation in Docker for reproducibility.

For business-record trajectories the same pattern translates directly: store an expected-state file or a list of state assertions per episode, the snapshot id the episode started from, and the recorded responses so the episode can run in replay mode when the live system is no longer available. Write replay instructions as numbered steps: load snapshot, install declared tool versions, feed recorded results in order, compare final state with the expected-state file.

Test the instructions before accepting the delivery. Ask someone who did not build the dataset to replay a small random sample of episodes from the written steps alone, and count how many reach the recorded final state. Episodes that diverge point to a missing snapshot, an undeclared tool version or a nondeterministic tool, and each of those is cheaper to fix with the supplier before the full dataset ships than after.

How should the files be packaged and versioned?

Use JSON Lines with one episode per file or many episodes per file, never one giant JSON array. The JSON Lines format has three rules: UTF-8 encoding without a byte order mark, each line a valid JSON value with no blank lines, and a newline as the line terminator.

Add a manifest beside the data listing every file with its record count, episode count, schema_version and a SHA-256 hash, plus the expected-state files and the validator version used. When the schema changes, bump schema_version and keep the old validator available, so episodes delivered under version 1.0 can still be checked after version 1.1 ships.

Applying the schema to trajectories built from business records

Many useful trajectories are reconstructed from operational histories, such as ticket events, order changes and approval logs, rather than recorded live. Reconstruction is where gaps, merged steps and unverifiable outcomes creep in, so put the schema and validator in the request before any records are converted, and ask for the failing-episode report alongside the data.

Trajectory data reconstructed this way can be requested from SourceX, which builds each delivery from the records of U.S. businesses found for that request rather than from inventory. Attach the schema and validator to the request-data form alongside scope, volume, format, timing and licensing needs, so conversion is held to them from the first episode.

Frequently asked questions

Why JSON Lines instead of one JSON file per dataset?

JSON Lines puts one record per line, so files can be streamed, split, concatenated and validated line by line without loading everything into memory. A single malformed line is easy to locate and reject, while a malformed array can break the whole file.

Should failed tool calls be kept in the dataset?

Yes. Record them as results with is_error set to true and the error text in content. Recovery after a failure is often what a model most needs to learn, and filtering failures out makes the data look cleaner than the work it represents.

Where does the agent's reasoning go?

If you have it, put it in a separate optional field or record linked by step, and label who or what produced it. Keep it out of the observation, action and result records so validation and training masks do not depend on it.

What counts as a verified outcome?

An outcome is verified when a mechanical check or a named reviewer confirmed it against the system's state, such as a comparison with an expected-state file or a passing test. An agent's own message saying the task is done is not verification.

Can trajectories reconstructed from ticket histories pass this validator?

They can, if the conversion keeps every event as its own step, pairs each system action with its recorded response and stores the end state. Reconstructed episodes often fail the step-sequence and call-pairing checks first, which is exactly what the validator is for.

Sources

  • The Model Context Protocol tools specification (version 2026-07-28) distinguishes protocol errors returned as JSON-RPC errors from tool execution errors reported in tool results with isError set to true. Source
  • Anthropic's tool-use documentation has a tool_result block reference the original call through tool_use_id and allows an is_error flag for failed calls. Source
  • OpenTelemetry's GenAI semantic conventions define gen_ai.tool.call.id as the tool call identifier supplied by the model or provider. Source
  • tau-bench's reward function replays ground-truth actions on freshly loaded data, compares database hashes with the agent's run and checks required outputs in the agent's responses. Source
  • SWE-bench task instances include base_commit, patch, test_patch, FAIL_TO_PASS and PASS_TO_PASS fields. Source
  • The SWE-bench README states that SWE-bench uses Docker for reproducible evaluations. Source
  • The JSON Lines format requires UTF-8 encoding without a byte order mark, one valid JSON value per line with no blank lines, and a newline line terminator. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies