Skip to content

Systems and records

Agent trajectory cross-system joins: how to validate them before you train

By SourceX Editorial · Updated

9 sections

Short answer

Validate joins in an agent trajectory dataset edge by edge. List every link the episode depends on (step to episode, tool call to result, step to artifact, episode to outcome, one system's key to another's), then measure orphan and unmatched rates for each with explicit numerators and denominators. Compare them with thresholds fixed in advance, check timestamps for order and offset, and have a second team replay a sample from the shipped files alone.

Agent run

Agent trajectory cross

This page covers agent trajectory cross-system joins: how to validate them before you train.Illustrative

Key takeaways

  • A trajectory is only as complete as its weakest join; a single broken edge can strip outcomes or tool results from thousands of steps.
  • Measure orphan rates per join edge, per source system and per month, never as one blended number.
  • Protocol request IDs are only unique while a request is outstanding, so join tool calls to results on a composite key.
  • Null keys pass many referential integrity tests by default; test for missing keys separately.
  • Legitimate gaps, such as redacted or out-of-scope steps, need reason codes rather than silent deletion.
  • A replay of a stratified sample by a second team is the final check that the joins carry the state and outcome.

How do you validate joins within an agent trajectory dataset?

You validate them one edge at a time and with numbers. Write down every relationship an episode needs, run a referential test on each in both directions, express each failure as a rate over a stated denominator, and compare it with a threshold agreed before the data arrived. Then confirm by replay that another team can rebuild the same task state and outcome from the files alone.

Cross-system trajectories are assembled, not recorded. A help desk ticket, a billing adjustment, an email thread and an agent's tool calls each live in different systems with different identifiers and clocks. The assembly step is where most quality loss happens, and it is invisible in a sample of ten pretty episodes.

The failure mode that ruins an otherwise valuable dataset is rarely dramatic. A key format changes mid-year, a pseudonymization salt is rotated, or one export truncates at a row limit. Every episode still parses. A slice of them simply lost their outcome or their tool results, and models trained on that slice learn from actions with no consequences.

Which joins hold a trajectory together?

Most trajectory datasets depend on six or seven join edges, and each one fails differently. Map them before testing anything, because the map tells you which denominator belongs to which check.

Telemetry gives a useful model for the internal edges. In the OpenTelemetry trace API, each span has zero or one parent, a trace has a single root span, and a child span carries the same trace ID as its parent. A trajectory's steps should obey the same rules relative to their episode: one episode, one root, and no step whose episode ID points nowhere.

Draw the map as a small graph with each edge labelled by its key and its owner, the team or system that produces the child records. When a check fails later, the owner column tells you who to ask, and the key column tells you which transformation to inspect first.

Which joins hold a trajectory together?
Join edgeChild and parentTypical keyHow it usually breaks
Step to episodeEach event or step; the episode headerEpisode IDEpisodes split across export files; header dropped by a filter
Tool call to tool resultResult record; the call that produced itEpisode ID plus call IDCall IDs reused across sessions; results logged by a different service
Step to artifactStep; the file, message or record it created or readArtifact ID or content hashAttachments purged under retention; links point to live URLs
Episode to source caseEpisode; the ticket, order or claim it worked onCase ID in the system of recordCase IDs reformatted after a migration or merge
Episode to outcomeOutcome record; the episode it gradesCase ID plus outcome typeOutcome pulled from a later extract with different pseudonyms
System A key to system B keyCrosswalk row; both systems' recordsPseudonymized shared identifierSalt or mapping table changed between extracts
Step to actorStep; the person, bot or agent roleActor ID with roleService accounts shared by people and automation

Why a tool call ID is not a join key by itself

A tool call ID is unique only inside a narrow scope, so on its own it can attach a result to the wrong call. Join calls to results on a composite key, usually episode or session ID plus call ID, and confirm the order of events.

The Model Context Protocol shows why. Its base specification requires each request to carry a string or integer ID that is not null and does not match any other request the sender has issued and is still waiting on. A response must carry the same ID as its request. Nothing in that rule stops the same ID from being used again later in the session, or in a different session, so a dataset-wide join on ID alone can pair a result with an unrelated call.

Model APIs have their own pairing rules. In Anthropic's tool use documentation, each tool_result block carries a tool_use_id that names the tool_use block it answers, and the results must immediately follow the turn that made the calls. A trajectory exported from such a conversation should preserve both the IDs and that adjacency; if the export reorders messages, the adjacency check fails even when the IDs still match.

Telemetry conventions add a further caution. The OpenTelemetry GenAI conventions say gen_ai.tool.call.id should be set to the identifier from the model or provider only if one was supplied. A pipeline that relies on that attribute will therefore meet calls with no ID at all, and those must be counted as missing keys rather than silently dropped.

The join integrity checklist with orphan event rates

This is the asset. Each check is a rate with a named numerator and denominator, computed per source system and per calendar month as well as overall. The warn and error thresholds shown are illustrative starting points for a team to replace with its own, agreed before the delivery is inspected.

At a glance

The join integrity checklist with orphan event rates

The join integrity checklist with orphan event rates
CheckNumerator over denominatorHow to computeIllustrative warn / errorAction on failure
Orphan stepsSteps whose episode ID has no header, over all stepsAnti-join steps to episode headersAbove 0.1% / above 1%Find the export file or filter that dropped headers
Orphan tool resultsResults with no matching call on the composite key, over all resultsAnti-join results to calls on episode plus call IDAbove 0.1% / above 0.5%Check for reused IDs and results logged by another service
Unanswered tool callsCalls with no result, error or timeout record, over all callsAnti-join calls to results; exclude calls ended by a logged timeoutAbove 0.5% / above 2%Recover results or mark the episode as truncated
Duplicate parent keysParent keys appearing more than once, over distinct parent keysGroup by key with count above oneAny / above 0.1%Stop: every downstream join fans out until fixed
Missing keysRows with a null or empty join key, over all rows on that edgeSeparate null test, since many relationship tests skip nullsAbove 0.1% / above 1%Trace the producer that fails to set the key
Dangling artifact linksArtifact references that resolve to nothing in the package, over all referencesResolve each reference against the shipped manifestAbove 1% / above 5%Ship the artifact, a hash with a redaction code, or drop the step
Orphan outcomesOutcome records with no matching episode, over all outcome recordsAnti-join outcomes to episodes on case IDAbove 1% / above 3%Compare pseudonym mappings between the two extracts
Outcome coverage gapEpisodes with no outcome and no censoring code, over all episodesAnti-join episodes to outcomes and to censoring recordsAbove 2% / above 10%Request the outcome extract or a censoring reason per episode
Crosswalk missesKeys in system A with no crosswalk row to system B, over keys in AAnti-join A keys to the crosswalk, by monthAbove 1% / above 5%Look for a mapping change at the month the rate jumps
Order and offset violationsResults stamped before their call, or timestamps without an explicit offset, over all eventsCompare call and result times; parse offsetsAny / above 0.1%Normalize clocks and record the original offset field

Counting orphans so the rate cannot hide them

Report every check as a rate with its counts beside it, broken out by source system and month, because a healthy average can hide one broken month. A 0.3% overall orphan rate that is 0% for eleven months and near 4% for one is a migration bug, not noise.

Run each test in both directions. Child-to-parent finds orphans; parent-to-child finds coverage gaps, such as episodes that never received an outcome. They are different failures with different owners, and a dataset can pass one while failing the other.

Watch how your tooling treats nulls. dbt's relationships test, a common way to check referential integrity in warehouse pipelines, validates that every child record has a parent but automatically excludes null values, and its documentation says to add a separate not_null test if nulls should fail. A join key that was never populated therefore passes the orphan test, which is why missing keys are a separate row in the checklist.

Thresholds should be written down as configuration, not judgment on the day. dbt's tests return a count of failures and can be set to warn or error on conditions such as a count above a limit through warn_if and error_if. Whatever tool you use, store thresholds next to the checks, with the date they were agreed, so the acceptance decision can be audited later.

Timestamps need their own test because order checks are only as good as the offsets. RFC 9557, which updates RFC 3339, says Z should now indicate that the local offset is unknown, while +00:00 means UTC is the preferred reference. If two systems emit local times without offsets, a result can appear to precede its call; require explicit offsets and keep the original value.

Keep the failing rows, not just the rates. Every check should write its failures to a table with the record ID, the edge, the source file and the run date, so a reviewer can open twenty examples and see the pattern. Rates tell you where to look; the rows tell you why.

Replaying a sample: do the joins carry the right state and outcome?

Rates prove the joins exist; only replay proves they are the right joins. Draw a stratified sample of episodes, give a second team nothing but the shipped files and the documentation, and ask them to rebuild each episode's starting state, the sequence of tool calls and results, and the graded outcome.

At a glance

Replaying a sample: do the joins carry the right state and outcome

Stratify the sample by source system, month and outcome type, and over-sample the months where any orphan rate spiked. Thirty to fifty episodes is a practical illustrative size for a first pass; the point is to find systematic join errors, not to estimate a rate precisely.

For each sampled episode, the replay team records four things: whether every step attached to the right episode, whether each tool result matches the call it answers in content as well as ID, whether the outcome refers to the same case, and any step whose meaning depended on a file that was not shipped. Content mismatches, such as a refund result attached to a lookup call, reveal reused-ID errors that ID-level checks miss.

Write the replay result back into the acceptance record next to the rates. A dataset whose rates pass but whose replay fails is rejected for that edge, because the rates were measuring the wrong thing.

  • Pass: state, call and result sequence, and outcome rebuilt and consistent for every sampled episode.
  • Conditional pass: inconsistencies confined to one documented edge and month, with a fix and a re-run planned.
  • Fail: any content-level mismatch between a call and its result, or an outcome attached to the wrong case.

Illustrative audit: a help desk and billing trajectory set

This example is illustrative; the counts are invented to show the method and do not describe any real dataset. A buyer receives 2,000 support episodes assembled from a help desk, a billing system and an agent's tool logs, totalling about 41,000 events and 9,000 tool calls.

The overall numbers look acceptable: 0.2% orphan tool results, 0.6% unanswered calls and 1.4% orphan outcomes. Broken out by month, the orphan outcome rate is under 0.5% for eleven months and about 12% for one. The crosswalk miss rate jumps in the same month.

The supplier confirms that the pseudonymization mapping for billing accounts was regenerated that month, so outcome records from the later billing extract no longer matched help desk cases. Replay of eight sampled episodes from that month finds three outcomes attached to the wrong customer, which a rate check alone could not reveal.

The decision follows the written rule: the affected month fails, the rest pass. The buyer accepts eleven months, asks for that month to be re-linked with the original mapping, and requires the crosswalk to ship with a version field so a future change is visible.

When an orphan is legitimate rather than a defect

Some unmatched records are correct, and deleting them would bias the dataset. The rule is to keep them with a reason code and exclude them from the defect rate, not to drop them quietly.

Common legitimate cases include steps whose artifacts were redacted for privacy, tool calls to systems outside the licensed scope, records removed under a retention schedule before export, and human work done outside any logged system, such as a phone call. Each should carry a code such as redacted, out_of_scope, retention_deleted or offline_step, and the checklist should report coded and uncoded orphans separately.

The judgment call is when coded gaps are so frequent that the episode no longer teaches anything. If most of an episode's tool results are redacted, a model sees actions without observations. Set a per-episode limit, for example no more than an illustrative one in five steps coded, and move episodes over it to a separate evaluation-only or excluded set.

Reason codes should be assigned by the supplier's pipeline at the point the gap is created, not reconstructed afterwards by the buyer. A code added later is a guess about why a record is missing; a code written when a step was redacted or filtered is a fact about what happened, and only the second can be audited.

What to ask a supplier before accepting linked trajectories

Ask for the join map and the rates before you ask for a sample. A supplier that can produce the checklist with per-month counts has assembled the data deliberately; one that can only show example episodes has not yet measured it.

SourceX finds datasets for buyers among U.S. businesses on request rather than holding stock, and buyers describe scope, volume, format, timing and licensing needs through the request form, so join requirements like these can be stated up front. Delivery waits until the supplying company has approved the scope and signed, and from then on the de-identified dataset is licensed only on that agreement's terms.

Checklist · 6 items

  1. The join edge map, with the key used on each edge and how it was constructed.
  2. Per-edge orphan and coverage rates by source system and month, with counts.
  3. The pseudonymization approach and whether the same mapping was used for every extract.
  4. Timestamp sources, original offsets and any clock normalization applied.
  5. Reason codes for legitimate gaps, and their counts.
  6. Replay instructions sufficient for a team that has never seen the source systems.

0 of 6 ticked. Tick each item as you go.

Frequently asked questions

What is an orphan event in trajectory data?

An orphan event is a record whose parent cannot be found, such as a tool result with no matching call or a step whose episode header is missing. Orphan rates measure how often that happens on each join edge, relative to all records on that edge.

What orphan rate is acceptable in an agent trajectory dataset?

There is no published standard; the threshold depends on how the data will be used. A workable illustrative starting point treats any duplicate parent key as a blocker and keeps orphan rates on core edges well under one percent, but the important step is fixing thresholds in writing before inspecting the delivery.

Why join tool calls to results on more than the call ID?

Protocols such as MCP only require a request ID to be unique among requests still awaiting a response, so the same ID can recur later or in another session. Adding the episode or session ID and checking event order prevents pairing a result with the wrong call.

Do database foreign keys solve this?

They help inside one database, but trajectories are usually assembled from several systems and files where no constraint is enforced. Many referential tests also ignore null keys, so missing keys need their own check.

Is a replay check necessary if all orphan rates are zero?

Yes. A zero orphan rate shows every child found a parent, not that it found the right one. Replay catches content mismatches, such as a result attached to the wrong call after an ID was reused or a mapping changed.

Sources

  • MCP request IDs must not be null and must not match another of the sender's requests still awaiting a response; responses carry the request's ID. Source
  • Each tool_result names its tool_use block by tool_use_id, and tool results must immediately follow the turn that made the calls. Source
  • The gen_ai.tool.call.id attribute is set to the model or provider's tool call identifier only if one was supplied. Source
  • In OpenTelemetry, each span has zero or one parent, each trace has a single root span, and a child span shares its parent's trace ID. Source
  • dbt's relationships test checks referential integrity from child to parent and excludes null values, so not_null must be tested separately. Source
  • dbt tests return a failure count and can warn or error on conditional thresholds set with warn_if and error_if. Source
  • RFC 9557 updates RFC 3339 so that Z indicates an unknown local offset and +00:00 indicates UTC as the preferred reference. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies