Skip to content

AI uses for records

Failed agent trajectories dataset: when to buy failures, and how many

By SourceX Editorial · Updated

10 sections

Short answer

Include failed agent trajectories whenever the dataset will train a preference, contrastive or recovery objective, calibrate a grader or reward model, or evaluate whether an agent notices its own mistakes. Buy failures paired with successes on the same tasks, labeled by fault owner and failure class, with the first wrong step marked and enough state to replay the episode. Success-only sets fit plain imitation and little else.

Agent run

Failed agent trajectories

This page covers failed agent trajectories dataset: when to buy failures, and how many.Illustrative

Key takeaways

  • Failed trajectories matter for preference training, grader calibration, recovery training and failure detection evaluation.
  • Pair every failed episode with a verified success on the same task, or the failure cannot teach a contrast.
  • Assign a fault owner first: environment, user and grader faults are not agent negatives.
  • Use a short, fixed failure taxonomy so classes can be counted, balanced and audited.
  • Report the natural failure rate separately from the resampled training mix.
  • Deliver negatives in a separate, labeled file with replay instructions.

When should a trajectory dataset include failed attempts as well as successes?

A trajectory dataset should include failed attempts whenever the buyer needs the model to learn what not to do, to score attempts, or to recover. That covers preference and contrastive training, reward model and grader calibration, recovery training, and evaluation of failure detection. Plain behavior cloning on verified successes is the one common use where failures add little.

Published agent research supports keeping the failures. The authors of Learning From Failure (arXiv 2402.11651) note that the usual practice discards trajectories that do not finish the task, and report that adding negative examples, marked as such, improved results over training on positives alone. Their project README adds that marking negatives worked better than simply mixing them into the training set unmarked.

Exploration-based trajectory optimization (ETO, arXiv 2403.02502) goes further: it collects the agent's failed attempts during exploration, pairs each with a successful trajectory for the same task, and trains on those contrastive pairs with a preference method. In both cases the failure is only useful because it sits next to a success on a comparable task and carries a trustworthy label.

What does a success-only sample hide from the team that buys it?

A success-only sample hides the base rate, the grader's errors and the boundary between acceptable and unacceptable behavior. If a vendor delivers only episodes that passed, the buyer cannot tell whether those passes were one in two or one in twenty, and cannot test whether the success check itself was wrong.

The success label is whatever the checker says it is. In the tau-bench reference code, an episode's reward is set to zero when the database state after the agent's run does not hash to the state produced by replaying the ground-truth actions, or when a required output string is missing from the agent's replies. A different checker would draw the line elsewhere, so a failure label is only as good as the rule that produced it, and the buyer needs that rule in writing.

Success-only sets also erase near misses: episodes that reached the right end state by a path that broke a policy, or that stalled one step short. Those are often the most informative records for a reward model, because they sit right on the decision boundary the model has to learn.

Finally, success-only data gives a misleading picture of difficulty. Tasks that the source agent never solved disappear from the sample entirely, so a model trained and evaluated on that sample looks competent on exactly the tasks that were easy all along.

Which failure classes should label each failed episode?

Each failed episode should carry one fault owner and one primary failure class, chosen from a short, fixed list. The tau-bench project ships an automated error-identification tool that does exactly this split: it assigns the fault to the user, the agent or the environment, then classifies the fault type, using the classes goal partially completed, used wrong tool, used wrong tool argument and took unintended action. Its README warns that the tool relies on an LLM and may misidentify errors, so treat its output as a first pass for human review.

The taxonomy below extends that split for tool-calling business agents. It separates tool execution errors from protocol errors the way the Model Context Protocol specification does: protocol errors are standard JSON-RPC errors about the request itself, while tool execution errors come back inside the tool result with isError set to true and give the model something it can act on. For multi-agent systems, the MAST study (arXiv 2503.13657) offers a larger taxonomy of 14 failure modes in three categories, built from annotated traces across seven frameworks.

Which failure classes should label each failed episode?
Code and classEvidence that must appear in the traceUsual fault ownerTraining use
F1 Wrong tool selectedAction record names a tool that cannot satisfy the step; the correct tool exists in the tool list shown to the agentAgentNegative example for tool choice; preference pair with the successful call
F2 Wrong or missing argumentCall parameters differ from the values available in earlier observations; tool result reflects the wrong recordAgentArgument-level contrast; grader test for silent wrong-record updates
F3 Tool execution error ignoredTool result carries an error flag or message; the next action proceeds as if the call succeededAgentRecovery training; failure detection evaluation
F4 Malformed requestProtocol-level error such as an unknown tool or invalid parameters, returned outside the tool resultAgent or harnessSchema adherence training; check harness before blaming the model
F5 Unintended state changeEnd state differs from the reference state on a field the task did not ask to changeAgentHigh-severity negative; safety and irreversible-action evaluation
F6 Goal partially completedSome required end-state fields or outputs match the reference and others do notAgentPartial-credit grading; near-miss preference pairs
F7 Policy breach on a correct outcomeEnd state matches, but a written rule was skipped, such as an approval or identity checkAgentPolicy-following training; tests whether the grader checks process
F8 False success claimAgent's final message reports completion while the verified end state shows otherwiseAgentCalibration of self-reports; key negative for reward models
F9 Environment faultTimeout, outage or stale fixture recorded in tool responses before any agent mistakeEnvironmentKeep out of agent negatives; use for robustness tests only
F10 User or simulator faultSimulated or real user gave contradictory or impossible instructionsUserExclude from agent negatives; flag the task for repair

Why must the fault owner be assigned before the failure class?

Assign the fault owner before the failure class, because the owner decides whether the episode can be a negative example at all. A trajectory that failed because a sandbox API timed out teaches the agent nothing about its own policy; trained as a negative, it teaches the model to distrust behavior that was correct.

Add a fourth owner that most datasets forget: the grader. When adjudicators find that a failed episode actually met the business requirement, record it as a grader fault and keep it. A pile of grader-fault episodes is the best test set for the next version of the checker.

Keep the class list short and stable. OpenTelemetry's semantic conventions give the same advice for the error.type attribute: values should be predictable and low in cardinality, a fallback value of _OTHER covers cases the list does not define, and the attribute is not set at all when the operation succeeded. A failure taxonomy with 60 free-text reasons cannot be balanced, sampled or audited.

In practice, two reviewers should label a calibration batch independently before the full set is labeled. Where they disagree on the owner, the episode usually lacks a tool response or a timestamp that would settle it, which is a data gap to fix at the source rather than a judgment call to average away.

How do you keep failed episodes out of the wrong split?

Failed and successful attempts at the same task must land in the same split. If a task's success sits in training and its failure sits in the evaluation set, the model has effectively seen the answer to the test item, and a failure-detection score will look better than it is.

Split by task instance or task template first, then by time if the episodes span several months of agent versions. Within the evaluation split, keep the natural failure rate rather than the resampled training mix, so that measured precision and recall reflect what the deployed agent will meet.

Grader-fault episodes deserve their own holdout. They are the cases where the old checker was wrong, so they are the fairest test of whether a new checker or reward model has improved, and they lose that value the moment any of them is used to tune the checker.

What goes in the balanced sample specification a buyer hands a supplier?

A balanced sample specification fixes how failures and successes are drawn, paired and labeled, so two deliveries built from it can be compared. The ratios below are illustrative starting points for a supplier quote, not recommendations from any study; set the real values from the training objective and the observed failure rate in the source system.

At a glance

What goes in the balanced sample specification a buyer hands a supplier

Send the specification with the request, not after the first delivery. A supplier who knows the class floors and pairing rule in advance can sample from the full history; one who learns them later can only filter what was already exported, which usually leaves the rare classes empty.

What goes in the balanced sample specification a buyer hands a supplier?
ParameterSpecificationWhy it mattersAcceptance check
Sampling unitThe task instance, not the episode: every sampled task contributes all of its recorded attemptsKeeps pairs intact and stops easy tasks dominatingNo task id appears with only part of its attempts
Pairing ruleEach failed episode is paired with at least one verified success on the same task or a task from the same templateContrastive and preference methods need comparable pairsShare of failures with a pair at or above the agreed floor
Outcome mixIllustrative: 40 to 60 percent failures per task family, recorded alongside the natural failure rateTraining mix and the real-world base rate are different numbers; both must be knownNatural rate reported per family before resampling
Class floorsIllustrative: at least 30 episodes per agent-owned failure class in each task family, or the class is reported as thinRare classes vanish under proportional samplingPer-class counts table delivered with the data
Owner separationEnvironment, user and grader faults delivered in separate files, never mixed into agent negativesPrevents punishing correct agent behaviorZero F9 or F10 episodes in the agent-negative file
First divergenceStep index where the episode first departs from any valid solution path, set by a reviewerLets training target the step that mattered, not the whole episodePresent on every agent-owned failure
Near missesF6 and F7 episodes kept at their natural share or aboveThey define the boundary a reward model must learnCount of near misses per family reported
Outcome verificationEach label names the check used: end-state comparison, required output, human adjudicationA label without its check cannot be auditedCheck name and version on every episode
DeduplicationRetries of the same attempt collapsed unless the retry changed an actionDuplicate failures inflate class countsNear-identical action sequences flagged and reviewed

What does a second team need to replay a failed episode and reach the same verdict?

A second team can reconstruct a failed episode only if the dataset keeps the full chain of observation, action, tool response and outcome, plus the starting state and the version of every tool. Failures are harder to replay than successes, because the error often depends on a specific fixture, timeout or record that a fresh environment does not reproduce.

SWE-bench shows what re-executable outcome evidence looks like: each instance records the repository's base commit, the gold patch and test patch, and two test lists, FAIL_TO_PASS and PASS_TO_PASS, so anyone can rerun the tests and see the same verdict. Business trajectories need the equivalent: the reference end state, the comparison rule and the fixture snapshot.

  • Case id, episode id and a gap-free step index on every record
  • Actor on every step: agent, user, tool, environment or human reviewer
  • Timestamp on every step, in one time zone, with the clock source named
  • Observation exactly as the agent saw it, including truncation markers
  • Action with tool name, arguments and the call id that links it to its result
  • Tool response, including error flags, status codes and latency
  • Links to artifacts the episode touched, such as the record ids changed
  • Verified outcome, the check that produced it, and the fault owner and failure class
  • Replay instructions: environment snapshot id, tool versions and seed data needed to reach the same state

Illustrative example: sampling a refund-handling agent's episodes

Illustrative only; every figure here is invented. A buyer is training a customer-service agent that processes refund requests in a sandbox copy of an order system. The supplier holds 4,000 recorded episodes across 600 task instances, with a natural failure rate of 35 percent.

Applying the specification, the supplier first removes 180 environment-fault episodes (mostly sandbox timeouts) into a separate file and flags 25 episodes where reviewers judged the checker wrong. It then samples 300 task instances that each have at least one success and one agent-owned failure, giving 1,200 episodes at a 50 percent failure share. Class counts show F8 false success claims at only 14 episodes, below the illustrative floor of 30, so the delivery reports F8 as thin rather than padding it with weak examples.

The exception is worth stating. Episodes recorded against an older version of the refund tool, whose arguments have since changed, fail for reasons the current agent can no longer reproduce. They stay out of the training sample even though they are labeled correctly, because a negative example built on a retired interface teaches nothing about the deployed one.

Before signing off, the buyer replays 50 sampled failures in a fresh sandbox. In this invented case, 44 reproduce the same end state and verdict; the 6 that do not are traced to a seed record that had been edited after recording, and the supplier reissues them with the correct fixture.

When do failed trajectories stop being worth buying?

Failed trajectories stop paying for themselves when the failure cannot be attributed, cannot be replayed or comes from a setup unlike the buyer's deployment. Narrow the scope rather than accept a large file of unexplained failures.

Three conditions call for a specialist review before purchase. First, when a large share of failures carries no fault owner, an agent data engineer should audit a sample before any training run. Second, when the failures involve regulated actions such as payments, account changes or health records, a domain expert should confirm what counts as a policy breach. Third, when the supplier's success check was an LLM judge, the buyer should ask for an adjudicated subset that measures how often that judge was wrong.

Unmarked negatives are the other trap. Learning From Failure reports that its marked approach beat directly including negative samples, which is a reason to deliver failures in a separate, labeled file and let the training team decide how to use them.

A practical stopping rule helps here. If the supplier cannot name the check behind a label, cannot replay a sample, or cannot separate agent faults from environment faults, buy the successes now and defer the failures until the source logs are fixed. A smaller set of attributable failures is worth more than a large set that nobody can explain.

Where can paired successes and failures from real business records come from?

Many business systems already record failed and corrected attempts: reopened tickets, reversed journal entries, rejected claims that were later resubmitted, and orders cancelled after a wrong fulfillment step. Converted into trajectories, they give the success and failure pairs this specification asks for, with outcomes that were verified by what happened next.

Paired success and failure histories are not held in stock at SourceX; they are sourced on request from U.S. businesses. A buyer who needs paired trajectories can describe the scope, volume, format, timing and licensing terms through the request form, including the failure taxonomy and sample specification above as part of the request.

Frequently asked questions

Should failed agent trajectories be used for supervised fine-tuning?

Only with care. Mixed into a fine-tuning set without a marker, a failed trajectory looks like a demonstration to imitate. Research such as Learning From Failure marks negatives explicitly, and preference methods pair them with successes, so deliver failures in a separate labeled file.

What ratio of failed to successful trajectories should a dataset have?

There is no universal ratio. Record the natural failure rate in the source system first, then resample to the mix the training objective needs, with minimum counts per failure class. The 40 to 60 percent band in this article is an illustrative starting point for a quote, not a finding.

Who decides whether an episode failed?

A written check decides, and every label should name it: an end-state comparison, a required output, or human adjudication. When adjudicators overturn the check, record the episode as a grader fault rather than deleting it.

Are environment failures useless?

Not useless, but they are not agent negatives. Timeouts and outages recorded in tool responses are good material for robustness tests and for training recovery behavior, as long as they sit in their own file with the environment named as the fault owner.

How do I know a failed trajectory can be replayed?

Ask the supplier to replay a random sample in a fresh copy of the environment and report how many reached the same end state and verdict. Episodes that cannot be reproduced can still be useful for log-based analysis, but should be labeled as non-replayable.

Sources

  • Learning From Failure reports that standard practice discards unsuccessful agent trajectories and that adding marked negative examples improved performance over positives alone. Source
  • The NAT project README reports that its negative-aware method outperformed directly including negative samples. Source
  • ETO pairs failed exploration trajectories with successful ones and trains on the contrastive pairs with a preference method. Source
  • The MAST study defines 14 failure modes in three categories for multi-agent LLM systems, built from annotated traces across seven frameworks. Source
  • tau-bench's auto error identification assigns fault to the user, agent or environment and classifies fault types; the README warns that it uses an LLM and may be inaccurate. Source
  • tau-bench sets reward to zero when the post-run database hash differs from the ground-truth replay or a required output is missing. Source
  • MCP distinguishes protocol errors returned as JSON-RPC errors from tool execution errors reported in tool results with isError true. Source
  • OpenTelemetry says error.type should be predictable and low-cardinality, offers _OTHER as a fallback, and is not set on success. Source
  • SWE-bench instances record the base commit, patches and FAIL_TO_PASS and PASS_TO_PASS test lists. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies