Skip to content

AI uses for records

Agent trajectory failed tool calls: labeling errors so they teach recovery

By SourceX Editorial · Updated

10 sections

Short answer

Failed tool calls should give an agent trajectory dataset four linked things: the error exactly as the agent saw it, what the agent concluded, what it did next, and whether the recovery was verified in system state. Label each failed step with its fault owner and training use, so it can serve as context, a rejected example or a recovery to imitate, never as a mistake to copy.

Agent run

Agent trajectory failed tool

This page covers agent trajectory failed tool calls: labeling errors so they teach recovery.Illustrative

Key takeaways

  • Keep the model-visible error text verbatim; many logs record an error class but not the message the agent read.
  • Label who caused each failure before judging the recovery: agent arguments, tool choice, environment, permission or injected fault.
  • Decide per step whether a failed call is imitated, contrasted, used as context only or excluded.
  • Common SFT settings train on every assistant turn, so an unlabeled failed call becomes a target.
  • A second team should reproduce each failure and reach the same labels from the pack alone.

What should failed tool calls contribute to an agent trajectory dataset?

A failed tool call should contribute four linked things: the error exactly as the agent saw it, what the agent concluded from it, what it did next and whether that recovery was verified against system state. Label each failed step with who caused the failure and how the step may be used in training, so the same trace can teach recovery without teaching the mistake.

Clean demonstrations show an agent that picks the right tool with the right arguments every time. Real agents spend a large share of their steps reading errors, empty results and partial successes, and that is where good and bad agents part ways. A dataset that drops failed calls, or keeps them without labels, either hides the most instructive steps or trains the model to repeat them.

This page defines a labeled error recovery trace pack: the step fields and label values, how each failed step should be used for imitation, contrast or context only, a worked example, and a replay test that a second team can run.

Why a clean demo hides the decisions that matter

In a clean demo the agent never has to interpret anything; in production it constantly does. Tool interfaces are designed with that in mind. The Model Context Protocol reports tool execution errors, such as API failures, input validation errors and business logic errors, inside the tool result with isError set to true, and says those errors carry actionable feedback that a model can use to self-correct and retry. It says clients should pass them to the model.

Anthropic's tool-use documentation makes the same assumption from the client side. A tool result can be flagged with is_error, the documentation recommends instructive error messages that say what went wrong and what to try next, and it notes that when a request is missing parameters Claude will retry two to three times with corrections before apologizing to the user. Those retries are behavior worth learning from, but only if the trace keeps the error text and the corrected call side by side.

A success-only trace loses three kinds of evidence: whether the agent read the error at all, whether its fix addressed the real cause, and whether it checked that a failed write really failed before retrying. Each is a separate skill, and none can be learned from steps where nothing went wrong.

Is the error in the trace the one the agent actually saw?

Often it is not, so check the capture layer before labeling anything. The trace must hold the model-visible error text verbatim, because a fix can only be judged against what the agent was told. Several common capture layers record something else.

OpenTelemetry's generative AI conventions, which are still marked as in development, define an execute_tool span with a conditionally required error.type attribute, described as a low-cardinality class of error such as timeout or 500. The opt-in gen_ai.tool.call.result attribute is described as the result returned if execution was successful, so a trace built only from these spans may record that a call failed without the message the model received. Benchmarks differ too: tau-bench converts a tool exception into an observation beginning with Error: followed by the exception text, which is what the agent sees, while a separate harness might log a stack trace the agent never saw.

Is the error in the trace the one the agent actually saw?
Capture layerWhat it records about a failureWhat can be missingFix
Model API transcripttool_result with is_error and the content sent to the modelUpstream status code and headersStore the raw tool response alongside the model-visible text
MCP client logProtocol errors as JSON-RPC errors, execution errors as results with isErrorProtocol errors the client chose not to show the modelRecord whether each error was shown to the model
OpenTelemetry execute_tool spanerror.type class on the spanThe error message text, unless captured separatelyAdd the model-visible message as its own field
Benchmark environmentObservation string built from the exceptionOriginal exception type and side effectsLog both, keyed by the same call ID
Application server logFull stack trace and requestWhich agent step caused itJoin on call ID or request ID before labeling

The labeled error recovery trace pack: step fields and labels

Each failed step in the pack carries the fields below in addition to the ordinary step record (step ID, call ID, tool, arguments, response). The allowed values are deliberately short lists so two labelers can agree; anything that fits none of them goes to review rather than into a catch-all value.

At a glance

The labeled error recovery trace pack: step fields and labels

The labeled error recovery trace pack: step fields and labels
FieldAllowed valuesHow to decideUsed for
error_seenVerbatim model-visible text plus is_error flagCopy from the transcript, never reconstructJudging every later label
fault_owneragent_arguments, agent_tool_choice, environment, permission, injectedWould a correct agent with the same knowledge have avoided it?Separating agent mistakes from bad luck
agent_readingcorrect, misread, ignoredCompare the next action with what the error actually saidTraining error interpretation
recovery_actionfix_arguments, reread_state, wait_retry, alternate_tool, ask_user, stop, false_successName the first action after the errorRecovery policy statistics
state_checked_before_retryyes, no, not_applicableDid the agent verify a write had not landed before repeating it?Duplicate side-effect risk
recovery_verifiedverified_in_state, unverified, failedCheck the system record, not the agent's messageOutcome credit
first_wrong_stepStep ID or noneEarliest step a correct agent would have done differentlyCredit assignment
training_useimitate, contrast, context_only, excludeSee the next sectionBuilding SFT and preference sets
replay_moderecorded_response, injected_fault with seed, liveHow the failure will be reproducedReplay test

Should the model imitate, contrast or merely read a failed step?

Decide per step, because the default in common training tools is to learn from every assistant turn. Hugging Face TRL's SFT trainer, for example, can compute loss on assistant messages only, which still includes an assistant turn that issued a bad tool call. Leave a misread error in a conversational SFT example and the model is trained to misread it.

There are cleaner routes. TRL also supports prompt-completion datasets, where loss is computed on the completion by default and the prompt is context; placing the failed call and its error in the prompt and the recovery in the completion teaches the model to recover from the mistake without imitating it. Preference datasets pair a chosen and a rejected completion for the same prompt, which suits a failed step that had a clear correct alternative. Unpaired preference data, a single completion with a label, fits failed steps where no clean alternative was recorded.

Should the model imitate, contrast or merely read a failed step?
Step situationtraining_useHow to package itWhy
Agent caused the error, then recovered correctlycontext_only for the error, imitate for the recoveryPrompt-completion: error in prompt, recovery as completionTeaches recovery without teaching the mistake
Agent caused the error and a correct alternative is knowncontrastPreference pair at that step: correct call chosen, failed call rejectedGives a direct signal about the mistake
Environment failure, agent waited or retried correctlyimitateKeep the whole span as a demonstrationCorrect handling of bad luck is the target behavior
Agent claimed success after a failurecontrastUnpaired example labeled not preferred, or pair with a verified-success replyFalse success is the most costly pattern
Harness bug or unrealistic injected faultexcludeDrop from training; keep in an audit fileTeaches nothing transferable

Worked example: an empty search that led to a duplicate ticket (illustrative)

This example is invented to show the labels in use; it is not drawn from a real system or dataset. A support agent is asked to add a customer's new information to their open ticket. Its search tool filters by status using an enumerated field, and the agent passes a status value the tool does not accept. Instead of an error, the tool returns an empty list, a silent failure that a labeler has to catch from the arguments.

The agent concludes no ticket exists and creates one. Two steps later, a duplicate-detection rule in the ticketing sandbox returns an execution error naming the existing ticket. The agent reads it correctly, merges the new information into the original ticket, closes the duplicate and verifies both records. The labels show why this one trace yields both a contrast example and an imitation example.

Compare that with the clean demonstration a team might have recorded for the same request: one search with a valid status value, one update, done. It teaches the happy path in three steps and nothing else. The failed version teaches that an empty result is not proof of absence, that a duplicate-detection error points to the record that should have been found, and that a recovery is finished only when both records have been read back. None of those lessons exist in the clean run.

Notice also what the labeler had to do. Step 1 carried no error flag at all, so fault_owner and first_wrong_step were assigned by comparing the arguments with the tool schema. A pipeline that selects failed steps by is_error alone would have missed the step that caused the whole episode.

Worked example: an empty search that led to a duplicate ticket (illustrative)
StepCall and response (illustrative)LabelsTraining use
1search_tickets with an unsupported status value; returns an empty listfault_owner agent_arguments; first_wrong_step 1contrast: correct status value chosen, this call rejected
2Agent reasons that no ticket existsagent_reading misread (empty result taken as absence)context_only
3create_ticket succeedsConsequence of step 1; state now holds a duplicatecontext_only
4Duplicate rule returns isError naming the existing ticketerror_seen verbatim; agent_reading correctcontext_only
5Agent merges information into the original, closes the duplicaterecovery_action reread_state then fix; state_checked_before_retry yesimitate
6Agent reads both records backrecovery_verified verified_in_stateimitate

Can a second team reproduce the failure and reach the same labels?

Only if the failure is recorded in a form that recurs on replay. Many tool failures are environmental, such as timeouts, rate limits and transient server errors, and will not happen again when the same call is reissued. Store either the recorded response so the replay serves it back, or an injected fault with its seed and trigger, and say which in replay_mode.

Store the state alongside each step as well. SWE-agent's trajectory files keep, for each step, the action, the observation that resulted and environment state extracted after the action, and its documentation says a query field holds the exact input the model saw for the next step. That is the level of detail a replay needs: the labeler of a failed step must see what the model saw.

Retries need one more check. HTTP semantics in RFC 9110 define idempotent methods by their intended effect on the server and caution against automatically retrying non-idempotent requests unless they are known to be safe. A trace that retries a failed create or payment call without checking state should be labeled state_checked_before_retry no, even if the retry happened to be harmless in that run.

  • Give a second team the pack, the tool schemas and the replay environment, but not the labels.
  • Have them replay each episode and confirm every failed step reproduces with the same model-visible error.
  • Have them label fault_owner, agent_reading, recovery_action and first_wrong_step independently.
  • Compare labels step by step and send disagreements to the task expert, not to a majority vote.
  • Reject the pack if any recovery_verified value cannot be confirmed from replayed state.

What should a buyer check in a delivered trace pack?

Check the pack on a sample before accepting volume, and check structure before content: if the error text, call IDs and state are not linked, no amount of labeling will make the failed steps usable. Ask the supplier for a short summary of the pack as well as the files, so that the checks below can be run against stated numbers rather than impressions.

Most problems show up in the first few dozen failed steps. The most common are error text reconstructed after the fact, fault owners assigned by a script that only looked at status codes, recoveries marked successful from the agent's final message, and silent failures missing entirely because selection keyed on an error flag.

  • Every failed step has a call ID that matches its response, and the error_seen text matches the transcript exactly.
  • The pack reports the share of failed steps, the share injected rather than observed, and the distribution of fault owners.
  • Silent failures, such as empty or partial results caused by bad arguments, appear and are labeled from arguments and later state.
  • recovery_verified values point to a state record or read-back call, not to the agent's closing message.
  • training_use values are present on every failed step, and contrast steps have a stated correct alternative.
  • Error payloads went through the same de-identification as the main records, checked on a raw sample.
  • A replay of a random sample of episodes reproduces each failed step with the same model-visible error.

Which failed calls are not worth keeping?

Keep failures that a deployed agent will meet and can handle better or worse; drop the rest. Repeated identical errors from one misconfigured tool add volume without new information, so keep a few and count the rest. Failures caused by the harness itself, such as a broken mock or an expired test credential, teach the model about your test rig rather than about the world, and belong in an audit file.

Injected faults are useful for coverage but distort the mix if overused. If most failures in a pack are injected, report the injection rate and keep evaluation splits separate so that results on injected faults are not read as results on real ones.

Error payloads also need their own privacy pass. Error messages often echo the input that caused them, including names, account numbers or internal hostnames, and they are easy to miss because de-identification rules are usually written for the main record fields. When failures involve regulated records, payments or credentials, narrow the scope and have the data owner's privacy and security reviewers look at a sample of raw error payloads before the pack is used.

Where real failure and recovery traces come from

Synthetic fault injection gives coverage, but realistic failure distributions come from systems where people and integrations already met errors and recovered: ticket histories with merged duplicates, integration run logs with retries, and finance records with corrected and resubmitted documents. Those records can seed both the faults to inject and the recoveries to imitate.

For trajectory builders who want such records, SourceX acts as the go-between: a business that agrees to license its error-and-recovery history has the rights review, contracting and preparation handled by SourceX, and the records themselves are located at U.S. businesses only after a data request has set out scope, volume, language, format, timing and licensing needs.

Frequently asked questions

Should failed tool calls be removed from agent training data?

Not removed, but labeled and packaged deliberately. A failed call is valuable as context for a recovery or as the rejected side of a preference pair; it is harmful only when left as an unlabeled assistant turn that the model is trained to imitate.

What is the difference between a protocol error and a tool execution error?

In the Model Context Protocol, protocol errors are JSON-RPC errors for problems such as unknown tools or malformed requests, while tool execution errors are returned inside the tool result with isError set to true. Execution errors are meant to be shown to the model so it can correct itself; protocol errors are less likely to be fixable by the model.

How do you label who caused a tool failure?

Ask whether a correct agent with the same information would have avoided it. If yes, the fault is the agent's arguments or tool choice; if no, it is the environment, a permission boundary or an injected fault. Record that as fault_owner before judging the recovery.

Why isn't an OpenTelemetry trace enough for failed tool calls?

The generative AI conventions record an error class on the execute_tool span, but the opt-in result attribute is described for successful executions, so the exact error text the model saw may be missing. Capture the model-visible message as its own field.

How many failed steps should a trajectory dataset contain?

There is no fixed ratio; it depends on the objective. Report the share of failed steps, the share that were injected rather than observed, and the distribution of fault owners, so buyers can judge whether the mix matches their deployment.

Can a silent failure, such as an empty result, count as a failed call?

Yes, and it is often the most instructive kind. Label it from the arguments and the later state rather than from an error flag, and mark the step where the agent drew the wrong conclusion as the first wrong step.

Sources

  • The Model Context Protocol separates JSON-RPC protocol errors from tool execution errors returned with isError true, and says clients should give execution errors to the model so it can self-correct. Source
  • Anthropic's tool-use documentation defines the is_error flag on tool results, recommends instructive error messages and notes Claude retries two to three times with corrections when parameters are missing. Source
  • OpenTelemetry GenAI execute_tool spans record a conditionally required error.type, while the opt-in tool call result attribute is described for successful executions. Source
  • tau-bench turns a tool exception into an observation that starts with Error: and the exception text. Source
  • SWE-agent trajectory files keep each step's action, observation, extracted environment state and the exact query given to the model. Source
  • TRL's SFT trainer can compute loss on assistant messages only, and computes loss on completions only by default for prompt-completion datasets. Source
  • TRL documents prompt-completion, preference (chosen and rejected) and unpaired preference (completion with a label) dataset types. Source
  • RFC 9110 defines idempotent methods by their intended effect on the server and cautions against automatically retrying non-idempotent requests. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies