AI uses for records
Agent trajectory failed tool calls: labeling errors so they teach recovery
By SourceX Editorial · Updated
10 sections
Short answer
Failed tool calls should give an agent trajectory dataset four linked things: the error exactly as the agent saw it, what the agent concluded, what it did next, and whether the recovery was verified in system state. Label each failed step with its fault owner and training use, so it can serve as context, a rejected example or a recovery to imitate, never as a mistake to copy.
Agent run
Agent trajectory failed tool
Key takeaways
- Keep the model-visible error text verbatim; many logs record an error class but not the message the agent read.
- Label who caused each failure before judging the recovery: agent arguments, tool choice, environment, permission or injected fault.
- Decide per step whether a failed call is imitated, contrasted, used as context only or excluded.
- Common SFT settings train on every assistant turn, so an unlabeled failed call becomes a target.
- A second team should reproduce each failure and reach the same labels from the pack alone.
What should failed tool calls contribute to an agent trajectory dataset?
A failed tool call should contribute four linked things: the error exactly as the agent saw it, what the agent concluded from it, what it did next and whether that recovery was verified against system state. Label each failed step with who caused the failure and how the step may be used in training, so the same trace can teach recovery without teaching the mistake.
Clean demonstrations show an agent that picks the right tool with the right arguments every time. Real agents spend a large share of their steps reading errors, empty results and partial successes, and that is where good and bad agents part ways. A dataset that drops failed calls, or keeps them without labels, either hides the most instructive steps or trains the model to repeat them.
This page defines a labeled error recovery trace pack: the step fields and label values, how each failed step should be used for imitation, contrast or context only, a worked example, and a replay test that a second team can run.
Why a clean demo hides the decisions that matter
In a clean demo the agent never has to interpret anything; in production it constantly does. Tool interfaces are designed with that in mind. The Model Context Protocol reports tool execution errors, such as API failures, input validation errors and business logic errors, inside the tool result with isError set to true, and says those errors carry actionable feedback that a model can use to self-correct and retry. It says clients should pass them to the model.
Anthropic's tool-use documentation makes the same assumption from the client side. A tool result can be flagged with is_error, the documentation recommends instructive error messages that say what went wrong and what to try next, and it notes that when a request is missing parameters Claude will retry two to three times with corrections before apologizing to the user. Those retries are behavior worth learning from, but only if the trace keeps the error text and the corrected call side by side.
A success-only trace loses three kinds of evidence: whether the agent read the error at all, whether its fix addressed the real cause, and whether it checked that a failed write really failed before retrying. Each is a separate skill, and none can be learned from steps where nothing went wrong.
Is the error in the trace the one the agent actually saw?
Often it is not, so check the capture layer before labeling anything. The trace must hold the model-visible error text verbatim, because a fix can only be judged against what the agent was told. Several common capture layers record something else.
OpenTelemetry's generative AI conventions, which are still marked as in development, define an execute_tool span with a conditionally required error.type attribute, described as a low-cardinality class of error such as timeout or 500. The opt-in gen_ai.tool.call.result attribute is described as the result returned if execution was successful, so a trace built only from these spans may record that a call failed without the message the model received. Benchmarks differ too: tau-bench converts a tool exception into an observation beginning with Error: followed by the exception text, which is what the agent sees, while a separate harness might log a stack trace the agent never saw.
| Capture layer | What it records about a failure | What can be missing | Fix |
|---|---|---|---|
| Model API transcript | tool_result with is_error and the content sent to the model | Upstream status code and headers | Store the raw tool response alongside the model-visible text |
| MCP client log | Protocol errors as JSON-RPC errors, execution errors as results with isError | Protocol errors the client chose not to show the model | Record whether each error was shown to the model |
| OpenTelemetry execute_tool span | error.type class on the span | The error message text, unless captured separately | Add the model-visible message as its own field |
| Benchmark environment | Observation string built from the exception | Original exception type and side effects | Log both, keyed by the same call ID |
| Application server log | Full stack trace and request | Which agent step caused it | Join on call ID or request ID before labeling |
The labeled error recovery trace pack: step fields and labels
Each failed step in the pack carries the fields below in addition to the ordinary step record (step ID, call ID, tool, arguments, response). The allowed values are deliberately short lists so two labelers can agree; anything that fits none of them goes to review rather than into a catch-all value.
At a glance
The labeled error recovery trace pack: step fields and labels
| Field | Allowed values | How to decide | Used for |
|---|---|---|---|
| error_seen | Verbatim model-visible text plus is_error flag | Copy from the transcript, never reconstruct | Judging every later label |
| fault_owner | agent_arguments, agent_tool_choice, environment, permission, injected | Would a correct agent with the same knowledge have avoided it? | Separating agent mistakes from bad luck |
| agent_reading | correct, misread, ignored | Compare the next action with what the error actually said | Training error interpretation |
| recovery_action | fix_arguments, reread_state, wait_retry, alternate_tool, ask_user, stop, false_success | Name the first action after the error | Recovery policy statistics |
| state_checked_before_retry | yes, no, not_applicable | Did the agent verify a write had not landed before repeating it? | Duplicate side-effect risk |
| recovery_verified | verified_in_state, unverified, failed | Check the system record, not the agent's message | Outcome credit |
| first_wrong_step | Step ID or none | Earliest step a correct agent would have done differently | Credit assignment |
| training_use | imitate, contrast, context_only, exclude | See the next section | Building SFT and preference sets |
| replay_mode | recorded_response, injected_fault with seed, live | How the failure will be reproduced | Replay test |
Should the model imitate, contrast or merely read a failed step?
Decide per step, because the default in common training tools is to learn from every assistant turn. Hugging Face TRL's SFT trainer, for example, can compute loss on assistant messages only, which still includes an assistant turn that issued a bad tool call. Leave a misread error in a conversational SFT example and the model is trained to misread it.
There are cleaner routes. TRL also supports prompt-completion datasets, where loss is computed on the completion by default and the prompt is context; placing the failed call and its error in the prompt and the recovery in the completion teaches the model to recover from the mistake without imitating it. Preference datasets pair a chosen and a rejected completion for the same prompt, which suits a failed step that had a clear correct alternative. Unpaired preference data, a single completion with a label, fits failed steps where no clean alternative was recorded.
| Step situation | training_use | How to package it | Why |
|---|---|---|---|
| Agent caused the error, then recovered correctly | context_only for the error, imitate for the recovery | Prompt-completion: error in prompt, recovery as completion | Teaches recovery without teaching the mistake |
| Agent caused the error and a correct alternative is known | contrast | Preference pair at that step: correct call chosen, failed call rejected | Gives a direct signal about the mistake |
| Environment failure, agent waited or retried correctly | imitate | Keep the whole span as a demonstration | Correct handling of bad luck is the target behavior |
| Agent claimed success after a failure | contrast | Unpaired example labeled not preferred, or pair with a verified-success reply | False success is the most costly pattern |
| Harness bug or unrealistic injected fault | exclude | Drop from training; keep in an audit file | Teaches nothing transferable |
Worked example: an empty search that led to a duplicate ticket (illustrative)
This example is invented to show the labels in use; it is not drawn from a real system or dataset. A support agent is asked to add a customer's new information to their open ticket. Its search tool filters by status using an enumerated field, and the agent passes a status value the tool does not accept. Instead of an error, the tool returns an empty list, a silent failure that a labeler has to catch from the arguments.
The agent concludes no ticket exists and creates one. Two steps later, a duplicate-detection rule in the ticketing sandbox returns an execution error naming the existing ticket. The agent reads it correctly, merges the new information into the original ticket, closes the duplicate and verifies both records. The labels show why this one trace yields both a contrast example and an imitation example.
Compare that with the clean demonstration a team might have recorded for the same request: one search with a valid status value, one update, done. It teaches the happy path in three steps and nothing else. The failed version teaches that an empty result is not proof of absence, that a duplicate-detection error points to the record that should have been found, and that a recovery is finished only when both records have been read back. None of those lessons exist in the clean run.
Notice also what the labeler had to do. Step 1 carried no error flag at all, so fault_owner and first_wrong_step were assigned by comparing the arguments with the tool schema. A pipeline that selects failed steps by is_error alone would have missed the step that caused the whole episode.
| Step | Call and response (illustrative) | Labels | Training use |
|---|---|---|---|
| 1 | search_tickets with an unsupported status value; returns an empty list | fault_owner agent_arguments; first_wrong_step 1 | contrast: correct status value chosen, this call rejected |
| 2 | Agent reasons that no ticket exists | agent_reading misread (empty result taken as absence) | context_only |
| 3 | create_ticket succeeds | Consequence of step 1; state now holds a duplicate | context_only |
| 4 | Duplicate rule returns isError naming the existing ticket | error_seen verbatim; agent_reading correct | context_only |
| 5 | Agent merges information into the original, closes the duplicate | recovery_action reread_state then fix; state_checked_before_retry yes | imitate |
| 6 | Agent reads both records back | recovery_verified verified_in_state | imitate |
Can a second team reproduce the failure and reach the same labels?
Only if the failure is recorded in a form that recurs on replay. Many tool failures are environmental, such as timeouts, rate limits and transient server errors, and will not happen again when the same call is reissued. Store either the recorded response so the replay serves it back, or an injected fault with its seed and trigger, and say which in replay_mode.
Store the state alongside each step as well. SWE-agent's trajectory files keep, for each step, the action, the observation that resulted and environment state extracted after the action, and its documentation says a query field holds the exact input the model saw for the next step. That is the level of detail a replay needs: the labeler of a failed step must see what the model saw.
Retries need one more check. HTTP semantics in RFC 9110 define idempotent methods by their intended effect on the server and caution against automatically retrying non-idempotent requests unless they are known to be safe. A trace that retries a failed create or payment call without checking state should be labeled state_checked_before_retry no, even if the retry happened to be harmless in that run.
- Give a second team the pack, the tool schemas and the replay environment, but not the labels.
- Have them replay each episode and confirm every failed step reproduces with the same model-visible error.
- Have them label fault_owner, agent_reading, recovery_action and first_wrong_step independently.
- Compare labels step by step and send disagreements to the task expert, not to a majority vote.
- Reject the pack if any recovery_verified value cannot be confirmed from replayed state.
What should a buyer check in a delivered trace pack?
Check the pack on a sample before accepting volume, and check structure before content: if the error text, call IDs and state are not linked, no amount of labeling will make the failed steps usable. Ask the supplier for a short summary of the pack as well as the files, so that the checks below can be run against stated numbers rather than impressions.
Most problems show up in the first few dozen failed steps. The most common are error text reconstructed after the fact, fault owners assigned by a script that only looked at status codes, recoveries marked successful from the agent's final message, and silent failures missing entirely because selection keyed on an error flag.
- Every failed step has a call ID that matches its response, and the error_seen text matches the transcript exactly.
- The pack reports the share of failed steps, the share injected rather than observed, and the distribution of fault owners.
- Silent failures, such as empty or partial results caused by bad arguments, appear and are labeled from arguments and later state.
- recovery_verified values point to a state record or read-back call, not to the agent's closing message.
- training_use values are present on every failed step, and contrast steps have a stated correct alternative.
- Error payloads went through the same de-identification as the main records, checked on a raw sample.
- A replay of a random sample of episodes reproduces each failed step with the same model-visible error.
Which failed calls are not worth keeping?
Keep failures that a deployed agent will meet and can handle better or worse; drop the rest. Repeated identical errors from one misconfigured tool add volume without new information, so keep a few and count the rest. Failures caused by the harness itself, such as a broken mock or an expired test credential, teach the model about your test rig rather than about the world, and belong in an audit file.
Injected faults are useful for coverage but distort the mix if overused. If most failures in a pack are injected, report the injection rate and keep evaluation splits separate so that results on injected faults are not read as results on real ones.
Error payloads also need their own privacy pass. Error messages often echo the input that caused them, including names, account numbers or internal hostnames, and they are easy to miss because de-identification rules are usually written for the main record fields. When failures involve regulated records, payments or credentials, narrow the scope and have the data owner's privacy and security reviewers look at a sample of raw error payloads before the pack is used.
Where real failure and recovery traces come from
Synthetic fault injection gives coverage, but realistic failure distributions come from systems where people and integrations already met errors and recovered: ticket histories with merged duplicates, integration run logs with retries, and finance records with corrected and resubmitted documents. Those records can seed both the faults to inject and the recoveries to imitate.
For trajectory builders who want such records, SourceX acts as the go-between: a business that agrees to license its error-and-recovery history has the rights review, contracting and preparation handled by SourceX, and the records themselves are located at U.S. businesses only after a data request has set out scope, volume, language, format, timing and licensing needs.
Frequently asked questions
Should failed tool calls be removed from agent training data?
Not removed, but labeled and packaged deliberately. A failed call is valuable as context for a recovery or as the rejected side of a preference pair; it is harmful only when left as an unlabeled assistant turn that the model is trained to imitate.
What is the difference between a protocol error and a tool execution error?
In the Model Context Protocol, protocol errors are JSON-RPC errors for problems such as unknown tools or malformed requests, while tool execution errors are returned inside the tool result with isError set to true. Execution errors are meant to be shown to the model so it can correct itself; protocol errors are less likely to be fixable by the model.
How do you label who caused a tool failure?
Ask whether a correct agent with the same information would have avoided it. If yes, the fault is the agent's arguments or tool choice; if no, it is the environment, a permission boundary or an injected fault. Record that as fault_owner before judging the recovery.
Why isn't an OpenTelemetry trace enough for failed tool calls?
The generative AI conventions record an error class on the execute_tool span, but the opt-in result attribute is described for successful executions, so the exact error text the model saw may be missing. Capture the model-visible message as its own field.
How many failed steps should a trajectory dataset contain?
There is no fixed ratio; it depends on the objective. Report the share of failed steps, the share that were injected rather than observed, and the distribution of fault owners, so buyers can judge whether the mix matches their deployment.
Can a silent failure, such as an empty result, count as a failed call?
Yes, and it is often the most instructive kind. Label it from the arguments and the later state rather than from an error flag, and mark the step where the agent drew the wrong conclusion as the first wrong step.
Sources
- The Model Context Protocol separates JSON-RPC protocol errors from tool execution errors returned with isError true, and says clients should give execution errors to the model so it can self-correct. Source
- Anthropic's tool-use documentation defines the is_error flag on tool results, recommends instructive error messages and notes Claude retries two to three times with corrections when parameters are missing. Source
- OpenTelemetry GenAI execute_tool spans record a conditionally required error.type, while the opt-in tool call result attribute is described for successful executions. Source
- tau-bench turns a tool exception into an observation that starts with Error: and the exception text. Source
- SWE-agent trajectory files keep each step's action, observation, extracted environment state and the exact query given to the model. Source
- TRL's SFT trainer can compute loss on assistant messages only, and computes loss on completions only by default for prompt-completion datasets. Source
- TRL documents prompt-completion, preference (chosen and rejected) and unpaired preference (completion with a label) dataset types. Source
- RFC 9110 defines idempotent methods by their intended effect on the server and cautions against automatically retrying non-idempotent requests. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.