Skip to content

AI uses for records

How to test whether an agent notices a truncated tool result

By SourceX Editorial · Updated

10 sections

Short answer

To test whether an agent recognizes a truncated tool result, build tasks where the tool returns a realistically cut result and the correct outcome depends on the missing part. Freeze the data so the full answer is known, then score separately whether the agent called the result incomplete, whether its follow-up fit the truncation mechanism, and whether the records ended in the right state.

Agent run

Tool use dataset result

This page covers how to test whether an agent notices a truncated tool result.Illustrative

Key takeaways

  • Truncation comes from timeouts, size caps, lower-bound counts, client output limits and silent text clipping, and each leaves a different signal.
  • Unlike pagination, truncation rarely offers a way to continue the same call, so the right follow-up is to change the request or report a partial answer.
  • A task set needs controls: a flag set with nothing lost, and a complete result that should not trigger extra calls.
  • Score the completeness claim, the follow-up and the final store state separately, over repeated trials.
  • Only a read of the system of record after the episode proves the outcome; the final message is a claim.

How do you test whether an agent recognizes a truncated tool result?

Build tasks in which the tool returns a result that is cut short in a realistic way, and in which the correct business outcome depends on the part that was cut. Freeze the data behind the tool so the full answer is known, then score three things separately: whether the agent said the result was incomplete, whether its next action was a sensible recovery, and whether the records in the system ended up in the right state.

The last of those is the one that matters to a business. An agent that writes a confident summary from the first part of a result can still produce a tidy final message, and a status field may say done. The test only means something when it checks the store after the episode and compares it with the state the full result would have required.

Five ways a tool result gets cut short in real systems

Truncation comes from several layers, and each leaves a different trace for the agent to notice. Some are explicit flags in the API response, some are lower-bound counts that look like exact ones, and some happen in the client before the model sees anything. A task set should include each kind, because an agent that handles one well can miss another entirely.

Five ways a tool result gets cut short in real systems
MechanismDocumented exampleSignal the agent can seeWhat a careless agent concludes
Search time limitGitHub's search API returns matches found before a query timeout and sets incomplete_results to trueA boolean in the response bodyThe matches listed are all the matches
Size-capped listingGitHub's Git trees endpoint sets truncated to true when a recursive tree exceeds its size limitsA boolean next to the arrayThe repository contains only the listed files
Lower-bound countElasticsearch counts hits accurately up to 10,000 by default and reports a gte relation above thatA relation field beside the totalThere are exactly 10,000 matching records
Client output capClaude Code warns when an MCP tool output exceeds 10,000 tokens and, above its cap, saves the result to a file and names the pathA message in place of the contentThe message is the result
Server-side text cutA tool that clips long text fields to fit a response, with or without a marker (illustrative)A marker, an unclosed record or nothing at allThe clipped text is the whole record

Truncation is not pagination, so the recovery differs

A paginated result hands the agent a way to continue the same call, such as a cursor or a next link; a truncated result usually does not. When a search times out or a listing hits a size cap, repeating the call returns the same cut, so the correct follow-up is to change the request: narrow the filter, split the date range, list one subtree at a time, read the saved file, or tell the user the answer is partial.

That makes truncation a test of judgment rather than of loop mechanics. The agent has to notice a signal that is easy to skip, infer that the missing part could change the answer, and pick a recovery that the tool contract actually supports. GitHub's own documentation adds a useful twist: a timeout does not necessarily mean results are missing, so a well-built task set includes cases where the flag is set but nothing was lost.

The result completeness and follow-up action task set

Use these eight task cards as a starting set. Each pairs a truncation mechanism with a business request, the signal the agent sees, the recovery a competent operator would choose, and the state check that decides the outcome. Values are illustrative; replace them with records and caps from the environment you are modelling.

The result completeness and follow-up action task set
Task cardRequest and tool behaviorSignal shownExpected follow-upState check
T1 Timed-out searchClose duplicate support tickets for one customer; search returns some matches before timeoutincomplete_results trueRe-run with a narrower filter per product or month until the flag clearsEvery duplicate closed, no originals closed
T2 Flag set, nothing lostSame request; search times out but all matches were already returnedincomplete_results trueNarrow once, confirm the same set, then actSame as T1; extra calls are not penalized
T3 Capped treeFind every config file that sets a deprecated flag; recursive listing is cuttruncated trueList subtrees one at a timeAll affected files flagged in a tracking issue
T4 Lower-bound countReport how many overdue invoices exceed a thresholdrelation gte beside the totalRequest an exact count or aggregate by segmentReported figure equals the store count
T5 Client capSummarize a long contract export returned by an MCP toolMessage naming a saved fileRead the file in parts before summarizingSummary cites clauses that appear only past the cap
T6 Silent text clipClassify refund requests whose reason text was clipped without a markerRecord ends mid-sentence; length equals a round limitFetch the full record by ID before classifyingLabels match those set from full text
T7 ControlSame as T1 but the search completesincomplete_results falseAct on the result without extra callsSame as T1; no needless retries
T8 No recovery pathSame as T3 but the agent lacks permission to list subtreestruncated true and an access error on narrower callsStop and report the answer as partial, naming what is missingNo changes made outside the visible set

Scoring the claim, the follow-up and the final state separately

Score each episode on three axes and keep them apart in the report, because they fail independently. The completeness claim is whether the agent's statement about the result being whole or partial was true. The follow-up is whether its next action fit the mechanism. The state outcome is whether the records ended where the full answer required.

For the state check, follow the approach popularized by tau-bench, which grades an episode by comparing the database at the end of a conversation with an annotated goal state and introduced pass^k to measure consistency over repeated trials. Truncation failures are often intermittent, since timeouts and caps depend on load and output size, so repeated trials matter more here than in most tool-use tests.

  • Recognized and recovered: claim true, follow-up fits, state correct.
  • Recognized and reported partial: claim true, no recovery available or chosen, no wrong changes made; correct for T8.
  • Missed: agent treated a cut result as complete; state wrong or incomplete.
  • False alarm: agent declared a complete result partial and stalled or made needless calls; matters most on T7.
  • Lucky pass: state correct but the agent never noticed the signal; record it, because it will fail when the cut moves.

Illustrative worked task: refunds checked against a capped order export

The records in this case are invented for illustration. A support operations agent is asked to flag every order from one merchant that qualifies for an automatic refund under a written policy, and to set a review flag on each. The order tool returns orders as text and the client caps tool output, so the full list is saved to a file and the model receives a short message naming the file path.

In the frozen snapshot, 140 orders match the merchant filter and 9 qualify, of which 6 appear only in the part of the output past the cap (counts illustrative). An agent that reads the message, opens the file in parts and applies the policy to all 140 sets 9 flags. An agent that treats the visible preview as the result flags 3 and reports success.

  • Starting state: 140 matching orders, 9 qualifying, 0 flags set; tool version and client cap recorded.
  • Evidence the agent should cite: the file path message, then order IDs from each part of the file it read.
  • Applicable rule: the refund policy text supplied with the task.
  • Decision: flag the qualifying orders and state that the full export was read.
  • Outcome check: a query on the store after the episode returns exactly the 9 qualifying order IDs with the flag set.
  • Counterexample: a final message saying all eligible orders were flagged while the store shows 3 flags; scored missed regardless of wording.

What proves the business state changed rather than the message?

Only a read of the system of record after the episode proves it. The agent's final message, a tool's success response and a status field are all claims; the test compares the store with the goal state computed from the full, untruncated data. Keep that query outside the agent's tools so the agent cannot influence what is checked.

Link each decision to the record that supports it. For every flag or change, the trace should show which tool call returned the record that justified it. When an action points at a record the agent could not have seen in the truncated output, the trace shows it recovered the missing part; when it cannot, the reviewer can see the agent guessed.

Building tool contracts that make truncation testable

Write the tool contracts so the truncation signal is part of the documented interface, then decide per task whether the agent sees it. The Model Context Protocol lets a tool declare an outputSchema; servers must then return structuredContent that conforms to it and clients should validate it, so a field such as truncated or incomplete can be declared rather than buried in text.

The same specification lets a tool return a resource link instead of inlining a large payload, and reports execution failures in the result with isError set to true so the model can adjust and retry. Both are useful recovery paths in a task set: a resource link turns an oversized result into a fetchable object, and an error result tests whether the agent reads the message before retrying the same call.

Record everything that sets a cap, because caps move between versions: client output limits, server page and size limits, query timeouts and the tool version. A truncation task without those settings cannot be rerun, and a result from a different cap is a different task.

Store the full, untruncated result next to each task as well. It is the reference for computing the goal state, and it lets a reviewer confirm that the part the agent missed actually changed the answer, which is the condition that makes the task worth scoring.

When a truncation test is the wrong instrument

Skip a dedicated truncation test when the harness never truncates, for example when results are always written to files and the agent must read them anyway; test file reading instead. It is also the wrong tool for measuring retrieval quality, because a search that returns the wrong records in full is a ranking problem, not a completeness one.

Be careful with live systems. Real timeouts and caps vary with load, so a task that truncates on one run may not on the next. Simulate the mechanism inside a frozen environment for scoring, and keep live runs for checking that the simulated signal matches what the real API sends.

Records that make truncation tasks realistic

Truncation tasks need business records large enough to hit real limits, with long text fields, many matches per query and a known correct outcome. Order histories, ticket archives, contract repositories and code configuration all qualify, provided the outcome of each case is recorded so the goal state can be computed.

Records of that kind can be requested through SourceX, which sources them from U.S. businesses on demand rather than from stock and takes care of rights review and preparation. A lab building a truncation task set can describe the record types, volumes and formats it needs through the request form.

Frequently asked questions

Why not just raise the output limit so nothing is truncated?

Limits exist for cost, latency and context size, and agents meet them in production anyway. Raising them hides the behavior you need to measure. Keep at least one task family where the cap is realistic and the signal is the only clue.

Is a truncated result the same as a tool error?

No. A truncated result usually succeeds and returns valid data, just not all of it, while an execution error reports that the call failed. The agent should treat them differently: recover the missing part for truncation, and read the error before retrying for a failure.

Should agents always retry when they see an incomplete flag?

Not blindly. Repeating the same call often returns the same cut, so the better move is to change the request. GitHub notes that a timed-out search may in fact be complete, so a good agent narrows once and confirms rather than looping.

How many trials does a truncation task need?

Enough to see consistency, because missed truncation tends to be intermittent. Report a pass^k style figure across repeated runs rather than a single pass rate, and log the caps and versions for every run.

What should the agent say when it cannot recover the missing part?

It should state that the result is partial, name what is missing and why, and avoid acting on records it could not see. In scoring, that is a correct outcome when no recovery path exists.

Sources

  • GitHub's REST search API limits how long a query can run, returns the matches already found when a query times out with incomplete_results set to true, and notes that a timeout does not necessarily mean results are incomplete. Source
  • GitHub's Git trees endpoint returns truncated set to true when the tree array exceeds its maximum size, and recommends fetching subtrees non-recursively. Source
  • Elasticsearch counts total hits accurately up to 10,000 by default and marks the total with a gte relation when it is a lower bound. Source
  • Claude Code shows a warning when an MCP tool output exceeds 10,000 tokens and, for results above its output cap, saves the result to a file and replaces it in the conversation with a message naming the file path. Source
  • The MCP tools specification lets a tool declare an outputSchema that structuredContent must conform to, return resource links instead of inline data, and report execution errors in the result with isError set to true. Source
  • tau-bench evaluates agents by comparing the database state at the end of a conversation with an annotated goal state and introduces pass^k for reliability across trials. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies