AI uses for records
How to test whether an agent notices a truncated tool result
By SourceX Editorial · Updated
10 sections
Short answer
To test whether an agent recognizes a truncated tool result, build tasks where the tool returns a realistically cut result and the correct outcome depends on the missing part. Freeze the data so the full answer is known, then score separately whether the agent called the result incomplete, whether its follow-up fit the truncation mechanism, and whether the records ended in the right state.
Agent run
Tool use dataset result
Key takeaways
- Truncation comes from timeouts, size caps, lower-bound counts, client output limits and silent text clipping, and each leaves a different signal.
- Unlike pagination, truncation rarely offers a way to continue the same call, so the right follow-up is to change the request or report a partial answer.
- A task set needs controls: a flag set with nothing lost, and a complete result that should not trigger extra calls.
- Score the completeness claim, the follow-up and the final store state separately, over repeated trials.
- Only a read of the system of record after the episode proves the outcome; the final message is a claim.
How do you test whether an agent recognizes a truncated tool result?
Build tasks in which the tool returns a result that is cut short in a realistic way, and in which the correct business outcome depends on the part that was cut. Freeze the data behind the tool so the full answer is known, then score three things separately: whether the agent said the result was incomplete, whether its next action was a sensible recovery, and whether the records in the system ended up in the right state.
The last of those is the one that matters to a business. An agent that writes a confident summary from the first part of a result can still produce a tidy final message, and a status field may say done. The test only means something when it checks the store after the episode and compares it with the state the full result would have required.
Five ways a tool result gets cut short in real systems
Truncation comes from several layers, and each leaves a different trace for the agent to notice. Some are explicit flags in the API response, some are lower-bound counts that look like exact ones, and some happen in the client before the model sees anything. A task set should include each kind, because an agent that handles one well can miss another entirely.
| Mechanism | Documented example | Signal the agent can see | What a careless agent concludes |
|---|---|---|---|
| Search time limit | GitHub's search API returns matches found before a query timeout and sets incomplete_results to true | A boolean in the response body | The matches listed are all the matches |
| Size-capped listing | GitHub's Git trees endpoint sets truncated to true when a recursive tree exceeds its size limits | A boolean next to the array | The repository contains only the listed files |
| Lower-bound count | Elasticsearch counts hits accurately up to 10,000 by default and reports a gte relation above that | A relation field beside the total | There are exactly 10,000 matching records |
| Client output cap | Claude Code warns when an MCP tool output exceeds 10,000 tokens and, above its cap, saves the result to a file and names the path | A message in place of the content | The message is the result |
| Server-side text cut | A tool that clips long text fields to fit a response, with or without a marker (illustrative) | A marker, an unclosed record or nothing at all | The clipped text is the whole record |
Truncation is not pagination, so the recovery differs
A paginated result hands the agent a way to continue the same call, such as a cursor or a next link; a truncated result usually does not. When a search times out or a listing hits a size cap, repeating the call returns the same cut, so the correct follow-up is to change the request: narrow the filter, split the date range, list one subtree at a time, read the saved file, or tell the user the answer is partial.
That makes truncation a test of judgment rather than of loop mechanics. The agent has to notice a signal that is easy to skip, infer that the missing part could change the answer, and pick a recovery that the tool contract actually supports. GitHub's own documentation adds a useful twist: a timeout does not necessarily mean results are missing, so a well-built task set includes cases where the flag is set but nothing was lost.
The result completeness and follow-up action task set
Use these eight task cards as a starting set. Each pairs a truncation mechanism with a business request, the signal the agent sees, the recovery a competent operator would choose, and the state check that decides the outcome. Values are illustrative; replace them with records and caps from the environment you are modelling.
| Task card | Request and tool behavior | Signal shown | Expected follow-up | State check |
|---|---|---|---|---|
| T1 Timed-out search | Close duplicate support tickets for one customer; search returns some matches before timeout | incomplete_results true | Re-run with a narrower filter per product or month until the flag clears | Every duplicate closed, no originals closed |
| T2 Flag set, nothing lost | Same request; search times out but all matches were already returned | incomplete_results true | Narrow once, confirm the same set, then act | Same as T1; extra calls are not penalized |
| T3 Capped tree | Find every config file that sets a deprecated flag; recursive listing is cut | truncated true | List subtrees one at a time | All affected files flagged in a tracking issue |
| T4 Lower-bound count | Report how many overdue invoices exceed a threshold | relation gte beside the total | Request an exact count or aggregate by segment | Reported figure equals the store count |
| T5 Client cap | Summarize a long contract export returned by an MCP tool | Message naming a saved file | Read the file in parts before summarizing | Summary cites clauses that appear only past the cap |
| T6 Silent text clip | Classify refund requests whose reason text was clipped without a marker | Record ends mid-sentence; length equals a round limit | Fetch the full record by ID before classifying | Labels match those set from full text |
| T7 Control | Same as T1 but the search completes | incomplete_results false | Act on the result without extra calls | Same as T1; no needless retries |
| T8 No recovery path | Same as T3 but the agent lacks permission to list subtrees | truncated true and an access error on narrower calls | Stop and report the answer as partial, naming what is missing | No changes made outside the visible set |
Scoring the claim, the follow-up and the final state separately
Score each episode on three axes and keep them apart in the report, because they fail independently. The completeness claim is whether the agent's statement about the result being whole or partial was true. The follow-up is whether its next action fit the mechanism. The state outcome is whether the records ended where the full answer required.
For the state check, follow the approach popularized by tau-bench, which grades an episode by comparing the database at the end of a conversation with an annotated goal state and introduced pass^k to measure consistency over repeated trials. Truncation failures are often intermittent, since timeouts and caps depend on load and output size, so repeated trials matter more here than in most tool-use tests.
- Recognized and recovered: claim true, follow-up fits, state correct.
- Recognized and reported partial: claim true, no recovery available or chosen, no wrong changes made; correct for T8.
- Missed: agent treated a cut result as complete; state wrong or incomplete.
- False alarm: agent declared a complete result partial and stalled or made needless calls; matters most on T7.
- Lucky pass: state correct but the agent never noticed the signal; record it, because it will fail when the cut moves.
Illustrative worked task: refunds checked against a capped order export
The records in this case are invented for illustration. A support operations agent is asked to flag every order from one merchant that qualifies for an automatic refund under a written policy, and to set a review flag on each. The order tool returns orders as text and the client caps tool output, so the full list is saved to a file and the model receives a short message naming the file path.
In the frozen snapshot, 140 orders match the merchant filter and 9 qualify, of which 6 appear only in the part of the output past the cap (counts illustrative). An agent that reads the message, opens the file in parts and applies the policy to all 140 sets 9 flags. An agent that treats the visible preview as the result flags 3 and reports success.
- Starting state: 140 matching orders, 9 qualifying, 0 flags set; tool version and client cap recorded.
- Evidence the agent should cite: the file path message, then order IDs from each part of the file it read.
- Applicable rule: the refund policy text supplied with the task.
- Decision: flag the qualifying orders and state that the full export was read.
- Outcome check: a query on the store after the episode returns exactly the 9 qualifying order IDs with the flag set.
- Counterexample: a final message saying all eligible orders were flagged while the store shows 3 flags; scored missed regardless of wording.
What proves the business state changed rather than the message?
Only a read of the system of record after the episode proves it. The agent's final message, a tool's success response and a status field are all claims; the test compares the store with the goal state computed from the full, untruncated data. Keep that query outside the agent's tools so the agent cannot influence what is checked.
Link each decision to the record that supports it. For every flag or change, the trace should show which tool call returned the record that justified it. When an action points at a record the agent could not have seen in the truncated output, the trace shows it recovered the missing part; when it cannot, the reviewer can see the agent guessed.
Building tool contracts that make truncation testable
Write the tool contracts so the truncation signal is part of the documented interface, then decide per task whether the agent sees it. The Model Context Protocol lets a tool declare an outputSchema; servers must then return structuredContent that conforms to it and clients should validate it, so a field such as truncated or incomplete can be declared rather than buried in text.
The same specification lets a tool return a resource link instead of inlining a large payload, and reports execution failures in the result with isError set to true so the model can adjust and retry. Both are useful recovery paths in a task set: a resource link turns an oversized result into a fetchable object, and an error result tests whether the agent reads the message before retrying the same call.
Record everything that sets a cap, because caps move between versions: client output limits, server page and size limits, query timeouts and the tool version. A truncation task without those settings cannot be rerun, and a result from a different cap is a different task.
Store the full, untruncated result next to each task as well. It is the reference for computing the goal state, and it lets a reviewer confirm that the part the agent missed actually changed the answer, which is the condition that makes the task worth scoring.
When a truncation test is the wrong instrument
Skip a dedicated truncation test when the harness never truncates, for example when results are always written to files and the agent must read them anyway; test file reading instead. It is also the wrong tool for measuring retrieval quality, because a search that returns the wrong records in full is a ranking problem, not a completeness one.
Be careful with live systems. Real timeouts and caps vary with load, so a task that truncates on one run may not on the next. Simulate the mechanism inside a frozen environment for scoring, and keep live runs for checking that the simulated signal matches what the real API sends.
Records that make truncation tasks realistic
Truncation tasks need business records large enough to hit real limits, with long text fields, many matches per query and a known correct outcome. Order histories, ticket archives, contract repositories and code configuration all qualify, provided the outcome of each case is recorded so the goal state can be computed.
Records of that kind can be requested through SourceX, which sources them from U.S. businesses on demand rather than from stock and takes care of rights review and preparation. A lab building a truncation task set can describe the record types, volumes and formats it needs through the request form.
Frequently asked questions
Why not just raise the output limit so nothing is truncated?
Limits exist for cost, latency and context size, and agents meet them in production anyway. Raising them hides the behavior you need to measure. Keep at least one task family where the cap is realistic and the signal is the only clue.
Is a truncated result the same as a tool error?
No. A truncated result usually succeeds and returns valid data, just not all of it, while an execution error reports that the call failed. The agent should treat them differently: recover the missing part for truncation, and read the error before retrying for a failure.
Should agents always retry when they see an incomplete flag?
Not blindly. Repeating the same call often returns the same cut, so the better move is to change the request. GitHub notes that a timed-out search may in fact be complete, so a good agent narrows once and confirms rather than looping.
How many trials does a truncation task need?
Enough to see consistency, because missed truncation tends to be intermittent. Report a pass^k style figure across repeated runs rather than a single pass rate, and log the caps and versions for every run.
What should the agent say when it cannot recover the missing part?
It should state that the result is partial, name what is missing and why, and avoid acting on records it could not see. In scoring, that is a correct outcome when no recovery path exists.
Sources
- GitHub's REST search API limits how long a query can run, returns the matches already found when a query times out with incomplete_results set to true, and notes that a timeout does not necessarily mean results are incomplete. Source
- GitHub's Git trees endpoint returns truncated set to true when the tree array exceeds its maximum size, and recommends fetching subtrees non-recursively. Source
- Elasticsearch counts total hits accurately up to 10,000 by default and marks the total with a gte relation when it is a lower bound. Source
- Claude Code shows a warning when an MCP tool output exceeds 10,000 tokens and, for results above its output cap, saves the result to a file and replaces it in the conversation with a message naming the file path. Source
- The MCP tools specification lets a tool declare an outputSchema that structuredContent must conform to, return resource links instead of inline data, and report execution errors in the result with isError set to true. Source
- tau-bench evaluates agents by comparing the database state at the end of a conversation with an annotated goal state and introduces pass^k for reliability across trials. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.