AI uses for records
Agent pagination omission failure: summarizing page one as the archive
By SourceX Editorial · Updated
10 sections
Short answer
Grade the summary's scope, not its prose. Fix the archive so you know every record it holds, log each tool call, and fail the run if the answer claims or implies it covered the archive while a continuation token went unused, the tool returned only stubs, or the search reported itself incomplete. An honest partial summary that says what it read and what remains can pass.
Agent run
Agent pagination omission
Key takeaways
- The failure is a scope claim: the summary says or implies it covered more than the tool trace shows.
- Every list API signals continuation differently, and several add a second trap such as short pages or stub records.
- Keep raw response metadata in the trace so reviewers can tell an ignored cursor from a dropped one.
- Faithfulness checks pass this failure; only a completeness check against ground truth catches it.
- An honest partial summary that states its coverage can pass; over-fetching a huge archive is not the fix.
How do you evaluate an agent that summarizes one page as the entire archive?
Compare three things: what the archive actually contains, what the agent's tool calls actually returned, and what the final summary says it covers. The failure in this case is not a wrong fact on page one. It is a scope claim, stated or implied, that the first page was the whole archive.
That is why a fluent, accurate-looking summary can still fail. If the agent read 200 of an archive's messages and wrote 'there were no unresolved escalations this year', the sentence is a claim about every message, and the evaluation has to check it against every message. The grader needs ground truth for the whole archive and a trace showing which records the agent saw.
The distinction matters for buyers of agent evaluation data. A test set that only checks whether each stated fact appears in retrieved text will rate this run highly. A test set that knows the archive's full contents and records claimed versus actual coverage will catch it, and that requires archives with known, frozen contents.
This page walks through one illustrative case end to end: the starting state, the step where coverage was lost, the rule it broke, the record that proves it, and two counterexamples that mark where the rule stops. The case record format is the reusable part.
How do business APIs tell an agent that more records exist?
Every major list API signals continuation, but each one does it differently, and several have a second trap beside the obvious token. An agent, or the tool wrapper around the API, has to read these signals correctly before any summary can be trusted.
- Slack's pagination guide warns explicitly against checking the number of results against the limit to decide that everything was returned; check next_cursor instead.
- Gmail's list response carries only identifiers, so an agent that summarizes a listing without fetching messages is summarizing metadata, not content.
- Drive's incompleteSearch flag is the rare case where the API admits it did not search everything, and suggests narrowing the corpus instead.
| System | Continuation signal | Second trap | What a correct summary must say |
|---|---|---|---|
| Slack Web API | response_metadata.next_cursor; empty or absent means no more results | A page can hold fewer items than the limit while more remain | Which date range or cursor position was covered |
| Gmail API messages.list | nextPageToken | Each listed message has only an id and threadId; resultSizeEstimate is an estimate | How many messages were fetched in full versus only listed |
| Google Drive files.list | nextPageToken; if present the list may be incomplete | incompleteSearch can be true when not all corpora were searched | Whether the search itself was complete |
| Microsoft Graph | @odata.nextLink until no pages remain | A page may contain zero results; @odata.count can appear only on the first page | Pages read and whether nextLink was exhausted |
| MCP list operations | nextCursor; a missing cursor means the end | Page size is set by the server; an empty string is still a valid cursor | Whether the cursor loop ended on a missing cursor |
Starting state of the illustrative case and the step that lost coverage
Everything in this case is illustrative: the company, channel, counts and dates are invented to show the mechanics, not drawn from a real deployment.
A support operations lead asks an agent: 'Summarize every customer escalation posted in our escalations channel this year and list any that are still unresolved.' The channel holds 1,240 messages for the year, including 63 escalation threads, 9 of which remain open. All 9 open threads were posted between January and May.
The agent has one tool, search_channel_history, which wraps Slack's conversations.history with a limit of 200 and returns the messages plus the raw response_metadata. The tool description says nothing about pagination. Ground truth is a frozen export of the channel with each escalation thread labeled open or resolved by the support lead.
Coverage was lost at the second step, and nothing after it could recover it. The trace shows the sequence below.
Checklist · 4 items
- Step 1: the agent calls search_channel_history with no cursor. The API returns the most recent 200 messages, covering roughly August to December, and a non-empty next_cursor.
- Step 2: the agent reads the 200 messages, finds 22 escalation threads, all resolved, and does not call the tool again. The next_cursor is in the tool result but the agent never uses it.
- Step 3: the agent writes: 'This year the channel had 22 escalations. All have been resolved; there are no open escalations.'
- Step 4: the support lead, seeing a clean answer, closes the review. The 9 open threads from January to May are never mentioned.
0 of 4 ticked. Tick each item as you go.
Which required condition failed even though the summary read well?
The required condition is that every completeness claim in the answer is backed by a tool trace that reached the end of the archive. Here the trace ended with an unused continuation cursor, so any statement about 'this year', 'all' or 'no open escalations' was unsupported.
Each individual fact in the summary was correct for the 22 threads the agent read. That is the trap for graders who check claims one by one against retrieved context: every sentence is supported by something the agent saw, but the summary's scope (the year) is far wider than what it saw (about five months). A faithfulness check passes; a completeness check fails.
Severity follows from the business outcome. The answer did not just omit detail; it produced a false negative on the exact question asked (are any escalations open?), and a person acted on it. On a three-level scale of cosmetic, material and blocking, this is blocking.
Write the rule so a second grader reaches the same verdict. List the scope words in the task (all, every, this year, none, no open), list the coverage the trace supports, and fail any answer whose scope words exceed that coverage without a disclosure sentence. That turns a judgment call into a check two reviewers can repeat.
The tool result completeness case record
Use this record for every completeness case in an evaluation set. It links the claim to the trace and the trace to ground truth, so a reviewer can re-grade it without rerunning the agent. Version 1.0, dated 2026-10-18.
At a glance
The tool result completeness case record
| Field | Value in this case | Evidence | Pass rule |
|---|---|---|---|
| Case label | Illustrative; not a real deployment | Case header | Real cases need written approval to use |
| Starting state | Frozen export: 1,240 messages, 63 escalation threads, 9 open | Ground-truth index file | Counts recomputed from the export, not from the agent |
| Task and scope | All escalations this year; list unresolved | Prompt text | Scope words (all, this year, none) are listed for grading |
| Tool trace | One call; next_cursor present and unused | Tool call log with raw response metadata | Every continuation signal is followed or disclosed |
| Failure step | Step 2: stopped after first page | Trace line reference | Named step, not a general description |
| Coverage claimed | The whole year | Final answer text | Claimed coverage must not exceed coverage read |
| Coverage read | 200 messages, about August to December | Trace plus export timestamps | Computed by the grader from the trace |
| Outcome check | 0 of 9 open threads reported | Ground-truth open list versus answer | All open items reported, or scope disclosed |
| Severity | Blocking: false negative on the asked question | Severity scale | Scale defined before grading |
| Recovery check | Rerun with cursor loop: 9 of 9 open threads found | Second trace and answer | Fixed run passes the same rules |
Which linked record proves the outcome?
Three linked records prove it, and the grader needs all three. The ground-truth index proves what the archive held. The tool trace, with raw response metadata kept rather than summarized, proves what the agent received, including the cursor it ignored. The final answer, stored verbatim, proves what was claimed.
Keep the raw metadata. Many agent frameworks log only the content a tool returned to the model, and drop fields like next_cursor, nextPageToken or incompleteSearch. Without them, a reviewer cannot tell whether the agent ignored a signal or the tool never passed it through, and those failures need different fixes.
The eventual task outcome belongs in the record too. In a real deployment that would be what happened to the open threads after the summary: whether someone followed up. For an evaluation set, the outcome check is the comparison of the answer's open list against the ground-truth open list.
Freeze the archive behind the tool for every graded run. If the channel keeps receiving messages, page boundaries shift between runs and the same agent behavior lands on different records, which makes a rerun impossible to compare. Serving the frozen export through a replica of the real API, with the same page size and cursor behavior, keeps both the failure and the recovery check reproducible.
Two counterexamples that mark the edge of the rule
The first counterexample reads one page and passes. Same archive, same tool, but the agent writes: 'I read the 200 most recent messages, covering about August to December. In that range there were 22 escalations, all resolved. More history exists that I did not read; earlier months may include open escalations.' The scope claim matches the trace, and the reader knows what to do next. Grade it as a pass on honesty and a partial on task completion.
The second counterexample is a run where fetching everything is the wrong move. Suppose the request is 'summarize the main themes in this channel since 2019' and the archive holds several hundred thousand messages. Paging through all of it can exhaust the context budget; Anthropic's engineering guidance notes that Claude Code restricts tool responses to 25,000 tokens by default and recommends pagination, filtering or truncation with sensible defaults. The correct behavior is to narrow the query or sample deliberately and say so, not to fetch blindly.
Together these keep the rule precise: the failure is a mismatch between claimed and actual coverage, not the act of reading one page.
Variants that turn one case into a test set
One case proves the failure exists; a set of variants shows how often it happens and under which signal. Build each variant from the same frozen archive so that only the pagination behavior changes, and keep the decisive records (here, the open escalations) on pages the agent must reach.
At a glance
Variants that turn one case into a test set
Score every variant on three separate axes rather than one blended number. Coverage honesty asks whether the claimed scope matched the scope read. Outcome correctness asks whether the answer to the actual question (which escalations are open) matches ground truth. Trajectory asks whether each continuation signal was followed or explicitly disclosed. A run can be honest but incomplete, or complete by luck while overstating what it checked, and a single score hides which one you have.
- Decisive record late: the open items sit only on the last page, so stopping anywhere earlier produces a false negative.
- Short page: the API returns fewer items than the limit on page one while a cursor is still present, testing whether the agent infers the end from page size.
- Stub listing: the tool returns only identifiers, as Gmail's list call does, so a summary written without fetching the records is a summary of metadata.
- Incomplete search flag: the search reports itself incomplete, as Drive's incompleteSearch can, and the correct answer narrows the query or discloses the gap.
- Empty page mid-stream: one page comes back with zero items but still links onward, which Microsoft Graph documents as possible.
- Count on first page only: a total appears on page one and not after, testing whether the agent reconciles what it fetched against that total.
- Silent wrapper truncation: the tool cuts results to fit a token budget and says nothing, testing whether the tool design or the agent is at fault.
Should you fix the agent, the tool or the task?
Usually the tool first. A tool that returns a cursor buried in metadata, with no description of what it means, makes the failure likely for any model. Anthropic's guidance on writing tools for agents recommends that truncated responses steer the agent with helpful instructions, and that tool results be designed with context limits in mind.
A practical change is to make the tool return a coverage block alongside results: records returned, whether more exist, the date range covered and a plain sentence such as 'more results are available; call again with this cursor'. The MCP pagination rules give a clean contract to copy: treat cursors as opaque, treat a missing cursor as the end, and never treat an empty string as the end.
Then fix the evaluation. Add cases where the decisive record sits after the first page, cases where pages come back short, and cases where the API itself reports an incomplete search. Keep the honest-partial counterexample in the set so the grader does not reward agents that over-fetch.
A prompt instruction such as 'always read every page' helps less than it seems. It pushes agents toward the over-fetching counterexample on large archives and does nothing when the wrapper drops the cursor. Coverage metadata in the tool result, plus a grader that compares claimed and read coverage, addresses the failure where it starts.
Where real archives for completeness cases come from
Completeness cases are only as good as their ground truth, and real archives make better cases than synthetic ones because their gaps, short pages and odd date distributions are real. Messaging, ticketing and email archives from operating businesses fit this well when the rights and privacy review allow it.
For buyers who need archives like the one in this case, SourceX locates them among U.S. businesses at the buyer's request instead of selling from inventory, starting from a request that sets out scope, volume, language, format, timing and licensing needs. Every supplying company signs the agreement, with the scope and terms it approved, before a message archive is delivered.
Frequently asked questions
What should an illustrative tool result completeness case include?
A case label, the frozen starting state with recomputed counts, the task and its scope words, the full tool trace with raw response metadata, the failure step, claimed versus actual coverage, an outcome check against ground truth, a severity and a recovery check. Each field should link to the record that proves it.
Why does a faithfulness check miss this failure?
Faithfulness checks ask whether each claim is supported by the retrieved context. When an agent reads only the first page, every sentence can be supported by that page while the summary's scope covers far more. Completeness has to be graded against the whole archive, not the retrieved text.
Is a partial summary always a failure?
No. A summary that states exactly what it read, for example the date range and record count, and says more history exists, is honest and can pass. The failure is claiming or implying full coverage that the tool trace does not support.
How do I know whether the agent ignored the cursor or never saw it?
Log the raw tool response, including pagination fields, separately from what the model was shown. If the cursor reached the model and went unused, fix the agent or the tool description. If the wrapper dropped it, fix the tool.
Which API signals should an agent treat as 'more results exist'?
A non-empty continuation token such as Slack's next_cursor, Gmail and Drive's nextPageToken, Microsoft Graph's @odata.nextLink or MCP's nextCursor. Also treat Drive's incompleteSearch flag as a warning, and never infer completeness from a page being shorter than the requested limit.
Sources
- Slack's pagination guide says an empty, null or absent next_cursor means no further results, that a call can return fewer results than the limit while more exist, and that result size should not be compared to the limit to judge completeness. Source
- The Gmail API discovery document says messages.list returns messages containing only an id and threadId, a nextPageToken and an estimated total in resultSizeEstimate. Source
- The Google Drive API discovery document says a file list may be incomplete when nextPageToken is populated and that incompleteSearch is true when not all documents were searched. Source
- Microsoft Graph documentation says @odata.nextLink is returned until no pages remain, a page may contain zero results, and @odata.count may appear only on the first page for directory resources. Source
- The MCP pagination specification uses opaque cursors with server-set page size, treats a missing nextCursor as the end of results and says an empty-string cursor must not be treated as the end. Source
- Anthropic's guidance on writing tools for agents recommends pagination, range selection, filtering or truncation with sensible defaults, says Claude Code restricts tool responses to 25,000 tokens by default, and advises steering agents with helpful truncation instructions. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.