AI uses for records
Tool-use pagination completeness: testing agents that stop at page one
By SourceX Editorial · Updated
9 sections
Short answer
To test an agent that stops searching after the first results page, build tasks whose correct answer depends on records beyond page one, fix the data behind each tool so the full answer set is known, and place required items after page boundaries, empty pages and silent page-size caps. Score evidence recall, whether the agent's claim of completeness was true, and the final business state, not the agent's summary.
Agent run
Tool use dataset pagination
Key takeaways
- Agents stop early because APIs signal more results in different ways, and several of those signals are easy to miss or are silently capped.
- A completeness benchmark needs a frozen data snapshot behind every tool so the full set of required evidence is known before the agent runs.
- Required items should sit beyond page one, after an empty page, past a page-size cap and on the last page, so each failure mode is tested.
- Score the claim as well as the answer: an agent that says it found everything when it did not should fail harder than one that reports a partial search.
- When the task changes records, query the store afterward; a confirmation message on screen is not proof that every intended record changed.
How do you test an agent that stops searching after the first results page?
You test it by giving the agent tasks where the right answer cannot be produced from the first page, and by knowing the full answer in advance. Freeze the records behind each tool, compute the complete set of required evidence from the store itself, and then check what the agent retrieved, what it claimed and what changed.
The failure is common because nothing breaks when it happens. The first page comes back well formed, the agent sees plausible results and writes a confident answer. Unless the task was built so that a missing record changes the outcome, a benchmark scores that run as a pass.
A useful test therefore has three parts: tasks whose required evidence spans pages, a tool environment whose pagination behaves like the real APIs the agent will meet, and scoring that separates retrieval, the completeness claim and the end state.
How do real APIs signal that more results exist?
Every API family signals continuation differently, and each one sets a different trap. The table summarizes six widely used patterns from their own documentation. A benchmark should copy these behaviors in its mock tools rather than inventing a single tidy pagination style that agents rarely meet in production.
Mirror more than one pattern in the benchmark. An agent tuned on cursor-style tools may handle nextCursor well and still ignore a done flag or a link header, because it learned to look for one field name rather than to ask whether the result is finished. Vary the field names, page sizes and end signals across tools in the same suite, and keep each tool's behavior faithful to the API it imitates.
| API pattern | More results signal | End signal | Trap for an agent |
|---|---|---|---|
| Model Context Protocol list calls | nextCursor in the result | nextCursor missing | Page size is set by the server and the cursor is opaque, so guessing sizes or editing cursors fails |
| GitHub REST API | link header with rel next | link header omitted or no rel next | per_page above the maximum, 100 for most endpoints, is reduced silently with no error |
| Microsoft Graph | @odata.nextLink URL | @odata.nextLink no longer returned | A page can hold zero results while more pages remain, and for directory resources the count appears only on page one |
| Stripe list endpoints | has_more set to true | has_more false | Default limit is 10, so an agent that never sets limit sees ten items and may stop |
| Salesforce REST query | done false with nextRecordsUrl | done true and no nextRecordsUrl | The first batch looks like a full result unless the agent reads the done flag |
| Elasticsearch search | More hits than returned; next request uses search_after | Fewer hits than the page size | from and size paging stops at 10,000 hits by default, and order can shift between pages without a point in time |
Six ways an agent declares a search complete too early
Each failure below maps to one of the API behaviors above. Build at least one task for each, because an agent that handles one well can still fail another.
Log the full tool-call trace for every episode, not just the final answer. The trace shows which continuation signals the agent saw and ignored, which is what a team needs in order to fix the behavior, and it lets reviewers separate a model that never looked for more pages from one that looked, hit an error and gave up without saying so.
- Treats the first page as the whole result because it never looks for a continuation field.
- Trusts a count instead of iterating, including counts that are only returned on the first page or that describe the query rather than what was retrieved.
- Asks for a large page, receives a silently capped page and assumes the request was honored.
- Stops at an empty page even though a continuation link is still present.
- Builds its own next request by editing a cursor or offset instead of using the returned token or URL whole.
- Hits a deep-paging ceiling, gets an error or a truncated result, and reports success instead of a partial search.
The pagination and evidence completeness benchmark brief
Use this brief as the specification for a task family. Every element is required; leaving one out usually lets an early-stopping agent pass. Values in the third column are illustrative.
At a glance
The pagination and evidence completeness benchmark brief
Write the brief before building any task, and have the business process owner sign off the task intent and the ground truth rule. Most disputes about a completeness benchmark turn out to be disputes about what should have been in the answer set, not about the agent. Freezing that rule first keeps later arguments about scoring short.
| Brief element | Specification | Illustrative setting |
|---|---|---|
| Task intent | A business request whose correct answer depends on every matching record | List every open invoice more than 60 days overdue for one customer |
| Data snapshot | Frozen records behind each tool, versioned and hashed | Billing records as of one cut-off date |
| Ground truth set | Required evidence computed from the store, not from tool output | IDs of all matching invoices |
| Tool contract | Tool schema, version, page-size rule and continuation semantics copied from a real API pattern | Cursor style with server-set page size |
| Trap placement | Required items placed on page one, a middle page, after an empty page, past a size cap and on the last page | One required item in each position |
| Permissions | Read and write scopes the agent holds; items it cannot see are excluded from ground truth | Read billing, write reminder flag only |
| Action and state check | What the agent must do with the found records, and the query run against the store afterward | Set a reminder flag on each overdue invoice; verify the flag is set on exactly those invoices |
| Completeness claim | The agent must state whether it believes it found all matches | Required final field: complete or partial |
| Outcome labels | Per-episode label set fixed before runs | Complete and correct, partial and declared, partial and claimed complete, over-inclusive |
| Trials and versions | Repeated trials per task and recorded model, prompt and tool versions | Several trials per task, versions logged |
Scoring evidence recall, completeness claims and the final state
Score retrieval, honesty about coverage and the end state separately, then combine them only for reporting. The tau-bench benchmark grades agents by comparing the database state at the end of an episode with an annotated goal state and reports pass^k, the chance that all of k trials succeed. The same state-first approach fits pagination tasks, with two additions specific to completeness.
Weight a false completeness claim as the worst outcome. A partial search that says it is partial lets a person or a supervising process continue the work. A partial search presented as complete hides the gap.
Report results per trap position and per API pattern, not only as one overall score. A single number can look acceptable while hiding that every run failed after empty pages or under one tool's page-size cap. The breakdown is what tells a model team which behavior to train and which records to collect more of.
| Metric | Definition | Pass condition |
|---|---|---|
| Evidence recall | Required items retrieved divided by required items in ground truth | All required items retrieved |
| Evidence precision | Retrieved items that match the task divided by items the agent used | No non-matching item used in the answer or action |
| Claim accuracy | Whether the complete or partial claim matches actual recall | Claim matches recall in every trial |
| Boundary recall | Recall measured separately for each trap position | No trap position below full recall |
| End state | Store state after the episode compared with the goal state | Exact match on required records, no other records changed |
| Consistency | pass^k across repeated trials of the same task | Reported alongside single-trial results |
Proving every flagged record changed in the store, not just on screen
Only a query against the system of record proves that every intended record changed. An agent's message that it flagged twelve invoices, or a screen showing a success banner, tells you what the agent believes. The store tells you which records actually changed.
Run the verification query after the episode with credentials independent of the agent, compare it with the goal state, and record both the missing changes and the unexpected ones. For pagination tasks the typical failure is a clean, correct change to every record on page one and no change at all to the rest, which looks like success in the interface and in the agent's summary.
Illustrative task: overdue invoices across four pages
This task is illustrative; the records and counts are invented to show the method, and no results are reported. A billing tool returns invoices for one customer in cursor-paged results with a server-set page size of 25. The frozen snapshot holds 92 invoices across four pages of results, the server also returns one empty page with a live cursor between pages two and three, and 7 of the invoices are open and more than 60 days overdue.
The required items are placed by design: 2 on page one, 2 on page two, 1 on page three right after that empty page, and 2 on the last page. The agent is asked to set a reminder flag on every qualifying invoice and to state whether its list is complete.
An agent that reads only page one flags 2 invoices and claims completion: recall 2 of 7, a false completeness claim and an end-state mismatch on 5 records. An agent that stops at the empty page flags 4 and fails boundary recall at the empty-page position. Only an agent that follows the cursor to the end flags all 7, and the verification query confirms that no other invoice changed.
The episode record for this task keeps the task text, the snapshot version, every tool call with its cursor and returned page size, the agent's final list and completeness claim, and the verification query result. Those fields are enough for a reviewer to replay the episode and to see exactly where an early stop happened.
Where a completeness benchmark is the wrong test
Some tasks should not demand exhaustive retrieval, and the brief should say so. If the user asks for a few examples, a representative sample or the most recent item, stopping early is correct, and scoring it as a failure trains the wrong habit. Write those tasks with a different ground truth: a minimum count or a specific item, not a full set.
Exhaustive search is also unreasonable when the source caps it. If a search API stops paging at a fixed depth, the right behavior is to narrow the query or report a partial result, and the benchmark should reward that. Live data is the third exception: if records change during the episode, completeness has no fixed answer. Run against a frozen snapshot, or record the source's own consistency mechanism, such as a point in time, as part of the task.
Stop and bring in the business process owner when the ground truth itself is disputed, for example when the definition of overdue depends on payment terms that vary by contract. A benchmark cannot settle a policy question; it can only measure against the answer the owner approved.
Sourcing paginated business records for completeness tasks
Completeness tasks work best on real record shapes: invoice lists, ticket queues, order histories and CRM activity, with realistic volumes per customer so that answers naturally span pages. Synthetic lists tend to be too uniform to reproduce empty pages, uneven page sizes or near-duplicate records.
Records like these reach AI teams through SourceX only after a specific request: it locates them at U.S. businesses, then clears rights and prepares them before delivery. A request can specify the record types, volumes per entity and snapshot dates a pagination benchmark needs.
Frequently asked questions
Why not give the agent a tool that returns every result at once?
Because the agent will meet paginated APIs in production, and a single-call tool hides the behavior you need to test. Large result sets also exceed context limits. Keep at least one tool per task family that paginates the way a real API does.
Should an agent trust a total count field to know when it is done?
Not on its own. Some APIs return counts only on the first page or only on request, and a count does not prove the agent retrieved each item. Use the continuation signal to decide when to stop and the count, where available, as a cross-check.
Does the Model Context Protocol fix a page size for tool and resource lists?
No. The specification leaves page size to the server and tells clients not to assume a fixed size, to treat cursors as opaque and to treat a missing nextCursor as the end of results. Mock servers in a benchmark should vary page size to test that.
How should a partial answer be scored if the agent says it is partial?
Score recall as measured, but record the claim as accurate. Treat a declared partial result as recoverable and a false completeness claim as a hard failure, because only the second hides missing work from the person relying on it.
How many required items should sit beyond page one?
Enough that every trap position has at least one required item: a middle page, after an empty page, past a page-size cap and the last page. One item per position is the minimum; more trials per task matter more than more items per page.
Sources
- MCP list operations paginate with opaque cursors; the server sets page size, clients must not assume a fixed size or modify cursors, and a missing nextCursor marks the end of results. Source
- The GitHub REST API includes a link header with rel next on paginated responses, omits it when everything fits on one page, and silently reduces per_page values above the maximum, which is 100 for most endpoints. Source
- Microsoft Graph returns @odata.nextLink until no pages remain, a page may contain zero results, and for directory resources @odata.count is returned only on the first page. Source
- Stripe list endpoints return has_more, accept a limit from 1 to 100 with a default of 10, and fetch the next page with starting_after set to the last object ID. Source
- A Salesforce REST query returns done false and a nextRecordsUrl when more records remain, and the last batch has done true with no nextRecordsUrl. Source
- Elasticsearch limits from and size paging to 10,000 hits through index.max_result_window and recommends search_after with a point in time for deeper, consistent paging. Source
- tau-bench evaluates agents by comparing the database state at the end of a conversation with an annotated goal state and introduces pass^k for reliability across trials. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.