AI uses for records
Tool use dataset rate limit recovery: how agents wait and resume
By SourceX Editorial · Updated
10 sections
Short answer
To train an agent to respect rate limits without losing the task, give it episodes where a real throttling response arrives mid-task and grade three things separately: whether it waited as the server instructed, whether it resumed from its last verified step without repeating completed writes, and whether the records ended in the intended state. The retry policy and checkpoint evaluation specification below defines each field.
Agent run
Tool use dataset rate limit
Key takeaways
- Grade wait compliance, repeated writes and end state separately; an agent can pass one and fail the others.
- Read the status code, then any Retry-After or reset time, then quota headers, because APIs send different combinations.
- After a limit, pause every call to the same quota, not only the one that failed.
- A checkpoint must list verified steps by business key so a resumed run cannot redo them.
- Include episodes where stopping and handing back beats waiting, such as a daily quota that outlasts the deadline.
How do you train an agent to respect rate limits and still finish the task?
Train it on episodes where a real throttling response interrupts work that is already half done, and grade the recovery in three separate parts: did the agent wait as long as the server told it to, did it resume from the last step it had verified instead of starting over, and did the records it was changing end up in the intended state. An agent that passes only one of those is not yet safe to run against a production API.
Rate limits are not an edge case for tool-using agents. Any agent that pages through a ticket queue, updates a batch of CRM records or reconciles invoices will meet a quota eventually, and the moment it does is the moment its behavior is least visible to the person who launched it.
Three failure patterns show up again and again. The agent hammers, retrying at once and burning more of the quota it just ran out of. The agent abandons, reporting that the task failed, or worse that it succeeded, when it was only paused. Or the agent restarts, redoing writes it had already completed and creating duplicates. A useful dataset contains all three as negative examples and shows the recovery that avoids them.
Which throttling signals must the agent read before it decides to wait?
The agent should read the status code, then any explicit wait instruction, then the quota headers, in that order, because servers disagree about which of these they send. Treating every 429 the same way loses information the server went to the trouble of providing.
The 429 Too Many Requests status code was defined in RFC 6585 for a client that has sent too many requests in a given amount of time, and the response may include a Retry-After header saying how long to wait. Many APIs go further. GitHub's REST API documentation says a client over its primary limit receives a 403 or 429 with x-ratelimit-remaining at 0 and should not retry until the time in x-ratelimit-reset. For secondary limits it says to honor retry-after when present, otherwise to wait at least one minute, then to back off exponentially and stop after a set number of retries.
A standardized pair of headers is still being worked out. The IETF HTTPAPI working group's RateLimit header fields draft defines a RateLimit-Policy field that describes a quota and a RateLimit field whose r parameter gives the available quota and whose t parameter gives the effective window in seconds. It is an Internet-Draft, not a finished standard, so a training set should record which header family each API actually sent rather than assume one.
| Signal | Where it appears | What the agent should do | What the episode must log |
|---|---|---|---|
| 429 status | RFC 6585 and most public APIs | Treat as a throttle, not a task failure; look for a wait instruction next | Status code, endpoint, timestamp of the response |
| 403 with a rate limit message | Some APIs, including GitHub, use 403 for limits | Read the message and headers before deciding it is a permission problem | Response body text and every rate limit header |
| Retry-After header | Optional on 429 per RFC 6585; GitHub secondary limits | Wait at least that long before the same call | Header value and the time the next call was actually sent |
| Remaining quota at 0 with a reset time | GitHub x-ratelimit-remaining and x-ratelimit-reset | Pause the whole queue, not just one call, until the reset time | Remaining, reset and the resource the call counted against |
| RateLimit and RateLimit-Policy fields | IETF draft, adopted by some APIs | Pace calls so the available quota is not exhausted again | Raw field values and the draft version the API follows |
| No header at all | Older or internal APIs | Fall back to capped exponential backoff with jitter | Each wait chosen and the attempt count |
Why does an immediate retry make the throttle last longer?
Because failed calls usually count against the same quota, so retrying at once spends budget without making progress. OpenAI's cookbook notebook on rate limits says this plainly: unsuccessful requests contribute to the per-minute limit, so continuously resending a request will not work. Its recommended remedy is a random exponential backoff that increases the sleep after each failure up to a maximum number of retries.
Amazon's Builders' Library article on timeouts, retries and backoff explains why the randomness matters. If many clients back off to the same moment, they collide again; jitter spreads the retries out. The same article warns that retries at several layers of a call stack multiply load, and recommends retrying at one point in the stack.
Agents add a twist that ordinary clients do not have: they may run tool calls in parallel and may also wrap each tool in its own retry logic. GitHub's best practices page advises making requests serially rather than concurrently to avoid secondary limits, for example through a queue. For a dataset, that means the episode should record whether the agent itself throttled its own concurrency after the first limit, not only whether one call eventually went through.
The retry policy an agent should follow, rule by rule
The policy below is the first half of the asset. It is written as rules an annotator can check against a logged episode, and each rule names the evidence that shows it was followed. Teams can tighten the numbers for their own APIs; the order of the rules is the part that matters.
| Rule | What the agent does | Why | Evidence of compliance |
|---|---|---|---|
| R1 Classify | Decide whether the error is a throttle, a permission denial, a validation error or a server fault before retrying | Only throttles and some server faults are worth waiting out | A logged classification tied to the response body and headers |
| R2 Obey explicit waits | Use Retry-After or the reset time when the server gives one | The server knows its window; a guess does not | Next call timestamp is at or after the instructed time |
| R3 Back off with jitter otherwise | Use capped exponential backoff with random jitter when no wait is given | Avoids synchronized retries and wasted quota | Increasing, non-identical gaps between attempts |
| R4 Pause the queue | Hold all calls to the same quota, not just the failed one | Parallel calls would hit the same limit | No other call to that resource during the wait |
| R5 Re-send writes only with the same idempotency key | Repeat a create or update only where the API supports a key and the same key and parameters are reused | Prevents a second side effect if the first call partly ran | Identical key and body on the original and the retry |
| R6 Respect a retry budget | Stop after a set number of attempts or when the wait passes the task deadline | Endless retries hide a stuck task | Attempt count and deadline in the checkpoint |
| R7 Report the pause honestly | Tell the user or orchestrator the task is paused, why, and when it will resume | Prevents false success and silent abandonment | A status message that names the remaining steps |
| R8 Resume from the checkpoint | Continue from the first unverified step after the wait | Completed work stays done | No repeated write for any step already verified |
What should the agent's checkpoint hold while it waits?
Enough to resume without guessing: what the task is, what is already verifiably done, what is in flight, and what the limit says about when work can continue. A checkpoint that holds only the chat history forces the agent to reconstruct progress from its own earlier messages, which is how completed writes get repeated.
At a glance
What should the agent's checkpoint hold while it waits
Agent frameworks already treat this as infrastructure. The LangGraph checkpoint library describes checkpointers that save graph state at every superstep to support durable execution, and says that when a node fails mid-execution it stores pending writes from the nodes that completed at that superstep so they are not re-run on resume. That handles the runtime side. A training or evaluation episode needs the business side too: which records changed, under which permission, with which idempotency key.
The fields below are the minimum. They let a grader tell the difference between an agent that resumed correctly and one that happened to finish by redoing everything.
- Task intent: the user's request in its original words plus the agent's restatement, so a resumed run cannot drift to an easier goal.
- Scope and permission: the account, role or token scope the work is running under, which must be unchanged after the wait.
- Completed steps: each step id with the business key it touched and the read-back that verified it.
- In-flight step: the call that was throttled, with its full parameters and idempotency key if it was a write.
- Remaining steps: the ordered list still to do, so the agent does not skip items after resuming.
- Rate limit state: the header values received, the computed resume time and the attempt count.
- Deadline: the time by which the task must finish or be handed back.
- Tool versions: the API version or tool schema in use, because a changed contract after a long wait is a different task.
The retry policy and checkpoint evaluation specification
This is the second half of the asset: the record each episode in the dataset carries and how a grader scores it. One row describes one field; a complete episode has all of them. The pass conditions are written so that two graders reading the same logs reach the same verdict.
| Field | What it records | Pass condition | Evidence source |
|---|---|---|---|
| episode_id and task_spec | The task, its start state and its goal state | Goal state is defined in business keys before the run | Seeded database or sandbox snapshot |
| throttle_injections | When, where and how the limit was triggered, including headers sent | At least one throttle lands after a completed write | Mock server or proxy log |
| classification | The agent's reading of each error | Every throttle classified as a throttle; no permission error retried | Agent trace matched to response bodies |
| wait_compliance | Gap between the throttle and the next call to the same quota | Gap at or above the instructed wait; backoff increasing when none given | Request log timestamps |
| concurrency_after_limit | Parallel calls to the same resource during the wait | Zero calls to the limited resource before resume time | Request log grouped by resource |
| write_repeats | Writes re-sent after resume and their keys | No verified step re-executed; re-sent writes reuse the original key | Request log and idempotency keys |
| status_honesty | What the agent told the user during and after the pause | No success claim while steps remain; pause and resume reported | Agent messages |
| end_state | Records in the system after the run | Matches the goal state exactly, with no duplicates or missing items | Post-run query by business key |
| repeat_trials | Outcome across several runs of the same task | Passes in every trial counted, not just the best | Per-trial results |
What proves the records changed, rather than the transcript saying so?
A query against the system of record after the run, compared with a goal state written down before it. The agent's final message is a claim; the records are the evidence. This is how the tau-bench authors graded their tool-agent-user benchmark: the database state at the end of a conversation is compared with the annotated goal state.
The same paper introduced pass^k, a measure of whether an agent succeeds on every one of k independent trials of the same task. Rate limit recovery is a good place to use it, because throttles change timing and timing changes which calls land first. An agent that recovers cleanly in one trial and duplicates a record in the next has not learned the policy.
Two checks are specific to rate limits. First, count the records by business key, such as ticket number or order number, so a duplicate created on resume is visible even if both copies look correct. Second, line up the request log against the reset times the server sent, so wait compliance is measured from timestamps rather than from the agent's account of how long it waited.
Illustrative worked episode: a ticket sync throttled at item 23
Everything in this example is illustrative; the counts and durations were chosen to show the scoring, not taken from any system. The task asks an agent to copy a priority field from 40 help desk tickets into matching records in a CRM, then post one summary note. The CRM sandbox is configured to return a 429 with a Retry-After of 60 seconds on the 23rd update.
A passing run looks like this. The agent classifies the 429 as a throttle, writes a checkpoint listing 22 verified updates and 18 remaining, tells the user the sync is paused for about a minute, sends nothing to the CRM for 60 seconds, then resumes at item 23. The post-run query finds 40 updated records, each updated once, and one summary note.
Three failing variants make the negatives for the dataset. In the first, the agent retries item 23 five times within ten seconds and receives five more 429 responses. In the second, it reports the sync complete after item 22. In the third, it restarts from item 1 after the wait; the field values end correct, but the audit history shows 22 duplicate updates and, if the summary note had been posted mid-run, a second note. The end-state check alone would pass the third variant on field values, which is why write_repeats is scored separately.
| Run | wait_compliance | write_repeats | end_state |
|---|---|---|---|
| Passing run | Pass: first retry at 60 seconds | Pass: none | Pass: 40 records, 1 note |
| Hammering | Fail: 5 retries in 10 seconds | Pass: none | Pass after a later success |
| False completion | Not reached | Pass: none | Fail: 18 records unchanged |
| Restart from item 1 | Pass | Fail: 22 repeated updates | Pass on values, fail on audit history |
When is waiting out the limit the wrong decision?
When the wait would outlast the task's usefulness, when the error is not really a throttle, or when the quota is shared with people who need it more. The dataset should contain episodes where the correct action is to stop and hand back, so the agent does not learn that waiting is always right.
A daily quota that resets tomorrow is the clearest case: an agent asked to prepare a report for a meeting in an hour should report what it finished and what remains, not sleep for twenty hours. A 403 caused by a missing scope will never clear on its own, and retrying it wastes the budget of every other integration sharing the token. Writes need care even where idempotency keys exist. Stripe's documentation, for example, says requests rejected before execution, such as rate-limited ones, are not cached against their key, while a request that returns a server error has an indeterminate outcome and should be resolved before any retry with a new key. Against an API with no key support at all, the agent should read the record back before trying again.
These episodes need a business process owner on the review panel alongside the evaluation engineer, because only the owner knows which deadlines are hard and which partial results are usable.
Where do realistic throttling episodes come from?
From integration run histories that already contain throttles: nightly CRM syncs, help desk exports, ERP reconciliations and marketing automation jobs log their 429 responses, waits and retries alongside the records they touched. Those logs show real header values, real reset windows and real recovery mistakes, which synthetic mocks tend to smooth over.
SourceX keeps no throttling datasets on the shelf. It looks for integration run histories among U.S. businesses when a lab asks, and the request form is where the lab names scope, volume, language, format, timing and licensing terms. A business that agrees still approves the scope and terms and signs before a single log line is delivered.
Frequently asked questions
What is rate limit recovery in a tool use dataset?
It is the part of a tool-use dataset that shows an agent meeting an API quota mid-task and continuing correctly. Each episode records the throttling response, the wait the agent chose, the checkpoint it kept and the final state of the records it was changing.
Should an agent always wait the full Retry-After time?
It should not send another call to the same quota before that time, because the server has said when capacity returns. It may decide not to resume at all if the wait would pass the task deadline, in which case it should report what is done and what remains.
How can an agent tell a rate limit from a permission error?
Read the response body and headers, not just the status code. Some APIs, GitHub among them, return 403 for rate limits as well as for missing permissions, and only the message and rate limit headers distinguish the two. A permission error will not clear by waiting.
Is it safe to retry a write after a 429?
Usually, if the API supports idempotency keys and the retry reuses the original key and identical parameters. Without a key, the agent should read back the record first to confirm the write did not land before sending it again.
How many trials of each throttled task are enough?
Enough to see whether recovery is consistent, which in practice means several runs per task scored with a measure such as pass^k that requires success in every trial. A single clean run can hide a timing-dependent duplicate.
Sources
- RFC 6585 defines the 429 Too Many Requests status code for rate limiting and allows a Retry-After header saying how long to wait. Source
- GitHub's REST API documentation describes 403 or 429 rate limit responses, the x-ratelimit headers and when a client may retry, including exponential backoff for secondary limits. Source
- GitHub recommends making requests serially rather than concurrently, for example through a queue, to avoid secondary rate limits. Source
- The IETF HTTPAPI draft defines RateLimit-Policy and RateLimit header fields, with r for available quota and t for the effective window in seconds. Source
- OpenAI's cookbook notes that unsuccessful requests count toward the per-minute limit and recommends random exponential backoff with a maximum number of retries. Source
- Amazon's Builders' Library explains that jitter spreads retries in time and that retries at several layers of a call stack multiply load. Source
- The LangGraph checkpoint library saves graph state at every superstep and keeps pending writes from completed nodes so they are not re-run on resume. Source
- Stripe documents idempotency keys, says rate-limited requests are not cached against the key, and says a 500 outcome is indeterminate and should be resolved before retrying with a new key. Source
- The tau-bench paper grades agents by comparing the final database state with an annotated goal state and introduces the pass^k reliability measure. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.