Skip to content

AI uses for records

Tool use dataset rate limit recovery: how agents wait and resume

By SourceX Editorial · Updated

10 sections

Short answer

To train an agent to respect rate limits without losing the task, give it episodes where a real throttling response arrives mid-task and grade three things separately: whether it waited as the server instructed, whether it resumed from its last verified step without repeating completed writes, and whether the records ended in the intended state. The retry policy and checkpoint evaluation specification below defines each field.

Agent run

Tool use dataset rate limit

This page covers tool use dataset rate limit recovery: how agents wait and resume.Illustrative

Key takeaways

  • Grade wait compliance, repeated writes and end state separately; an agent can pass one and fail the others.
  • Read the status code, then any Retry-After or reset time, then quota headers, because APIs send different combinations.
  • After a limit, pause every call to the same quota, not only the one that failed.
  • A checkpoint must list verified steps by business key so a resumed run cannot redo them.
  • Include episodes where stopping and handing back beats waiting, such as a daily quota that outlasts the deadline.

How do you train an agent to respect rate limits and still finish the task?

Train it on episodes where a real throttling response interrupts work that is already half done, and grade the recovery in three separate parts: did the agent wait as long as the server told it to, did it resume from the last step it had verified instead of starting over, and did the records it was changing end up in the intended state. An agent that passes only one of those is not yet safe to run against a production API.

Rate limits are not an edge case for tool-using agents. Any agent that pages through a ticket queue, updates a batch of CRM records or reconciles invoices will meet a quota eventually, and the moment it does is the moment its behavior is least visible to the person who launched it.

Three failure patterns show up again and again. The agent hammers, retrying at once and burning more of the quota it just ran out of. The agent abandons, reporting that the task failed, or worse that it succeeded, when it was only paused. Or the agent restarts, redoing writes it had already completed and creating duplicates. A useful dataset contains all three as negative examples and shows the recovery that avoids them.

Which throttling signals must the agent read before it decides to wait?

The agent should read the status code, then any explicit wait instruction, then the quota headers, in that order, because servers disagree about which of these they send. Treating every 429 the same way loses information the server went to the trouble of providing.

The 429 Too Many Requests status code was defined in RFC 6585 for a client that has sent too many requests in a given amount of time, and the response may include a Retry-After header saying how long to wait. Many APIs go further. GitHub's REST API documentation says a client over its primary limit receives a 403 or 429 with x-ratelimit-remaining at 0 and should not retry until the time in x-ratelimit-reset. For secondary limits it says to honor retry-after when present, otherwise to wait at least one minute, then to back off exponentially and stop after a set number of retries.

A standardized pair of headers is still being worked out. The IETF HTTPAPI working group's RateLimit header fields draft defines a RateLimit-Policy field that describes a quota and a RateLimit field whose r parameter gives the available quota and whose t parameter gives the effective window in seconds. It is an Internet-Draft, not a finished standard, so a training set should record which header family each API actually sent rather than assume one.

Which throttling signals must the agent read before it decides to wait?
SignalWhere it appearsWhat the agent should doWhat the episode must log
429 statusRFC 6585 and most public APIsTreat as a throttle, not a task failure; look for a wait instruction nextStatus code, endpoint, timestamp of the response
403 with a rate limit messageSome APIs, including GitHub, use 403 for limitsRead the message and headers before deciding it is a permission problemResponse body text and every rate limit header
Retry-After headerOptional on 429 per RFC 6585; GitHub secondary limitsWait at least that long before the same callHeader value and the time the next call was actually sent
Remaining quota at 0 with a reset timeGitHub x-ratelimit-remaining and x-ratelimit-resetPause the whole queue, not just one call, until the reset timeRemaining, reset and the resource the call counted against
RateLimit and RateLimit-Policy fieldsIETF draft, adopted by some APIsPace calls so the available quota is not exhausted againRaw field values and the draft version the API follows
No header at allOlder or internal APIsFall back to capped exponential backoff with jitterEach wait chosen and the attempt count

Why does an immediate retry make the throttle last longer?

Because failed calls usually count against the same quota, so retrying at once spends budget without making progress. OpenAI's cookbook notebook on rate limits says this plainly: unsuccessful requests contribute to the per-minute limit, so continuously resending a request will not work. Its recommended remedy is a random exponential backoff that increases the sleep after each failure up to a maximum number of retries.

Amazon's Builders' Library article on timeouts, retries and backoff explains why the randomness matters. If many clients back off to the same moment, they collide again; jitter spreads the retries out. The same article warns that retries at several layers of a call stack multiply load, and recommends retrying at one point in the stack.

Agents add a twist that ordinary clients do not have: they may run tool calls in parallel and may also wrap each tool in its own retry logic. GitHub's best practices page advises making requests serially rather than concurrently to avoid secondary limits, for example through a queue. For a dataset, that means the episode should record whether the agent itself throttled its own concurrency after the first limit, not only whether one call eventually went through.

The retry policy an agent should follow, rule by rule

The policy below is the first half of the asset. It is written as rules an annotator can check against a logged episode, and each rule names the evidence that shows it was followed. Teams can tighten the numbers for their own APIs; the order of the rules is the part that matters.

The retry policy an agent should follow, rule by rule
RuleWhat the agent doesWhyEvidence of compliance
R1 ClassifyDecide whether the error is a throttle, a permission denial, a validation error or a server fault before retryingOnly throttles and some server faults are worth waiting outA logged classification tied to the response body and headers
R2 Obey explicit waitsUse Retry-After or the reset time when the server gives oneThe server knows its window; a guess does notNext call timestamp is at or after the instructed time
R3 Back off with jitter otherwiseUse capped exponential backoff with random jitter when no wait is givenAvoids synchronized retries and wasted quotaIncreasing, non-identical gaps between attempts
R4 Pause the queueHold all calls to the same quota, not just the failed oneParallel calls would hit the same limitNo other call to that resource during the wait
R5 Re-send writes only with the same idempotency keyRepeat a create or update only where the API supports a key and the same key and parameters are reusedPrevents a second side effect if the first call partly ranIdentical key and body on the original and the retry
R6 Respect a retry budgetStop after a set number of attempts or when the wait passes the task deadlineEndless retries hide a stuck taskAttempt count and deadline in the checkpoint
R7 Report the pause honestlyTell the user or orchestrator the task is paused, why, and when it will resumePrevents false success and silent abandonmentA status message that names the remaining steps
R8 Resume from the checkpointContinue from the first unverified step after the waitCompleted work stays doneNo repeated write for any step already verified

What should the agent's checkpoint hold while it waits?

Enough to resume without guessing: what the task is, what is already verifiably done, what is in flight, and what the limit says about when work can continue. A checkpoint that holds only the chat history forces the agent to reconstruct progress from its own earlier messages, which is how completed writes get repeated.

At a glance

What should the agent's checkpoint hold while it waits

Agent frameworks already treat this as infrastructure. The LangGraph checkpoint library describes checkpointers that save graph state at every superstep to support durable execution, and says that when a node fails mid-execution it stores pending writes from the nodes that completed at that superstep so they are not re-run on resume. That handles the runtime side. A training or evaluation episode needs the business side too: which records changed, under which permission, with which idempotency key.

The fields below are the minimum. They let a grader tell the difference between an agent that resumed correctly and one that happened to finish by redoing everything.

  • Task intent: the user's request in its original words plus the agent's restatement, so a resumed run cannot drift to an easier goal.
  • Scope and permission: the account, role or token scope the work is running under, which must be unchanged after the wait.
  • Completed steps: each step id with the business key it touched and the read-back that verified it.
  • In-flight step: the call that was throttled, with its full parameters and idempotency key if it was a write.
  • Remaining steps: the ordered list still to do, so the agent does not skip items after resuming.
  • Rate limit state: the header values received, the computed resume time and the attempt count.
  • Deadline: the time by which the task must finish or be handed back.
  • Tool versions: the API version or tool schema in use, because a changed contract after a long wait is a different task.

The retry policy and checkpoint evaluation specification

This is the second half of the asset: the record each episode in the dataset carries and how a grader scores it. One row describes one field; a complete episode has all of them. The pass conditions are written so that two graders reading the same logs reach the same verdict.

The retry policy and checkpoint evaluation specification
FieldWhat it recordsPass conditionEvidence source
episode_id and task_specThe task, its start state and its goal stateGoal state is defined in business keys before the runSeeded database or sandbox snapshot
throttle_injectionsWhen, where and how the limit was triggered, including headers sentAt least one throttle lands after a completed writeMock server or proxy log
classificationThe agent's reading of each errorEvery throttle classified as a throttle; no permission error retriedAgent trace matched to response bodies
wait_complianceGap between the throttle and the next call to the same quotaGap at or above the instructed wait; backoff increasing when none givenRequest log timestamps
concurrency_after_limitParallel calls to the same resource during the waitZero calls to the limited resource before resume timeRequest log grouped by resource
write_repeatsWrites re-sent after resume and their keysNo verified step re-executed; re-sent writes reuse the original keyRequest log and idempotency keys
status_honestyWhat the agent told the user during and after the pauseNo success claim while steps remain; pause and resume reportedAgent messages
end_stateRecords in the system after the runMatches the goal state exactly, with no duplicates or missing itemsPost-run query by business key
repeat_trialsOutcome across several runs of the same taskPasses in every trial counted, not just the bestPer-trial results

What proves the records changed, rather than the transcript saying so?

A query against the system of record after the run, compared with a goal state written down before it. The agent's final message is a claim; the records are the evidence. This is how the tau-bench authors graded their tool-agent-user benchmark: the database state at the end of a conversation is compared with the annotated goal state.

The same paper introduced pass^k, a measure of whether an agent succeeds on every one of k independent trials of the same task. Rate limit recovery is a good place to use it, because throttles change timing and timing changes which calls land first. An agent that recovers cleanly in one trial and duplicates a record in the next has not learned the policy.

Two checks are specific to rate limits. First, count the records by business key, such as ticket number or order number, so a duplicate created on resume is visible even if both copies look correct. Second, line up the request log against the reset times the server sent, so wait compliance is measured from timestamps rather than from the agent's account of how long it waited.

Illustrative worked episode: a ticket sync throttled at item 23

Everything in this example is illustrative; the counts and durations were chosen to show the scoring, not taken from any system. The task asks an agent to copy a priority field from 40 help desk tickets into matching records in a CRM, then post one summary note. The CRM sandbox is configured to return a 429 with a Retry-After of 60 seconds on the 23rd update.

A passing run looks like this. The agent classifies the 429 as a throttle, writes a checkpoint listing 22 verified updates and 18 remaining, tells the user the sync is paused for about a minute, sends nothing to the CRM for 60 seconds, then resumes at item 23. The post-run query finds 40 updated records, each updated once, and one summary note.

Three failing variants make the negatives for the dataset. In the first, the agent retries item 23 five times within ten seconds and receives five more 429 responses. In the second, it reports the sync complete after item 22. In the third, it restarts from item 1 after the wait; the field values end correct, but the audit history shows 22 duplicate updates and, if the summary note had been posted mid-run, a second note. The end-state check alone would pass the third variant on field values, which is why write_repeats is scored separately.

Illustrative worked episode: a ticket sync throttled at item 23
Runwait_compliancewrite_repeatsend_state
Passing runPass: first retry at 60 secondsPass: nonePass: 40 records, 1 note
HammeringFail: 5 retries in 10 secondsPass: nonePass after a later success
False completionNot reachedPass: noneFail: 18 records unchanged
Restart from item 1PassFail: 22 repeated updatesPass on values, fail on audit history

When is waiting out the limit the wrong decision?

When the wait would outlast the task's usefulness, when the error is not really a throttle, or when the quota is shared with people who need it more. The dataset should contain episodes where the correct action is to stop and hand back, so the agent does not learn that waiting is always right.

A daily quota that resets tomorrow is the clearest case: an agent asked to prepare a report for a meeting in an hour should report what it finished and what remains, not sleep for twenty hours. A 403 caused by a missing scope will never clear on its own, and retrying it wastes the budget of every other integration sharing the token. Writes need care even where idempotency keys exist. Stripe's documentation, for example, says requests rejected before execution, such as rate-limited ones, are not cached against their key, while a request that returns a server error has an indeterminate outcome and should be resolved before any retry with a new key. Against an API with no key support at all, the agent should read the record back before trying again.

These episodes need a business process owner on the review panel alongside the evaluation engineer, because only the owner knows which deadlines are hard and which partial results are usable.

Where do realistic throttling episodes come from?

From integration run histories that already contain throttles: nightly CRM syncs, help desk exports, ERP reconciliations and marketing automation jobs log their 429 responses, waits and retries alongside the records they touched. Those logs show real header values, real reset windows and real recovery mistakes, which synthetic mocks tend to smooth over.

SourceX keeps no throttling datasets on the shelf. It looks for integration run histories among U.S. businesses when a lab asks, and the request form is where the lab names scope, volume, language, format, timing and licensing terms. A business that agrees still approves the scope and terms and signs before a single log line is delivered.

Frequently asked questions

What is rate limit recovery in a tool use dataset?

It is the part of a tool-use dataset that shows an agent meeting an API quota mid-task and continuing correctly. Each episode records the throttling response, the wait the agent chose, the checkpoint it kept and the final state of the records it was changing.

Should an agent always wait the full Retry-After time?

It should not send another call to the same quota before that time, because the server has said when capacity returns. It may decide not to resume at all if the wait would pass the task deadline, in which case it should report what is done and what remains.

How can an agent tell a rate limit from a permission error?

Read the response body and headers, not just the status code. Some APIs, GitHub among them, return 403 for rate limits as well as for missing permissions, and only the message and rate limit headers distinguish the two. A permission error will not clear by waiting.

Is it safe to retry a write after a 429?

Usually, if the API supports idempotency keys and the retry reuses the original key and identical parameters. Without a key, the agent should read back the record first to confirm the write did not land before sending it again.

How many trials of each throttled task are enough?

Enough to see whether recovery is consistent, which in practice means several runs per task scored with a measure such as pass^k that requires success in every trial. A single clean run can hide a timing-dependent duplicate.

Sources

  • RFC 6585 defines the 429 Too Many Requests status code for rate limiting and allows a Retry-After header saying how long to wait. Source
  • GitHub's REST API documentation describes 403 or 429 rate limit responses, the x-ratelimit headers and when a client may retry, including exponential backoff for secondary limits. Source
  • GitHub recommends making requests serially rather than concurrently, for example through a queue, to avoid secondary rate limits. Source
  • The IETF HTTPAPI draft defines RateLimit-Policy and RateLimit header fields, with r for available quota and t for the effective window in seconds. Source
  • OpenAI's cookbook notes that unsuccessful requests count toward the per-minute limit and recommends random exponential backoff with a maximum number of retries. Source
  • Amazon's Builders' Library explains that jitter spreads retries in time and that retries at several layers of a call stack multiply load. Source
  • The LangGraph checkpoint library saves graph state at every superstep and keeps pending writes from completed nodes so they are not re-run on resume. Source
  • Stripe documents idempotency keys, says rate-limited requests are not cached against the key, and says a 500 outcome is indeterminate and should be resolved before retrying with a new key. Source
  • The tau-bench paper grades agents by comparing the final database state with an annotated goal state and introduces the pass^k reliability measure. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies