AI uses for records
RL environment API failure simulation: how to train agents on outages
By SourceX Editorial · Updated
10 sections
Short answer
To simulate API outages in an agent training environment, put a fault layer between the agent's tool calls and the mock backend, drive it from a seeded schedule that names the call, fault type, duration and whether the write committed, and grade recovery on the final business state rather than on the error message. Score duplicate side effects, retry discipline and honest escalation separately, and run zero-fault baselines first.
Grader rubric
RL environment API failure
Key takeaways
- Inject faults from a seeded schedule stored with the episode, so every failure can be replayed exactly.
- The hardest and most useful fault is the timeout that hides a committed write, because a naive retry duplicates the action.
- Grade recovery on the final state of the system, then separately on duplicates, retry behavior and what the agent told the user.
- An episode cut short by an outage budget is a truncation, not a task failure, and the training code should treat it that way.
- Run each task with no faults first; a task the agent cannot finish cleanly tells you nothing about recovery.
How do you simulate API outages in an agent training environment?
You place a fault layer between the agent's tool calls and the simulated service, and you let a seeded schedule decide which call fails, how and for how long. The agent sees what a real client would see: a 503 with a Retry-After value, a 429, a connection reset, a slow response or a timeout. The grader then checks the business state after the episode, not whether the agent produced a polite apology.
Three design choices decide whether the exercise is worth running. The schedule must be part of the episode record, so a failure can be replayed. Some faults must change state behind the agent's back, because real outages do. And recovery has to be scored on several separate dimensions, since an agent can reach the right final state while hammering the API, or stay perfectly polite while leaving a refund issued twice.
This page covers the failures to include, where to inject them, the schedule format, the scoring sheet and the assumptions that quietly change results. It is written for teams building tool-use environments around business systems such as billing, ticketing or order management.
Which API failures belong in the fault catalog?
Include the failures whose correct recovery differs, not every status code. A rate limit calls for waiting, an indeterminate timeout calls for checking state before retrying, and a permanent error calls for stopping and escalating. If two faults have the same correct response, one of them is enough to start with.
The protocol references matter because the agent should be judged against the contract a real client would follow. HTTP semantics in RFC 9110 let a 503 carry a Retry-After field holding either a date or a number of seconds, and RFC 6585 defines 429 Too Many Requests with an optional Retry-After. Google's canonical RPC error codes add the case most environments miss: DEADLINE_EXCEEDED can be returned for a state-changing call even when the operation actually completed, and UNAVAILABLE is usually transient but not always safe to retry for non-idempotent operations.
Two rows in the catalog look identical to the agent and differ only in hidden state: the ambiguous timeout and the lost write. That pairing is deliberate. An agent cannot tell them apart from the error, so the only winning strategy is to read the system before acting again, which is exactly the habit you want it to carry into production.
| Fault type | What the agent observes | Hidden state change | Correct recovery |
|---|---|---|---|
| Transient outage | 503 with Retry-After | None | Wait the stated interval, retry, cap attempts |
| Rate limit | 429, sometimes with Retry-After | None | Back off, reduce call rate, do not switch tools to dodge it |
| Ambiguous timeout | Timeout or 504 on a write | Write committed | Read state or reuse the idempotency key before any retry |
| Lost write | Timeout or 504 on a write | Write not committed | Confirm absence, then retry once with the same key |
| Connection reset | Socket closed mid-response | Either, set by schedule | Treat as ambiguous and verify state |
| Slow dependency | Response after a long delay | None | Respect its own timeout, avoid parallel duplicate calls |
| Truncated payload | Partial or unparsable body | None | Retry the read, never act on half a record |
| Precondition failure | 409 or validation error after an outage | Another actor changed the record | Re-read, re-plan, ask before overwriting |
| Hard outage | Repeated 503 past the budget | None | Stop, tell the user what is pending, hand off |
Where should the fault layer sit: proxy, gateway or mock service?
Put network-level faults in a proxy and state-dependent faults inside the mock service; most environments need both. A proxy cannot know whether a write committed, and a mock service is a poor place to model a reset socket.
Off-the-shelf tools cover the network side well. Shopify's Toxiproxy sits between client and upstream and offers toxics for latency with jitter, a downed service, bandwidth limits, timeouts, TCP resets, sliced packets, data limits and packet loss, each applied with a toxicity probability. Envoy's fault injection filter injects delays and aborts with chosen HTTP or gRPC codes, can be switched per request by headers and percentages, and is explicit that it cannot emulate CPU or disk failures on the local host.
Probability settings are convenient for load tests but awkward for training data, because the same seed must produce the same failure on replay. Drive the proxy from your schedule rather than from its random sampler, or record every sampled decision in the episode log.
- Proxy (Toxiproxy-style): latency, resets, truncation, downed upstream. Cheap and realistic at the wire, blind to business state.
- Gateway filter (Envoy-style): status-code aborts and delays per route or header. Good for 429 and 503 shapes, still blind to commits.
- Inside the mock service or tool wrapper: commit-then-timeout, lost writes, concurrent edits by another actor. The only place the ambiguous cases can be modeled faithfully.
The fault injection schedule, row by row
Each scheduled fault is one row tied to an episode seed, and the full schedule is stored with the episode so a reviewer can replay it. The agent never sees the schedule; only the environment and the grader do.
At a glance
The fault injection schedule, row by row
Start with one fault per episode on a fixed call, then move to compound schedules where a rate limit follows an outage. Compound faults are where agents loop, so keep their share small until single-fault recovery is stable.
Write schedules as data files, not as code paths in the mock service, so that a reviewer can read them and a new fault can be added without touching the environment. A schedule file per task family, versioned alongside the task list, also makes it easy to answer later questions such as which agents ever saw a hard outage on the billing tool. Keep the commit_behavior field explicit even for read-only tools; an empty value invites someone to assume the safe case.
| Field | What to record | Illustrative value |
|---|---|---|
| fault_id and episode_seed | Stable IDs so the row replays with the episode | F-017, seed 4411 |
| trigger | Tool name and call index, or simulated clock time | payments.create_refund, call 1 |
| fault_type | One entry from the catalog | Ambiguous timeout |
| duration | Number of calls or simulated seconds the fault lasts | First 2 calls |
| response shape | Status, headers and body the agent receives | 504, no body |
| commit_behavior | Whether the write lands before the error | Committed on call 1 |
| expected_recovery | The behavior the grader rewards | Look up refund by order before retrying |
| attempt_budget | Maximum tool calls before the episode is truncated | 6 calls on this tool |
| held_out | Whether the fault type is reserved for evaluation | No |
The recovery scoring sheet: six things a grader checks
Score the final state first and everything else second. tau-bench is a useful model: it rewards an episode by comparing a hash of the final database with the hash produced by replaying the ground-truth actions, so a fluent transcript earns nothing if the records are wrong. Faulted episodes need that check plus dimensions a state hash cannot see.
Retry discipline deserves its own line because it is where real damage happens. The Amazon Builders' Library notes that retries at several layers multiply load on a failing service, prefers capped exponential backoff with jitter over immediate retries, and says APIs with side effects should be idempotent before they are retried. An agent that retries fifteen times in two seconds can pass the state check while teaching itself a habit that worsens production incidents.
Keep the gates and the scored dimensions apart in reports. A single blended number hides the difference between an agent that recovers slowly and one that refunds twice, and those call for very different fixes. Report per fault type as well, since an agent can be excellent on rate limits and dangerous on ambiguous timeouts at the same time.
| Dimension | Evidence the grader reads | Pass rule | Weight in the episode verdict |
|---|---|---|---|
| Final state | State diff or hash against the reference run | Matches the reference exactly | Gate: a fail fails the episode |
| No duplicate side effects | Count of writes per business key | One refund, one email, one ticket update | Gate: a duplicate fails the episode |
| Retry discipline | Call log with timestamps | Honors Retry-After, backs off, stays within budget | Scored, logged per fault type |
| State check before retry | Read calls between an ambiguous error and the next write | A lookup precedes any repeated write | Scored on ambiguous faults only |
| Honest communication | Agent messages to the user | States what is done, pending or unknown; no false confirmation | Scored, reviewed on a sample |
| Escalation on hard outage | Final message and handoff action | Stops within budget and records the pending action | Scored on hard-outage rows |
Which environment assumptions change the result of the recovery test?
The mock service's idempotency behavior, the episode-ending rule and the simulated clock change scores more than the faults themselves. Write each one down before comparing two agents.
At a glance
Which environment assumptions change the result of the recovery test
Idempotency should mirror the real API the environment imitates. Stripe, for example, stores the first result for an idempotency key and returns it on retries with that key, including 500 errors, and rejects a reused key sent with different parameters. A mock that ignores keys makes a careful agent look no better than a careless one; a mock that deduplicates everything hides the duplicate-refund failure entirely.
How an outage ends an episode matters for training. Gymnasium separates terminated from truncated and bootstraps value targets on terminated only. An episode stopped because the attempt budget ran out during an outage is a truncation, and labeling it as a terminal failure teaches the policy that outages are the agent's fault.
- Clock: use simulated time for Retry-After and backoff, or an agent that waits correctly will make every episode slow and get cut by a wall-clock limit.
- Reset: the fault state, counters and partially committed writes must reset with the episode.
- Error bodies: copy the real API's error schema; agents learn to read fields like a retry hint.
- Budget: set attempt limits per tool, not per episode, so one flaky dependency cannot starve the rest of the task.
Illustrative run: a refund desk with one flaky payments API
Illustrative example, not measured results: a refund environment with 40 tasks, each run once with no faults and three times with a scheduled fault on the payments tool. The zero-fault pass shows 36 of 40 tasks completed cleanly, so the four failing tasks are removed from the recovery analysis because they measure task difficulty, not recovery.
On the 108 faulted episodes that remain, the 503 and 429 rows mostly recover. The ambiguous-timeout rows tell the real story: in this illustration, 9 of 36 episodes issue a second refund because the agent retried without checking whether the first had landed. Each of those fails the duplicate gate even though the customer-facing message looks correct.
The exception is the lost-write row. There the right move is to retry, and an agent trained only on commit-then-timeout cases may over-correct by refusing to retry at all and escalating every time. Keep both variants in the schedule in roughly equal numbers so the agent learns to check state, not to fear writes.
Dosing faults for training without teaching the agent to give up
Keep faults a minority of training episodes and raise their share only after clean-task performance is stable. A policy that sees outages everywhere learns that ending early with an apology is the best available reward.
Watch for the shortcut where a fault becomes an excuse. If the escalation dimension is rewarded more than finishing, an agent can learn to declare an outage at the first slow response. Score escalation only on hard-outage rows, and penalize escalation on rows where recovery was possible within budget.
Hold out at least one fault type, such as the precondition failure after a concurrent edit, for evaluation only. Recovery that generalizes to an unseen fault is the result worth reporting; recovery on the exact schedule the agent trained on is closer to memorization.
Schedule faults at different points in a task, not only on the first call. A failure on the last write of a multi-step task is harder than one at the start, because the agent has already changed state and must decide whether to roll back, retry or report a partial result. Spread trigger positions across early, middle and final calls so the policy sees all three situations.
Faults to model only with the API owner's help
Some failure modes cannot be imitated from public documentation alone, and guessing them produces confident but wrong training signal. Bring in the owner of the real integration, or narrow the environment, in these cases.
Partial commits across several systems, such as a refund that posts in the payment processor but not in the ledger, depend on reconciliation jobs that only the operating team understands. Webhook retries and out-of-order deliveries depend on the vendor's documented delivery behavior and the customer's handler. Rate-limit buckets shared across tenants depend on account configuration. If you cannot describe the correct recovery for a fault in one sentence that the process owner agrees with, leave it out of the schedule until you can.
A short interview with the owner usually settles these questions faster than reading logs: what the team does when the processor times out, how they find out whether a payment posted, and who is told. Write the answers into the expected_recovery field and have the owner sign off on that column before training starts.
Calibrating fault rates from real call histories
The most defensible fault schedule is calibrated to what a business's integrations actually experienced: the mix of error codes, how long outages lasted and how support staff recovered. That history usually lives in API gateway logs, integration error queues and the tickets opened when a sync failed, and it has to be licensed and de-identified before it shapes training.
Error histories like these come from operating businesses: SourceX runs the rights review, contracting, preparation and delivery when a business licenses selected records, and for AI teams it locates such records at U.S. companies on request rather than selling from a catalog. If your environment needs real error histories or the tickets that show how people recovered, describe the scope, volume, format, timing and licensing terms you need in a data request.
Frequently asked questions
What is RL environment API failure simulation?
It is the practice of injecting realistic tool and API failures, such as outages, rate limits and timeouts, into a reinforcement learning environment so an agent learns and is tested on recovery. The failures follow a seeded schedule so each episode can be replayed, and a grader checks the final business state afterward.
Should faults be random or scheduled?
Scheduled, or random with every sampled decision logged. Training data and evaluations need replay, and a probability setting with no record of which calls failed makes a failed episode impossible to reproduce or debug.
Which fault teaches agents the most?
The timeout that hides a committed write. The correct response is to check state or reuse an idempotency key before retrying, and agents that skip that step create duplicate refunds, orders or messages while appearing to recover.
Is an episode that ends in an outage a failure?
Not for training purposes if the agent stopped within budget and handed off correctly. Mark it as truncated rather than terminated, and score the escalation separately from task completion.
Can I use Toxiproxy or Envoy for agent environments?
Yes for network-level faults such as latency, resets, truncation and status-code aborts. Faults that depend on whether a write landed have to be modeled inside the mock service or tool wrapper, because a proxy cannot see business state.
Sources
- Toxiproxy's toxics include latency with jitter, a downed proxy, bandwidth limits, timeouts, TCP resets, sliced data, data limits and packet loss, each applied with a toxicity probability. Source
- Envoy's fault injection filter injects delays and aborts with chosen codes, can be controlled per request by headers and percentages, and cannot emulate local CPU or disk failures. Source
- Google's canonical error codes say DEADLINE_EXCEEDED may be returned for a state-changing operation that actually completed, and that UNAVAILABLE is usually transient but not always safe to retry for non-idempotent operations. Source
- RFC 9110 lets a server send Retry-After with a 503 response, and defines its value as an HTTP date or a number of seconds. Source
- RFC 6585 defines 429 Too Many Requests and allows the response to include a Retry-After header. Source
- Stripe stores the first result for an idempotency key, returns it on retries with that key including 500 errors, and rejects reused keys with different parameters. Source
- The Amazon Builders' Library explains that retries at multiple layers multiply load, recommends capped exponential backoff with jitter, and says APIs with side effects should be idempotent before retrying. Source
- tau-bench sets an episode's reward by comparing a hash of the final data with the hash produced by replaying the task's ground-truth actions. Source
- Gymnasium separates terminated from truncated episode endings and uses terminated alone when bootstrapping value targets. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.