AI uses for records
Long horizon agent recovery cost: how much work gets repeated
By SourceX Editorial · Updated
10 sections
Short answer
Measure repeated work after an agent failure by fixing the last verified checkpoint, then labeling every later step as redo, repair, reorientation or new progress. Redo shows whether finished work survived, repair shows unhandled side effects, and reorientation shows lost context. Count a run complete only when independent end-state checks pass, and report the ratios across repeated runs rather than from one trace.
Agent run
Long horizon agent recovery
Key takeaways
- Split post-failure work into redo, repair, reorientation and new progress, measured from the last verified checkpoint.
- High redo points at checkpointing, high repair at side-effect handling, high reorientation at lost state or constraints.
- Record the harness configuration, because checkpoints and redrive decide much of the repeated work.
- Duplicate payments, messages or bookings are severity flags, whatever the other ratios say.
- A run is complete only when independent end-state checks pass, and ratios belong in distributions across repeated runs.
How do you measure repeated work after an agent failure?
Measure repeated work by fixing the last verified checkpoint before the failure, then classifying every step the agent takes afterward as redo (repeating work that was already done and verified), repair (undoing or reconciling side effects), reorientation (re-reading context and re-planning) or new progress. Recovery overhead is the redo, repair and reorientation steps together, set against the work completed before the failure, and the run only counts as complete when an independent check of the end state passes.
The decision rule that follows is simple. If most recovery cost is redo, the harness is not preserving finished work and checkpointing should be fixed first. If most is repair, side effects are not idempotent or not reconciled. If most is reorientation, the agent is losing its constraints and state between attempts, which is a model or memory problem. Each answer calls for a different investment.
This page gives the worksheet: the inputs and their definitions, the formulas, the assumptions that change the answer and an illustrative scenario. It reports no measurements for any agent or model.
Why recovery cost needs its own number
Success rate says whether an agent finished; it does not say what finishing cost after something went wrong. Two agents with the same success rate can differ widely in how much work they redo, how many duplicate side effects they leave and how long a person waits. On long tasks those differences decide whether an agent is usable. An agent that finishes a 60-step reconciliation after a timeout by redoing 50 steps, and another that redoes 2, report the same success, yet only one of them is cheap enough to leave running overnight.
Long-horizon evaluation already points this way. METR's paper on measuring AI ability to complete long tasks defines time horizon by how long tasks take human professionals, and attributes the growth in agents' horizons mainly to greater reliability, ability to adapt to mistakes, logical reasoning and tool use. If adapting to mistakes drives progress, how much an adaptation costs deserves a measurement of its own.
Reliability also needs repetition. The tau-bench paper grades agents by comparing the final database state with an annotated goal and introduces pass^k, the chance that all k trials of a task succeed. Recovery overhead should be reported the same way: across repeated runs of the same task and failure, as a distribution rather than one lucky or unlucky trace.
What the runtime already saves before the agent decides anything
Part of the repeated work is set by infrastructure, not by the agent, so record the harness configuration with every measurement. LangGraph's documentation says checkpoints let a graph restart from the last successful step after node failures, and that when one node fails mid-step, writes from nodes that completed in the same step are stored so they are not re-run on resume. Its fault-tolerance guide adds per-node retries, timeouts and error handlers that run after retries are exhausted.
Workflow engines make the same promise at a different grain. AWS documents that redriving a failed Step Functions Standard workflow restarts from the failed step, keeping the results and history of the steps that succeeded. Under either system, a well-configured harness should drive measured redo toward zero for completed steps; any redo that remains comes from the agent choosing to repeat work, or from steps the harness could not checkpoint.
Side effects need their own guard. Stripe's documentation describes idempotency keys that let a client retry a request without performing the operation twice, and advises treating a 500 response as indeterminate because side effects may already have happened, so the outcome should be reconciled before retrying under a new key. An agent that resumes and repeats a payment, an email or a booking without that check creates repair work that no checkpoint can prevent.
The recovery overhead and verified completion worksheet: inputs
Fill one row set per failure event. Units are steps by default because they are easiest to count from a trace; tool calls, tokens and wall-clock minutes can be added as parallel columns. Whatever unit you choose, keep it fixed across agents being compared.
At a glance
The recovery overhead and verified completion worksheet: inputs
| Input | Definition | Where it comes from | Assumption to record |
|---|---|---|---|
| S_before | Steps completed and verified before the failure | Trace up to the last verified checkpoint | What counts as verified |
| Failure type | Tool error, timeout, crash, wrong action caught later, human interruption | Trace and harness logs | One primary type per event |
| S_redo | Steps after failure that repeat already verified work | Matching post-failure steps to pre-failure step IDs | Matching rule for near-identical steps |
| S_repair | Steps that undo, reconcile or compensate for side effects | Calls that reverse, cancel or query the outcome of earlier writes | Whether a status check counts as repair |
| S_reorient | Steps spent re-reading context, re-planning or re-asking for information already given | Reads and messages with no new information | Threshold for no new information |
| S_new | Steps that make progress beyond the checkpoint | Remaining steps that advance the task | Final step that ends the task |
| Duplicates | Side effects that happened twice, such as two emails or two charges | System of record after the run | Which duplicates are harmless |
| Lost constraints | Requirements or approvals present before the failure and violated after | Gold constraint list compared with end state | Constraint list frozen before the run |
| Verified completion | Independent end-state checks passed, out of total checks | Grader reading the system of record | Who wrote the checks and when |
Formulas and the decision each one drives
Compute these per failure event, then summarize per agent and harness configuration across repeated runs. Report medians and spread, not only means, because a few runs that restart from scratch can dominate an average.
- Redo ratio = S_redo / S_before. Near zero means finished work survived the failure; near one means the agent or harness effectively restarted. A high ratio points at checkpointing first.
- Repair share = S_repair / (S_redo + S_repair + S_reorient). A high share means side effects are not idempotent or not reconciled; fix keys and status checks before tuning the model.
- Reorientation share = S_reorient / (S_redo + S_repair + S_reorient). A high share means context and constraints are not carried across the failure; look at memory, state handoff and prompts.
- Recovery overhead = (S_redo + S_repair + S_reorient) / S_new. The cost of getting back on track, relative to the work still to do.
- Duplicate rate = duplicate side effects per failure event. Any duplicate on a payment, message or booking is a severity flag regardless of the other numbers.
- Constraint retention = 1 minus lost constraints over constraints in force at the failure.
- Verified completion = end-state checks passed over total checks, required at 100% for a run to count as complete. A run that ends with the agent declaring success but fails a check is incomplete, not partly complete.
Planting failures so the measurement can be repeated
Natural failures are uneven: they cluster at a few flaky tools and rarely land after an irreversible write. To compare agents or harnesses fairly, inject failures at chosen points in otherwise identical runs and leave natural failures for a separate, observational report.
Choose injection points by what they test. A failure before any write tests whether the agent re-plans cheaply. A failure just after a write whose response was lost tests whether it reconciles before repeating. A failure after a human approval tests whether it notices the approval may have lapsed. A crash of the whole process, rather than one tool call, tests what the harness actually persisted.
- Freeze the task, tool versions, harness settings and gold end-state checks before the first run.
- Log step IDs that stay stable across attempts so post-failure steps can be matched to earlier ones.
- Inject each failure type at each chosen point, and run every combination several times.
- Label post-failure steps with two annotators, using the definitions in the worksheet.
- Grade every run against the same end-state checks, read from the system of record.
What a resumed attempt forgets: constraints, approvals and edits made while it was down
Whatever lived only in the failed attempt's working context is at risk on resume. The most fragile items are recent ones: a constraint the user added three steps before the crash, an approval granted in a message, or the fact that two systems disagreed and the agent had chosen one. A checkpoint of graph state does not hold them unless someone put them there.
At a glance
What a resumed attempt forgets: constraints, approvals and edits made while it was down
Permissions are a special case. A token or approval valid before the failure may have expired by the time the agent resumes, and an agent that replays its earlier plan can act without authority it no longer holds. Shared state can also move: a person may have edited the record, reassigned the ticket or completed the step manually while the agent was down.
Anthropic's guidance on building agents says agents should gain ground truth from the environment at each step and can pause for human feedback at checkpoints or when blocked. On resume, that means rereading the current state rather than trusting the checkpoint. Count those reads as reorientation, but do not penalize them: a resume that skips them is cheaper and wrong. Keep the following in the handoff record for each checkpoint:
- Constraints in force, each with the step where it was introduced.
- Authority held, who granted it and when it expires.
- Side effects issued, with idempotency keys or external IDs so their outcome can be queried.
- Open conflicts between sources and which one the plan relies on.
- Human edits expected or possible while the task is paused.
Illustrative: a procurement agent fails at step 26
This scenario is invented to show the worksheet; every count is illustrative. A procurement agent runs a 40-step task: gather three supplier quotes, check them against a budget code, draft a purchase order, get approval and send the order. At step 26, after approval and while sending, the ERP call times out. The last verified checkpoint is step 25, so S_before is 25.
Recovery A restarts the task from scratch. It redoes 22 steps, sends a second quote request to all three suppliers, and finishes the 15 remaining steps from 26 to 40: redo ratio 0.88, duplicates 3, overhead 22 over 15. Recovery B resumes from the checkpoint, replays the send without checking, and creates a second purchase order: redo ratio 0, repair 4 steps to cancel the duplicate, duplicates 1. Recovery C resumes, queries the ERP by the purchase order's external ID, finds the order was not created, rereads the approval, notices it expired at the end of the day and asks for re-approval before sending: redo 0, repair 1, reorientation 5, duplicates 0, constraint retention 100%.
C costs more steps than B and passes every end-state check; B looks efficient and fails the duplicate check. Without the worksheet, a step count alone would rank them the wrong way round.
Step definitions, verification rules and other assumptions that swing the ratios
Three assumptions move the numbers most. The first is what counts as a step: tool calls, model turns and plan items give different S values, so choose one and state it. The second is what counts as verified before the failure; a lenient definition inflates S_before and shrinks the redo ratio. The third is the matching rule for redo; whether a re-read of the same record is redo or reorientation shifts the shares between the two.
Show sensitivity rather than hiding it. Recompute the headline ratios under a strict and a lenient version of each assumption and report both. Where two annotators classify recovery steps, measure their agreement on the redo, repair and reorientation labels and rewrite the definitions where agreement is poor.
Report uncertainty from repetition as well. Run each task and failure several times, report the share of runs with zero duplicates and full verified completion, and give intervals for the ratios from resampling runs. Stop and bring in an operations domain expert when the end-state checks depend on business rules annotators do not own, such as when an expired approval must be renewed.
Where real failure and recovery history comes from
Planted failures make the worksheet repeatable; business records show how recovery actually goes. Workflow run histories with retries, failed integration syncs and their reprocessing, reissued or cancelled purchase orders in ERPs, duplicate-invoice corrections in accounting systems and reopened jobs in field service software all record a failure, the repeated or corrective work and the final state.
Those records carry their own conditions. System access, privacy permission to process them, ownership and a contractual right to license them for AI are separate, and run histories often hold identifiers that need removing.
SourceX can look for that history on an agent team's behalf. It holds no inventory; it approaches U.S. businesses with a request naming the scope, volume, language, format, timing and licensing terms needed, and records leave the business only after it has agreed the scope and terms and signed.
Frequently asked questions
What is recovery overhead for an AI agent?
It is the work an agent does after a failure that is not new progress: repeating finished steps, repairing side effects and re-orienting in the task. Measured against the work still to do, it shows how costly a failure was beyond whether the task eventually finished.
Does checkpointing remove repeated work?
It removes much of it when finished steps are saved and resume starts from the last good point. It does not remove repeated side effects that were sent but not recorded, or constraints that lived only in the agent's context, so measure both.
How do you avoid double side effects when an agent retries?
Use idempotency keys or external IDs for writes, and query the outcome of an uncertain call before repeating it. Count any duplicate payment, message or booking as a severity flag in the results.
Should re-reading context after a failure count against the agent?
Count it as reorientation, but do not treat it as waste by default. Rereading current state after a pause is often what prevents acting on stale approvals or records that a person changed.
What unit should recovery cost use?
Steps are easiest to count from traces. Tool calls, tokens and wall-clock minutes can be added, but keep one unit fixed when comparing agents and state which one each figure uses.
Sources
- METR's paper measures agent capability as a time horizon based on how long tasks take human professionals and attributes horizon growth mainly to reliability, ability to adapt to mistakes, logical reasoning and tool use. Source
- The tau-bench paper compares final database state with an annotated goal state and introduces pass^k, the probability that all k trials succeed. Source
- LangGraph's checkpointer documentation says checkpoints allow restarting from the last successful step and that pending writes from completed nodes are not re-run on resume. Source
- LangGraph's fault-tolerance guide describes per-node retries, timeouts and error handlers that run after retries are exhausted. Source
- AWS Step Functions redrive restarts a failed Standard workflow from the failed step while keeping the results of successful steps. Source
- Stripe's idempotent requests let a client retry an API request without performing the same operation twice. Source
- Stripe advises treating a 500 response as indeterminate and reconciling the outcome before retrying under a new idempotency key. Source
- Anthropic's agent guidance says agents gain ground truth from the environment at each step and can pause for human feedback at checkpoints or blockers. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.