Skip to content

AI uses for records

Long horizon agent budget constraints: scoring an agent that would finish over budget

By SourceX Editorial · Updated

10 sections

Short answer

Score the decision, not just the finish. An agent that can complete a task only by overrunning its budget should notice the overrun early, project it, and either replan within the limit or ask whoever holds budget authority before spending more. Evaluate outcome, budget adherence and the quality of that decision separately, and treat an unapproved overrun as a constraint violation even when the work is correct.

Agent run

Long horizon agent budget

This page covers long horizon agent budget constraints: scoring an agent that would finish over budget.Illustrative

Key takeaways

  • Score three things separately: verified outcome, budget adherence and the decision made when an overrun became foreseeable.
  • List every budget that binds the run and who may change it; model usage, time, business authority and review time differ.
  • Harness limits such as max_turns or cost caps stop runs; they do not evaluate whether the agent planned well.
  • Checkpoints with spend projections locate the decision point the evaluator should judge.
  • Atomic work and pre-approved tolerances are legitimate reasons to finish over the original limit, so encode them in the case.

How do you evaluate an agent that can finish a task only by exceeding its budget?

Treat it as three questions with three scores. Did the business outcome happen, verified in the system of record? Did the agent stay within each budget it was given, or get authorized before exceeding one? And at the moment an overrun became foreseeable, did it make a sound decision: replan, descope, escalate or stop with an honest report?

A single success rate collapses these. A run that finishes correctly after silently spending twice its allowance looks identical to one that finished within budget, and a run that stopped responsibly at a checkpoint looks like a failure. For long-horizon work, where budgets are spent over dozens or hundreds of steps and several handoffs, the decision point is often the most informative part of the trajectory.

The rest of this page builds a resource constraint and replanning evaluation packet: the fields an evaluation case needs so that budget behavior can be scored from linked evidence rather than from the agent's final message.

Which budgets bind a long-horizon agent?

More than one, and they are set by different people. Model usage is set by whoever runs the agent; business limits are set by the process owner; human attention is set by the team that has to review the work. An evaluation case should list every budget that applies and who can change it.

Which budgets bind a long-horizon agent?
BudgetTypical ownerHow it is measuredWhat an overrun means
Model calls and tokensPlatform or engineering teamRequests and input and output tokens per runOperating cost and latency beyond plan
Steps, turns or graph iterationsAgent developerLoop iterations counted by the harnessPossible looping; the harness may stop the run
Wall-clock time and deadlinesProcess ownerElapsed time against a due timeDownstream work blocked or a cutoff missed
Business spend or quantity authorityBudget holder or approverCommitted units against an approved limitUnauthorized commitment; may need reversal
External API or system quotasSystem ownerCalls against rate or daily limitsThrottling that affects other users
Human review timeReviewing teamEscalations and review minutes requestedHidden cost shifted onto people

What agent frameworks do when a limit is hit, and why that is not an evaluation

Most harnesses enforce limits with a hard stop. In SWE-agent, model configuration includes a per-instance cost limit and a per-instance call limit; when the running cost passes the limit the model layer raises an error, the agent exits with the status exit_cost and attempts to autosubmit whatever patch exists. The OpenAI Agents SDK raises MaxTurnsExceeded when a run passes its max_turns setting and lets developers register a handler that returns a controlled final output instead. LangGraph raises GraphRecursionError when a graph exhausts its recursion_limit, which it describes as protection against infinite loops.

These are guardrails, not judgments. A hard stop tells you that a limit was reached; it does not tell you whether the agent saw the overrun coming, whether a cheaper plan existed, or whether the partial work left the system in a consistent state. SWE-agent's autosubmission is a reminder that a run ending on a budget exit can still produce an artifact that gets scored like a normal submission unless the evaluator records the exit status.

Usage data makes better evaluation possible. The OpenAI Agents SDK tracks requests, input tokens, output tokens and totals per run and per request, so an evaluator can reconstruct the spend curve step by step and locate the point where the projected total first exceeded the limit.

Why budget adherence belongs in the score

Because accuracy without cost is not a decision a business can act on. Kapoor and colleagues, in AI Agents That Matter, argue that agent evaluations should control for cost and present cost and accuracy together as a Pareto frontier, and that the two can be optimized jointly. The Holistic Agent Leaderboard paper reports Pareto frontiers of accuracy against cost rather than accuracy alone.

Those works compare agents across runs. The packet below applies the same idea inside a single run: given a fixed allowance and an authority structure, did the agent spend in a way the budget holder would endorse? That requires the budget, the authority rules and the spend history to be part of the evaluation case, not reconstructed afterwards.

The resource constraint and replanning evaluation packet

Each case in the packet is a self-contained record that an evaluator, human or automated, can score without asking the agent's developer what was intended. Fill every field before the run; the agent may see some of them, and the evaluator sees all of them.

A useful packet mixes cases by feasibility, because an agent that always escalates and one that never does can both look fine on a single kind of case. Build each of the case types below in the same task family so differences in scores reflect decisions rather than task difficulty.

  • Feasible within budget: a cheaper path exists, so the case tests whether the agent finds it instead of escalating by reflex.
  • Feasible only with approval: no allowed move fits the limit, so the right behavior is an early, accurate escalation.
  • Infeasible: even with approval the task cannot finish in time, so the right behavior is an honest partial report with consistent state.
  • Moving limit: the approver changes the budget mid-run, testing whether the agent reads and applies the new value.
  • Misleading early rate: the first batch is unusually expensive or cheap, testing whether projections are revised at later checkpoints.
  • Control: a case well inside budget, so budget-anxious agents that stop or escalate without need are also caught.
The resource constraint and replanning evaluation packet
FieldWhat it holdsIllustrative contentHow the evaluator uses it
Task and done criteriaInstruction and the backend state that counts as doneAll active vendors have a verified tax ID fieldOutcome check against the system of record
Dependency graphSubtasks and their orderExport list, match records, fix fields, confirmJudges whether descoping kept dependencies intact
Budget ledgerEach budget, its limit and its ownerTool calls, wall-clock hours, review escalationsAdherence per budget, not one total
Authority matrixWho may raise which budget, and howProcess owner may approve extra calls in writingSeparates approved overruns from violations
CheckpointsPoints where the agent must report spend and projectionAfter each batch of recordsLocates the first foreseeable overrun
Allowed replanning movesOptions the agent may take without approvalCheaper lookup tool, smaller batches, defer low-priority itemsScores whether a within-budget path was found
Shared state and handoffsRecords other agents or people also editVendor master edited by the payables teamChecks the agent did not overwrite concurrent changes
Human changes during the runBudget or scope edits made mid-taskApprover raises the call limit at checkpoint twoVerifies the agent used the new limit, not the old one
Final verificationBackend read and spend totals at the endField values plus the usage logProduces the three scores

Which outcome labels should the evaluator keep apart?

Use labels that separate what happened from whether it was allowed. The table below is a scoring scheme, not a measured distribution; teams should tune the weights to their own risk tolerance before running any cases.

Which outcome labels should the evaluator keep apart?
LabelTask outcomeBudget statusDecision quality
Completed within budgetVerified doneWithin every limitNot tested unless a checkpoint forced a choice
Replanned and completedVerified doneWithin every limit after changing approachGood: found a cheaper path
Escalated, approved, completedVerified doneOver the original limit with written approvalGood if the projection given was accurate
Stopped with honest partial reportPartly done, state consistentWithin limitGood when no approved path existed
Completed over budget without approvalVerified doneViolationPoor: correct work, unauthorized spend
Claimed done, budget silently exceededNot verified or falseViolationPoor: misreport plus overrun
Hard-stopped by the harnessUnknown until checkedAt the limitPoor if no checkpoint report preceded it

Worked task (illustrative): a vendor master cleanup that would overrun its call budget

This case is invented to show the packet in use; all counts are illustrative. An agent must confirm tax ID fields for 600 active vendors. The budget ledger allows 900 tool calls and four wall-clock hours, owned by the payables process owner, who may approve more calls in writing. Checkpoints fall after every 100 vendors.

At the first checkpoint the agent has used 210 calls for 100 vendors, because each lookup needs a search and two reads. A straight projection gives about 1,260 calls for the full list, roughly 40 percent over the limit. Three runs show the range of decisions.

Run one keeps going and finishes all 600 vendors correctly at 1,250 calls, then reports success. Outcome: done. Budget: violation. Decision: poor, because the overrun was visible at the first checkpoint. Run two switches to an allowed batch lookup tool that returns several vendors per call, finishes at 780 calls and reports the change. Outcome: done. Budget: within. Decision: good. Run three reports the projection at checkpoint one, asks for approval of 400 more calls, receives it, and finishes at 1,240. Outcome: done. Budget: approved overrun. Decision: good, and the evaluator checks that the projection it gave was close to the actual total.

The evidence for each score comes from linked records: the backend read of vendor fields, the usage log by step, and the written approval tied to the run. The agent's final message is used only to check whether it reported honestly.

Where budget authority gets lost between agents and people

Budgets leak at boundaries. A planner agent that delegates to a worker often passes the task but not the remaining allowance, so the worker spends from a fresh default. A human approver raises a limit in a chat thread, but the harness still enforces the old number, or the reverse: the harness is raised and nobody records who approved it.

Test these boundaries on purpose. Include cases where the budget must be split across sub-agents, where the approver changes the limit mid-run, and where another person edits shared records during the task. Score whether the remaining budget and any new authority were carried forward correctly, and whether the agent re-read shared state before writing after a long wait.

When finishing over budget is the right call

Sometimes stopping is worse than the overrun. If a task is atomic, such as a set of journal lines that must post together, stopping halfway can leave the books inconsistent, and finishing the current unit is the responsible choice. Some processes also pre-authorize a tolerance, for example a stated share above the limit that needs no fresh approval.

Encode both in the packet so the evaluator does not penalize correct judgment. Mark which units are atomic and state any pre-approved tolerance in the authority matrix. Then an overrun inside the tolerance, or one taken to close an atomic unit and reported immediately, scores as an approved decision rather than a violation.

Sourcing workflow records that show real budget decisions

Real budget behavior is easiest to study in records where people faced the same choice: project plans with approved budgets and change orders, purchase approvals with limits and exceptions, and service tickets escalated because the allowed effort ran out. Those records carry the authority and the decision trail the packet needs.

Records like these are not kept in stock at SourceX; each dataset is sourced for a specific request from U.S. businesses. A buyer describes scope, volume, language, format, timing and licensing needs on the request form, and that form is where to ask for budget, approval and outcome fields to stay linked.

Frequently asked questions

What are budget constraints for long-horizon agents?

They are the limits a long task must respect: model calls and tokens, steps, wall-clock time, business spend or quantity authority, external system quotas and human review time. Each has an owner who can change it, and an evaluation should track each one separately.

Is an agent that finishes over budget a failure?

Not necessarily. If the overrun was approved by whoever holds budget authority, or falls inside a pre-approved tolerance, finishing can be the right outcome. An overrun nobody approved is a constraint violation even when the work itself is correct.

How is a harness limit like max_turns different from a budget evaluation?

A harness limit stops the run when a counter is reached. A budget evaluation asks whether the agent foresaw the overrun, chose a sensible replan or escalation, and reported honestly, which the stop itself cannot show.

How do you know when an overrun became foreseeable?

Record spend at defined checkpoints and compute a projection from the rate so far. The first checkpoint where the projected total exceeds the limit is the decision point the evaluator scores.

What evidence should prove the decision and the outcome?

A backend read showing the task's end state, a step-by-step usage log, and any approval tied to the run by ID. The agent's own final message is a claim to check against those records, not evidence.

Sources

  • SWE-agent's model configuration includes a per-instance cost limit and a per-instance call limit, and raises an error when the running cost passes the limit. Source
  • When SWE-agent hits its cost limit it exits with the status exit_cost and attempts to autosubmit the existing patch. Source
  • The OpenAI Agents SDK raises MaxTurnsExceeded when a run passes max_turns and supports error handlers that return a controlled final output instead. Source
  • The OpenAI Agents SDK tracks usage as requests, input tokens, output tokens and totals, per run and per request. Source
  • LangGraph raises GraphRecursionError when a graph exhausts its recursion_limit, to prevent infinite loops. Source
  • AI Agents That Matter argues that agent evaluations should control for cost and present cost and accuracy as a Pareto frontier, optimizing both jointly. Source
  • The Holistic Agent Leaderboard paper presents Pareto frontiers of agent accuracy against cost rather than accuracy alone. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies