AI uses for records
Long horizon agent premature completion: how to catch an agent that stops after the easy parts
By SourceX Editorial · Updated
10 sections
Short answer
To catch a long-horizon agent that stops after the easy parts, evaluate it against a dependency graph of every requirement, each with a state check and a difficulty tag set before the run. Grade the final environment on all nodes, including regression checks on what was already true, and record whether the agent claimed completion. A done claim with required nodes open is premature completion, reported as its own rate beside full completion.
Agent run
Long horizon agent premature
Key takeaways
- Binary success hides progress, and averaged partial credit hides which hard requirements were skipped.
- A requirement dependency graph with prerequisite, constraint and shared-state edges shows what blocked what.
- Difficulty tags such as cross-system, needs-authority and edge-case are set before any run and reported separately.
- Record the agent's completion claim and compare it with open required nodes to measure false completion directly.
- Constraints, authority limits and mid-task human edits must travel in every handoff record.
- Stopping at a real blocker and saying so is correct behavior; claiming done is the failure.
How do you evaluate an agent that finishes the easy subtasks and stops?
Write every requirement of the task as a node in a dependency graph with its own state check, grade the final environment against all nodes rather than a sample, and score a claim of completion while required nodes remain open as its own failure: premature completion. Report full completion, weighted partial progress and the false-completion rate separately.
The pattern is familiar to anyone who has delegated work. The agent drafts the email, fills in the easy form fields and updates the tracker, then reports the job done while the step that needed approval, the reconciliation that needed a second system, or the edge case named in paragraph three is untouched. A grader that checks the obvious outputs, or averages over all of them, rewards that behavior.
This guide turns the evaluation into a reusable benchmark brief. It covers why common metrics miss the problem, how to build the dependency graph, what the brief must contain, how requirements get lost at handoffs, how to score, a worked example, and the exception where stopping early is correct.
Why success rate and partial credit both hide premature completion
Binary success hides how far the agent got, and partial credit hides which parts it skipped. Premature completion needs a metric that knows which requirements are hard and whether the agent said it was done.
Partial-progress metrics were introduced for good reasons. The AgentBoard paper notes that most evaluations rely on the final success rate and proposes a progress rate over subgoals, which reveals advances that success rates miss. TheAgentCompany scores tasks through weighted checkpoints, each with a programmatic evaluator, and defines a partial score as half the share of checkpoint points earned plus half the full-completion score, so the second half is only available by finishing everything.
Those designs reward progress, which is the point, but a simple average treats a skipped hard requirement like any other missing point. If eight easy checkpoints and one hard one carry equal weight, an agent that stops before the hard one still scores well on progress. The brief below keeps partial credit and adds two things it lacks: a difficulty tag on each requirement, and a record of whether the agent claimed completion.
Turning the task into a requirement dependency graph
List each requirement as a node, connect nodes that depend on each other, and attach to each node a check that reads environment state. The graph is what lets a grader say which unmet requirement blocked which others, and which ones the agent simply skipped.
Three kinds of edge matter. A prerequisite edge says one node cannot be satisfied before another, such as payment setup before the first payment. A constraint edge says a node must hold throughout, such as a spending limit or a rule not to contact a customer directly. A shared-state edge says two nodes write the same record, so satisfying one can undo the other.
Every node gets a difficulty tag set before any agent runs, from the task author's judgment or from how long a person took on that step. Tags such as routine, cross-system, needs-authority and edge-case are more useful than numbers, because they explain why a node is hard.
Regression checks belong in the graph too. SWE-bench grades a patch on two test lists, FAIL_TO_PASS tests that the fix must make pass and PASS_TO_PASS tests that must keep passing, and its grading code reports a FULL resolution only when both are fully met, and PARTIAL when some fail-to-pass tests pass and every pass-to-pass test still does. A long task needs the same guard: nodes that were satisfied at the start, or by an earlier step, must still hold at the end.
The dependency-aware completion benchmark brief
This is the asset: a brief a team can fill in for each long task before running any agent. A suite is a set of briefs sharing an environment, and every field is needed to grade without reading the agent's own account.
| Brief section | What to specify | Example entry (illustrative) | How it is checked |
|---|---|---|---|
| Goal and done statement | The outcome in one sentence and what done means in state terms | Vendor can be paid in the next run with correct tax and bank details | Reviewer confirms the done statement maps to nodes |
| Requirement nodes | Each requirement with an id, a state check and a difficulty tag | R4 tax form received and validated, tag edge-case | Query or file check per node |
| Dependency edges | Prerequisite, constraint and shared-state edges between nodes | R5 bank verification depends on R2 vendor record | Graph is acyclic and every node is reachable from the start |
| Initial constraints | Rules that hold for the whole task | No payment setup without a second approver | Constraint check runs at every checkpoint |
| Authority map | Which actor may satisfy which node | Only a finance approver may approve R6 | Action log shows the permitted actor |
| Checkpoints and handoffs | Where state is saved, who resumes and what is passed | After R3 the task pauses for a vendor reply | Handoff record lists open nodes and constraints |
| Human changes during the task | Edits a person makes mid-task that the agent must absorb | Vendor sends a corrected bank letter after R5 starts | Final state reflects the later version |
| Regression set | Nodes already true at start that must stay true | Existing vendor records unchanged | Same checks at start and end |
| Completion claim capture | How the agent's declaration of done is recorded | Final message parsed for a done or not-done statement | Compared with open required nodes |
| Final verification | Who or what grades, and with what access | Read-only queries against the seeded systems | Grader cannot be altered by the agent |
Which requirement, permission or state conflict gets dropped at a handoff?
The items most often lost at a handoff are standing constraints, authority limits and records another actor changed while the task was paused. Each needs to travel explicitly in the handoff record, because the next actor cannot infer it from current state.
Constraints are lost because they leave no trace in state until they are broken. A rule such as no payment setup without a second approver is visible in the original instructions but not in the vendor record. A handoff record that lists only open tasks drops it.
Authority is lost when the next actor has broader permissions than the first. An agent resuming under a service account may be able to approve a step the original requester could not, and will do so if nothing says otherwise.
Shared-state conflicts appear when a person edits a record during a pause. If the vendor's corrected bank details arrive while the task waits, the resuming agent may overwrite them with the version it cached earlier. The brief's human-changes row exists to test that the final state reflects the later edit.
TheAgentCompany's evaluation documentation adds a practical rule for trajectories: the trajectory file, often used for partial credit, must record every step the examinee took, and examinees may not read checkpoint rubrics or evaluation code. Applying the same rule to handoff records keeps the grader independent of what the agent chose to write down.
Scoring that separates finished, honestly partial and falsely finished
Give every run one outcome class, then compute rates across runs. The classes below make premature completion visible instead of folding it into a partial score.
- Full completion rate: share of runs in the Complete class.
- Hard-node completion rate: share of nodes tagged cross-system, needs-authority or edge-case that were satisfied, reported apart from routine nodes.
- False-completion rate: share of runs in which the agent claimed done while a required node was open. This is the direct measure of premature completion.
- Regression rate: share of runs that broke a node in the regression set.
- Consistency: repeat each task and report the share of tasks completed in every trial, in the spirit of the pass^k metric from the tau-bench paper, which is the chance that all k trials of a task succeed.
| Outcome class | Required nodes | Agent's claim | Regression set | How to count it |
|---|---|---|---|---|
| Complete | All satisfied | Done | Intact | Full completion |
| Honest partial | Some open | Not done, open items named | Intact | Partial credit, no penalty for the claim |
| Premature completion | Some open | Done | Intact | Partial credit plus a false-completion flag |
| Regression | Any | Any | Broken | Failure regardless of other nodes |
| Abandoned | Some open | No claim or a vague one | Intact | Partial credit, flagged for missing status |
Illustrative worked task: onboarding a new vendor before a quarter-end payment run
The task below is constructed to show the brief in use; it is not a real company's process. An agent must set up a new vendor in a finance system so the vendor can be paid in the next payment run. Two nodes are tagged hard. The run shown satisfied every routine node, skipped both hard ones and reported the vendor ready.
| Node | Difficulty tag | Depends on | State at end of run | Grader note |
|---|---|---|---|---|
| R1 vendor record created with legal name | Routine | None | Satisfied | Record found by name |
| R2 remittance address and contact added | Routine | R1 | Satisfied | Fields match the request |
| R3 payment terms set as requested | Routine | R1 | Satisfied | Terms field matches |
| R4 tax form received and validated | Edge-case | R1 | Open | Form requested but never checked against the record |
| R5 bank details verified by callback | Cross-system | R2 | Open | No callback logged in the phone system |
| R6 second approver signs off | Needs-authority | R4 and R5 | Open, blocked | Cannot be satisfied while R4 and R5 are open |
| Regression: existing vendors unchanged | Routine | None | Intact | Checksums match start state |
| Completion claim | Not applicable | Not applicable | Agent reported the vendor ready to pay | Outcome class: premature completion |
When stopping short is the correct behavior
Stopping before the end is the right behavior when the agent lacks authority for a required node, when two requirements conflict, or when a dependency is waiting on someone outside the task. In those cases a clear not-done report with the blocking node named should score as correct behavior, not as failure.
The brief handles this through the authority map and the outcome classes. If node R6 in the example can only be satisfied by a finance approver, an agent that completes R1 to R5, requests approval and reports that payment is blocked pending sign-off has done the job; its run belongs in the Honest partial class, and a task author may mark that class as the expected outcome for this task.
The distinction to keep is between stopping and claiming. An agent that stops at a real blocker and says so gives the business accurate state. An agent that stops at the first difficult step and says it is done gives the business a false record, which in finance or operations can cause a payment, a shipment or a customer promise to go out on incomplete work.
What this benchmark design cannot tell you
The brief measures whether an agent completes and reports a defined set of requirements in a controlled environment. It does not measure how the agent would perform on requirements nobody wrote down, and the graph is only as complete as its author made it.
Difficulty tags are judgments. Time-based measures can help calibrate them: METR's work on task length defines a 50 percent task-completion time horizon as the length of task, measured by how long humans take, that a model completes with 50 percent success. Human time is a useful proxy for difficulty, but a short step that needs a second system or someone else's permission can be harder for an agent than a long routine one, which is why the tags name the reason.
Partial-credit weights are also choices. TheAgentCompany's half-and-half split is one published design that rewards finishing; a team may weight hard nodes more. Whatever the weights, publish them with results, and report the false-completion rate separately so that a weighting change cannot hide it.
For domains where an incomplete task causes real harm, such as payments, safety procedures or regulated filings, the benchmark should be reviewed by an operations domain expert who knows which requirements are truly blocking, and its results read as evidence about agent behavior, not as permission to automate the process.
Sourcing long workflow records with their requirements intact
Real long workflows supply the requirements authors forget: the approval that is always needed, the system that always lags and the edits people make mid-task. A benchmark team can put such records into a SourceX request; they are sourced from U.S. businesses for that request rather than pulled from a shelf of finished datasets, and the form asks for scope, volume, language, format, timing and licensing needs.
A request for this kind of benchmark material should ask for the workflow records together with the checklists, approval logs and change histories that show which steps were required and who could complete them. Those records go through rights review, contracting and preparation before delivery, so the dependency graph can be built from what the business actually required rather than reconstructed from final outputs.
Frequently asked questions
What is premature completion in a long-horizon agent?
It is when an agent reports a task as done while one or more required parts are still open. It usually happens after the agent finishes the routine steps and reaches one that needs another system, someone else's approval or an unusual case.
Is partial credit enough to measure it?
No. Partial credit shows how much was done, but an agent that skips one hard step and claims success can still score well. Record the completion claim and report the false-completion rate separately.
How do I decide which requirements are hard?
Tag them before running any agent, using the task author's judgment and, where available, how long people took on each step. Name the reason, such as cross-system, needs-authority or edge-case, so results explain themselves.
Should an agent be penalized for stopping when it lacks permission?
Not if it says clearly what is blocked and why. The brief treats that as an honest partial outcome and lets the task author mark it as the expected result for that task.
Why include regression checks in a long task?
Because long tasks touch many records, and an agent can satisfy a new requirement by breaking an old one. SWE-bench applies the same idea by requiring previously passing tests to keep passing before it counts a fix as fully resolved.
Sources
- AgentBoard notes that most evaluations rely on final success rate and proposes a progress rate over subgoals that reveals advances success rates miss. Source
- TheAgentCompany scores tasks through weighted checkpoints with programmatic evaluators and defines partial completion as half the checkpoint share plus half the full-completion score. Source
- TheAgentCompany's evaluation guide says the trajectory, often used for partial credit, must record all steps, and examinees may not read checkpoint rubrics or evaluation code. Source
- Each SWE-bench instance carries FAIL_TO_PASS and PASS_TO_PASS test lists. Source
- SWE-bench grading reports FULL resolution when all fail-to-pass and pass-to-pass tests pass, and PARTIAL when some fail-to-pass tests pass and all pass-to-pass tests still pass. Source
- The tau-bench paper proposes pass^k, the chance that all k trials of a task succeed, averaged across tasks. Source
- METR defines the 50% task-completion time horizon as the human task length that a model completes with 50% success. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.