AI uses for records
Long horizon agent deadline awareness: scoring correct but late work
By SourceX Editorial · Updated
9 sections
Short answer
Score a correct but late agent run on two axes. Grade correctness with the usual verifier, then multiply by a deadline utility fixed before the run: zero after a hard cutoff, a decaying value for a service target, full value for an internal goal. Report on-time rate, tardiness and breaches beside the combined score, and test whether the deadline survives every handoff.
Agent run
Long horizon agent deadline
Key takeaways
- Completion-only benchmarks score a correct late run the same as a correct timely one, so deadline failures stay invisible.
- Classify each deadline by what lateness costs: hard, firm, soft, paused service target, internal goal or dependent milestone.
- Freeze the clock definition, pause rules, deadline authority and utility curve before any agent runs.
- Report correctness, on-time rate, tardiness, breaches and notice rate together rather than one blended number.
- Test the deadline like any other requirement: it must survive handoffs, restarts and edits by people without authority.
How should an evaluation score correct work that misses a business deadline?
Score it on two axes, never one. Grade correctness with the same verifier you would use without a deadline, then multiply by a deadline utility that depends on what the business loses when the work is late: zero for a hard cutoff, a decaying value for a service target, full value for an internal goal. Report on-time rate and tardiness beside the combined score so a reader can see why a run lost credit.
The reason is simple. A reconciliation that balances perfectly but lands after the lender's covenant reporting cutoff is worth less to the business than one that lands on time, and sometimes worth nothing. A benchmark that counts both as a pass teaches the team, and any model trained on its labels, that time does not matter.
This guide is a method for evaluation designers at AI labs and agent teams. It gives the deadline classes, the fields an evaluation plan needs, the handoff checks that catch lost deadlines, and a worked example with illustrative numbers. It reports no model results.
Why completion-only agent benchmarks cannot see a missed deadline
Most agent benchmarks grade the end state and ignore the clock. tau-bench, for example, compares the database at the end of a conversation with an annotated goal state and adds pass^k, the probability that all of k trials succeed, to measure reliability. That design is right for what it measures, but a deadline is not part of the goal state, so a correct but late run looks identical to a correct and timely one.
TheAgentCompany goes further on partial work. Each task is split into checkpoints worth points, and its partial completion score gives half credit for the fraction of checkpoint points earned plus half for full completion, so only a fully completed task reaches 1.0. That is a useful model for incomplete work, and it still says nothing about when each checkpoint finished.
Time does appear in evaluation research, but usually as a measure of difficulty rather than a business constraint. METR's time horizon expresses task length as the time human professionals need to complete a task, and relates it to the length of tasks a model completes with a given success rate. TimeArena is one of the few environments that simulates action durations, dependencies and occupancy, and asks agents to finish several tasks as quickly as possible. Neither tells you how to score a correct answer delivered after the moment the business needed it.
The gap matters most for long-horizon work. Over many steps an agent can lose track of a due time stated once at the start, wait on a tool or a person without noticing the clock, or polish an answer past the point where it still has value.
Which kind of deadline is the agent facing?
Classify each deadline by what lateness costs, because the class decides the utility curve. Real-time systems offer a useful vocabulary: the Linux kernel's deadline scheduler documentation defines tardiness as the maximum of zero and the difference between a job's finishing time and its absolute deadline. Tardiness is the input; the class decides what a given amount of it costs.
Service tools show that business clocks are rarely simple wall time. Zendesk's requester wait time counts time in the New, Open and On-hold statuses and pauses while a ticket is Pending, and Jira Service Management lets teams add optional pause conditions, for example while waiting for a customer, and stop conditions that end the measurement. An evaluation that ignores those rules will mark runs late that the business would count as on time, and the reverse.
When one task carries several deadlines, score each one separately and let the strictest class govern the combined result. A vendor onboarding task might have an internal goal for the paperwork, a soft target for the first reply to the vendor and a hard cutoff for the payment run; averaging them would let a breach of the cutoff disappear inside two on-time results.
| Deadline class | What lateness costs | Utility after the deadline | Illustrative business case |
|---|---|---|---|
| Hard cutoff | The output cannot be used or creates a breach | Zero, and log a breach flag | A filing window or a bid submission portal that closes |
| Firm, value expires | The output is useless but no harm follows | Zero, no penalty beyond lost value | A same-day price quote after the customer has bought elsewhere |
| Soft, decaying | Value falls with each hour of delay | Declines from 1 to 0 over a stated window | A first reply to a priority support ticket |
| Paused service target | Only counted time matters | Soft curve applied to counted time only | A ticket clock that stops while waiting on the customer |
| Internal goal | Schedule slips but nothing external breaks | Full value; report tardiness only | A team's target to finish a draft by Friday |
| Dependent milestone | Lateness pushes later steps past their own deadlines | Scored through the downstream deadline it threatens | Vendor setup that must finish before the payment run |
The deadline and completion utility evaluation plan
The plan is a per-task record written before any agent runs, so the scoring rule cannot drift toward the results. Each table row is a field the evaluation designer fills in; the third column shows an illustrative entry for a single accounts payable task, and the list names the metrics every report should carry.
At a glance
The deadline and completion utility evaluation plan
- Combined task score: correctness score multiplied by deadline utility, where correctness may use partial checkpoint credit.
- On-time rate: share of runs whose verified completion falls at or before the deadline on the defined clock.
- Tardiness: zero for on-time runs, otherwise counted time past the deadline; report the median and the worst case.
- Breach count: runs that crossed a hard cutoff, reported even when the combined score already shows zero.
- Notice rate: share of late runs that warned the deadline owner before the deadline passed.
| Field | What to record | Illustrative entry | Why it is needed |
|---|---|---|---|
| Deadline source and authority | Who set the deadline and who may extend it | Controller; only the controller can extend | Separates a real extension from an agent moving its own goalposts |
| Clock definition | Calendar, time zone, business hours, pause and stop rules | Business hours, US Eastern, paused while awaiting vendor reply | Matches how the business itself measures lateness |
| Deadline class and curve | One class from the table above and its curve parameters | Soft, linear decay to zero over 16 counted hours | Fixes the utility before results exist |
| Completion verifier | End-state check, independent of time | Invoice batch posted and matched to purchase orders | Keeps correctness and timeliness separable |
| Checkpoints and intermediate deadlines | Ordered checkpoints, each with its own due time if any | Vendor verified by 12:00; batch posted by 17:00 | Locates where time was lost |
| Time accounting | Simulated or wall-clock time, and the cost of each action | Simulated: each tool call 2 minutes, each human reply 3 hours | Makes runs comparable across infrastructure |
| Notice and escalation credit | Whether a timely warning of a likely miss earns separate credit | Yes, if sent to the controller before the deadline | Rewards the behavior a business wants when a miss is unavoidable |
| Trials and reliability | Number of trials per task and the reliability metric | Four trials; report all-on-time across trials | One lucky fast run hides an unreliable agent |
| Reporting set | Metrics that must appear together | Correctness, on-time rate, median and worst tardiness, combined utility | Prevents a single blended number from hiding the cause |
| Version and date | Plan version and the date it was frozen | v1.0, frozen before the first run | Lets reviewers audit later changes |
Which clock is the agent racing, and who can stop it?
Decide whether the agent races simulated business time or the real wall clock, and write it down. Wall-clock time mixes model speed, rate limits and infrastructure into the result; simulated time charges each action a fixed duration, which is what TimeArena does with its action durations, and lets two systems be compared fairly. For most business-task evaluations, simulated time is the better default, with wall-clock latency reported separately.
Then define who can pause or extend the clock. A business clock pauses when work is legitimately blocked on someone else, such as a customer reply, and the pause rule must come from the business's own definition rather than the agent's claim. An extension counts only when it comes from the person named as the deadline authority and is recorded in the environment, for example as a ticket field or an approval message.
Anthropic's guidance on building agents notes that agents can pause for human feedback at checkpoints or when they hit blockers, and that runs often include stopping conditions such as a maximum number of iterations. In a deadline evaluation those pauses are scoring events: the plan must say whether waiting for a human stops the clock, and an iteration cap should be set so that an agent that loops is scored as late rather than silently truncated.
Which deadline, permission or shared-state change can be lost at the next handoff?
The deadline itself is often the first thing dropped when work moves between agents, tools or people. Test it the way you would test any other requirement: plant the due time once, then check whether it survives each handoff, and whether later changes to it are respected only when they come from someone with authority.
At a glance
Which deadline, permission or shared-state change can be lost at the next handoff
- Restatement check: after each handoff, does the receiving agent's plan or message carry the deadline, its time zone and its class?
- Authority check: if a colleague without authority asks for a later delivery, does the agent keep the original deadline and route the request to the owner?
- Human edit check: if the deadline owner moves the date in the shared record mid-task, does the agent adopt the new date and drop the old one?
- Shared-state check: when two agents work in parallel, does each read the deadline from one shared record rather than from its own copy?
- Dependency check: when an upstream step slips, does the agent recompute whether downstream deadlines can still be met and say so?
- Recovery check: after a failed tool call or a restart from a checkpoint, does the resumed run still know how much counted time remains?
- Final verification check: is the completion timestamp taken when the verifier passes, not when the agent declares it is done?
Illustrative: three runs on a month-end accrual package
The example below is illustrative and uses invented durations. The task: prepare and post a month-end accrual package due at 17:00 on the third business day, on a business-hours clock that pauses while waiting for a department head's confirmation. The plan classes it as soft, with utility falling linearly from 1 to 0 over 16 counted hours, and uses checkpoint credit in the TheAgentCompany style: a fully verified run scores 1, otherwise half the checkpoint fraction.
Run A posts a fully verified package at 15:40. Correctness 1, utility 1, combined score 1, tardiness zero. Run B posts a fully verified package 4 counted hours late because it re-ran a reconciliation it had already completed. Correctness 1, utility 0.75, combined 0.75. Run C completes three of four checkpoints by the deadline, cannot get the last confirmation, and emails the controller at 14:00 that the package will be late and why. Correctness 0.375, utility 1, combined 0.375, with notice credit recorded separately.
Change one field and the ranking changes. If the controller's plan said the package feeds a lender report with a hard cutoff, Run B drops to zero and records a breach, while Run C still shows partial work and a timely warning. That is the point of fixing the class before running: the same three runs tell a different and more honest story depending on what lateness actually costs.
One exception belongs in every plan. If the department head, as the named authority, moves the deadline by a day in the shared record before it passes, all three runs are scored against the amended deadline, and the evaluation logs the change. If the agent itself asserts an extension that no authority granted, the original deadline stands.
When a deadline-weighted score misleads
A deadline-weighted score is only as good as the deadline data behind it. Where the real due time is unknown, contested or set by law, a curve chosen by an evaluation designer is a guess, and the plan should say so or drop the task. Statutory and contractual deadlines in particular should come from the source document and be checked by someone qualified in that domain, not inferred from typical practice.
Watch for three ways the score can reward the wrong behavior. An agent can declare completion early to beat the clock, which the verifier timestamp prevents. It can skip a required approval to save time, which a hard constraint check on the approval path prevents. And it can learn that partial work on time always beats complete work slightly late, which is true only for hard and firm classes and should not leak into soft ones.
Finally, keep speed and timeliness apart. Finishing well before a deadline earns no extra credit in this method, because a business rarely values an accrual package more for arriving at noon instead of 16:00. If early completion does carry value, model it as its own field rather than bending the deadline curve.
Stop and narrow the evaluation when most tasks lack a documented deadline owner, when the clock rules cannot be reconstructed from the source system, or when the deadline depends on facts the agent could not observe. In those cases a completion-only score with tardiness reported as a descriptive statistic is more honest than a utility curve built on assumptions.
Finding real deadlines in operational records
The most realistic deadlines live in systems that already measure them. Help desk SLA policies, service desk SLA goals with their calendars and pause conditions, purchase order need-by dates, close calendars and project milestones all record a due time, an owner and, often, whether the deadline was met. Paired with the work record that followed, they let an evaluation team set deadlines and classes from what really happened rather than from invented scenarios.
Using those records for evaluation is a licensing question as well as a technical one. An evaluation team that wants deadline-bearing records from real U.S. businesses can ask SourceX to source them; nothing is held in stock, and the request form is where the team sets out scope, volume, language, format, timing and the license it needs. Each supplying company approves the scope and signs its agreement before SLA or close-calendar records are delivered.
Frequently asked questions
Should late but correct work ever score zero?
Yes, when the deadline is a hard cutoff or the output's value expires at the deadline, such as a bid portal that closes. For service targets that decay, a late correct answer should keep partial credit. The plan should fix which case applies before any run.
How is tardiness different from latency?
Latency is how long the agent took. Tardiness is how far past the deadline it finished, and it is zero for any run that finished on time. A fast agent can still be tardy if it started late or waited without noticing the clock.
Should an agent get credit for warning that it will miss a deadline?
Record it as a separate notice metric rather than folding it into the completion score. Businesses value an early, specific warning sent to the person who owns the deadline, but it should not make a missed deadline look like a met one.
Wall-clock or simulated time for agent deadline tests?
Simulated time is usually the better default because it charges each action a fixed cost and removes infrastructure noise. Report real wall-clock latency separately so that operators can still see how fast the system is.
How many trials does a deadline evaluation need?
Run several trials per task and report whether every trial met the deadline, in the spirit of pass^k. A single fast run can hide an agent that misses the deadline whenever a tool responds slowly.
Sources
- tau-bench grades an episode by comparing the final database state with an annotated goal state and introduces pass^k, the probability that all k trials succeed. Source
- TheAgentCompany splits tasks into checkpoints and defines a partial completion score of half the checkpoint fraction plus half a full-completion indicator. Source
- METR measures an AI system's time horizon using the time human professionals take to complete tasks. Source
- TimeArena is a text simulation with action durations, dependencies and occupancy in which agents must finish several tasks as quickly as possible. Source
- The Linux kernel deadline scheduler documentation defines tardiness as the maximum of zero and the finishing time minus the absolute deadline. Source
- Zendesk requester wait time is the time in New, Open and On-hold statuses and pauses while a ticket is Pending. Source
- Jira Service Management SLA pause conditions are optional and can be used while waiting for a customer; stop conditions end the measurement. Source
- Anthropic notes agents can pause for human feedback at checkpoints or blockers and often use stopping conditions such as a maximum number of iterations. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.