Skip to content

AI uses for records

Long horizon agent deadline awareness: scoring correct but late work

By SourceX Editorial · Updated

9 sections

Short answer

Score a correct but late agent run on two axes. Grade correctness with the usual verifier, then multiply by a deadline utility fixed before the run: zero after a hard cutoff, a decaying value for a service target, full value for an internal goal. Report on-time rate, tardiness and breaches beside the combined score, and test whether the deadline survives every handoff.

Agent run

Long horizon agent deadline

This page covers long horizon agent deadline awareness: scoring correct but late work.Illustrative

Key takeaways

  • Completion-only benchmarks score a correct late run the same as a correct timely one, so deadline failures stay invisible.
  • Classify each deadline by what lateness costs: hard, firm, soft, paused service target, internal goal or dependent milestone.
  • Freeze the clock definition, pause rules, deadline authority and utility curve before any agent runs.
  • Report correctness, on-time rate, tardiness, breaches and notice rate together rather than one blended number.
  • Test the deadline like any other requirement: it must survive handoffs, restarts and edits by people without authority.

How should an evaluation score correct work that misses a business deadline?

Score it on two axes, never one. Grade correctness with the same verifier you would use without a deadline, then multiply by a deadline utility that depends on what the business loses when the work is late: zero for a hard cutoff, a decaying value for a service target, full value for an internal goal. Report on-time rate and tardiness beside the combined score so a reader can see why a run lost credit.

The reason is simple. A reconciliation that balances perfectly but lands after the lender's covenant reporting cutoff is worth less to the business than one that lands on time, and sometimes worth nothing. A benchmark that counts both as a pass teaches the team, and any model trained on its labels, that time does not matter.

This guide is a method for evaluation designers at AI labs and agent teams. It gives the deadline classes, the fields an evaluation plan needs, the handoff checks that catch lost deadlines, and a worked example with illustrative numbers. It reports no model results.

Why completion-only agent benchmarks cannot see a missed deadline

Most agent benchmarks grade the end state and ignore the clock. tau-bench, for example, compares the database at the end of a conversation with an annotated goal state and adds pass^k, the probability that all of k trials succeed, to measure reliability. That design is right for what it measures, but a deadline is not part of the goal state, so a correct but late run looks identical to a correct and timely one.

TheAgentCompany goes further on partial work. Each task is split into checkpoints worth points, and its partial completion score gives half credit for the fraction of checkpoint points earned plus half for full completion, so only a fully completed task reaches 1.0. That is a useful model for incomplete work, and it still says nothing about when each checkpoint finished.

Time does appear in evaluation research, but usually as a measure of difficulty rather than a business constraint. METR's time horizon expresses task length as the time human professionals need to complete a task, and relates it to the length of tasks a model completes with a given success rate. TimeArena is one of the few environments that simulates action durations, dependencies and occupancy, and asks agents to finish several tasks as quickly as possible. Neither tells you how to score a correct answer delivered after the moment the business needed it.

The gap matters most for long-horizon work. Over many steps an agent can lose track of a due time stated once at the start, wait on a tool or a person without noticing the clock, or polish an answer past the point where it still has value.

Which kind of deadline is the agent facing?

Classify each deadline by what lateness costs, because the class decides the utility curve. Real-time systems offer a useful vocabulary: the Linux kernel's deadline scheduler documentation defines tardiness as the maximum of zero and the difference between a job's finishing time and its absolute deadline. Tardiness is the input; the class decides what a given amount of it costs.

Service tools show that business clocks are rarely simple wall time. Zendesk's requester wait time counts time in the New, Open and On-hold statuses and pauses while a ticket is Pending, and Jira Service Management lets teams add optional pause conditions, for example while waiting for a customer, and stop conditions that end the measurement. An evaluation that ignores those rules will mark runs late that the business would count as on time, and the reverse.

When one task carries several deadlines, score each one separately and let the strictest class govern the combined result. A vendor onboarding task might have an internal goal for the paperwork, a soft target for the first reply to the vendor and a hard cutoff for the payment run; averaging them would let a breach of the cutoff disappear inside two on-time results.

Which kind of deadline is the agent facing?
Deadline classWhat lateness costsUtility after the deadlineIllustrative business case
Hard cutoffThe output cannot be used or creates a breachZero, and log a breach flagA filing window or a bid submission portal that closes
Firm, value expiresThe output is useless but no harm followsZero, no penalty beyond lost valueA same-day price quote after the customer has bought elsewhere
Soft, decayingValue falls with each hour of delayDeclines from 1 to 0 over a stated windowA first reply to a priority support ticket
Paused service targetOnly counted time mattersSoft curve applied to counted time onlyA ticket clock that stops while waiting on the customer
Internal goalSchedule slips but nothing external breaksFull value; report tardiness onlyA team's target to finish a draft by Friday
Dependent milestoneLateness pushes later steps past their own deadlinesScored through the downstream deadline it threatensVendor setup that must finish before the payment run

The deadline and completion utility evaluation plan

The plan is a per-task record written before any agent runs, so the scoring rule cannot drift toward the results. Each table row is a field the evaluation designer fills in; the third column shows an illustrative entry for a single accounts payable task, and the list names the metrics every report should carry.

At a glance

The deadline and completion utility evaluation plan

  • Combined task score: correctness score multiplied by deadline utility, where correctness may use partial checkpoint credit.
  • On-time rate: share of runs whose verified completion falls at or before the deadline on the defined clock.
  • Tardiness: zero for on-time runs, otherwise counted time past the deadline; report the median and the worst case.
  • Breach count: runs that crossed a hard cutoff, reported even when the combined score already shows zero.
  • Notice rate: share of late runs that warned the deadline owner before the deadline passed.
The deadline and completion utility evaluation plan
FieldWhat to recordIllustrative entryWhy it is needed
Deadline source and authorityWho set the deadline and who may extend itController; only the controller can extendSeparates a real extension from an agent moving its own goalposts
Clock definitionCalendar, time zone, business hours, pause and stop rulesBusiness hours, US Eastern, paused while awaiting vendor replyMatches how the business itself measures lateness
Deadline class and curveOne class from the table above and its curve parametersSoft, linear decay to zero over 16 counted hoursFixes the utility before results exist
Completion verifierEnd-state check, independent of timeInvoice batch posted and matched to purchase ordersKeeps correctness and timeliness separable
Checkpoints and intermediate deadlinesOrdered checkpoints, each with its own due time if anyVendor verified by 12:00; batch posted by 17:00Locates where time was lost
Time accountingSimulated or wall-clock time, and the cost of each actionSimulated: each tool call 2 minutes, each human reply 3 hoursMakes runs comparable across infrastructure
Notice and escalation creditWhether a timely warning of a likely miss earns separate creditYes, if sent to the controller before the deadlineRewards the behavior a business wants when a miss is unavoidable
Trials and reliabilityNumber of trials per task and the reliability metricFour trials; report all-on-time across trialsOne lucky fast run hides an unreliable agent
Reporting setMetrics that must appear togetherCorrectness, on-time rate, median and worst tardiness, combined utilityPrevents a single blended number from hiding the cause
Version and datePlan version and the date it was frozenv1.0, frozen before the first runLets reviewers audit later changes

Which clock is the agent racing, and who can stop it?

Decide whether the agent races simulated business time or the real wall clock, and write it down. Wall-clock time mixes model speed, rate limits and infrastructure into the result; simulated time charges each action a fixed duration, which is what TimeArena does with its action durations, and lets two systems be compared fairly. For most business-task evaluations, simulated time is the better default, with wall-clock latency reported separately.

Then define who can pause or extend the clock. A business clock pauses when work is legitimately blocked on someone else, such as a customer reply, and the pause rule must come from the business's own definition rather than the agent's claim. An extension counts only when it comes from the person named as the deadline authority and is recorded in the environment, for example as a ticket field or an approval message.

Anthropic's guidance on building agents notes that agents can pause for human feedback at checkpoints or when they hit blockers, and that runs often include stopping conditions such as a maximum number of iterations. In a deadline evaluation those pauses are scoring events: the plan must say whether waiting for a human stops the clock, and an iteration cap should be set so that an agent that loops is scored as late rather than silently truncated.

Which deadline, permission or shared-state change can be lost at the next handoff?

The deadline itself is often the first thing dropped when work moves between agents, tools or people. Test it the way you would test any other requirement: plant the due time once, then check whether it survives each handoff, and whether later changes to it are respected only when they come from someone with authority.

At a glance

Which deadline, permission or shared-state change can be lost at the next handoff

  • Restatement check: after each handoff, does the receiving agent's plan or message carry the deadline, its time zone and its class?
  • Authority check: if a colleague without authority asks for a later delivery, does the agent keep the original deadline and route the request to the owner?
  • Human edit check: if the deadline owner moves the date in the shared record mid-task, does the agent adopt the new date and drop the old one?
  • Shared-state check: when two agents work in parallel, does each read the deadline from one shared record rather than from its own copy?
  • Dependency check: when an upstream step slips, does the agent recompute whether downstream deadlines can still be met and say so?
  • Recovery check: after a failed tool call or a restart from a checkpoint, does the resumed run still know how much counted time remains?
  • Final verification check: is the completion timestamp taken when the verifier passes, not when the agent declares it is done?

Illustrative: three runs on a month-end accrual package

The example below is illustrative and uses invented durations. The task: prepare and post a month-end accrual package due at 17:00 on the third business day, on a business-hours clock that pauses while waiting for a department head's confirmation. The plan classes it as soft, with utility falling linearly from 1 to 0 over 16 counted hours, and uses checkpoint credit in the TheAgentCompany style: a fully verified run scores 1, otherwise half the checkpoint fraction.

Run A posts a fully verified package at 15:40. Correctness 1, utility 1, combined score 1, tardiness zero. Run B posts a fully verified package 4 counted hours late because it re-ran a reconciliation it had already completed. Correctness 1, utility 0.75, combined 0.75. Run C completes three of four checkpoints by the deadline, cannot get the last confirmation, and emails the controller at 14:00 that the package will be late and why. Correctness 0.375, utility 1, combined 0.375, with notice credit recorded separately.

Change one field and the ranking changes. If the controller's plan said the package feeds a lender report with a hard cutoff, Run B drops to zero and records a breach, while Run C still shows partial work and a timely warning. That is the point of fixing the class before running: the same three runs tell a different and more honest story depending on what lateness actually costs.

One exception belongs in every plan. If the department head, as the named authority, moves the deadline by a day in the shared record before it passes, all three runs are scored against the amended deadline, and the evaluation logs the change. If the agent itself asserts an extension that no authority granted, the original deadline stands.

When a deadline-weighted score misleads

A deadline-weighted score is only as good as the deadline data behind it. Where the real due time is unknown, contested or set by law, a curve chosen by an evaluation designer is a guess, and the plan should say so or drop the task. Statutory and contractual deadlines in particular should come from the source document and be checked by someone qualified in that domain, not inferred from typical practice.

Watch for three ways the score can reward the wrong behavior. An agent can declare completion early to beat the clock, which the verifier timestamp prevents. It can skip a required approval to save time, which a hard constraint check on the approval path prevents. And it can learn that partial work on time always beats complete work slightly late, which is true only for hard and firm classes and should not leak into soft ones.

Finally, keep speed and timeliness apart. Finishing well before a deadline earns no extra credit in this method, because a business rarely values an accrual package more for arriving at noon instead of 16:00. If early completion does carry value, model it as its own field rather than bending the deadline curve.

Stop and narrow the evaluation when most tasks lack a documented deadline owner, when the clock rules cannot be reconstructed from the source system, or when the deadline depends on facts the agent could not observe. In those cases a completion-only score with tardiness reported as a descriptive statistic is more honest than a utility curve built on assumptions.

Finding real deadlines in operational records

The most realistic deadlines live in systems that already measure them. Help desk SLA policies, service desk SLA goals with their calendars and pause conditions, purchase order need-by dates, close calendars and project milestones all record a due time, an owner and, often, whether the deadline was met. Paired with the work record that followed, they let an evaluation team set deadlines and classes from what really happened rather than from invented scenarios.

Using those records for evaluation is a licensing question as well as a technical one. An evaluation team that wants deadline-bearing records from real U.S. businesses can ask SourceX to source them; nothing is held in stock, and the request form is where the team sets out scope, volume, language, format, timing and the license it needs. Each supplying company approves the scope and signs its agreement before SLA or close-calendar records are delivered.

Frequently asked questions

Should late but correct work ever score zero?

Yes, when the deadline is a hard cutoff or the output's value expires at the deadline, such as a bid portal that closes. For service targets that decay, a late correct answer should keep partial credit. The plan should fix which case applies before any run.

How is tardiness different from latency?

Latency is how long the agent took. Tardiness is how far past the deadline it finished, and it is zero for any run that finished on time. A fast agent can still be tardy if it started late or waited without noticing the clock.

Should an agent get credit for warning that it will miss a deadline?

Record it as a separate notice metric rather than folding it into the completion score. Businesses value an early, specific warning sent to the person who owns the deadline, but it should not make a missed deadline look like a met one.

Wall-clock or simulated time for agent deadline tests?

Simulated time is usually the better default because it charges each action a fixed cost and removes infrastructure noise. Report real wall-clock latency separately so that operators can still see how fast the system is.

How many trials does a deadline evaluation need?

Run several trials per task and report whether every trial met the deadline, in the spirit of pass^k. A single fast run can hide an agent that misses the deadline whenever a tool responds slowly.

Sources

  • tau-bench grades an episode by comparing the final database state with an annotated goal state and introduces pass^k, the probability that all k trials succeed. Source
  • TheAgentCompany splits tasks into checkpoints and defines a partial completion score of half the checkpoint fraction plus half a full-completion indicator. Source
  • METR measures an AI system's time horizon using the time human professionals take to complete tasks. Source
  • TimeArena is a text simulation with action durations, dependencies and occupancy in which agents must finish several tasks as quickly as possible. Source
  • The Linux kernel deadline scheduler documentation defines tardiness as the maximum of zero and the finishing time minus the absolute deadline. Source
  • Zendesk requester wait time is the time in New, Open and On-hold statuses and pauses while a ticket is Pending. Source
  • Jira Service Management SLA pause conditions are optional and can be used while waiting for a customer; stop conditions end the measurement. Source
  • Anthropic notes agents can pause for human feedback at checkpoints or blockers and often use stopping conditions such as a maximum number of iterations. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies