Skip to content

AI uses for records

Agent business deadline failure: when the right action comes after the decision window

By SourceX Editorial · Updated

10 sections

Short answer

Score a task finished after its decision window closed against the state the agent actually acted on. Once the window ends, a default outcome has taken effect, so performing the original action is a failure even if the output looks right. Label the run window closed, record which required condition failed, and give credit only to runs that detected the closure and took the post-window path the business allows.

Agent run

Agent business deadline

This page covers agent business deadline failure: when the right action comes after the decision window.Illustrative

Key takeaways

  • A decision window ends an option; a delivery deadline only makes work late.
  • After a window closes, the originally requested action is usually wrong, not just late.
  • Write each case as a card with the window field, boundary rule, required conditions, resulting state and recovery check.
  • Diagnose which condition failed before scoring, because the same outcome can come from different gaps.
  • Include the authorized exception, so the case tests who can reopen a window rather than teaching that late action is always wrong.

How do you score a task completed after its decision window closes?

Score it against the world as it stood when the agent acted, not as it stood when the task was assigned. Once a decision window closes, the business has usually moved to a default outcome: a discount lapsed, a renewal ran, a hold released. The action the task originally asked for is then no longer the correct one, so a run that performs it anyway is a failure, however clean its output. Label the run window closed, record which required condition failed, and give credit only if the agent detected the closure and took the post-window path the business allows.

This is a different failure from late delivery of a document or report. A late report still has some value and can be scored with a decaying curve. An action taken after a decision window has closed can have no value at all, or a negative one, because it is applied to a state that no longer exists. A payment taken with a discount the vendor no longer offers creates a short payment and a dispute, not a slightly worse version of the right answer.

This page is for evaluation teams at AI labs and the business operators who help them build agent test cases. It defines decision windows, gives a failure case card you can fill in for your own tasks, walks through one illustrative case, and shows how to tell which condition failed. It reports no model results.

A decision window is not a delivery deadline

A decision window is the period during which an option exists; when it ends, something happens without anyone acting. A delivery deadline is the time by which work is due; when it passes, the work is late but the task is unchanged. The difference decides how a late run is graded.

Accounts payable has a clear, documented example. In Microsoft Dynamics 365 Finance, a cash discount defined on a vendor or customer invoice is taken if the invoice is paid within the cash discount date, and a series of discount codes can be set up so that each succeeds the previous one as its date expires. Microsoft's own walkthrough uses an invoice offering a 1 percent discount if paid within 14 days: when the payment is created after the discount date and the invoice is marked for settlement, no cash discount is taken.

Other business systems are full of the same structure: a quote with an expiration date, an auto-renewing subscription with a cancellation notice period, a purchase order with a cancel-without-charge date, an inventory hold that releases at a set time, a price lock, a return authorization that expires. In each case the record holds a date field after which the available options change. Those fields are exactly what an agent must read, and exactly what a completion-only test does not check.

A decision window is not a delivery deadline
PropertyDelivery deadlineDecision window
What endsThe expected time for the workThe availability of an option
What happens at the endNothing automatic; the work is lateA default outcome takes effect
Value of the original action after the endReduced, often partialOften zero or harmful
Correct behavior after the endFinish and report the delayDetect closure, stop the original action, follow the post-window path
Typical evidence in recordsDue date and completion timestampExpiry or cutoff field, effective date of the action, resulting state

The illustrative deadline and outcome utility task

Write each case as a card with fixed fields, so that the failure is defined before any agent runs and every reviewer grades it the same way. The card below is the asset; the right-hand column fills it in for the illustrative accounts payable case discussed next. Every value in it is invented.

At a glance

The illustrative deadline and outcome utility task

The illustrative deadline and outcome utility task
FieldWhat it recordsIllustrative entry
Case labelIllustrative or approved real case, with version and dateIllustrative, v1, 2026-10-18
Starting stateRecords and clock at task startInvoice with a 1 percent discount for 14 days; 2 days of the window left; payment run not yet built
Window definitionThe field, time zone and boundary rule that close the optionCash discount date, end of day, company time zone, inclusive
Required conditionsEverything a correct run must satisfyCorrect vendor and invoice; approval obtained; payment effective on or before the discount date
Failure stepWhere the reference failing run went wrongWaited for approval past the discount date, then submitted the discounted amount
Applicable ruleThe business rule or system behavior after the windowNo discount is taken on settlement after the discount date unless the vendor approves
Resulting stateSystem state after the failing runInvoice partly open; vendor shows a short payment
SeverityBusiness impact on a fixed scaleModerate: recoverable, creates a vendor dispute
Recovery checkWhat a good run does once it sees the window closedPays the full amount, or asks the vendor and records written approval before taking the discount
Scoring outcomePass, fail or partial, with the reason codeFail, reason window closed, condition 3 not met

Walking the case: the discount run that cleared after the discount date

The illustrative task: an agent is asked to prepare the week's vendor payments and take every discount that is still available. One invoice offers 1 percent for payment within 14 days, and when the task starts there are two days left in that window. Payments above a set threshold need a controller's approval.

The reference failing run does most things right. It identifies the invoice, computes the discounted amount correctly, builds the payment and requests approval. The controller is out for a day, and the agent waits. When approval arrives the next morning, the discount date has passed. The agent submits the payment it built two days earlier, at the discounted amount, and reports that all available discounts were taken.

Everything in that final report sounds plausible, and a grader that reads only the transcript or checks only that a payment exists will pass it. The settlement tells a different story: the discount is no longer available, so the invoice remains partly open and the vendor sees a short payment. The task asked for available discounts, and by the time the agent acted, this one was not available.

A passing run looks different at only one step. Before submitting, it re-reads the invoice and the current date, sees that the discount date has passed, drops the discount, and either pays the full amount or contacts the vendor to ask whether the discount can still be taken. Its report says which discount was missed and why.

Which required condition failed behind the agent's plausible final answer?

Diagnose the failure before scoring it, because the same wrong outcome can come from different causes, and each points to a different fix in the agent. Check the conditions in the order below and stop at the first one that failed.

Which required condition failed behind the agent's plausible final answer?
Condition that failedWhat it looks like in the traceEvidence to checkWhat the case should test next
Window never readNo access to the expiry field before actingTool calls and fields retrievedA variant where the window is the only field that changes the answer
Window read once, never re-checkedField read at the start, action taken much laterTime between the read and the actionA variant with a long wait between planning and acting
Clock or time zone misreadCorrect date, wrong day boundary or offsetTime zone used and boundary ruleA window that closes near midnight or a daylight saving change
Wait treated as paused timeAgent assumed the clock stopped while waiting for approvalMessages during the waitA case where approval arrives after the window closes
Stale plan reusedAction built before the wait is submitted unchangedDiff between the built action and the submitted oneA case where the right amount changes at closure
Unauthorized extensionAgent assumed the window could be stretchedAny approval record in the environmentA case where only a named counterparty can extend
Misreported outcomeReport claims success the system state does not showReport text against the final stateGrade the report separately from the action

Grading the world after the action, not the agent's report of it

Grade the system state that results from the action, plus the time the action took effect, and treat the agent's report as a separate item. The tau-bench benchmark already grades by state: it compares the database at the end of a conversation with an annotated goal state, and it reports pass^k, the chance that all k independent trials of a task succeed. A decision window case adds one assertion to that end-state check: the effective time of the action must fall inside the window, or the state must show the post-window path.

Run several trials with the clock starting at different points relative to the window: well before it closes, just before, and just after. A good agent passes all three, but for different reasons, taking the option in the first two and the post-window path in the third. An agent that passes only when the window is wide open has learned to perform the action, not to check whether it is still available.

Waiting deserves its own design. Anthropic's guidance on building agents notes that agents can pause for human feedback at checkpoints or when they hit blockers, and that it is common to add stopping conditions such as a maximum number of iterations. In a decision window case, the pause for human approval is the risk. Service desk tools let teams pause an SLA clock, for example while waiting for a customer to respond, as Atlassian documents for Jira Service Management, but a vendor's discount date or a renewal cutoff does not pause because someone inside the company is busy. The case should make that explicit, so an agent that treats a wait as stopped time fails for the right reason.

When acting after the window is still the right call

Sometimes the business can still take the option after the window, and the case must say exactly when. Microsoft documents this for Dynamics 365: when a payment is made after the discount date but the vendor approves taking the discount anyway, the user can set the Use cash discount field to Always, and the discount is applied at settlement. A second scenario uses a Date to use for calculating discounts setting to calculate the discount from a selected date inside the discount period.

That turns a single rule into a test of authorization. A run that takes a late discount with a recorded vendor approval in the environment passes. A run that takes the same late discount with no such record, or that writes an approval note itself, fails, and the failure should be labeled as an unauthorized extension rather than a missed window. The difference between the two is not in the final amount but in the evidence the agent relied on.

Include this exception in the case set deliberately. Without it, an evaluation teaches that late action is always wrong, and an agent may refuse a legitimate post-window action that the business would want. With it, the case measures the real skill: knowing who can reopen a closed window and what proof of that looks like.

Clocks, time zones and the exact edge of a window

Define the window boundary precisely enough that two reviewers cannot disagree about a run that acts close to it. Write down the field that holds the window, the time zone it is read in, whether the last day is inclusive, and whether the boundary is the end of a business day or a calendar day.

Time zones cause quiet failures. Python's zoneinfo module, which supports the IANA time zone database, notes that during offset transitions that create ambiguous times, such as the change from daylight saving to standard time, the offset from before the transition is used when the fold attribute is 0 and the offset after it when fold is 1. A window that closes at 01:30 local time on that night happens twice. A good case either avoids those instants or states which one counts.

Record times with their offsets in every log the grader reads, and run the grader in the business's time zone, not the evaluation server's. A run that acts at 23:30 in the vendor's time zone may be on time there and an hour late in the company's, and the case must say whose clock governs.

Limits of a single failure case, and when to involve an operator

One failure case shows that a gap exists; it does not measure how often an agent falls into it. Treat each card as a probe and build families of variants around it, changing the time left, the length of the wait and the window type, before drawing conclusions about an agent.

Bring in a business operator when the window rule is not written down in the system, when the post-window path depends on a relationship rather than a rule, or when the consequence of acting late is legal or regulatory rather than commercial. Filing deadlines, statutory notice periods and contract termination windows should be specified from the source documents by someone qualified, not inferred from typical practice.

Stop and narrow the case when nobody can say what the resulting state should be after the window closes. A case without a defined post-window path cannot be graded consistently, and an evaluation built on it will reward guessing.

Finding real decision windows in business records

Real decision windows sit in ordinary business records: invoice discount dates and their settlement history, quote expirations and the orders that followed, renewal notice dates and cancellation records, hold expiries and release logs. Paired with the action timestamps and the resulting state, these records let evaluation teams build windows with real rules and real exceptions instead of invented ones.

AI teams that need such records from U.S. businesses can ask SourceX to source them. Nothing is held in stock: the request sets out scope, volume, language, format, timing and the license required. The business that owns the records approves the scope and signs the agreement before anything is delivered.

Frequently asked questions

Is a task completed after its decision window a partial success?

Usually not. If the option no longer exists, the original action is applied to a state that has changed, so it is wrong rather than late. Partial credit belongs to runs that detect closure and take the correct post-window path.

How is a decision window different from an SLA?

An SLA is a target for how quickly work is done, and many tools let its clock pause, for example while waiting for a customer. A decision window belongs to an option, such as a discount or a cancellation right, and it closes whether or not anyone is working on it.

Should an agent be penalized for waiting on a human approval?

Not for waiting, but for ignoring the window while it waits. A good run notices that the approval may arrive after the window closes, warns the approver, and re-checks the window before acting on a plan built earlier.

How many variants does one failure case need?

At least three clock starts relative to the window: well before, just before and just after closure, plus the authorized exception. Run several trials of each, because an agent that passes only with plenty of time left has not learned the check.

Can these cases use real company records?

Yes, if the records are licensed for evaluation use and personal data is handled under a reviewed plan. Real records give real window rules and exceptions, which invented cases tend to simplify.

Sources

  • In Dynamics 365 Finance a cash discount is taken if the invoice is paid within the cash discount date, and a series of discounts can succeed each other as dates expire. Source
  • Microsoft's walkthrough shows no cash discount taken for a payment after the discount date, and how Use cash discount set to Always or a selected calculation date applies it when the vendor approves. Source
  • tau-bench grades by comparing the database state at the end of a conversation with an annotated goal state and reports pass^k reliability. Source
  • Anthropic's agent guidance notes agents can pause for human feedback at checkpoints or blockers and often include stopping conditions such as a maximum number of iterations. Source
  • Jira Service Management SLA pause conditions are optional and can be used, for example, while waiting for a customer to respond. Source
  • Python's zoneinfo module supports the IANA time zone database and resolves ambiguous times at offset transitions with the fold attribute. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies