AI uses for records
Agent business deadline failure: when the right action comes after the decision window
By SourceX Editorial · Updated
10 sections
Short answer
Score a task finished after its decision window closed against the state the agent actually acted on. Once the window ends, a default outcome has taken effect, so performing the original action is a failure even if the output looks right. Label the run window closed, record which required condition failed, and give credit only to runs that detected the closure and took the post-window path the business allows.
Agent run
Agent business deadline
Key takeaways
- A decision window ends an option; a delivery deadline only makes work late.
- After a window closes, the originally requested action is usually wrong, not just late.
- Write each case as a card with the window field, boundary rule, required conditions, resulting state and recovery check.
- Diagnose which condition failed before scoring, because the same outcome can come from different gaps.
- Include the authorized exception, so the case tests who can reopen a window rather than teaching that late action is always wrong.
How do you score a task completed after its decision window closes?
Score it against the world as it stood when the agent acted, not as it stood when the task was assigned. Once a decision window closes, the business has usually moved to a default outcome: a discount lapsed, a renewal ran, a hold released. The action the task originally asked for is then no longer the correct one, so a run that performs it anyway is a failure, however clean its output. Label the run window closed, record which required condition failed, and give credit only if the agent detected the closure and took the post-window path the business allows.
This is a different failure from late delivery of a document or report. A late report still has some value and can be scored with a decaying curve. An action taken after a decision window has closed can have no value at all, or a negative one, because it is applied to a state that no longer exists. A payment taken with a discount the vendor no longer offers creates a short payment and a dispute, not a slightly worse version of the right answer.
This page is for evaluation teams at AI labs and the business operators who help them build agent test cases. It defines decision windows, gives a failure case card you can fill in for your own tasks, walks through one illustrative case, and shows how to tell which condition failed. It reports no model results.
A decision window is not a delivery deadline
A decision window is the period during which an option exists; when it ends, something happens without anyone acting. A delivery deadline is the time by which work is due; when it passes, the work is late but the task is unchanged. The difference decides how a late run is graded.
Accounts payable has a clear, documented example. In Microsoft Dynamics 365 Finance, a cash discount defined on a vendor or customer invoice is taken if the invoice is paid within the cash discount date, and a series of discount codes can be set up so that each succeeds the previous one as its date expires. Microsoft's own walkthrough uses an invoice offering a 1 percent discount if paid within 14 days: when the payment is created after the discount date and the invoice is marked for settlement, no cash discount is taken.
Other business systems are full of the same structure: a quote with an expiration date, an auto-renewing subscription with a cancellation notice period, a purchase order with a cancel-without-charge date, an inventory hold that releases at a set time, a price lock, a return authorization that expires. In each case the record holds a date field after which the available options change. Those fields are exactly what an agent must read, and exactly what a completion-only test does not check.
| Property | Delivery deadline | Decision window |
|---|---|---|
| What ends | The expected time for the work | The availability of an option |
| What happens at the end | Nothing automatic; the work is late | A default outcome takes effect |
| Value of the original action after the end | Reduced, often partial | Often zero or harmful |
| Correct behavior after the end | Finish and report the delay | Detect closure, stop the original action, follow the post-window path |
| Typical evidence in records | Due date and completion timestamp | Expiry or cutoff field, effective date of the action, resulting state |
The illustrative deadline and outcome utility task
Write each case as a card with fixed fields, so that the failure is defined before any agent runs and every reviewer grades it the same way. The card below is the asset; the right-hand column fills it in for the illustrative accounts payable case discussed next. Every value in it is invented.
At a glance
The illustrative deadline and outcome utility task
| Field | What it records | Illustrative entry |
|---|---|---|
| Case label | Illustrative or approved real case, with version and date | Illustrative, v1, 2026-10-18 |
| Starting state | Records and clock at task start | Invoice with a 1 percent discount for 14 days; 2 days of the window left; payment run not yet built |
| Window definition | The field, time zone and boundary rule that close the option | Cash discount date, end of day, company time zone, inclusive |
| Required conditions | Everything a correct run must satisfy | Correct vendor and invoice; approval obtained; payment effective on or before the discount date |
| Failure step | Where the reference failing run went wrong | Waited for approval past the discount date, then submitted the discounted amount |
| Applicable rule | The business rule or system behavior after the window | No discount is taken on settlement after the discount date unless the vendor approves |
| Resulting state | System state after the failing run | Invoice partly open; vendor shows a short payment |
| Severity | Business impact on a fixed scale | Moderate: recoverable, creates a vendor dispute |
| Recovery check | What a good run does once it sees the window closed | Pays the full amount, or asks the vendor and records written approval before taking the discount |
| Scoring outcome | Pass, fail or partial, with the reason code | Fail, reason window closed, condition 3 not met |
Walking the case: the discount run that cleared after the discount date
The illustrative task: an agent is asked to prepare the week's vendor payments and take every discount that is still available. One invoice offers 1 percent for payment within 14 days, and when the task starts there are two days left in that window. Payments above a set threshold need a controller's approval.
The reference failing run does most things right. It identifies the invoice, computes the discounted amount correctly, builds the payment and requests approval. The controller is out for a day, and the agent waits. When approval arrives the next morning, the discount date has passed. The agent submits the payment it built two days earlier, at the discounted amount, and reports that all available discounts were taken.
Everything in that final report sounds plausible, and a grader that reads only the transcript or checks only that a payment exists will pass it. The settlement tells a different story: the discount is no longer available, so the invoice remains partly open and the vendor sees a short payment. The task asked for available discounts, and by the time the agent acted, this one was not available.
A passing run looks different at only one step. Before submitting, it re-reads the invoice and the current date, sees that the discount date has passed, drops the discount, and either pays the full amount or contacts the vendor to ask whether the discount can still be taken. Its report says which discount was missed and why.
Which required condition failed behind the agent's plausible final answer?
Diagnose the failure before scoring it, because the same wrong outcome can come from different causes, and each points to a different fix in the agent. Check the conditions in the order below and stop at the first one that failed.
| Condition that failed | What it looks like in the trace | Evidence to check | What the case should test next |
|---|---|---|---|
| Window never read | No access to the expiry field before acting | Tool calls and fields retrieved | A variant where the window is the only field that changes the answer |
| Window read once, never re-checked | Field read at the start, action taken much later | Time between the read and the action | A variant with a long wait between planning and acting |
| Clock or time zone misread | Correct date, wrong day boundary or offset | Time zone used and boundary rule | A window that closes near midnight or a daylight saving change |
| Wait treated as paused time | Agent assumed the clock stopped while waiting for approval | Messages during the wait | A case where approval arrives after the window closes |
| Stale plan reused | Action built before the wait is submitted unchanged | Diff between the built action and the submitted one | A case where the right amount changes at closure |
| Unauthorized extension | Agent assumed the window could be stretched | Any approval record in the environment | A case where only a named counterparty can extend |
| Misreported outcome | Report claims success the system state does not show | Report text against the final state | Grade the report separately from the action |
Grading the world after the action, not the agent's report of it
Grade the system state that results from the action, plus the time the action took effect, and treat the agent's report as a separate item. The tau-bench benchmark already grades by state: it compares the database at the end of a conversation with an annotated goal state, and it reports pass^k, the chance that all k independent trials of a task succeed. A decision window case adds one assertion to that end-state check: the effective time of the action must fall inside the window, or the state must show the post-window path.
Run several trials with the clock starting at different points relative to the window: well before it closes, just before, and just after. A good agent passes all three, but for different reasons, taking the option in the first two and the post-window path in the third. An agent that passes only when the window is wide open has learned to perform the action, not to check whether it is still available.
Waiting deserves its own design. Anthropic's guidance on building agents notes that agents can pause for human feedback at checkpoints or when they hit blockers, and that it is common to add stopping conditions such as a maximum number of iterations. In a decision window case, the pause for human approval is the risk. Service desk tools let teams pause an SLA clock, for example while waiting for a customer to respond, as Atlassian documents for Jira Service Management, but a vendor's discount date or a renewal cutoff does not pause because someone inside the company is busy. The case should make that explicit, so an agent that treats a wait as stopped time fails for the right reason.
When acting after the window is still the right call
Sometimes the business can still take the option after the window, and the case must say exactly when. Microsoft documents this for Dynamics 365: when a payment is made after the discount date but the vendor approves taking the discount anyway, the user can set the Use cash discount field to Always, and the discount is applied at settlement. A second scenario uses a Date to use for calculating discounts setting to calculate the discount from a selected date inside the discount period.
That turns a single rule into a test of authorization. A run that takes a late discount with a recorded vendor approval in the environment passes. A run that takes the same late discount with no such record, or that writes an approval note itself, fails, and the failure should be labeled as an unauthorized extension rather than a missed window. The difference between the two is not in the final amount but in the evidence the agent relied on.
Include this exception in the case set deliberately. Without it, an evaluation teaches that late action is always wrong, and an agent may refuse a legitimate post-window action that the business would want. With it, the case measures the real skill: knowing who can reopen a closed window and what proof of that looks like.
Clocks, time zones and the exact edge of a window
Define the window boundary precisely enough that two reviewers cannot disagree about a run that acts close to it. Write down the field that holds the window, the time zone it is read in, whether the last day is inclusive, and whether the boundary is the end of a business day or a calendar day.
Time zones cause quiet failures. Python's zoneinfo module, which supports the IANA time zone database, notes that during offset transitions that create ambiguous times, such as the change from daylight saving to standard time, the offset from before the transition is used when the fold attribute is 0 and the offset after it when fold is 1. A window that closes at 01:30 local time on that night happens twice. A good case either avoids those instants or states which one counts.
Record times with their offsets in every log the grader reads, and run the grader in the business's time zone, not the evaluation server's. A run that acts at 23:30 in the vendor's time zone may be on time there and an hour late in the company's, and the case must say whose clock governs.
Limits of a single failure case, and when to involve an operator
One failure case shows that a gap exists; it does not measure how often an agent falls into it. Treat each card as a probe and build families of variants around it, changing the time left, the length of the wait and the window type, before drawing conclusions about an agent.
Bring in a business operator when the window rule is not written down in the system, when the post-window path depends on a relationship rather than a rule, or when the consequence of acting late is legal or regulatory rather than commercial. Filing deadlines, statutory notice periods and contract termination windows should be specified from the source documents by someone qualified, not inferred from typical practice.
Stop and narrow the case when nobody can say what the resulting state should be after the window closes. A case without a defined post-window path cannot be graded consistently, and an evaluation built on it will reward guessing.
Finding real decision windows in business records
Real decision windows sit in ordinary business records: invoice discount dates and their settlement history, quote expirations and the orders that followed, renewal notice dates and cancellation records, hold expiries and release logs. Paired with the action timestamps and the resulting state, these records let evaluation teams build windows with real rules and real exceptions instead of invented ones.
AI teams that need such records from U.S. businesses can ask SourceX to source them. Nothing is held in stock: the request sets out scope, volume, language, format, timing and the license required. The business that owns the records approves the scope and signs the agreement before anything is delivered.
Frequently asked questions
Is a task completed after its decision window a partial success?
Usually not. If the option no longer exists, the original action is applied to a state that has changed, so it is wrong rather than late. Partial credit belongs to runs that detect closure and take the correct post-window path.
How is a decision window different from an SLA?
An SLA is a target for how quickly work is done, and many tools let its clock pause, for example while waiting for a customer. A decision window belongs to an option, such as a discount or a cancellation right, and it closes whether or not anyone is working on it.
Should an agent be penalized for waiting on a human approval?
Not for waiting, but for ignoring the window while it waits. A good run notices that the approval may arrive after the window closes, warns the approver, and re-checks the window before acting on a plan built earlier.
How many variants does one failure case need?
At least three clock starts relative to the window: well before, just before and just after closure, plus the authorized exception. Run several trials of each, because an agent that passes only with plenty of time left has not learned the check.
Can these cases use real company records?
Yes, if the records are licensed for evaluation use and personal data is handled under a reviewed plan. Real records give real window rules and exceptions, which invented cases tend to simplify.
Sources
- In Dynamics 365 Finance a cash discount is taken if the invoice is paid within the cash discount date, and a series of discounts can succeed each other as dates expire. Source
- Microsoft's walkthrough shows no cash discount taken for a payment after the discount date, and how Use cash discount set to Always or a selected calculation date applies it when the vendor approves. Source
- tau-bench grades by comparing the database state at the end of a conversation with an annotated goal state and reports pass^k reliability. Source
- Anthropic's agent guidance notes agents can pause for human feedback at checkpoints or blockers and often include stopping conditions such as a maximum number of iterations. Source
- Jira Service Management SLA pause conditions are optional and can be used, for example, while waiting for a customer to respond. Source
- Python's zoneinfo module supports the IANA time zone database and resolves ambiguous times at offset transitions with the fold attribute. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.