AI uses for records
Business timeline reasoning benchmark: a protocol for moving deadlines
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
This is a protocol, not a result. A business timeline reasoning benchmark should give the model an event log, a calendar and the policy versions in force, then ask for a specific deadline and the evidence behind it. Tasks should vary what moves the deadline: business-day counting, holidays, time zones, amended events and policy changes. Grade the date exactly, check the cited trigger and rule, and reward abstention when evidence is missing.
Evaluation split
Business timeline reasoning
Key takeaways
- This page proposes a method and reports no results, scores or rankings.
- Each task supplies three kinds of evidence: an event log, a calendar and the policy text in force; the answer is a date plus the trigger and rule that produced it.
- Task families are defined by what changes the deadline, such as a holiday, a time zone, an amended event or a new policy version.
- Grade the date by exact match against a reference calculation, then check that the cited trigger and rule are the right ones, and score abstention when evidence is missing.
- Report accuracy per task family with intervals, and hold out task families and source organizations, not just random tasks.
How should a benchmark test deadlines that depend on a changing timeline?
It should give the system the same evidence a careful operations analyst would use, an event log, a business calendar and the policy versions in force, and ask for one deadline with the trigger event and rule that produced it. Then it should change one thing at a time in the timeline and check whether the deadline moves the way the rule requires.
This page is a protocol for that business timeline reasoning benchmark. It describes the question, the task suite, the schema, the grader, the baselines and the pre-registration. It reports no measured findings, and every example is labeled illustrative.
The research question is: given a record of dated events and the written rules that govern a deadline, does an AI system compute the correct deadline, identify the event that triggered it, and recognize when the evidence is insufficient?
Why do business deadlines depend on the timeline?
Business deadlines are computed from a trigger event under a counting rule, and both can change after the fact. Published rules show how much the counting method alone matters.
Federal Rule of Civil Procedure 6(a)(1) excludes the day of the triggering event, counts every day including weekends and legal holidays, and includes the last day unless it falls on a Saturday, Sunday or legal holiday, in which case the period runs to the next day that is not. Under Rule 6(a)(5), the next day is found by counting backward when a period is measured before an event, so a backward-counted period that lands on a weekend moves earlier, not later.
Regulation E's error resolution rule mixes units. A consumer's notice must arrive within 60 days after the institution sent the periodic statement, the institution generally has 10 business days to investigate, and it may take up to 45 days if it provisionally credits the account within those 10 business days, with longer periods in specified cases. Whether the longer clock applies depends on an event in the timeline, the provisional credit.
Calendars and clocks move too. The Office of Personnel Management explains that when a federal holiday falls on a Saturday, the in-lieu-of holiday is generally the preceding workday, and on a Sunday the following workday. The IANA time zone database records the history of local time, including offsets and daylight saving rules, and is updated when governments change them, so a cutoff at 5 p.m. local time depends on which rules applied on that date.
| Rule example | Trigger | Counting unit | What can move the deadline |
|---|---|---|---|
| Court filing period (FRCP 6) | Service or entry of an order | Calendar days with weekend and holiday roll-forward | A last day on a weekend or legal holiday; backward counting |
| Error investigation (Reg E 1005.11) | Receipt of a notice of error | Business days, then calendar days | Provisional credit within 10 business days extends the period |
| Error notice window (Reg E 1005.11) | Sending of the periodic statement | Calendar days | A reissued or corrected statement changes the trigger |
| Observed holiday (OPM) | Holiday falling on a weekend | Workdays | Saturday holidays observed the day before, Sunday the day after |
| Local-time cutoff (tz database) | Submission time stamp | Hours in a named time zone | Daylight saving changes and historical offset changes |
What do existing temporal reasoning benchmarks leave out?
Existing benchmarks test temporal reasoning well in isolation but rarely combine dated business events with written policy. Test of Time, from Google researchers, uses synthetic data to avoid problems with pretraining exposure and anonymization, and separates temporal semantics and logic from temporal arithmetic. That separation is worth copying.
What business deadlines add is evidence handling. The system must pick the right trigger from a log with several candidates, apply the policy version in force on the trigger date rather than today's, apply the right calendar, and notice when a later event, such as a corrected statement or a provisional credit, changes which rule applies. A benchmark that hands the model the trigger and rule as clean inputs skips the part that fails in practice.
Which deadline decisions go in, and how are the one-change variants built?
The population is deadline decisions from operational records whose outcome is known: a recorded due date that a reviewer confirmed, or a missed or met deadline with the computation documented. Sources include dispute and claims handling, contract notice periods, service-level timers in ticket systems, and payment terms. Synthetic tasks fill families where real records are thin, and are labeled synthetic in the data.
Stratify by task family, by rule source and by contributing organization, and cap each organization's share. Hold out entire task families for a generalization test and entire organizations for a transfer test, because an organization's calendar and policy wording repeat across its records.
Pin the calendar data. Record which holiday calendar and which time zone database release each task was computed against, so a later release cannot silently change the gold answer.
Build perturbation sets on purpose. For each base task, create variants that change exactly one input, such as moving the trigger by one day, adding an observed holiday, amending the event or swapping the policy version, and record the expected new deadline for each. Comparing a system's answers across a base task and its variants shows whether it is applying the rule or matching a familiar answer, and it makes every family's failures easy to diagnose.
Record the denominator for real records: how many deadline decisions each source had in the window, how many met the inclusion rules, and how many were dropped at rights review, de-identification or gold verification.
The date and policy evidence reasoning suite
The suite is organized by what changes the deadline, so results show which kind of change a system mishandles. Examples are illustrative.
| Task family | What changes | Evidence given | Illustrative question |
|---|---|---|---|
| 1. Base count | Nothing; a single trigger and rule | One event, one rule, a calendar | Notice received on a stated date; when is the 10-business-day deadline? |
| 2. Unit mixing | Business days versus calendar days | Rule with both units | Investigation deadline versus extended deadline for the same notice |
| 3. Weekend and holiday roll | Last day lands on a non-working day | Calendar with observed holidays | Period ends on a Saturday holiday; which day is the deadline? |
| 4. Backward counting | Deadline counted before an event | Event date and rule | Filing due a stated number of days before a hearing that falls after a weekend |
| 5. Time zone and cutoff | Local cutoff across a daylight saving change | Time stamps, zone name, tz release | Was a submission at a given UTC time before the local 5 p.m. cutoff? |
| 6. Amended event | Trigger event corrected or reissued | Event log with an amendment | Statement reissued with a correction; which date starts the window? |
| 7. Conditional extension | An action in the log changes the rule | Log with or without a provisional credit | Does the longer investigation period apply? |
| 8. Policy version | Rule changed between trigger and today | Two dated policy versions | Which notice period applies to a contract signed before the change? |
| 9. Competing triggers | Several events could start the clock | Log with near-duplicate events | Which of two receipts of the same complaint starts the period? |
| 10. Insufficient evidence | A needed fact is missing | Log without the trigger date or calendar | The correct answer is to abstain and name the missing fact |
What does one deadline task packet contain?
Every task is a packet of dated evidence and a question with one correct date, or a required abstention. Keep the reference computation out of the model input.
At a glance
What does one deadline task packet contain
| Field | Content | Grader use |
|---|---|---|
| task_id, family, split | Stable ID, family number, public or holdout | Per-family reporting; split enforcement |
| event_log | Dated events with types and time stamps, including amendments | Input; the grader knows which event is the trigger |
| calendar | Working days, observed holidays, time zone name and tz release | Input; pinned for reproducibility |
| policy_versions | Rule text with effective dates | Input; the grader knows which version applies |
| question | One deadline to compute, with the required output format | Input |
| gold_deadline | Date and, where relevant, local time and zone, or ABSTAIN | Exact-match check |
| gold_trigger and gold_rule | Event ID and policy version ID that produce the deadline | Evidence check |
| reference_computation | Step-by-step count produced by a reference calculator and checked by a reviewer | Audit of the gold; never shown to the model |
| provenance and canary | Source type (real or synthetic), rights record, canary string | Release gate and contamination detection |
How are the date, the cited evidence and abstentions graded?
Grade the date with code and the evidence with a rubric. Anthropic's guidance on agent evaluations describes combining code-based, model-based and human graders and calibrating model-based graders against human judgment; this plan uses that split.
- Date check (code): the answer date, and time and zone where required, must equal gold_deadline exactly. Near misses are wrong, because a deadline one day late is a missed deadline.
- Evidence check (code first, then rubric): the answer must cite the gold trigger event ID and the policy version ID. Where the answer is free text, a model grader calibrated on expert labels extracts the cited trigger and rule.
- Abstention check (code): on family 10 tasks, ABSTAIN with the missing fact named is correct and any date is wrong; on other families, abstaining is scored as wrong but reported separately from wrong dates.
- Adjudication (human): two reviewers with operations or compliance background resolve grader disagreements and audit every task where the system's date differs from gold by exactly one day, which is where gold errors most often hide.
Which baselines and metrics show where deadline errors come from?
Use three baselines. A deterministic reference calculator given the correct trigger and rule shows the ceiling when evidence selection is solved. A naive calendar-day baseline that ignores business days, holidays and amendments shows how much each family depends on those rules. Human analysts on a blind subset show achievable accuracy and expose ambiguous tasks.
At a glance
Which baselines and metrics show where deadline errors come from
The primary metric is exact deadline accuracy per family. Secondary metrics are trigger accuracy, rule-version accuracy, abstention precision and recall on family 10, and the share of errors that are late rather than early, because a late deadline is usually the costlier mistake. Report each proportion with a Wilson score interval as described in the NIST Engineering Statistics Handbook, and resample by organization.
Pre-register before running any system. The registration should include the items below.
- Research question and the exact output format.
- Population, sources, real versus synthetic share, strata and caps.
- Holdout families and organizations.
- Pinned calendars and tz database release.
- Task schema, grader and rubric versions.
- Baselines and model identifiers with dates.
- Primary and secondary metrics, interval method and resampling unit.
- Exclusion rules, and how tasks with disputed gold are handled.
- Conditions for releasing results.
Rights, release and limits
Real tasks come from operational records, so the same permissions apply as for any licensed dataset: system access, privacy rules, ownership and a license that permits benchmark use and publication are separate questions, and de-identification does not create a right to publish. Shifting dates to de-identify records must preserve weekdays and holiday positions, or the task changes its answer.
Mark every released file with a canary string, as BIG-bench asks for its task files, and keep a private holdout. Limits to state on the card: the suite covers rules written in English, chosen calendars and the sampled rule sources, and it does not measure whether a computed deadline is legally correct for a specific matter.
Teams that need real event logs and policy histories from other businesses can ask SourceX to source them from U.S. businesses on request, setting out scope, volume, format, timing and licensing needs, including whether a public benchmark split may be released.
Frequently asked questions
Does this page report results for any AI model?
No. It is a protocol for a benchmark that has not been run: task families, schema, grader, baselines and a pre-registration checklist. Any example is illustrative, and results should be published only after the grader is calibrated and the holdout is intact.
What should the date and policy evidence reasoning suite include?
Tasks that vary one thing that moves a deadline: units, weekend and holiday roll-forward, backward counting, time zones, amended events, conditional extensions, policy versions, competing triggers and missing evidence. Each task gives an event log, a calendar and dated policy text, and expects a date with the trigger and rule that produced it.
Why grade deadlines by exact match instead of partial credit?
A deadline that is one day late is a missed deadline, so near misses carry real cost. Partial credit is better spent on the evidence: report trigger and rule accuracy separately so a team can see whether errors come from selecting evidence or from counting.
Which benchmark result would remain valid on held-out companies and task families?
Only a result reported per family on organizations and families the system never saw during development. Pooled accuracy on random splits can reflect repeated calendars and policy wording from the same organization rather than general timeline reasoning.
Is this legal advice on computing deadlines?
No. Court rules and regulations appear only as examples of published counting rules. A specific deadline depends on the governing rule, jurisdiction and facts, which counsel or the responsible compliance owner should confirm.
Sources
- FRCP 6(a)(1) excludes the trigger day, counts every day including weekends and holidays, and rolls a last day on a weekend or legal holiday to the next day, which Rule 6(a)(5) counts backward for periods measured before an event. Source
- Regulation E sets a 60-day notice window, a 10-business-day investigation period, and up to 45 days with provisional credit. Source
- OPM explains in-lieu-of holidays: the preceding workday for a Saturday holiday and the following workday for a Sunday holiday. Source
- The tz database records the history of local time and is updated when governments change time zone and daylight saving rules. Source
- Test of Time uses synthetic data and separates temporal semantics and logic from temporal arithmetic. Source
- Agent evaluations combine code-based, model-based and human graders, with model graders calibrated against humans. Source
- BIG-bench asks that task files and publications include its canary string. Source
- The NIST handbook describes the Wilson score interval for a proportion. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.