Skip to content

AI uses for records

Business timeline reasoning benchmark: a protocol for moving deadlines

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

This is a protocol, not a result. A business timeline reasoning benchmark should give the model an event log, a calendar and the policy versions in force, then ask for a specific deadline and the evidence behind it. Tasks should vary what moves the deadline: business-day counting, holidays, time zones, amended events and policy changes. Grade the date exactly, check the cited trigger and rule, and reward abstention when evidence is missing.

Evaluation split

Business timeline reasoning

This page covers business timeline reasoning benchmark: a protocol for moving deadlines.Illustrative

Key takeaways

  • This page proposes a method and reports no results, scores or rankings.
  • Each task supplies three kinds of evidence: an event log, a calendar and the policy text in force; the answer is a date plus the trigger and rule that produced it.
  • Task families are defined by what changes the deadline, such as a holiday, a time zone, an amended event or a new policy version.
  • Grade the date by exact match against a reference calculation, then check that the cited trigger and rule are the right ones, and score abstention when evidence is missing.
  • Report accuracy per task family with intervals, and hold out task families and source organizations, not just random tasks.

How should a benchmark test deadlines that depend on a changing timeline?

It should give the system the same evidence a careful operations analyst would use, an event log, a business calendar and the policy versions in force, and ask for one deadline with the trigger event and rule that produced it. Then it should change one thing at a time in the timeline and check whether the deadline moves the way the rule requires.

This page is a protocol for that business timeline reasoning benchmark. It describes the question, the task suite, the schema, the grader, the baselines and the pre-registration. It reports no measured findings, and every example is labeled illustrative.

The research question is: given a record of dated events and the written rules that govern a deadline, does an AI system compute the correct deadline, identify the event that triggered it, and recognize when the evidence is insufficient?

Why do business deadlines depend on the timeline?

Business deadlines are computed from a trigger event under a counting rule, and both can change after the fact. Published rules show how much the counting method alone matters.

Federal Rule of Civil Procedure 6(a)(1) excludes the day of the triggering event, counts every day including weekends and legal holidays, and includes the last day unless it falls on a Saturday, Sunday or legal holiday, in which case the period runs to the next day that is not. Under Rule 6(a)(5), the next day is found by counting backward when a period is measured before an event, so a backward-counted period that lands on a weekend moves earlier, not later.

Regulation E's error resolution rule mixes units. A consumer's notice must arrive within 60 days after the institution sent the periodic statement, the institution generally has 10 business days to investigate, and it may take up to 45 days if it provisionally credits the account within those 10 business days, with longer periods in specified cases. Whether the longer clock applies depends on an event in the timeline, the provisional credit.

Calendars and clocks move too. The Office of Personnel Management explains that when a federal holiday falls on a Saturday, the in-lieu-of holiday is generally the preceding workday, and on a Sunday the following workday. The IANA time zone database records the history of local time, including offsets and daylight saving rules, and is updated when governments change them, so a cutoff at 5 p.m. local time depends on which rules applied on that date.

Why do business deadlines depend on the timeline?
Rule exampleTriggerCounting unitWhat can move the deadline
Court filing period (FRCP 6)Service or entry of an orderCalendar days with weekend and holiday roll-forwardA last day on a weekend or legal holiday; backward counting
Error investigation (Reg E 1005.11)Receipt of a notice of errorBusiness days, then calendar daysProvisional credit within 10 business days extends the period
Error notice window (Reg E 1005.11)Sending of the periodic statementCalendar daysA reissued or corrected statement changes the trigger
Observed holiday (OPM)Holiday falling on a weekendWorkdaysSaturday holidays observed the day before, Sunday the day after
Local-time cutoff (tz database)Submission time stampHours in a named time zoneDaylight saving changes and historical offset changes

What do existing temporal reasoning benchmarks leave out?

Existing benchmarks test temporal reasoning well in isolation but rarely combine dated business events with written policy. Test of Time, from Google researchers, uses synthetic data to avoid problems with pretraining exposure and anonymization, and separates temporal semantics and logic from temporal arithmetic. That separation is worth copying.

What business deadlines add is evidence handling. The system must pick the right trigger from a log with several candidates, apply the policy version in force on the trigger date rather than today's, apply the right calendar, and notice when a later event, such as a corrected statement or a provisional credit, changes which rule applies. A benchmark that hands the model the trigger and rule as clean inputs skips the part that fails in practice.

Which deadline decisions go in, and how are the one-change variants built?

The population is deadline decisions from operational records whose outcome is known: a recorded due date that a reviewer confirmed, or a missed or met deadline with the computation documented. Sources include dispute and claims handling, contract notice periods, service-level timers in ticket systems, and payment terms. Synthetic tasks fill families where real records are thin, and are labeled synthetic in the data.

Stratify by task family, by rule source and by contributing organization, and cap each organization's share. Hold out entire task families for a generalization test and entire organizations for a transfer test, because an organization's calendar and policy wording repeat across its records.

Pin the calendar data. Record which holiday calendar and which time zone database release each task was computed against, so a later release cannot silently change the gold answer.

Build perturbation sets on purpose. For each base task, create variants that change exactly one input, such as moving the trigger by one day, adding an observed holiday, amending the event or swapping the policy version, and record the expected new deadline for each. Comparing a system's answers across a base task and its variants shows whether it is applying the rule or matching a familiar answer, and it makes every family's failures easy to diagnose.

Record the denominator for real records: how many deadline decisions each source had in the window, how many met the inclusion rules, and how many were dropped at rights review, de-identification or gold verification.

The date and policy evidence reasoning suite

The suite is organized by what changes the deadline, so results show which kind of change a system mishandles. Examples are illustrative.

The date and policy evidence reasoning suite
Task familyWhat changesEvidence givenIllustrative question
1. Base countNothing; a single trigger and ruleOne event, one rule, a calendarNotice received on a stated date; when is the 10-business-day deadline?
2. Unit mixingBusiness days versus calendar daysRule with both unitsInvestigation deadline versus extended deadline for the same notice
3. Weekend and holiday rollLast day lands on a non-working dayCalendar with observed holidaysPeriod ends on a Saturday holiday; which day is the deadline?
4. Backward countingDeadline counted before an eventEvent date and ruleFiling due a stated number of days before a hearing that falls after a weekend
5. Time zone and cutoffLocal cutoff across a daylight saving changeTime stamps, zone name, tz releaseWas a submission at a given UTC time before the local 5 p.m. cutoff?
6. Amended eventTrigger event corrected or reissuedEvent log with an amendmentStatement reissued with a correction; which date starts the window?
7. Conditional extensionAn action in the log changes the ruleLog with or without a provisional creditDoes the longer investigation period apply?
8. Policy versionRule changed between trigger and todayTwo dated policy versionsWhich notice period applies to a contract signed before the change?
9. Competing triggersSeveral events could start the clockLog with near-duplicate eventsWhich of two receipts of the same complaint starts the period?
10. Insufficient evidenceA needed fact is missingLog without the trigger date or calendarThe correct answer is to abstain and name the missing fact

What does one deadline task packet contain?

Every task is a packet of dated evidence and a question with one correct date, or a required abstention. Keep the reference computation out of the model input.

At a glance

What does one deadline task packet contain

What does one deadline task packet contain?
FieldContentGrader use
task_id, family, splitStable ID, family number, public or holdoutPer-family reporting; split enforcement
event_logDated events with types and time stamps, including amendmentsInput; the grader knows which event is the trigger
calendarWorking days, observed holidays, time zone name and tz releaseInput; pinned for reproducibility
policy_versionsRule text with effective datesInput; the grader knows which version applies
questionOne deadline to compute, with the required output formatInput
gold_deadlineDate and, where relevant, local time and zone, or ABSTAINExact-match check
gold_trigger and gold_ruleEvent ID and policy version ID that produce the deadlineEvidence check
reference_computationStep-by-step count produced by a reference calculator and checked by a reviewerAudit of the gold; never shown to the model
provenance and canarySource type (real or synthetic), rights record, canary stringRelease gate and contamination detection

How are the date, the cited evidence and abstentions graded?

Grade the date with code and the evidence with a rubric. Anthropic's guidance on agent evaluations describes combining code-based, model-based and human graders and calibrating model-based graders against human judgment; this plan uses that split.

  • Date check (code): the answer date, and time and zone where required, must equal gold_deadline exactly. Near misses are wrong, because a deadline one day late is a missed deadline.
  • Evidence check (code first, then rubric): the answer must cite the gold trigger event ID and the policy version ID. Where the answer is free text, a model grader calibrated on expert labels extracts the cited trigger and rule.
  • Abstention check (code): on family 10 tasks, ABSTAIN with the missing fact named is correct and any date is wrong; on other families, abstaining is scored as wrong but reported separately from wrong dates.
  • Adjudication (human): two reviewers with operations or compliance background resolve grader disagreements and audit every task where the system's date differs from gold by exactly one day, which is where gold errors most often hide.

Which baselines and metrics show where deadline errors come from?

Use three baselines. A deterministic reference calculator given the correct trigger and rule shows the ceiling when evidence selection is solved. A naive calendar-day baseline that ignores business days, holidays and amendments shows how much each family depends on those rules. Human analysts on a blind subset show achievable accuracy and expose ambiguous tasks.

At a glance

Which baselines and metrics show where deadline errors come from

The primary metric is exact deadline accuracy per family. Secondary metrics are trigger accuracy, rule-version accuracy, abstention precision and recall on family 10, and the share of errors that are late rather than early, because a late deadline is usually the costlier mistake. Report each proportion with a Wilson score interval as described in the NIST Engineering Statistics Handbook, and resample by organization.

Pre-register before running any system. The registration should include the items below.

  • Research question and the exact output format.
  • Population, sources, real versus synthetic share, strata and caps.
  • Holdout families and organizations.
  • Pinned calendars and tz database release.
  • Task schema, grader and rubric versions.
  • Baselines and model identifiers with dates.
  • Primary and secondary metrics, interval method and resampling unit.
  • Exclusion rules, and how tasks with disputed gold are handled.
  • Conditions for releasing results.

Rights, release and limits

Real tasks come from operational records, so the same permissions apply as for any licensed dataset: system access, privacy rules, ownership and a license that permits benchmark use and publication are separate questions, and de-identification does not create a right to publish. Shifting dates to de-identify records must preserve weekdays and holiday positions, or the task changes its answer.

Mark every released file with a canary string, as BIG-bench asks for its task files, and keep a private holdout. Limits to state on the card: the suite covers rules written in English, chosen calendars and the sampled rule sources, and it does not measure whether a computed deadline is legally correct for a specific matter.

Teams that need real event logs and policy histories from other businesses can ask SourceX to source them from U.S. businesses on request, setting out scope, volume, format, timing and licensing needs, including whether a public benchmark split may be released.

Frequently asked questions

Does this page report results for any AI model?

No. It is a protocol for a benchmark that has not been run: task families, schema, grader, baselines and a pre-registration checklist. Any example is illustrative, and results should be published only after the grader is calibrated and the holdout is intact.

What should the date and policy evidence reasoning suite include?

Tasks that vary one thing that moves a deadline: units, weekend and holiday roll-forward, backward counting, time zones, amended events, conditional extensions, policy versions, competing triggers and missing evidence. Each task gives an event log, a calendar and dated policy text, and expects a date with the trigger and rule that produced it.

Why grade deadlines by exact match instead of partial credit?

A deadline that is one day late is a missed deadline, so near misses carry real cost. Partial credit is better spent on the evidence: report trigger and rule accuracy separately so a team can see whether errors come from selecting evidence or from counting.

Which benchmark result would remain valid on held-out companies and task families?

Only a result reported per family on organizations and families the system never saw during development. Pooled accuracy on random splits can reflect repeated calendars and policy wording from the same organization rather than general timeline reasoning.

Is this legal advice on computing deadlines?

No. Court rules and regulations appear only as examples of published counting rules. A specific deadline depends on the governing rule, jurisdiction and facts, which counsel or the responsible compliance owner should confirm.

Sources

  • FRCP 6(a)(1) excludes the trigger day, counts every day including weekends and holidays, and rolls a last day on a weekend or legal holiday to the next day, which Rule 6(a)(5) counts backward for periods measured before an event. Source
  • Regulation E sets a 60-day notice window, a 10-business-day investigation period, and up to 45 days with provisional credit. Source
  • OPM explains in-lieu-of holidays: the preceding workday for a Saturday holiday and the following workday for a Sunday holiday. Source
  • The tz database records the history of local time and is updated when governments change time zone and daylight saving rules. Source
  • Test of Time uses synthetic data and separates temporal semantics and logic from temporal arithmetic. Source
  • Agent evaluations combine code-based, model-based and human graders, with model graders calibrated against humans. Source
  • BIG-bench asks that task files and publications include its canary string. Source
  • The NIST handbook describes the Wilson score interval for a proportion. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies