Skip to content

AI uses for records

RL environment time-dependent tasks: adding real business deadlines

By SourceX Editorial · Updated

10 sections

Short answer

To add meaningful business deadlines to an enterprise RL task, give the environment its own replay clock, store each deadline with its start event and counting rule in state, show the agent the time remaining, charge actions for the time they take, and end or score the episode by whether deadlines were met. Keep harness step caps separate as truncation, and log every clock change for replay.

Grader rubric

RL environment time

This page covers RL environment time-dependent tasks: adding real business deadlines.Illustrative

Key takeaways

  • A business deadline is part of the task: put it in state and in the observation, and treat missing it as termination.
  • A turn or step cap added for cost is truncation; keep it out of the reward and the observation.
  • The environment, not the host machine, owns the clock, and every clock change is logged with its cause.
  • Model deadlines the way the source system counts them: business hours, calendar days, roll-forward rules and clock restarts.
  • Test time zones, calendar versions and start-time sweeps before a task ships.

How do you add meaningful business deadlines to enterprise RL tasks?

Put time inside the environment: a replay clock the environment owns, deadlines stored as state the agent can observe, and a rule for how much time each action consumes. Then make the deadline part of how the episode ends or is scored, so that acting late changes the outcome. A deadline that lives only in the prompt text, or a step cap the agent cannot see, does not teach the agent anything about time.

Business deadlines matter because they change which action is correct, not just how fast it should come. Returning a defective invoice is right on day 3 and pointless on day 40. Escalating a ticket is wasteful an hour after it arrives and mandatory just before a service target lapses. If the environment cannot express that difference, the agent will learn one policy for every point in the clock.

This page sets out how to do that: separating task deadlines from harness cutoffs, the time rules worth modeling, a time-aware task specification, a replay clock design, a worked invoice example and the acceptance tests that show whether the clock behaves.

Is the deadline part of the task or just a harness cutoff?

Decide this first, because the two need opposite treatment. Gymnasium's API separates them: terminated means the episode reached a terminal state defined by the environment, such as success or failure, while truncated means it was stopped by an outside condition, typically a time limit or step cap. Its time-limits tutorial adds that a time limit inherent to a finite-horizon task counts as termination, and that the remaining time must then be in the agent's observation to keep the Markov property.

The distinction changes learning, not only bookkeeping. The same tutorial explains that value targets bootstrap from the next state after truncation but not after termination, so treating a harness cutoff as a real ending, or a real deadline as a cutoff, trains the wrong values. Pardo and coauthors made the same argument in Time Limits in Reinforcement Learning: when time limits are part of the task, the agent should receive the remaining time as input; when they only bound training, the learner should bootstrap at the cut.

For enterprise tasks, a contractual or regulatory deadline is part of the task: the business world really does change at that moment, so it belongs in state and in the observation. A cap of a fixed number of agent turns that you added to keep runs affordable is a harness limit: record it as truncation and keep it out of the reward.

Which kinds of time actually change the right action?

Model only time rules that flip a decision, and model them the way the source system computes them. The six patterns below cover most business processes; each needs a different representation in state and a different check in the grader.

Which kinds of time actually change the right action?
Time ruleReal-world patternWhat the agent must observeHow the grader uses it
Window that closestau-bench airline policy: any reservation can be cancelled within 24 hours of bookingBooking time and current timeSame request is valid or invalid depending on the clock
Business-hours targetZendesk SLA targets run in calendar or business hours; business-hour targets pause outside the scheduleSchedule, holiday list, elapsed business timeTarget met or breached in business time
Calendar-day count with roll-forwardU.S. federal prompt payment: due 30 days after a proper invoice, calendar days, next business day if the date falls on a weekend or holidayStart event, day-count rule, holiday calendarPayment on time, early or late
Clock that restartsPrompt payment clock starts only when a proper invoice is received; an improper one is returned within 7 daysWhich event started the clockCredit for restarting correctly, penalty for counting from the wrong event
Aging and escalationA case moves to a new owner after a set ageCase age and escalation rulesWhether the agent acted before or after the handoff
Scheduled external eventA counterparty reply or system batch arrives at a set timePending events the agent knows aboutWhether waiting was a better action than acting

The time-aware task specification: fields to fill in

Every field below should be filled before the task enters a training or evaluation set. The values in the third column are illustrative and belong to the worked example later on this page; they are not taken from a real business.

At a glance

The time-aware task specification: fields to fill in

The time-aware task specification: fields to fill in
FieldWhat it holdsExample value (illustrative)Failure it prevents
clock.startEpisode start as a zoned timestamp, never the host timeTuesday 09:00 in America/Chicago, fixed per seedRuns drifting with the day they execute
clock.zone and calendar_versionIANA zone key and a versioned holiday and business-hours calendarAmerica/Chicago, cal-2026-r3Silent changes when a holiday list is edited
clock.advance_ruleHow time moves: per action cost, scheduled jumps, or explicit wait actionsRead 2 min, draft 10 min, wait advances to next eventAgents that act instantly and never pay for delay
deadlines[]Each deadline with its start event, count rule and roll-forward ruleReturn defect notice by day 7; pay by day 30 from proper invoiceDeadlines hard-coded in prompt prose
observation.time_fieldsWhat the agent sees: now, time remaining per deadline, calendar flagsnow, days_to_return_notice, is_business_dayState aliasing when remaining time is hidden
scheduled_events[]Future events and when they fireSupplier sends corrected invoice on day 5Agents rewarded for acting when waiting was right
terminationDeadline outcomes that end the episodePayment made, or day 30 passes unpaidTreating a real deadline as a cutoff
truncationHarness caps, kept separateTurn cap for cost controlPenalizing the agent for a budget it never saw
reward.time_termsHow lateness or earliness is scoredLate payment fails; early payment before validation failsA single success bit that ignores timing
clock_logEvery clock change with the action or event that caused itStep 4: wait, clock to day 5 08:00 (event)Replays that cannot reproduce the timeline

How should the replay clock advance during an episode?

Advance the clock only through declared rules and log every change, so that the same seed and the same actions always produce the same timeline. The wall clock of the machine running the episode should never reach the environment; tau-bench takes the simplest version of this approach by stating a fixed current time in the airline policy the agent reads, so every run sees the same date.

A fixed time is enough for single-decision tasks. Multi-step business tasks need the clock to move, and there are three workable ways to move it. Charge each action a declared duration, so reading a record and drafting a letter cost different amounts of time. Jump to the next scheduled event when the agent chooses an explicit wait action. Or combine the two, which is usually the most realistic. Testing libraries already support these modes for application code: freezegun freezes the datetime module at a chosen time, can restart time and let it tick, can auto-increment by a fixed number of seconds on each call, and offers manual tick and move_to controls.

Record the clock in the episode log, not only in memory. Each entry should name the step, the clock before and after, and the cause. That log is what lets a second team replay the episode and get the same deadline outcomes, and it is the first thing to check when a grader verdict looks wrong.

  • No call to the host clock anywhere in tools, graders or simulators.
  • One clock object owned by the environment and passed to every tool.
  • Each action has a declared duration; unknown actions are rejected, not timed at zero.
  • Wait actions jump to the next scheduled event or to a stated time, never beyond the next deadline without logging it.
  • Every deadline outcome is computed from the clock log, so it can be recomputed after the episode.

How should a grader score acting late, early or at the wrong moment?

Score timing as its own dimension instead of folding it into one success bit, because the timing mistakes agents make are different in kind. Missing a deadline is the obvious failure. Acting too early is the less obvious one: paying before an invoice is validated, closing a ticket before the customer's reply window ends, or sending a reminder before the counterparty could reasonably have answered. A grader that only checks whether the final state is correct will pass all of these whenever the outcome happens to land right.

At a glance

How should a grader score acting late, early or at the wrong moment

Compute every timing verdict from the clock log, never from the agent's own statements about time. Agents often claim an action is within the window while acting after it closes, and a grader that reads the transcript will believe them.

Keep the timing terms simple enough that a process owner can read them. A hard deadline is pass or fail. A target with a tolerance, such as a business-hours response goal, can be scored by the elapsed business time against the target. Waiting that was correct, because a scheduled event would supply missing information, should be credited explicitly; otherwise the agent learns that acting is always better than waiting.

  • Deadline met or missed, per deadline, computed from the clock log.
  • Premature action: a gated or irreversible step taken before its earliest valid time.
  • Wrong start event: a count that began from the wrong document or message.
  • Wasted time: actions that consumed clock without changing state, such as repeated polling.
  • Correct wait: a wait action that let a scheduled event resolve the task.

Worked example: a supplier invoice with a defect (illustrative)

This example is invented to show the specification in use. Its day counts are loosely modeled on the U.S. federal prompt payment rules described by the Treasury's Fiscal Service, which apply to federal agencies; a private company's terms would come from its own contracts and policies, and the task should cite whichever rule it models.

The agent works in accounts payable. On day 0 an invoice arrives that is missing a required purchase order reference. The policy in state says a defective invoice must be returned with the defect named within 7 days, and that payment is due 30 calendar days after a proper invoice is received, moving to the next business day when that date falls on a weekend or holiday. A corrected invoice is scheduled to arrive on day 5 if, and only if, the agent returns the original by day 4.

The table walks through one passing trajectory and shows where a time-blind agent would go wrong. All counts are illustrative.

Worked example: a supplier invoice with a defect (illustrative)
ClockAgent action or eventTime-aware state after the stepWhat a time-blind agent does
Day 0, 09:00Reads invoice, finds missing PO referenceReturn deadline day 7; payment clock not startedStarts the 30-day count from day 0
Day 0, 09:20Sends defect notice naming the missing fieldNotice sent within window; corrected invoice scheduledPays the defective invoice to clear the queue
Day 0 to day 5Waits for the scheduled eventClock jumps to day 5 08:00 via the eventPolls repeatedly, each poll costing time
Day 5, 08:00Corrected invoice received and validatedPayment clock starts; due date computed from day 5Keeps the original due date
Due date checkDue date falls on a Sunday in the calendarRolls to Monday per the calendar versionPays on Friday or misses Monday
Scheduled payment runSchedules payment for the rolled due dateTermination: paid on timeEpisode scored late or early

Which clock assumptions change the result of the acceptance test?

Five assumptions flip deadline outcomes often enough that each should be fixed in the specification and tested before the task ships. The first is the time zone. Python's zoneinfo documentation shows that during a daylight saving transition the same local time is ambiguous and is resolved by the fold attribute, so a deadline at a local clock time on a transition night can land an hour apart depending on how it was computed.

At a glance

Which clock assumptions change the result of the acceptance test

The second is the calendar version. Zendesk lets teams add holidays as exceptions to business hours, up to two years ahead, and treats each holiday as the full calendar day; an environment that copies such a schedule must pin which version it used, because adding one holiday moves every business-hour target that spans it. The third is the start-time distribution: a task that always starts on a Tuesday morning never tests the weekend roll-forward. The fourth is action duration: change the cost of a lookup and a borderline trajectory crosses a deadline. The fifth is any leak of host time through a tool, a log formatter or a simulator.

Turn those into acceptance tests that run on every task version.

  • Replay test: the same seed and action sequence reproduce the same clock log and deadline outcomes, byte for byte.
  • Start-time sweep: the task is solvable from every sampled start, including Friday afternoon and the day before a holiday.
  • Transition test: at least one sampled start crosses a daylight saving change in the configured zone.
  • Calendar swap: changing calendar_version changes outcomes only where the calendar says it should.
  • Host-clock trap: running the environment with the host clock set years ahead changes nothing.
  • Oracle check: a scripted policy that follows the rules meets every deadline; one that ignores time misses at least one.

When should time stay out of the task?

Leave deadlines out when no action's correctness depends on them. Adding a clock to a task whose right answer is the same at any time only adds noise and a new way for graders to disagree. The same applies to truncation: a step cap for cost control is not a business deadline and should not appear in the observation as if it were one.

Be careful with deadlines set by law or contract. A modeled rule is a simplification, and the environment should say which rule it models and from which version. If the task is meant to teach compliance with a specific statutory or contractual deadline, have someone who owns that process check the specification, and treat the environment as training material, not as a statement of what the rule requires in a particular case.

Narrow the scope when the timeline depends on events the environment cannot simulate believably, such as a counterparty's negotiating behavior over weeks. In that case, keep the deadline but script the events, and record that the counterparty is scripted so evaluators do not read the result as negotiation skill.

Where realistic timelines come from

The hardest inputs to invent are real timelines: how long invoices actually sit before someone notices a defect, how often corrected documents arrive late, how service targets interact with holidays. Timestamped histories from finance, service and operations systems supply those distributions, which can then set the scheduled events and start-time samples in the specification.

Environment builders who need those distributions can ask SourceX for timestamped histories. It has no catalog to sell from: it searches U.S. businesses against a request covering scope, volume, language, format, timing and licensing needs, then takes the records a business agrees to license through rights review, contracting and preparation.

Frequently asked questions

What is a time-dependent task in an RL environment?

It is a task where the correct action depends on when it is taken, such as returning a document before a window closes or escalating before a service target lapses. The environment has to own the clock and expose the relevant times to the agent for the task to be learnable.

Should the remaining time be part of the observation?

Yes, when the deadline is part of the task. Gymnasium's guidance and the Time Limits in Reinforcement Learning paper both say that a finite-horizon task needs a representation of remaining time in the observation; otherwise identical-looking states have different values.

What is the difference between termination and truncation for deadlines?

Termination means the episode reached an ending defined by the task, such as a payment made or a deadline missed. Truncation means an outside limit, like a turn cap, stopped it. Learning code bootstraps after truncation and not after termination, so mixing them up trains the wrong values.

How do you make deadline outcomes reproducible?

Use one environment-owned clock, declare how each action advances it, pin the time zone and calendar version, and log every clock change with its cause. A replay of the same seed and actions should then reproduce the same timeline and verdicts.

Can a fixed date in the prompt replace a clock?

For a single decision, yes; some benchmarks state a fixed current time in the policy the agent reads. For multi-step tasks where waiting or delay matters, the clock must advance under declared rules, or the agent never pays a cost for being slow.

Sources

  • Gymnasium returns terminated for task-defined endings and truncated for externally imposed limits such as a fixed number of timesteps. Source
  • Gymnasium's time-limits tutorial says finite-horizon time limits are termination and need remaining time in the observation, and that learning code bootstraps after truncation but not termination. Source
  • Time Limits in Reinforcement Learning (Pardo et al.) argues for giving agents the remaining time when limits are part of the task and bootstrapping at the end of partial episodes when they are not. Source
  • The tau-bench airline policy states a fixed current time and allows any reservation to be cancelled within 24 hours of booking. Source
  • Zendesk SLA targets run in calendar or business hours, and business-hour targets pause outside business hours. Source
  • Zendesk holidays are exceptions to business hours, can be set up to two years ahead and count as the full calendar day. Source
  • Under U.S. federal prompt payment rules, payment is due 30 calendar days after a proper invoice, weekend or holiday due dates move to the next business day, and improper invoices are returned within 7 days. Source
  • Python's zoneinfo resolves ambiguous local times during daylight saving transitions with the fold attribute. Source
  • freezegun freezes Python's datetime at a chosen time and supports ticking, auto-increment, manual tick and move_to. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies