Systems and records
RL environment task horizon: measuring it beyond the step count
By SourceX Editorial · Updated
10 sections
Short answer
Measure the task horizon of an RL environment per task, then as a distribution: record the fewest state-changing actions on a verified solution, the dependency depth of subgoals that must happen in order, and a reference solution's steps. Fix initial state, actions, reset and success first, compare each task with the step limit, and test for shortcuts that finish long-labeled tasks early.
Grader rubric
RL environment task horizon
Key takeaways
- Task horizon is a per-task property reported as a distribution; a suite average hides a pool dominated by short tasks.
- Minimal action count, dependency depth, reference steps and human time measure different things; name the one behind any number.
- Write the environment contract first: initial state, actions, seeded reset, termination and truncation limits.
- A task dependency diagram with a state check per subgoal gives the critical path and the checks a grader should enforce.
- Shortcut tests catch long labels on short tasks; the usual fix is a stricter success test or checkpoint scoring.
How do you measure the effective task horizon of an RL environment?
Measure it per task, then report the distribution across the task pool. For each task, record three numbers: the fewest state-changing actions on a verified solution path, the dependency depth (the longest chain of subgoals that must happen in order), and the steps a reference solution actually took. Compare those numbers with the environment's step limit and check that no shortcut finishes the task in fewer steps than its label claims.
A single average hides the shape that matters. A suite whose mean is 20 steps can be mostly 4-step tasks plus a few very long ones, and an agent trained on it learns short tasks. The asset on this page is a horizon distribution and a task dependency diagram, built after the environment contract is fixed, with a worked example and an exception.
The order matters: define the initial state, the allowed actions, the reset behavior and the observable success criteria before measuring horizon. Without them, a step count is a property of one run, not of the environment.
Step count, dependency depth and human time measure different horizons
Horizon has several defensible definitions, and a buyer should ask which one a supplier used. The measures in the table below can disagree sharply on the same task, so a dataset card should state each one it reports and how it was obtained.
- METR measures agent capability as the length of tasks, in skilled-human completion time, that a model completes about half the time; that is a property of the model against a task suite, not of a single task.
- Laidlaw, Russell and Dragan define the effective horizon as roughly the lookahead needed when leaf nodes are evaluated with random rollouts, and show it tracks deep RL performance across 155 deterministic benchmark environments better than earlier bounds.
| Horizon measure | Definition | How to obtain it | What it misses |
|---|---|---|---|
| Minimal action count | Fewest state-changing actions on any verified solution path | Search or hand-written shortest solution checked by the success test | Reads, waits and the difficulty of choosing each action |
| Dependency depth | Longest chain of subgoals where each needs the previous one done | Critical path through the task dependency diagram | Total work on parallel branches |
| Observed reference steps | Steps a competent reference solution actually took | Logged runs by a scripted or human solver | Detours and retries inflate it; skill differences shift it |
| Human time horizon | Time a skilled person needs to complete the task | Timed human baseliners in the same environment, as in METR's work | Whether that time is spent on many actions or a few hard ones |
| Effective horizon (theory) | Lookahead depth needed to pick the next optimal action when leaves are scored by random rollouts | Computed for tabular or small deterministic environments | Not directly computable for large tool-using business environments |
Fixing the environment contract before counting steps
Horizon numbers are only comparable when the contract is written down. The Gymnasium API is a useful reference point even for business environments, because it separates two ways an episode ends. Its step function returns terminated when the agent reaches a terminal state defined by the task, and truncated when a condition outside the task, typically a time limit, cuts the episode short. Its TimeLimit wrapper issues truncation after a maximum number of steps.
Steps
Fixing the environment contract before counting steps
That distinction is the first horizon check. If truncation fires before a reference solution can terminate, the environment caps the horizon below what the task requires, and no policy can succeed. Gymnasium also documents that reset should be called with a seed to make the random number generator reproducible; for a business environment the equivalent is a seeded or snapshot-based reset that rebuilds the same records, permissions and clock.
Checklist · 6 items
- Initial state: the records, documents, user permissions and clock time at reset, versioned and hashable.
- Allowed actions: the tool or API calls the agent may make, with arguments and side effects listed.
- Reset: deterministic from a seed or snapshot, or randomness documented by source and range.
- Termination: the success and failure states defined by the task, checked against system state.
- Truncation: the step, token and time limits, set from the horizon distribution rather than guessed.
- Replay log: every action, observation and state change, so a second team can rerun the episode.
0 of 6 ticked. Tick each item as you go.
Building the task dependency diagram for one task
Draw each task as subgoals with a verifiable state check, then mark which subgoals must finish before others can start. The longest chain is the dependency depth; the sum of minimal actions across all subgoals is the minimal action count. The table below is the diagram in list form for an illustrative accounts payable exception in a sandbox ERP; it is invented and not drawn from customer data.
Read it as a graph. S1 starts the task. S2 and S3 can run in parallel once S1 is done. S4 needs both. S5 and S6 form a chain that includes a vendor reply, and S7 to S9 close the case. The critical path runs S1, S2, S4, S5, S6, S7, S8, S9: a dependency depth of 8 subgoals. Summing the minimal actions gives 16 state-changing actions for the shortest path, before any reads.
| Subgoal | State check that proves it | Depends on | Minimal actions (illustrative) |
|---|---|---|---|
| S1 Open the exception and claim it | Exception owner set to the agent | None | 1 |
| S2 Retrieve the purchase order lines | PO lines attached to the exception record | S1 | 2 |
| S3 Retrieve the goods receipt | Receipt quantities attached to the exception record | S1 | 2 |
| S4 Record the quantity variance | Variance field equals invoice quantity minus received quantity | S2, S3 | 1 |
| S5 Request a credit memo from the vendor | Outbound message logged with variance and PO reference | S4 | 2 |
| S6 Apply the vendor's credit memo | Credit memo linked and invoice total adjusted | S5 and vendor reply | 3 |
| S7 Route for approval | Approval task created for the right approver | S6 | 2 |
| S8 Post the adjusted invoice | Invoice status posted with adjusted total | S7 approved | 2 |
| S9 Close the exception with a reason | Exception closed with reason code and note | S8 | 1 |
The horizon distribution: what to report across the task pool
Once every task has its numbers, report the pool as a distribution, banded by dependency depth, with minimal action counts and reference steps inside each band. The template below names the bands and what each row must show; the counts are filled from the environment, never estimated.
Add three summary lines under the table: the median and 90th percentile of minimal actions, the share of tasks whose reference solution needed more steps than the step limit allows, and the share of tasks where a shortcut search found a path shorter than the task's label. Report agent success per band, because a suite average dominated by short bands says little about long ones.
| Dependency depth band | Record for each band | What the band tells a buyer | Warning sign |
|---|---|---|---|
| 1 to 3 subgoals | Task count, minimal actions range, reference steps range | Single-system lookups and edits; useful for tool-use basics | Band holds most of the pool while the suite is sold as long-horizon |
| 4 to 7 subgoals | Same, plus number of parallel branches | Multi-step work inside one or two systems | Reference steps far above minimal actions, suggesting unclear tools |
| 8 to 15 subgoals | Same, plus external waits and handoffs per task | Case work that crosses approvals or outside parties | Step limit below the band's reference steps |
| 16 or more subgoals | Same, plus the longest single chain and its checkpoints | Projects and month-end style work | Few tasks, no partial credit, so success is near zero and uninformative |
| Any band, shortcut found | Task ids where a shorter verified path exists | Labels overstate horizon | Success check reads only the final field and ignores required intermediate states |
Where a 50-step label hides a 5-step task
A task is shorter than it looks when the environment lets the agent skip dependencies. The common causes are a tool that performs several subgoals in one call, an initial state that already contains half the work, and a success check that only reads the final status field. In each case the agent can terminate successfully without the chain the task was meant to exercise.
Test for shortcuts directly. Try to reach the success state with each subgoal removed in turn; if the check still passes, either the subgoal is unnecessary or the check is too weak. Run a random or scripted exploration baseline: the effective horizon research found that where acting greedily on a random policy's values already solves an environment, deep RL tends to succeed, which is a sign the effective horizon is short whatever the step count says.
The fix is usually in the success test, not the task. Require the intermediate state checks from the dependency diagram, such as the variance recorded and the approval granted, as part of termination, or score them as checkpoints.
Settings that move measured horizon and measured success together
Several environment settings move measured horizon and measured success together, so a buyer should see them fixed in the acceptance report rather than tuned per run.
At a glance
Settings that move measured horizon and measured success together
- Step and time limits: a limit below the 90th percentile of reference steps truncates long tasks into failures that say nothing about the agent.
- Reset determinism: if reset does not rebuild the same state from a seed or snapshot, the same task can have different minimal paths on different runs.
- Simulated users and outside parties: tau-bench uses a language-model user, and a vendor reply in a business task plays the same role; its behavior adds or removes steps, so fix its prompt and version.
- Partial credit: TheAgentCompany scores weighted checkpoints and gives full completion only when all pass, which keeps long tasks informative when few runs finish them.
- Repeated trials: tau-bench's pass^k is the chance that all k independent trials succeed; reliability on long tasks often falls faster than single-trial success suggests.
- Tool granularity: merging or splitting tools changes the minimal action count without changing the business work, so version the tool list with the horizon numbers.
Acceptance demonstrations a buyer can rerun
Ask for evidence, not a horizon claim. Each demonstration below should come with the commands, seeds and logs needed to repeat it, and with the environment and tool versions it used.
- Two resets with the same seed produce the same initial state hash; two different seeds produce documented differences only.
- For a sample from every depth band, a reference solution terminates successfully within the step limit, with its replay log.
- For the same sample, the subgoal-removal test fails when any required subgoal is skipped.
- The per-task horizon record lists task id, minimal actions, dependency depth, reference steps, step limit, truncation rate and checkpoints.
- The dependency diagram for each sampled task matches the state checks the grader actually runs.
- A changelog ties every horizon number to an environment, tool and policy version, with a refresh when any of them changes.
An exception: tasks with many valid orders and long waits
Some business tasks have no single critical path. A month-end close lets accountants reconcile accounts in many orders, and an insurance claim may wait days for a document that arrives at an unpredictable point. For these, a single dependency depth misleads; report the depth of the strictest ordering the success test enforces, the number of valid orderings where it is small enough to count, and wait states as events in the log rather than as steps.
When the environment is stochastic, when horizon depends heavily on a simulated party, or when the success test cannot be written without business judgment, bring in an RL environment engineer and the process owner who runs the workflow. They can decide which ordering constraints are real and which are habits, and that decision changes the diagram more than any measurement does.
Finding workflows long enough to need these measurements
Long, dependent tasks are easier to find in operational histories than to invent: approval chains, exception queues and multi-party cases already record the subgoals, their order and the state checks a grader needs. Turning them into an environment still needs the contract and acceptance tests above, plus a rights and privacy review of the records involved.
SourceX keeps no shelf of ready-made datasets; it sources them from U.S. businesses when a buyer asks. Its request form covers scope, volume, language, format, timing and licensing needs, and a target horizon distribution plus the subgoal checks for each task family fit naturally there.
Frequently asked questions
What should a horizon distribution and task dependency diagram include?
The diagram lists each subgoal with the state check that proves it, its dependencies and its minimal actions, and marks the critical path. The distribution bands the task pool by dependency depth and reports task counts, minimal and reference step ranges, step limits, truncation rates and shortcut findings for each band.
Is step count a good measure of RL environment task horizon?
Only partly. Step count depends on tool granularity and on detours, so the same business work can look short or long. Pair the minimal action count with dependency depth and, where available, human completion time, and state which one any headline number uses.
How does truncation differ from termination in an RL environment?
In the Gymnasium API, termination means the agent reached a terminal state defined by the task, such as success or failure. Truncation means a condition outside the task, usually a time or step limit, ended the episode early. A step limit set below what tasks require turns long tasks into truncations.
What is the effective horizon in reinforcement learning research?
Laidlaw, Russell and Dragan use it for roughly the lookahead needed to choose the next optimal action when leaf nodes are scored with random rollouts. It explains when deep RL succeeds on benchmark environments, but it is hard to compute for large tool-using business environments, so practical audits use the measures on this page.
Which environment settings shift horizon and success numbers?
Step and time limits, reset determinism, the behavior of simulated users or outside parties, partial credit rules, the number of repeated trials and tool granularity. Fix and version each one in the acceptance report so horizon and success numbers stay comparable.
Sources
- Gymnasium's Env.step returns terminated when the agent reaches a terminal state of the task and truncated when an outside condition, typically a time limit, ends the episode; reset should be seeded for reproducibility. Source
- Gymnasium's TimeLimit wrapper truncates an episode once a maximum number of steps is exceeded. Source
- METR measures AI capability as the human completion time of tasks that models complete with 50% success, using timed human baseliners. Source
- Laidlaw, Russell and Dragan introduce the effective horizon, roughly the lookahead needed when leaves are evaluated with random rollouts, and test it on 155 deterministic MDPs. Source
- tau-bench pairs agents with a language-model-simulated user and defines pass^k as the chance that all k trials of a task succeed. Source
- TheAgentCompany scores long workplace tasks with weighted checkpoints, giving partial credit and full completion only when all checkpoints pass. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.