AI uses for records
RL environment task solvability: is the task hard or impossible?
By SourceX Editorial · Updated
10 sections
Short answer
An RL task is hard, not impossible, when a reference solution passes the grader from the reset state and a qualified person working under the agent's limits can finish it. If no reference passes, the task is broken or impossible; if people fail on unstated rules, it is underspecified. Label tasks meant to be infeasible in advance and grade them on a correct refusal.
Grader rubric
RL environment task
Key takeaways
- Prove solvability with a reference run from reset on every shipped seed, not with an author's belief.
- A zero pass rate across many attempts usually points to a broken task before it points to a hard one.
- Human baselines count only when people have the agent's tools, observations and limits.
- Sweep step limits, permissions, seeds, clock and grader strictness; each can make a solvable task look impossible.
- Give every failing task a verdict and an action, and repeat the review when the environment changes.
How do you tell a hard RL task from an impossible one?
A task is hard, not impossible, when a reference solution passes the grader from the environment's reset state and at least one qualified person working under the agent's conditions can complete it. If no reference passes, the task is broken or impossible. If the reference passes but people fail for reasons the task text never stated, it is underspecified. Tasks meant to be impossible must be labeled and graded on a correct refusal.
This matters because an RL environment rewards whatever its tasks reward. A task that cannot be completed gives zero reward on every rollout and teaches nothing, or worse, teaches the agent to stop early. A task that only looks impossible because a grader rejects valid answers pushes the policy toward the grader's quirks. Anthropic's agent evaluation guidance puts the warning plainly: with frontier models, a 0% pass rate across many trials is most often a signal of a broken task, not a hard one.
The protocol below separates those cases with evidence a buyer can rerun: a reference solution executed in the environment, human attempts recorded under the same limits the agent has, and an independent review of every task where those two disagree.
Four verdicts for a task nobody passes
Give every task that fails all model attempts one of four verdicts, each backed by a different kind of evidence. Treating them as one bucket of hard tasks is how environments ship with dead tasks inside.
| Verdict | What it means | Evidence that establishes it | What to do |
|---|---|---|---|
| Hard | Solvable within the rules, but beyond current agents | Reference passes; at least one qualified human passes under agent conditions | Keep; it is the signal you want |
| Impossible by design | The correct behavior is to refuse or report infeasibility | Task labeled infeasible in advance; grader checks for a correct refusal | Keep, labeled, and graded on the refusal |
| Broken | The environment or grader prevents any valid completion | No reference passes from the reset state, or the grader rejects a verified correct end state | Fix the environment or grader, then re-review |
| Underspecified | Success depends on something the task text does not say | Reference passes; qualified humans fail or disagree on what was asked | Rewrite the task text, then re-review |
What the environment must pin down before solvability can be judged
Solvability is a property of a task in a specific environment state, so fix the state, actions, reset and success criteria first; otherwise a task can be solvable on Monday and impossible after a dependency update. Write down the initial state the task starts from, the actions an agent may take, how reset restores that state, and the observable condition the grader checks.
Distinguish ending from stopping. The Gymnasium API separates terminated, meaning the agent reached a terminal state of the task, from truncated, a condition outside the task such as a time limit. A task that always ends in truncation is not proven impossible; it may only need more steps than the limit allows. Record both flags on every attempt.
Note the randomness too. If reset draws seeded values, solvability has to hold across the seeds you ship, not only the one the reference was built on. Tasks whose solvability depends on a lucky seed are a separate failure from tasks that are hard.
The human baseline and solvability review protocol
Run the steps in order for every task, and store the evidence column so a buyer or a second team can rerun the review. A task advances only when its step passes; otherwise it gets a verdict from the table above and leaves the protocol.
| Step | Who | What they do | Passes when | Evidence kept |
|---|---|---|---|---|
| 1. Spec read | Task author and a second reviewer | Check the text states every condition the grader checks | Both agree nothing graded is missing from the text | Annotated task text |
| 2. Reference run | Environment engineer | Execute a reference solution from reset, through the agent's action interface | Grader passes on every shipped seed | Action log, final state, grader output per seed |
| 3. Negative check | Environment engineer | Run a do-nothing attempt and a plausible wrong answer | Grader fails both | Grader outputs for each |
| 4. Human attempt | Two qualified people, separately | Attempt the task with only the agent's tools, observations and limits | At least one completes it | Recording, time, steps, outcome, notes on where they stalled |
| 5. Verdict agreement | Two domain experts, separately | Judge each human end state pass or fail without seeing the grader | Experts agree with each other and with the grader | Both verdicts and any disagreement note |
| 6. Assumption sweep | Environment engineer | Rerun the reference with step limit, seed and clock varied within shipped ranges | Reference still passes | Results per setting |
| 7. Review board | Engineer, process owner, evaluator | Assign hard, impossible by design, broken or underspecified to any task that failed a step | A verdict and an action are recorded | Decision record with date and environment version |
Running human baselines under the agent's own conditions
A human baseline only shows solvability if the person had what the agent has: the same tools, the same observations, the same step and time limits, and no outside knowledge of the hidden state. METR's HCAST benchmark followed this rule, collecting 563 baselines from people skilled in the relevant domains working under conditions identical to the agents', which is why its time estimates can be compared with agent results.
Human failure is not proof of impossibility. Even on WebArena, a benchmark of realistic web tasks, the paper reports human annotators completing 78.24% of tasks; GAIA reports human respondents at 92% on questions designed to be simple for people. Some tasks in any pool will be missed by a given person on a given day, so use at least two people per task and treat a single failure as a prompt for review, not a verdict.
Record more than pass or fail. Time taken, steps, the point where the person stalled, and what they said they were missing are what separate a hard task from an underspecified one. A person who solved it after guessing an unstated rule is evidence of underspecification, even though the attempt passed.
Settings outside the task text that flip a solvability verdict
Several settings outside the task text can flip a task between solvable and not; sweep them in step 6 and record the range over which the verdict holds. A task that is only solvable at one setting should say so in its metadata. These are also the settings most likely to drift between the environment a vendor tested and the one a buyer runs, so the sweep results belong in the delivery package alongside the tasks themselves.
| Assumption | How it makes a task look impossible | Check |
|---|---|---|
| Step or time limit | Valid solution needs more steps than the limit; every attempt truncates | Compare the reference's step count with the limit; record truncated versus terminated |
| Permissions of the agent's role | Task requires an action the role cannot take | Confirm the reference used only the agent's role and credentials |
| Seeded randomness | Some seeds generate states with no valid path | Run the reference on every shipped seed |
| Environment clock | Deadlines, business hours or date rules block the action at reset time | Fix the clock in the reset state and test boundary dates |
| Grader strictness | Correct end states phrased or formatted differently are rejected | Score human end states that experts judged correct |
| Dependencies and versions | An updated package or app changes the interface the task relies on | Pin versions and rerun the reference on each upgrade |
| External services | Network calls fail or return different data in the sandbox | Stub or record external responses; fail closed if missing |
When the reference, the humans and the grader disagree
Disagreement is the most useful output of the protocol, because each pattern points at a different fix. When the reference passes and qualified humans fail, suspect the task text. When humans reach an end state that experts judge correct and the grader fails it, suspect the grader. When nobody, reference included, gets through, suspect the environment.
Public benchmarks have hit each of these. OpenAI's SWE-bench Verified effort had professional software developers screen samples of the original SWE-bench for issue descriptions that were underspecified and for unit tests that could reject valid solutions, leaving a 500-task subset that the SWE-bench project describes as confirmed solvable. The Agentic Benchmark Checklist paper frames the same split as task validity, whether success requires the capability being measured, and outcome validity, whether the grader detects success.
Anthropic's guidance gives a simple bar for the expert step: a good task is one where two domain experts would independently reach the same pass or fail verdict. If they cannot, the task is not ready for training, however hard it looks.
Illustrative review: a purchase order task that no attempt could finish
This review is illustrative; it does not describe a real environment. A procurement environment includes the task: approve the open purchase order for the requested supplier and notify the requester. Across many model attempts, none passes. The team runs the protocol.
The reference run fails at approval. The environment engineer finds that the order total exceeds the approval limit of the buyer role the agent is given, so the approve action returns a permission error from every path. Two human attempts with the same role fail at the same point. The verdict is broken, unless the author intended the agent to recognize the limit and route the order to a manager.
Asked, the process owner says that routing is exactly what a real buyer should do. The task is relabeled as impossible by design: success becomes a correct escalation to the manager queue with the reason, and approving it is now the failure. The rewritten task passes the reference, a do-nothing attempt fails, two humans complete it, and two experts agree. The exception is worth keeping: a task the protocol first marked broken became one of the environment's better tests of whether an agent respects its authority. OSWorld treats this category explicitly, keeping a list of infeasible tasks whose evaluator checks for infeasibility rather than completion.
Keep, rewrite, relabel or retire: deciding each task's fate
Every task ends the review with one action. Record it with the environment version, so a later change to tools or graders sends the task back through the protocol rather than silently changing its meaning.
At a glance
Keep, rewrite, relabel or retire: deciding each task's fate
- Keep: reference passes on all seeds, at least one of two humans passes, experts agree, negative checks fail.
- Rewrite: reference passes but humans fail or disagree because of something the text did not state; rerun from step 1.
- Relabel as impossible by design: the realistic correct behavior is a refusal or escalation; change the grader to score that behavior.
- Fix and re-review: the grader rejects expert-approved end states, or the reference fails because of the environment.
- Retire: the task cannot be made solvable without changing what it measures, or depends on a setting you cannot pin.
- Escalate for specialist review: the verdict depends on domain rules the review board cannot judge, such as regulated approvals or accounting treatment.
Workflow records that come with proof a task can be done
Tasks drawn from real workflows start with an advantage: the historical case shows that someone did complete it, under real permissions, and the outcome record shows what done looked like. That does not replace the reference run, since the environment may not reproduce every step, but it gives the review a concrete answer to check against.
An RL environment team can ask SourceX for workflow records that carry case histories, permissions context and outcomes; SourceX finds them at U.S. businesses on request rather than from a catalog, and handles rights review, contracting, preparation and delivery for the companies licensing them. Volume, format and licensing needs go in the data request.
Frequently asked questions
What should the human baseline and solvability review protocol include?
It should include a spec read, a reference run from the reset state on every shipped seed, negative checks, at least two human attempts under agent conditions, independent expert verdicts, an assumption sweep and a review board decision. Each step keeps evidence a second team can rerun. Every failing task leaves with a verdict and an action.
Does a 0% agent pass rate mean the task is too hard?
Usually not on its own. Anthropic's guidance says a 0% pass rate across many trials is most often a signal of a broken task. Run a reference solution and human attempts before calling it hard.
How many people should attempt each task?
At least two, working separately, because a single person can miss a solvable task on a given day. If one passes and one fails, look at where the second stalled before deciding the task is hard or underspecified.
What is the difference between an impossible task and an infeasible-by-design task?
An impossible task is a defect: nothing the agent does can satisfy the grader. An infeasible-by-design task is intentional: the right behavior is to refuse or report that it cannot be done, and the grader scores that behavior. The second kind must be labeled before agents are scored on it.
Which environment changes require repeating the review?
Any change to tools, permissions, step limits, seeds, the clock, dependencies or the grader can change solvability. Record the environment version with each verdict and rerun the reference after every such change.
Sources
- Anthropic's agent evaluation guidance says a 0% pass rate across many trials is most often a signal of a broken task, recommends a reference solution that passes all graders, and says two domain experts should independently reach the same pass or fail verdict. Source
- OSWorld keeps a list of infeasible tasks whose evaluator function is set to infeasible rather than a completion check. Source
- The SWE-bench project describes SWE-bench Verified as a subset of 500 problems that real software engineers confirmed are solvable. Source
- OpenAI's SWE-bench Verified effort had professional software developers screen samples for underspecified issue descriptions and unit tests that could reject valid solutions. Source
- METR's HCAST collected 563 human baselines from people skilled in the relevant domains working under conditions identical to those of AI agents. Source
- The WebArena paper reports that human annotators completed 78.24% of its tasks. Source
- The GAIA paper reports that human respondents scored 92% on its questions. Source
- Gymnasium's step API separates terminated, reaching a terminal state of the task, from truncated, a condition outside the task such as a time limit. Source
- The Agentic Benchmark Checklist paper defines task validity and outcome validity as the two conditions for a rigorous agentic evaluation. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.