Skip to content

AI uses for records

RL environment task solvability: is the task hard or impossible?

By SourceX Editorial · Updated

10 sections

Short answer

An RL task is hard, not impossible, when a reference solution passes the grader from the reset state and a qualified person working under the agent's limits can finish it. If no reference passes, the task is broken or impossible; if people fail on unstated rules, it is underspecified. Label tasks meant to be infeasible in advance and grade them on a correct refusal.

Grader rubric

RL environment task

This page covers RL environment task solvability: is the task hard or impossible.Illustrative

Key takeaways

  • Prove solvability with a reference run from reset on every shipped seed, not with an author's belief.
  • A zero pass rate across many attempts usually points to a broken task before it points to a hard one.
  • Human baselines count only when people have the agent's tools, observations and limits.
  • Sweep step limits, permissions, seeds, clock and grader strictness; each can make a solvable task look impossible.
  • Give every failing task a verdict and an action, and repeat the review when the environment changes.

How do you tell a hard RL task from an impossible one?

A task is hard, not impossible, when a reference solution passes the grader from the environment's reset state and at least one qualified person working under the agent's conditions can complete it. If no reference passes, the task is broken or impossible. If the reference passes but people fail for reasons the task text never stated, it is underspecified. Tasks meant to be impossible must be labeled and graded on a correct refusal.

This matters because an RL environment rewards whatever its tasks reward. A task that cannot be completed gives zero reward on every rollout and teaches nothing, or worse, teaches the agent to stop early. A task that only looks impossible because a grader rejects valid answers pushes the policy toward the grader's quirks. Anthropic's agent evaluation guidance puts the warning plainly: with frontier models, a 0% pass rate across many trials is most often a signal of a broken task, not a hard one.

The protocol below separates those cases with evidence a buyer can rerun: a reference solution executed in the environment, human attempts recorded under the same limits the agent has, and an independent review of every task where those two disagree.

Four verdicts for a task nobody passes

Give every task that fails all model attempts one of four verdicts, each backed by a different kind of evidence. Treating them as one bucket of hard tasks is how environments ship with dead tasks inside.

Four verdicts for a task nobody passes
VerdictWhat it meansEvidence that establishes itWhat to do
HardSolvable within the rules, but beyond current agentsReference passes; at least one qualified human passes under agent conditionsKeep; it is the signal you want
Impossible by designThe correct behavior is to refuse or report infeasibilityTask labeled infeasible in advance; grader checks for a correct refusalKeep, labeled, and graded on the refusal
BrokenThe environment or grader prevents any valid completionNo reference passes from the reset state, or the grader rejects a verified correct end stateFix the environment or grader, then re-review
UnderspecifiedSuccess depends on something the task text does not sayReference passes; qualified humans fail or disagree on what was askedRewrite the task text, then re-review

What the environment must pin down before solvability can be judged

Solvability is a property of a task in a specific environment state, so fix the state, actions, reset and success criteria first; otherwise a task can be solvable on Monday and impossible after a dependency update. Write down the initial state the task starts from, the actions an agent may take, how reset restores that state, and the observable condition the grader checks.

Distinguish ending from stopping. The Gymnasium API separates terminated, meaning the agent reached a terminal state of the task, from truncated, a condition outside the task such as a time limit. A task that always ends in truncation is not proven impossible; it may only need more steps than the limit allows. Record both flags on every attempt.

Note the randomness too. If reset draws seeded values, solvability has to hold across the seeds you ship, not only the one the reference was built on. Tasks whose solvability depends on a lucky seed are a separate failure from tasks that are hard.

The human baseline and solvability review protocol

Run the steps in order for every task, and store the evidence column so a buyer or a second team can rerun the review. A task advances only when its step passes; otherwise it gets a verdict from the table above and leaves the protocol.

The human baseline and solvability review protocol
StepWhoWhat they doPasses whenEvidence kept
1. Spec readTask author and a second reviewerCheck the text states every condition the grader checksBoth agree nothing graded is missing from the textAnnotated task text
2. Reference runEnvironment engineerExecute a reference solution from reset, through the agent's action interfaceGrader passes on every shipped seedAction log, final state, grader output per seed
3. Negative checkEnvironment engineerRun a do-nothing attempt and a plausible wrong answerGrader fails bothGrader outputs for each
4. Human attemptTwo qualified people, separatelyAttempt the task with only the agent's tools, observations and limitsAt least one completes itRecording, time, steps, outcome, notes on where they stalled
5. Verdict agreementTwo domain experts, separatelyJudge each human end state pass or fail without seeing the graderExperts agree with each other and with the graderBoth verdicts and any disagreement note
6. Assumption sweepEnvironment engineerRerun the reference with step limit, seed and clock varied within shipped rangesReference still passesResults per setting
7. Review boardEngineer, process owner, evaluatorAssign hard, impossible by design, broken or underspecified to any task that failed a stepA verdict and an action are recordedDecision record with date and environment version

Running human baselines under the agent's own conditions

A human baseline only shows solvability if the person had what the agent has: the same tools, the same observations, the same step and time limits, and no outside knowledge of the hidden state. METR's HCAST benchmark followed this rule, collecting 563 baselines from people skilled in the relevant domains working under conditions identical to the agents', which is why its time estimates can be compared with agent results.

Human failure is not proof of impossibility. Even on WebArena, a benchmark of realistic web tasks, the paper reports human annotators completing 78.24% of tasks; GAIA reports human respondents at 92% on questions designed to be simple for people. Some tasks in any pool will be missed by a given person on a given day, so use at least two people per task and treat a single failure as a prompt for review, not a verdict.

Record more than pass or fail. Time taken, steps, the point where the person stalled, and what they said they were missing are what separate a hard task from an underspecified one. A person who solved it after guessing an unstated rule is evidence of underspecification, even though the attempt passed.

Settings outside the task text that flip a solvability verdict

Several settings outside the task text can flip a task between solvable and not; sweep them in step 6 and record the range over which the verdict holds. A task that is only solvable at one setting should say so in its metadata. These are also the settings most likely to drift between the environment a vendor tested and the one a buyer runs, so the sweep results belong in the delivery package alongside the tasks themselves.

Settings outside the task text that flip a solvability verdict
AssumptionHow it makes a task look impossibleCheck
Step or time limitValid solution needs more steps than the limit; every attempt truncatesCompare the reference's step count with the limit; record truncated versus terminated
Permissions of the agent's roleTask requires an action the role cannot takeConfirm the reference used only the agent's role and credentials
Seeded randomnessSome seeds generate states with no valid pathRun the reference on every shipped seed
Environment clockDeadlines, business hours or date rules block the action at reset timeFix the clock in the reset state and test boundary dates
Grader strictnessCorrect end states phrased or formatted differently are rejectedScore human end states that experts judged correct
Dependencies and versionsAn updated package or app changes the interface the task relies onPin versions and rerun the reference on each upgrade
External servicesNetwork calls fail or return different data in the sandboxStub or record external responses; fail closed if missing

When the reference, the humans and the grader disagree

Disagreement is the most useful output of the protocol, because each pattern points at a different fix. When the reference passes and qualified humans fail, suspect the task text. When humans reach an end state that experts judge correct and the grader fails it, suspect the grader. When nobody, reference included, gets through, suspect the environment.

Public benchmarks have hit each of these. OpenAI's SWE-bench Verified effort had professional software developers screen samples of the original SWE-bench for issue descriptions that were underspecified and for unit tests that could reject valid solutions, leaving a 500-task subset that the SWE-bench project describes as confirmed solvable. The Agentic Benchmark Checklist paper frames the same split as task validity, whether success requires the capability being measured, and outcome validity, whether the grader detects success.

Anthropic's guidance gives a simple bar for the expert step: a good task is one where two domain experts would independently reach the same pass or fail verdict. If they cannot, the task is not ready for training, however hard it looks.

Illustrative review: a purchase order task that no attempt could finish

This review is illustrative; it does not describe a real environment. A procurement environment includes the task: approve the open purchase order for the requested supplier and notify the requester. Across many model attempts, none passes. The team runs the protocol.

The reference run fails at approval. The environment engineer finds that the order total exceeds the approval limit of the buyer role the agent is given, so the approve action returns a permission error from every path. Two human attempts with the same role fail at the same point. The verdict is broken, unless the author intended the agent to recognize the limit and route the order to a manager.

Asked, the process owner says that routing is exactly what a real buyer should do. The task is relabeled as impossible by design: success becomes a correct escalation to the manager queue with the reason, and approving it is now the failure. The rewritten task passes the reference, a do-nothing attempt fails, two humans complete it, and two experts agree. The exception is worth keeping: a task the protocol first marked broken became one of the environment's better tests of whether an agent respects its authority. OSWorld treats this category explicitly, keeping a list of infeasible tasks whose evaluator checks for infeasibility rather than completion.

Keep, rewrite, relabel or retire: deciding each task's fate

Every task ends the review with one action. Record it with the environment version, so a later change to tools or graders sends the task back through the protocol rather than silently changing its meaning.

At a glance

Keep, rewrite, relabel or retire: deciding each task's fate

  • Keep: reference passes on all seeds, at least one of two humans passes, experts agree, negative checks fail.
  • Rewrite: reference passes but humans fail or disagree because of something the text did not state; rerun from step 1.
  • Relabel as impossible by design: the realistic correct behavior is a refusal or escalation; change the grader to score that behavior.
  • Fix and re-review: the grader rejects expert-approved end states, or the reference fails because of the environment.
  • Retire: the task cannot be made solvable without changing what it measures, or depends on a setting you cannot pin.
  • Escalate for specialist review: the verdict depends on domain rules the review board cannot judge, such as regulated approvals or accounting treatment.

Workflow records that come with proof a task can be done

Tasks drawn from real workflows start with an advantage: the historical case shows that someone did complete it, under real permissions, and the outcome record shows what done looked like. That does not replace the reference run, since the environment may not reproduce every step, but it gives the review a concrete answer to check against.

An RL environment team can ask SourceX for workflow records that carry case histories, permissions context and outcomes; SourceX finds them at U.S. businesses on request rather than from a catalog, and handles rights review, contracting, preparation and delivery for the companies licensing them. Volume, format and licensing needs go in the data request.

Frequently asked questions

What should the human baseline and solvability review protocol include?

It should include a spec read, a reference run from the reset state on every shipped seed, negative checks, at least two human attempts under agent conditions, independent expert verdicts, an assumption sweep and a review board decision. Each step keeps evidence a second team can rerun. Every failing task leaves with a verdict and an action.

Does a 0% agent pass rate mean the task is too hard?

Usually not on its own. Anthropic's guidance says a 0% pass rate across many trials is most often a signal of a broken task. Run a reference solution and human attempts before calling it hard.

How many people should attempt each task?

At least two, working separately, because a single person can miss a solvable task on a given day. If one passes and one fails, look at where the second stalled before deciding the task is hard or underspecified.

What is the difference between an impossible task and an infeasible-by-design task?

An impossible task is a defect: nothing the agent does can satisfy the grader. An infeasible-by-design task is intentional: the right behavior is to refuse or report that it cannot be done, and the grader scores that behavior. The second kind must be labeled before agents are scored on it.

Which environment changes require repeating the review?

Any change to tools, permissions, step limits, seeds, the clock, dependencies or the grader can change solvability. Record the environment version with each verdict and rerun the reference after every such change.

Sources

  • Anthropic's agent evaluation guidance says a 0% pass rate across many trials is most often a signal of a broken task, recommends a reference solution that passes all graders, and says two domain experts should independently reach the same pass or fail verdict. Source
  • OSWorld keeps a list of infeasible tasks whose evaluator function is set to infeasible rather than a completion check. Source
  • The SWE-bench project describes SWE-bench Verified as a subset of 500 problems that real software engineers confirmed are solvable. Source
  • OpenAI's SWE-bench Verified effort had professional software developers screen samples for underspecified issue descriptions and unit tests that could reject valid solutions. Source
  • METR's HCAST collected 563 human baselines from people skilled in the relevant domains working under conditions identical to those of AI agents. Source
  • The WebArena paper reports that human annotators completed 78.24% of its tasks. Source
  • The GAIA paper reports that human respondents scored 92% on its questions. Source
  • Gymnasium's step API separates terminated, reaching a terminal state of the task, from truncated, a condition outside the task such as a time limit. Source
  • The Agentic Benchmark Checklist paper defines task validity and outcome validity as the two conditions for a rigorous agentic evaluation. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies