AI data market
Multi-step agent data investment: what 340 public agent tasks reveal
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Multi-step agent data is a different asset from single-turn examples because its value sits in step dependencies, intermediate states and recovery paths, not in more prompt-response pairs. In 340 public benchmark tasks we examined, 91 percent of TheAgentCompany tasks grade two or more checkpoints and 63 percent of τ-bench retail instructions contain an if-clause. Back suppliers that can rebuild those structures from real records and verify outcomes.
Agent run
Multi step agent data
Key takeaways
- The scarce input in multi-step work is structure: which step depended on which, the state between steps and how people recovered, none of which a prompt-response pair records.
- Public tasks are structured more than long: τ-bench retail tasks need a median of 5 reference actions, while 159 of 175 TheAgentCompany tasks grade two or more checkpoints.
- Step count is a poor value proxy: 20 of 50 τ-bench airline tasks expect no database change, so the value is one judgment, not many steps.
- Evaluation buyers pay for verifiable end states; training buyers pay for paths at decision points, as PRM800K's 800,000 step labels show.
- A record collection without a rebuildable starting state and a checkable outcome is text data, whatever it is called.
What does a multi-step agent data investment actually bet on?
It bets that the record of a task done in several dependent steps is a different asset from a pile of single-turn examples, with different suppliers, preparation work and buyers. If that is true, the scarce input is the structure of the work: which step depended on which, what the system looked like between steps, and how people recovered when a step failed.
Anthropic's January 2026 guide to agent evaluation explains why that structure matters. Single-turn evals are a prompt, a response and grading logic; agents instead work over many turns, calling tools, modifying state and adapting to intermediate results, so mistakes can propagate and compound. The guide separates the transcript, the complete record of a trial, from the outcome, the final state of the environment: an agent can say a flight is booked while no reservation exists in the database.
Four kinds of evidence would support the thesis: buyers grade or train on intermediate structure; real records can supply it; single-turn data cannot stand in for it; and simulated or synthetic episodes cannot match it at equal utility. The sections below test each one, starting with public task files anyone can inspect.
The question is sharpening because agent tasks are getting longer. METR reported in March 2025 that the length of tasks frontier agents complete with 50 percent reliability, measured in human working time, had doubled roughly every seven months over six years. That trend comes from METR's own suite of software, cybersecurity, reasoning and machine learning tasks, so it shows direction, not the shape of any business workflow.
Single-turn examples and multi-step episodes, side by side
A single-turn example holds one input and one judged output; a multi-step episode holds a chain of states, and most of its value sits between the first and the last. OpenAI's GDPval is a useful single-turn reference point: OpenAI says it spans 1,320 tasks across 44 occupations but is limited to one-shot evaluations, so it does not capture cases where a model must build context or improve through multiple drafts.
The comparison below does not argue that single-turn data is worthless. GDPval-style deliverables test expert judgment directly, and many agent steps are single decisions in disguise. It shows that the multi-step column needs inputs a prompt and response never record.
| Property | Single-turn example | Multi-step episode | Public evidence | What a buyer pays for |
|---|---|---|---|---|
| Dependencies | None: the answer depends on the prompt and files | Later actions use values found or created by earlier ones | τ-bench retail reference paths run a median of 5 actions, mostly lookups before a change | Correct ordering on real data |
| Intermediate states | One input state, one output | A system state after each action, some graded as checkpoints | 159 of 175 TheAgentCompany tasks grade two or more checkpoints | Partial credit and step-level diagnosis |
| Recovery paths | Not represented | Branches after a refusal, an error or a changed condition | 73 of 115 τ-bench retail instructions contain an if-clause | Behavior when the first plan fails |
| Outcome check | Compare the output with a reference answer | Compare the final environment state with an expected state | τ-bench replays reference actions and compares database hashes | A verifier the agent cannot talk past |
| Reliability | Accuracy on one attempt | Consistency across repeated trials, measured as pass^k | Anthropic: 75 percent per-trial success gives about 42 percent across three trials | Confidence the agent behaves the same way every time |
What 340 public agent tasks reveal about the shape of multi-step work
Public task files show multi-step work defined less by length than by branching and graded intermediate states. On 2026-10-26 we read every task in two permissively licensed benchmarks, τ-bench (tau-bench, from Sierra) and TheAgentCompany, both released under the MIT license, and counted the three structures this article is about.
Sampling frame: the τ-bench retail test split (115 tasks) and airline test split (50 tasks) at repository commit 59a200c of March 18, 2026, and all 175 task folders in TheAgentCompany at commit 98b68ef of November 17, 2025. Excluded: τ-bench's train and dev splits, which repeat the test domains, and benchmarks whose task files are gated or could not be downloaded here. A database-changing action is any reference action other than a lookup, a calculation, a think step or a transfer to a human; an if-clause is any instruction that contains the word if.
The example that challenges the headline sits in the airline split. Two in five tasks expect no change to the booking database at all. In one, a simulated customer is told to keep pushing to cancel a basic economy reservation even after the agent says it cannot be cancelled; the reference action list is empty, so the only passing outcome is an unchanged database. Step counts say nothing about the value of a task like that. Its value is one judgment at one decision point, made under pressure.
So the investable difference is not length. τ-bench retail tasks are short, but two in five need two or more database changes and most can branch, while TheAgentCompany grades work at intermediate states. Those are the properties to price, and the ones a supplier should be able to count in its own data.
| Benchmark and split | Tasks | What we counted | Result |
|---|---|---|---|
| τ-bench retail, test | 115 | Reference actions per task, tasks needing two or more database changes, instructions with an if-clause | Median 5 actions (range 0 to 14); 45 tasks (39 percent) need two or more changes; 73 (63 percent) contain an if-clause |
| τ-bench airline, test | 50 | Reference actions per task and tasks needing no database change | Median 2 actions; 20 tasks (40 percent) need no database change; 15 need two or more |
| TheAgentCompany, all tasks | 175 | Graded checkpoints per task | Median 3 checkpoints; 159 tasks (91 percent) have two or more; 106 (61 percent) have three or more; 16 have one |
| All three | 340 | What the grader checks | τ-bench checks the final database state; TheAgentCompany checks results first and checkpoints second |
Why intermediate states are worth paying for
Intermediate states are worth paying for because outcome-only scores hide how close an agent came, and buyers now pay to see it. TheAgentCompany's latest paper version reports that the best model it tested, Gemini 2.5 Pro, completed 30.3 percent of tasks fully but scored 39.3 percent once partial credit for checkpoints was counted. That nine-point gap is information an end-state check alone throws away.
Evaluation designers ask for it explicitly. Anthropic's guide recommends partial credit for tasks with several components: a support agent that identifies the problem and verifies the customer but fails to process the refund is meaningfully better than one that fails at once. Training teams have paid for the same structure at scale. OpenAI's PRM800K release contains 800,000 step-level correctness labels on model-generated math solutions, a process supervision dataset that grades individual steps rather than only the final answer.
Reliability is the second reason. Anthropic's guide shows how consistency falls across repeated trials: an agent that succeeds 75 percent of the time per trial passes all of three trials only about 42 percent of the time. The τ-bench paper found the pattern in practice, with gpt-4o's retail success falling from about 61 percent on one trial to about 25 percent when all eight trials had to succeed. Episodes that record intermediate states let a buyer find where those failures begin; episodes that keep only the final state do not.
Recovery paths: the branch most record collections discard
Recovery paths are the steps taken after something goes wrong or a condition changes, and they are the part of a business record most often lost in export or cleanup. τ-bench writes them into its tasks: its first two retail test tasks share the same customer and the same order and differ only in the fallback the customer wants if no matching keyboard exists, which changes the correct final state.
Model developers treat bad paths as a quality problem of their own. NVIDIA's Nemotron documentation says the Ultra pipeline rejects agentic rollouts that show edit-test thrashing, lost-in-exploration behavior or edits that were never tested, and that the Super pipeline screens synthetic tool-calling data with both a turn-level and a trajectory-level judge. The same page describes PivotRL, which reuses expert trajectories at informative pivot turns during reinforcement learning.
In company systems, recovery appears as a reopened ticket, a reversed entry, a rerouted approval or a rescheduled visit. The investor question is whether a supplier keeps the failed branch and the fix together, in order and with timestamps, or keeps only the clean final record. Only the first supports a premium for recovery paths.
The agent-task input map: eight inputs and where each comes from
A multi-step episode is investable when it carries eight inputs, and a supplier's edge sits in the ones it can source from real records instead of inventing them. Score any dataset or supplier by marking each row present, reconstructable or missing.
The figure turns the map into a value chain. Raw records become episodes when someone reconstructs their order and starting state; episodes become training and evaluation assets when checkpoints, branches and a verifier are added. Each link can be a different business: the record owner, the preparer that reconstructs and labels, the environment builder that makes episodes replayable, and the model developer that trains or evaluates. A supplier that claims the whole chain should show evidence at every link.
Methods note for the map: the eight inputs are drawn from the evaluation definitions in Anthropic's guide, the grading code of τ-bench and the checkpoint design of TheAgentCompany, read on 2026-10-26. It is an editorial framework and has not been validated against buyer acceptance results.
| Input | Single-turn data has it? | Where it lives in company records | Diligence test |
|---|---|---|---|
| Initial state | Partly: the prompt and files | System snapshot or audit log at task start | Can the supplier rebuild what the system showed before step one? |
| Goal and constraints | Yes, in the prompt | Request text plus the policy version in force | Is the policy dated to the episode rather than today's version? |
| Action space | No | Tool, API or screen actions the worker used | Are actions recorded as calls with arguments, not prose summaries? |
| Step dependencies | No | Links between records, such as ticket to order to refund | Can each action's inputs be traced to an earlier step? |
| Intermediate checkpoints | No | Status changes, approvals and saved drafts | Which states are graded, and who defined them? |
| Branches and recoveries | No | Reopens, reversals, escalations and retries | Is the failed branch kept beside the fix? |
| Verified outcome | Partly: a reference answer | Final system state plus a later confirmation | Can a program check the end state without a person reading prose? |
| Repeatability | No | Many similar cases under one policy | Are there enough near-identical cases to measure consistency? |
Is a record collection the wrong asset for agents?
Often it is. A collection of tickets, orders or messages can lack the two things an agent buyer needs most: a starting state that can be rebuilt and an outcome a program can verify. Without both, the collection is reference material for retrieval or text fine-tuning, not agent data, and it should be priced that way.
The benchmarks show what the missing evidence looks like. τ-bench can grade an episode because it holds the full database and replays reference actions to compute the expected end state. TheAgentCompany ships a self-hosted company with its own websites and data. A record export has neither by default, and rebuilding them is preparation labor, not something the company already owns.
Anthropic's guide adds a second caution. It recommends grading what the agent produced rather than the path it took, because agents find valid approaches designers did not anticipate. For evaluation buyers that lowers the value of exact paths and raises the value of verifiable outcomes; paths matter most to training buyers, as PRM800K and PivotRL show. Apply a three-way rule:
- Starting state and outcome both reconstructable: agent data. Price it on episode structure.
- Outcome verifiable but starting state lost: evaluation material only after someone builds an environment. Price the build separately.
- Neither: text data. Price it as documents, whatever the pitch calls it.
What could make the multi-step data thesis wrong
Four developments would shrink the premium for real multi-step records. Each has a public marker an investor can track, and a supplier that cannot say which one worries it most has not tested its own thesis.
Checklist · 4 items
- Synthetic episodes could close the gap. NVIDIA's Nemotron documentation describes synthetic tool-calling data screened by turn-level and trajectory-level judges. If such data passes buyers' acceptance tests at lower cost, real episodes keep their value mainly as evaluation holdouts.
- Simulation could replace records. τ-bench already uses one model to play the customer while the agent works, a design Anthropic's guide describes; environment builders that scale this approach compete directly with record suppliers.
- Harness lock-in could limit reuse. Nemotron's documentation says NVIDIA varies harnesses and tool formats over shared tasks to keep agentic data from being tied to one system, so episodes recorded in one tool format may need costly normalization before a second buyer can use them.
- Scores could mislead the market. Anthropic reports that Opus 4.5 first scored 42 percent on CORE-Bench and 95 percent after grading bugs and scaffold limits were fixed. Demand built on fragile benchmarks can reverse when the benchmarks are corrected.
0 of 4 ticked. Tick each item as you go.
Questions to ask a multi-step data supplier, SourceX included
Ask for evidence at the link of the value chain where a supplier claims its edge, using definitions you set, and apply the same questions to SourceX.
SourceX is one application of this thesis. For companies licensing operational records to AI developers, it covers the whole run from finding and vetting a supplier to rights review, contracting, data preparation, delivery and payment, so that selected records end up as licensed datasets. Delivery waits until the supplying company has signed the agreement and approved its scope and terms, so the rights question is settled before any episode is reconstructed.
SourceX has not published metrics on episode coverage, checkpoint density or buyer acceptance; it shares company and commercial metrics with investors directly. A Rule 506(c) raise from accredited investors is under way. Investors can request the investor overview and bring a specific diligence question, such as how a candidate dataset scores on the input map above.
Checklist · 6 items
- What share of your episodes have a starting state that can be rebuilt, and how is that checked?
- Which intermediate states do you record, and who defined the checkpoints?
- What share of episodes contain a recovery branch, and is the failed branch kept?
- Can a program verify each episode's outcome, or does a person have to read it?
- How many near-identical episodes exist per policy version, so a buyer can measure consistency?
- Which synthetic or simulated alternative has a buyer tested against your data, and with what result?
0 of 6 ticked. Tick each item as you go.
Methods note: sources, counts and what was not measured
Counts in this article come from our reading of public task files on 2026-10-26: the τ-bench test splits at commit 59a200c and TheAgentCompany's task folders at commit 98b68ef, parsed with a script that reads reference actions and checkpoint headings without running benchmark code. Percentages are of tasks in each split. Other figures come from Anthropic's evaluation guide (January 2026), OpenAI's GDPval page, the τ-bench and TheAgentCompany papers, OpenAI's PRM800K repository, NVIDIA's Nemotron documentation and METR (March 2025).
Not measured: how often real business records contain each input on the map, what buyers pay for episodes compared with single-turn examples, and whether real episodes beat synthetic ones on any acceptance test. Each needs company or buyer data that has not been published. The if-clause count is a text proxy for branching and overstates branches where a clause does not change the correct end state.
Frequently asked questions
What is multi-step agent data?
It is the record of a task completed through several dependent actions, including the starting state, each action with its inputs, the intermediate states and the final outcome. Its value comes from how the steps connect and how errors were handled, which single-turn examples do not capture.
How is multi-step data different from single-turn training data?
A single-turn example pairs one input with one judged output. A multi-step episode adds dependencies between actions, intermediate states that can be graded as checkpoints, recovery branches and an end state that a program can verify. OpenAI notes that its one-shot GDPval benchmark does not capture work where a model must build context over multiple drafts.
Why do agent benchmarks grade the final state instead of every step?
Agents often find valid paths that task designers did not anticipate, so Anthropic's evaluation guide recommends grading what the agent produced rather than the exact path. Benchmarks such as τ-bench compare the final database with an expected one, and TheAgentCompany adds checkpoints for partial credit.
Can synthetic environments replace real multi-step business records?
Partly, and that is the main risk to the thesis. Developers such as NVIDIA already screen synthetic tool-calling data with turn-level and trajectory-level judges, and benchmarks simulate users with a language model. Real records keep an edge where they hold policies, branches and outcomes that simulations do not reproduce, which a buyer should test head to head.
Does SourceX publish agent data metrics?
No. SourceX has not published episode, checkpoint or acceptance metrics and shares company and commercial metrics with investors directly. Materials are available on request at sourcex.si/investors for investors who want to test episode coverage; any offering is restricted to verified accredited investors.
Sources
- Anthropic's guide (January 2026) defines single-turn evals as a prompt, a response and grading logic, explains that agents modify state over many turns so mistakes propagate and compound, separates transcript from outcome, recommends partial credit and grading outcomes rather than paths, gives the 75 percent to about 42 percent pass^k example, describes τ-Bench's user-simulating model and reports Opus 4.5 moving from 42 to 95 percent on CORE-Bench after grading fixes. Source
- OpenAI says GDPval spans 1,320 tasks across 44 occupations and is limited to one-shot evaluations, so it does not capture cases where a model must build context or improve through multiple drafts. Source
- τ-bench (MIT license) retail and airline test task files list the reference actions and user instructions counted in this article (commit 59a200c, March 18, 2026). Source
- τ-bench's reward code replays a task's reference actions to compute the expected database state and gives zero reward when the agent's final database hash differs. Source
- The τ-bench paper reports gpt-4o function-calling success of about 61 percent on retail with pass^1, falling to about 25 percent for pass^8. Source
- TheAgentCompany (MIT license) contains 175 task folders whose checkpoint files define graded checkpoints; its README describes result-based evaluation as primary and subcheckpoints as secondary (commit 98b68ef, November 17, 2025). Source
- TheAgentCompany paper (latest version) describes a self-hosted simulated software company with internal websites and data, and reports the best model, Gemini 2.5 Pro, completed 30.3 percent of tasks and scored 39.3 percent with partial credit. Source
- OpenAI's PRM800K is a process supervision dataset of 800,000 step-level correctness labels for model-generated solutions to MATH problems. Source
- NVIDIA's Nemotron FAQ describes turn-level and trajectory-level judges for synthetic tool-calling data, rejection of agentic rollouts with edit-test thrashing or lost-in-exploration behavior, PivotRL's reuse of expert trajectories at pivot turns, and varying harnesses so agentic data is not tied to one system. Source
- METR (March 2025) found the length of tasks frontier agents complete with 50 percent reliability, in human working time, doubled roughly every seven months over six years, on a suite of software, cybersecurity, reasoning and ML tasks. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.