Deal economics
Long-horizon agent dataset economics: the cost moves to verification
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
In long-horizon agent dataset economics, the unit that matters is the verified episode, not the record. As tasks lengthen, assembling records stays cheap while checking outcomes grows with the hours a skilled person would need. HCAST's builders logged over 1,500 hours of human baselines for 189 tasks (March 2025). Records that already carry the outcome, such as reopen counts or payment status, cut that verification bill most where tasks run longest.
Agent run
Long horizon agent dataset
Key takeaways
- The tradable unit for long-horizon agents is a verified episode: starting state, actions, constraints and an outcome someone has checked.
- Verification, not collection, dominates episode cost once tasks run for hours; HCAST's builders spent over 1,500 hours on 563 human baselines (March 2025).
- Records that already contain the outcome act as a cheap verifier, and in the illustrative model the hours they save grow in step with task length.
- The thesis fails where automated checks are cheap, as in code with tests, because synthetic pipelines such as SWE-smith generate tasks at scale.
- Ask suppliers for verified-episode yield and reviewer hours per episode by horizon band, not for record counts.
What do long-horizon agents change about dataset economics?
They change the unit of value from the record to the verified episode, and they move most of the cost from collecting data to checking it. An episode is everything an agent needs to attempt one task and be judged on it: the starting state, the actions available, the constraints in force and an outcome someone has confirmed. For a five-minute task the outcome is often obvious. For a task that takes a skilled person a week, confirming the outcome can cost nearly as much as doing the work.
The pressure comes from capability. METR researchers defined an agent's 50% time horizon as the length of task, measured in skilled-human time, that a model completes half the time, and reported that it doubled roughly every seven months from 2019 to 2025 (arXiv 2503.14499, March 2025); a later revision says the trend may have sped up since 2024. Their tasks are software tasks, and the authors discuss limits on how far the results generalize.
If horizons keep lengthening, buyers will need episodes at lengths where verification is the expensive input. That is the thesis this page tests: in long-horizon agent dataset economics, value accrues to whoever can verify outcomes cheaply, and business records that already contain outcomes are one of the few cheap verifiers available outside code.
The horizon input map: what an episode needs at each task length
Each step up in task length adds state that must be captured and narrows the kinds of evidence that can confirm the result. The four bands in the map are editorial, drawn from the benchmark designs cited on this page, and meant to be argued with rather than taken as measurement.
| Horizon band (skilled-human time) | State the episode must capture | Evidence that can confirm the outcome | Typical verifier |
|---|---|---|---|
| Minutes | One record or screen and the request | The record's own final field | An automated check or a quick human look |
| Hours | Several systems' starting state, plus tool access | A final state in each system touched | A per-task execution script, as in OSWorld, or a reviewer |
| Days | State at each handoff and pause, and constraints that persist | Downstream records such as payment, reopen, shipment or sign-off | A reviewer with domain knowledge, or outcome records |
| Weeks | Changing state across people, approvals and revised sources | Outcomes observed weeks later, often partial | Several experts, or outcome records that have matured |
What public benchmark builders spent to verify long tasks
Mostly human hours, and the bill rose with task length. HCAST's authors at METR assembled 189 tasks and collected 563 human baselines totaling over 1,500 hours from 140 skilled people, an average above 2.6 hours per attempt, to calibrate tasks that take humans from about a minute to more than eight hours (arXiv 2503.17354, March 2025). Even with that effort, the paper reports 79 tasks without a successful human baseline.
RE-Bench, also from METR, rests on 71 eight-hour attempts by 61 human experts across seven research-engineering environments (arXiv 2411.15114, November 2024). Its results show why references for long tasks stay expensive: with a two-hour budget the best agents scored four times higher than the experts, but humans gained more from extra time and reached twice the top agent's score when both had 32 total hours.
Where tasks come from real paid work, the checking is still built by hand. SWE-Lancer's more than 1,400 real freelance tasks from Upwork are graded with end-to-end tests that experienced software engineers triple-verified (arXiv 2502.12115, February 2025). OSWorld gives each of its 369 computer tasks its own initial-state setup and custom execution-based evaluation script (arXiv 2404.07972, April 2024). Per-task setup and checking, not raw data, is the recurring construction cost.
An episode cost model you can rerun with your own inputs
In this illustrative model, cost per verified episode is assembly hours plus validation hours, and validation dominates once a task runs past a couple of hours. Assembly is 0.3 reviewer-hours per system the task touches. Validation is the task's skilled-human time multiplied by a review fraction and by the number of reviewers. With two reviewers, the review fraction is 0.5 when nothing in the records shows the outcome, because the reviewer must reconstruct and judge the end state, and 0.1 when the records already show it, because the reviewer only confirms it.
Every input is an assumption chosen to show the shape, not a measurement, and none describes SourceX's costs. Two results survive reasonable changes to the inputs. Validation's share of episode cost rises with task length whenever the review fraction stays fixed: here it is 45% of the no-record episode at 15 minutes and 96% at one week. And the hours saved by outcome-bearing records equal task length times reviewers times the gap between review fractions, 0.8 hours per hour of task, so the saving grows in a straight line, from 0.2 hours per episode at 15 minutes to 32 hours at one week.
Reviewer counts vary in practice; SWE-Lancer used three. Substitute your own fractions, reviewer counts and systems per band, and the table recomputes the same way.
| Horizon band (systems touched) | Assembly hours | Validation hours, no outcome record | Validation hours, outcome record | Episode total, no record vs record |
|---|---|---|---|---|
| 15 minutes (1) | 0.3 | 0.25 | 0.05 | 0.55 vs 0.35 (1.6x) |
| 2 hours (2) | 0.6 | 2.0 | 0.4 | 2.6 vs 1.0 (2.6x) |
| 1 day, 8 hours (4) | 1.2 | 8.0 | 1.6 | 9.2 vs 2.8 (3.3x) |
| 1 week, 40 hours (6) | 1.8 | 40.0 | 8.0 | 41.8 vs 9.8 (4.3x) |
Which links in the value chain gain as horizons lengthen?
The links that lower verification cost gain share, and the link that only assembles records loses it. That follows from the model only if buyers pay roughly in proportion to the cost of a usable episode, which is an assumption to test, not a finding.
Read the table as a map of bargaining power. A model developer can build environments and hire experts in-house; it cannot recreate years of real outcomes it never observed. Operating systems already store some of those outcomes: Zendesk's Ticket Metrics API, for example, reports the number of times a ticket was reopened and its full resolution time. Outcome-bearing records are the input hardest to substitute, which is why the record holder's position strengthens with horizon while the assembler's weakens.
| Link | What it contributes | How its weight changes as horizons grow | Evidence to ask for |
|---|---|---|---|
| Record holder (the operating business) | Real decisions, the state around them and the outcomes that followed | Rises where matured outcomes replace review hours | Share of episodes with an outcome field or downstream record |
| Episode assembler | Joins records across systems into ordered episodes | Grows slowly with systems touched; the easiest link to automate | Assembly hours per episode, by band |
| Environment builder | A restorable starting state and tools for the agent to act with | Rises with the number of systems and versions involved | Share of episodes that replay to the recorded end state |
| Outcome verifier | Experts, scripts or outcome records that confirm success | Rises with task length unless automated checks exist | Reviewer hours per verified episode and reviewer agreement |
| Model developer (the buyer) | Training and evaluation use, and the acceptance test | Pays for verified episodes; can bring any link in-house | Acceptance criteria stated per horizon band |
The strongest objection: records lack the environment and the outcome an agent needs
Often they do, and where they do the thesis fails. A support history shows what happened, but an agent needs a world to act in: tools that respond, state that changes, room to make a different choice. A CRM cannot be restored to last March just because its records describe last March, and records whose outcomes have not matured save no review hours at all.
Synthetic pipelines attack exactly this gap wherever automated checks exist. SWE-smith builds an execution environment for a Python codebase and then generates task instances that break existing tests, reaching 50,000 instances from 128 GitHub repositories, about ten times the size of earlier efforts (arXiv 2504.21798, April 2025). In software, tests are a cheap verifier, the review fraction falls toward zero and synthetic generation wins on cost.
So the claim narrows to domains with three properties: no cheap automated check, outcomes that real operations already record, and enough history to build episodes at long horizons. Claims handling, procurement, collections, field service and support escalation are candidates to test, and each still needs proof that its outcome fields mean what they appear to mean.
An investor can run the test directly. For a supplier's episodes in one domain, compare reviewer hours per verified episode with and without the outcome records at two or more horizon bands. If the gap does not widen as tasks lengthen, the thesis does not hold for that domain.
What could break the long-horizon data thesis
Six developments would weaken it, and each can be watched:
- Automated verifiers spread beyond code: if rule engines, simulators or model graders confirm business outcomes reliably, review fractions fall and outcome records lose their premium.
- Synthetic environments improve: pipelines like SWE-smith extend to business software and generate long tasks with checks built in.
- Horizon gains stall or fail to transfer: METR's measurements cover software tasks and its authors discuss the limits of generalizing them, so messier work may not follow the same curve.
- Rights and privacy narrow supply: outcome-bearing records often involve customers and employees, and every episode needs both authority to license and review of personal data.
- A few buyers set the price: if a handful of developers buy most long-horizon data, they can bring verification in-house and pay per record rather than per episode.
- Verification itself fails: HCAST lacked a successful human baseline for 79 of its 189 tasks, so a long-task episode can be expensive and still unverified.
Questions to put to a long-horizon data supplier, SourceX included
SourceX sits at the supply end of this chain. It finds and qualifies companies with operational records, reviews their rights, contracts with them, prepares the data, delivers it and pays the companies that license it to AI developers. Its commercial and operating figures are not published; SourceX shares them with investors directly, and materials can be requested through its investor page.
Whoever the supplier is, ask for measurements by horizon band with their denominators, not for record counts:
- What share of delivered episodes carry a matured outcome, by horizon band and domain?
- How many reviewer hours does one verified episode take, with and without outcome records?
- What share of episodes replay from the recorded starting state to the recorded end state?
- How is agreement between reviewers measured, and what happens to episodes they disagree on?
- In which domains could automated checks let synthetic tasks compete, and how is the supplier positioned there?
- Who holds the authority to license each source, and how is personal data handled before review?
How the episode cost model was built, and what it leaves out
Sources: the benchmark papers cited in each section, verified on arxiv.org on October 27, 2026 through search excerpts of their abstracts and summaries, plus Zendesk's developer documentation for outcome fields. Figures are reported as the papers state them; the 2.6-hour average per HCAST baseline is our arithmetic on the paper's totals.
Model: the assembly rate, review fractions, reviewer count and systems per band are illustrative assumptions, not measurements from any supplier, buyer or SourceX. The model leaves out compute for repeated agent trials, environment upkeep across software versions, legal and privacy review, and the wait before outcomes mature, any of which could reorder the links in the value chain. It estimates no prices, margins or revenues.
Not measured: how often real business records contain outcomes that survive validation, and whether buyers in fact pay per verified episode. Both are evidence a diligence process should request rather than assume.
Frequently asked questions
What is a verified episode in agent training data?
It is one task packaged so an agent can attempt it and be judged: the starting state, the available actions, the constraints in force and an outcome someone has confirmed. A collection of records becomes episodes only after this assembly and verification, and for long tasks verification is the costly part.
Why does verification cost grow with task length?
When no automated check exists, a reviewer has to reconstruct and judge the end state, and that effort scales with how long the task takes a skilled person. Benchmark builders show the scale: HCAST's authors logged over 1,500 hours of human baselines for 189 tasks.
Do synthetic environments make real business records unnecessary for agents?
Not where outcomes cannot be checked cheaply. Synthetic pipelines such as SWE-smith work well in code, where tests verify results. In domains without automated checks, records that already contain outcomes remain one of the cheaper ways to verify long tasks.
Which metrics should an investor request from an agent data supplier?
Verified-episode yield by horizon band, reviewer hours per verified episode with and without outcome records, replay success from recorded start to recorded end state, and agreement between reviewers. Record counts alone say little about long-horizon value.
Does SourceX publish episode or verification metrics?
No. SourceX does not publish its commercial or operating figures; it shares company and commercial metrics with investors directly. Investor materials can be requested at sourcex.si/investors.
Sources
- METR (Kwa et al., March 2025) defines the 50% task-completion time horizon and reports it doubled roughly every seven months from 2019 to 2025 on software tasks, with a later revision noting possible acceleration since 2024. Source
- HCAST (METR, March 2025) comprises 189 tasks calibrated with 563 human baselines totaling over 1,500 hours from 140 skilled people; 79 tasks lack a successful human baseline. Source
- RE-Bench (METR, November 2024) uses seven ML research-engineering environments and 71 eight-hour attempts by 61 experts; agents lead at 2 hours, humans reach twice the top agent's score at 32 hours. Source
- SWE-Lancer (February 2025) contains over 1,400 real freelance software tasks from Upwork, graded with end-to-end tests triple-verified by experienced software engineers. Source
- OSWorld (April 2024) has 369 computer tasks, each with a detailed initial state setup configuration and a custom execution-based evaluation script. Source
- SWE-smith (April 2025) builds execution environments and synthesizes 50,000 task instances from 128 GitHub repositories, about ten times earlier efforts. Source
- Zendesk's Ticket Metrics API reports outcome-related fields including the number of times a ticket was reopened and its full resolution time. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.