Skip to content

AI data market

Computer use agent data economics: screens are cheap, checks are not

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Computer use agent data economics turn on four layers: screens, actions, policies and resulting states. Screens and actions come free once an environment runs; the authored work is the starting state and the success check. In WebArena, 411 of 812 tasks are scored by inspecting pages after the run, and OSWorld's 369 tasks rely on 121 distinct checker functions. Investors should ask who owns states and checks.

Agent run

Computer use agent data

This page covers computer use agent data economics: screens are cheap, checks are not.Illustrative

Key takeaways

  • Split computer-use data into screens, actions, policies and resulting states before comparing companies; the layers have very different costs.
  • WebArena scores 411 of its 812 tasks by inspecting pages after the run, and 302 of those reopen a named page to read what was stored.
  • OSWorld's 369 tasks use 121 distinct checker functions, 78 of them for a single task, so checking logic carries over poorly.
  • Only 62 of 1,181 tasks across both benchmarks reward refusal, leaving the policy layer of business work largely untested.
  • Business records can supply realistic starting states and grounded checks, but not screens or a running application.

Where does the work go in computer-use agent data?

Mostly into the starting state and the success check, not into the screens and clicks people picture. Once an environment runs, the application draws the screens and the agent produces the actions. What someone has to build by hand is the world the agent starts in and the test that decides whether it finished in the right place.

The capability these inputs serve is no longer hypothetical. OpenAI's Computer-Using Agent, introduced in January 2025, perceives the screen as pixels and acts through mouse and keyboard, and the company reported 38.1 percent success on OSWorld and 58.1 percent on WebArena. The benchmark papers had put the best model at 12.24 percent on OSWorld against more than 72.36 percent for humans (April 2024), and a GPT-4 agent at 14.41 percent on WebArena against 78.24 percent for humans (2023).

For investors, computer use agent data economics come down to who produces the expensive inputs. This piece counts what two widely used public benchmarks had to author for 1,181 tasks, sorts those inputs into four layers, and asks which layers business records can supply and which they cannot.

Screens, actions, policies, resulting states: the input map

Every computer-use task draws on four layers, and they differ sharply in who produces them. Screens are what the agent sees; actions are what it does; policies are what it may and may not do, including refusing a task that cannot or should not be done; resulting states are what is true in the application afterwards.

The map below separates the four layers by their source in a benchmark, what business records can contribute, and how scarce each one is:

Screens, actions, policies, resulting states: the input map
LayerWhat it coversProduced in a benchmark byWhat business records can addScarcity
ScreensRendered pages, windows, accessibility treesThe running application, at no extra cost per runLittle; screens come from software, not archivesLow once an environment exists
ActionsClicks, keystrokes, scrolls, tool callsThe agent under test; human demonstrations for trainingLittle, unless screen recordings were keptLow to moderate
PoliciesPermissions, approval limits, tasks to refuseA handful of tasks marked infeasible or unanswerableWritten procedures, approval logs, exception recordsHigh; barely covered in these benchmarks
Resulting statesWhat the application holds after the taskSeeded files, populated sites, gold outputs and checker codeThe system of record before and after real workHigh; most authored effort sits here

What 1,181 public benchmark tasks had to author

Two benchmarks, read file by file, show where the authoring effort went. WebArena's task file at commit dce0468 holds 812 tasks built from 190 intent templates, an average of 4.27 tasks per template, across six site keys on self-hosted web applications, and every task starts from a logged-in session. OSWorld's repository at commit 044bc33 holds 369 task files in ten folders, one of them for multi-application work, matching the 369 tasks its paper describes, each with a setup configuration and an execution-based check.

Success is decided by inspecting what the application holds far more often than by reading the agent's answer. In WebArena, 411 tasks (50.6 percent) run page checks after the agent stops, 335 (41.3 percent) score the answer text and 66 (8.1 percent) look only at the final URL. In OSWorld, 179 tasks (48.5 percent) compare the agent's output with a prepared gold file, 145 (39.3 percent) apply a written rule, 26 (7.0 percent) are infeasible by design and 19 (5.1 percent) use other checks such as a live page or a test suite.

The ledger below lists the authored inputs side by side; the methods note gives the definitions:

What 1,181 public benchmark tasks had to author
Authored inputWebArena, 812 tasksOSWorld, 369 tasksLayer
Task design190 intent templates, up to 10 tasks eachSources recorded per task: 58 by the authors, 39 from YouTube, 39 from Super UserPolicies and states
Starting stateSix site keys on self-hosted apps; all tasks start logged in260 tasks seed files into the virtual machine; 1,045 setup steps in allResulting states (before)
Check on stored state411 tasks with page checks; 302 reopen a named page such as an admin screen179 tasks against a gold file; 145 against a written ruleResulting states (after)
Checker codeThree evaluator types, combined per task121 distinct checker functions; 78 used by a single taskResulting states (after)
Answer-only scoring335 tasks scored on answer text; 66 on the final URL alone19 tasks scored another wayActions
Refusal cases36 tasks whose reference answer is N/A26 tasks marked infeasiblePolicies
Two stacked bars showing how public computer-use tasks decide success: WebArena's 812 tasks split into 411 page checks, 335 answer-text checks and 66 URL-only checks; OSWorld's 369 tasks split into 179 gold-file checks, 145 written rules, 26 infeasible tasks and 19 other checks.
How 1,181 public benchmark tasks decide success, counted from WebArena (commit dce0468) and OSWorld (commit 044bc33) task files. Each task is counted once.

Why screens and clicks come cheap and checks do not

Screens and actions are generated at run time, while every new task family needs new state and new checking logic. In OSWorld, 78 of the 121 checker functions serve exactly one task, so most checking code does not carry over to the next task. In WebArena, 302 of the 411 page-checked tasks send the evaluator to a named page, such as a product's admin edit screen or a user's profile, to read what was stored rather than what the agent last saw. Somebody had to write each of those 651 page probes.

State also wears out. WebArena's README tells users to reset the environment to its initial state after evaluating the 812 examples, and many tasks edit products, profiles or repositories, so one run changes the data the next would see. Each of OSWorld's 179 gold-file checks needed a correct output prepared in advance. Neither cost shrinks when a model gets better; both grow with every new application and workflow a buyer wants covered.

Demonstrations, the screens-and-actions layer, are the thinner artifact. WebArena's maintainers released human trajectories for about 170 of its 812 tasks. Mind2Web (arXiv 2306.06070) gathered crowdsourced demonstrations for 2,350 tasks on 137 websites across 31 domains, which is valuable for learning paths through interfaces. A recording shows the route a person took; it does not show whether the application ended in the right state unless someone captured that as well.

The policy layer that benchmarks barely touch

Benchmarks test whether an agent can do a task far more than whether it should. Across both sets, 62 of 1,181 tasks, about 5 percent, reward declining: 36 WebArena tasks whose expected answer is N/A and 26 OSWorld tasks built to be infeasible. No field in either task format records approval limits, data-access rules or the records an agent must leave untouched.

Business work runs the other way. A procurement clerk's day is shaped less by whether the purchase screen can be operated than by spending thresholds, vendor rules and who must sign. Those constraints live in written procedures, approval histories and exception logs, which only the company holds. If computer-use agents move into regulated or high-value workflows, the policy layer is where demand for company-held evidence should rise first, and it is the layer least represented in these two benchmarks.

Can a pile of business records stand in for an environment?

No. A record collection holds resulting states and some policies, but no screens, no running application and no state frozen at the moment a task began. That is the strongest objection to any records-for-agents thesis, and the benchmark files support it: WebArena needed populated, self-hosted copies of whole applications and a reset procedure; OSWorld needed seeded files and setup steps for 260 tasks before an agent could start.

At a glance

Can a pile of business records stand in for an environment

Records still enter the chain at two points that public benchmarks fill with invented content. They supply starting states that look like a real business, with realistic volumes, odd edge cases and history, and they supply success checks grounded in what the system of record showed after the work was actually done. The chain from records to a usable task has five steps, and a seller should say which it performs:

  • Records as held: exports from the system of record, with field histories where they exist.
  • Reconstructed starting state: the data as it stood before a given task, rebuilt and loaded into a copy of the application.
  • Task and policy: an instruction plus the rules that bound it, including cases the agent should refuse.
  • Check on resulting state: a test that reads what the application stores afterwards and compares it with what the record shows a competent person produced.
  • Scored runs: agent attempts graded by that check, which a buyer can reproduce under licensed terms.

Risks that would undercut the states-and-checks thesis

Five developments could weaken the claim that states and checks are where value concentrates, and each suggests a question to ask before backing a company in this space:

  • Capability outruns the benchmarks. OSWorld scores rose from 12.24 percent in the April 2024 paper to 38.1 percent in OpenAI's January 2025 report; if public tasks saturate, buyers may value fresh tasks over any one supplier's states.
  • Checks get automated. If model-written checkers become reliable on stored-state tasks, the long tail of hand-written checker functions, 121 in OSWorld alone, could shrink sharply in cost.
  • Benchmarks are not businesses. These counts describe two research benchmarks built on open-source applications; enterprise software, with permissions and integrations, may shift effort toward environment engineering instead.
  • Screens carry personal data. Recordings and rendered pages from real work capture customer and employee details, which narrows what can be licensed and raises preparation effort.
  • Applications belong to someone else. A business can license its records, but the software they live in is a vendor's, and rebuilding it for agents may depend on terms the record holder does not control.

Diligence questions for a computer-use data company

Ask where a company's product sits among the four layers, and for counts that show it. A seller of screen recordings competes on volume in the cheap layer; a seller of reconstructed states and grounded checks competes in the scarce one, and should be able to show how much of its checking logic carries across tasks.

SourceX touches one end of this chain. It licenses selected business records that companies supply to AI developers, covering the rights review, contracting and preparation, and buyers commission datasets on request rather than choosing from stock. It has not published computer-use metrics. SourceX is raising from accredited investors under Rule 506(c), and investors can ask for its materials through the investors page. The same questions apply to it as to any other company:

  • Which of the four layers does the product supply, and which does the buyer still have to build?
  • How many tasks share each checker, and what share of checks read stored state rather than the agent's answer?
  • Can a starting state be rebuilt as of a chosen date, and how long does one rebuild take?
  • How many tasks test refusal or a policy limit, and where do those rules come from?
  • What happens to tasks and checks when the target application ships a new version?
  • Which uses of screens and records do the supplying companies' agreements permit?

Methods note: counting authored inputs in WebArena and OSWorld

Files: WebArena config_files/test.raw.json at commit dce0468 (November 26, 2025) from github.com/web-arena-x/webarena, and OSWorld evaluation_examples/examples/<app>/<id>.json at commit 044bc33 (October 24, 2026) from github.com/xlang-ai/OSWorld, both read from clones on October 27, 2026. The counting script is filed with this article's review materials.

Definitions for WebArena: a template is a distinct intent_template_id; a page check is an evaluator list that includes program_html; answer-only scoring includes string_match without program_html; a probe is one program_html entry; a named-page probe has a URL other than the agent's last page; an unanswerable task has the fuzzy-match answer N/A. For OSWorld: a seeded task has at least one download step in its setup; tasks are split once each, infeasible first, then gold file (an expected value of type cloud_file or local_file), then rule (an expected value of type rule or rule_relativeTime, so the 145 rule tasks include rule_relativeTime checks), then other; a checker function is a distinct evaluator func value.

Not measured: hours or money spent, how the benchmarks were built in practice, versions other than these commits (OSWorld has since published a verified variant), and enterprise environments. Model scores are as the cited papers and OpenAI report them. The four-layer map and the five-step chain are editorial frameworks built on these counts.

Frequently asked questions

What is computer-use agent data?

Inputs for agents that operate software through its interface: screens, the actions taken on them, the rules that bound the task, and the state the application holds afterwards. Benchmarks such as WebArena and OSWorld package these as tasks with a starting state and an automated check.

Why are screen recordings not enough to train or test a computer-use agent?

A recording shows the path a person took through an interface, not whether the application ended in the right state. To grade an agent, someone must capture or rebuild the stored result and write a check that reads it, which is where most authoring effort in public benchmarks sits.

How do WebArena and OSWorld decide whether a task succeeded?

WebArena combines answer matching, URL matching and page checks; at commit dce0468, 411 of 812 tasks inspect pages after the run. OSWorld runs a checker per task, comparing output with a gold file in 179 of 369 tasks and applying a written rule in 145.

Can business records replace a computer-use environment?

No. Records hold resulting states and some policies, but no screens and no running application. They add value as realistic starting states and as grounded checks once someone rebuilds the application state they describe.

Does SourceX build computer-use environments?

SourceX licenses selected business records from companies to AI developers, sourced on request; its public materials do not describe it operating agent environments, and it has published no computer-use metrics. Investors can request company materials through sourcex.si/investors.

Sources

  • WebArena's task configuration at commit dce0468 lists 812 tasks from 190 intent templates; 411 use page checks after the run (302 of them reopening a named page), 335 score answer text, 66 score only the final URL, and 36 have N/A as the reference answer. Source
  • WebArena's README tells users to reset the environment to its initial state after evaluating the 812 examples and notes released human trajectories for about 170 tasks. Source
  • OSWorld's 369 task files at commit 044bc33 include setup steps that seed files for 260 tasks, gold-file checks for 179, written rules for 145, 26 infeasible tasks and 121 distinct checker functions, 78 used by a single task. Source
  • The WebArena paper curates 812 examples and reports a GPT-4 agent at 14.41 percent end-to-end success against 78.24 percent for humans. Source
  • The OSWorld paper (April 2024) builds 369 computer tasks, each with a setup configuration and an execution-based evaluation script, and reports humans above 72.36 percent and the best model at 12.24 percent. Source
  • OpenAI's Computer-Using Agent perceives the screen as pixels, acts through mouse and keyboard, and was reported at 38.1 percent on OSWorld and 58.1 percent on WebArena. Source
  • Mind2Web collects 2,350 tasks from 137 websites across 31 domains with crowdsourced demonstrations. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies