Skip to content

AI data market

Economically weighted agent datasets: what 463 priced tasks reveal

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Economically weighted agent datasets count each task by what the work is worth, not how often it appears, and investors should accept one only with its weights and a sensitivity table attached. In SWE-Lancer's 463 public priced tasks, the top tenth holds 49.1% of listed value, price weighting leaves about 69 effective tasks, and an illustrative ranking flips under one of four weighting schemes.

Value drivers

Economically weighted agent

This page covers economically weighted agent datasets: what 463 priced tasks reveal.Illustrative

Key takeaways

  • Economic weighting changes which tasks a dataset is about: in SWE-Lancer's public list, the cheapest half of tasks holds only 17.6% of listed value.
  • Weighting buys relevance at the cost of precision; price weights shrink 463 tasks to about 69 effective ones.
  • A conclusion that holds under only one weighting scheme is a property of the scheme, so ask for results under equal, raw, capped and log weights.
  • Every weight needs a task beneath it with a state, allowed actions, constraints and a verifiable outcome, or the weight multiplies noise.
  • Public weight sources each carry limits: O*NET ratings have precision flags, OEWS leaves out the self-employed and listed prices show what a buyer paid, not the value created.

What does economic weighting change about an agent dataset?

It changes what a score is about. A count-weighted dataset treats a two-minute password reset and a two-day contract reconciliation as equals; an economically weighted one scales each task by an estimate of what the work is worth, so the score tracks the value an agent could take on rather than the number of chores it clears.

For an investor, the decision is whether a company's claim about agent data or agent performance rests on weights that are disclosed, sourced and stable. Useful evidence comes in four parts: the weight on each task and where it came from, the task's verifiable outcome, the effective number of tasks after weighting, and how the headline result moves under other reasonable weights.

The thesis worth testing is this: if buyers of agent data move from counting finished tasks to paying for outcomes that matter to a business, datasets whose tasks carry audited weights and verifiable outcomes should be worth more than larger collections counted task by task. It fails if weights stay unverifiable, if precision collapses under weighting, or if buyers keep paying per task regardless of value.

Candidate weights come in four families: how often a task occurs, how long it takes a qualified person, what the labor costs or the market pays, and what an error costs. Each can be estimated from enterprise records or public statistics, and each fails in its own predictable way.

Frequency weights overrate routine work that may already be automated. Time weights reward slow processes as readily as hard ones. Cost and price weights inherit whatever set the wage or the price. Loss weights hinge on rare events that a few years of records may not contain. No family is neutral, which is why the choice has to be visible.

Where public evidence already prices or ranks agent tasks

A handful of public benchmarks and statistical programs already attach economic information to tasks, and their designs show both the method and its limits.

Two uses of that information should be kept apart. Selection weighting decides which work enters a dataset, as GDPval does when it picks occupations by their contribution to wages. Score weighting decides how much each task counts once it is in, as SWE-Lancer does when it reports the value of the tasks a model solved alongside the share it solved. A dataset can be economically selected and still scored as if every task were equal, so ask which one a company means.

The five sources compare as follows:

Where public evidence already prices or ranks agent tasks
SourceEconomic signalWhat it weightsLimit to check
OpenAI GDPvalSectors above 5% of U.S. GDP; within each, the five knowledge-work occupations contributing most to total wagesOccupations chosen for 1,320 tasks, 220 of them in an open gold setSelection is weighted; how tasks combine into a headline score needs checking separately
SWE-Lancer (OpenAI researchers)The price listed for each real freelance software task from UpworkIndividual coding fixes and manager-style proposal choicesA price records what one client paid, not the value created or the cost of an error
Remote Labor Index (Mazeika et al., 2025)Whole freelance projects with a human gold deliverable; mean 28.9 and median 11.5 hours to completeProjects, through completion timeGraded by hand; the best agent automated 2.5% of projects, leaving little to weight
O*NET task ratingsFrequency, importance and relevance of each task within an occupationTask statements inside occupationsEach rating carries a sample size, standard error and a low-precision suppression flag
BLS OEWSEmployment and wages for about 830 occupationsOccupations, through the wage billWage and salary workers only; the self-employed are excluded

What 463 priced tasks reveal about concentration

Listed value in SWE-Lancer's public task file is heavily concentrated, so a price-weighted score is mostly a score on a few tasks. The file in OpenAI's preparedness repository holds 463 Diamond tasks: 198 individual-contributor coding tasks, in which a model writes a patch for a real issue, and 265 manager tasks, in which it picks the best of several freelancer proposals. The README says the repository keeps 198 of the paper's original 237 problems, the ones adjusted and verified to run offline.

Sorted from cheapest to most expensive, the tasks sit on nine distinct price levels, and the top level is 640 times the bottom one. The cheapest quarter of tasks holds 6.2% of total listed value and the cheapest half 17.6%, while the 46 most expensive tasks, a tenth of the list, hold 49.1%. The Gini coefficient of task value is 0.545.

The split between task types barely moves under price weighting: coding tasks make up 42.8% of tasks and 41.7% of listed value. What weighting changes here is concentration, not task type, which is why the next question is precision.

Enterprise workloads may well look similar wherever a few task types carry most of the money, such as large claims beside routine ones or major renewals beside small account changes. Treat that as an expectation to test in each dataset, not as a finding: the only distribution measured here is SWE-Lancer's.

A Lorenz curve of 463 public SWE-Lancer tasks sorted by listed price, with the cheapest half holding 17.6% of value, beside panels showing that the top 46 tasks hold 49.1% and that price weighting leaves 69 effective tasks out of 463.
Concentration of listed value in the public SWE-Lancer Diamond task file (openai/preparedness, commit b78dab1), computed for this article.

How far does the choice of weights move a result?

Far enough to reverse a ranking. Weighting by price cuts the effective number of tasks from 463 to 68.8, using the standard effective-sample-size formula (the squared sum of weights divided by the sum of squared weights), so a price-weighted score is about as noisy as an unweighted score on 69 tasks. Capping prices at their 90th percentile keeps 363.5 effective tasks, and weighting by the logarithm of price keeps 455.3.

Precision has a price an investor can read directly. For a score near 50%, a rough 95% interval spans about plus or minus 4.6 points with 463 equally weighted tasks and about plus or minus 11.8 points with 69 effective ones, using the usual binomial approximation that treats tasks as independent. A ten-point gap between two agents on a price-weighted score can sit inside that noise.

To see what that does to a comparison, take two illustrative agents. Agent A solves exactly the cheapest half of the tasks; agent B solves only the most expensive 15%. Neither is a real model result. Agent A leads under equal, log and capped weights; agent B leads only under raw price weights, where it scores 54.2% against A's 17.6%.

The lesson for diligence is not that one scheme is right. A headline built on economic weights should arrive with the same result under at least three alternatives, so a reader can see whether the conclusion belongs to the agents or to the weights. All four schemes, on the public task file:

How far does the choice of weights move a result?
Weighting schemeEffective tasksTop tenth's share of weightAgent A score (illustrative)Agent B score (illustrative)
Equal, one per task463.09.9%49.9%14.9%
Listed price68.849.1%17.6%54.2%
Price capped at the 90th percentile363.516.6%28.8%24.9%
Logarithm of price455.312.7%45.3%18.1%

The agent-task input map: what has to sit under every weight

A weight is only as good as the task it multiplies. Before a weighted score means anything, each task needs a starting state, a set of allowed actions, the constraints that bind them and an outcome that can be checked, and each weight input needs a source a reader can audit.

Enterprise records can supply most of these when they are complete. System snapshots give state, audit logs give actions, policies and approval limits give constraints, and the recorded resolution gives the outcome. SWE-Lancer's manager tasks show the outcome half working: a model's pick is graded against the proposal the original hiring manager chose, a decision already sitting in the record.

The map lists the seven inputs and the evidence that shows each one is really present:

The agent-task input map: what has to sit under every weight
InputWhat it recordsTypical enterprise sourceEvidence an investor should see
StateSystem and case facts when the task startsTicket, order or account snapshot as of the start timeSnapshot times earlier than the outcome, with no later edits leaked in
ActionsWhat a worker or agent could do, in orderAudit logs, tool calls, status transitionsShare of tasks with a complete action trail
ConstraintsRules that limit the allowed actionsPolicies, approval limits, contract terms in force at the timePolicy version attached to each task
Verifiable outcomeA result checkable without the agent's own claimRecorded resolution, payment, approval or reversalShare of tasks with an outcome a third party can verify
Frequency weightHow often the task occursEvent counts per period from the system of recordCounting window and denominator, with seasonality noted
Effort weightTime a qualified person needsHandle times, timestamps or occupational task ratingsMeasurement method and spread, not just the mean
Value or loss weightWhat the work is worth or what an error costsWage data, price paid or loss recordsWhich of the three it is, plus the sensitivity table

Could a record collection lack what an agent needs, whatever the weights?

Yes, and it is the strongest alternative explanation for a weak weighted result. Records of finished work often hold the outcome but not the environment: the screens, tools and intermediate states an agent would have to act in. A weight attached to a task the agent cannot attempt, or whose outcome nobody can check, multiplies noise rather than value.

The public evidence points the same way. The Remote Labor Index graded whole freelance projects by hand against human gold deliverables, and its best agent automated 2.5% of them. When an agent rarely succeeds, any weighting scheme is weighting a handful of successes, and the effective sample shrinks further still.

The test is mechanical. For every task in a weighted dataset, ask whether state, actions, constraints and outcome are all present, then report the weighted share of tasks that pass. A company that cannot produce that share is selling weights, not agent data.

Weighting can also hide missing evidence. If tasks without a verifiable outcome cluster among the high-value ones, which is plausible because complex work tends to leave messier records, a dataset can look complete by count and thin by value. Report the verifiable share both ways.

What could make the economic-weighting thesis wrong?

Several things could, and each deserves a place in the risk discussion of any company built on weighted agent data.

At a glance

What could make the economic-weighting thesis wrong

  • Price is not value. A listed price records what one buyer paid, which can sit well above or below the value created or the loss an error would cause.
  • Wages measure cost, not automation potential. OEWS covers about 830 occupations of wage and salary workers and leaves out the self-employed, so wage-bill weights miss freelance and owner-operated work.
  • Ratings are estimates. O*NET task ratings come with sample sizes, standard errors and a flag for low precision, and a weight built on a flagged rating inherits that uncertainty.
  • Precision collapses. Concentrated weights can turn hundreds of tasks into a few dozen effective ones, so differences that look large may sit inside the noise.
  • Weights invite gaming. Once a seller knows which tasks drive the score, the incentive is to add or inflate them; published weights and independent verification are the defense.
  • Weights age. Wages, prices and task frequencies drift, so a weight measured in one year can misstate value a few years later unless it carries a date and a refresh rule.
  • Capability may lag value. If agents rarely finish high-value work, as the Remote Labor Index found for whole projects, value-weighted scores stay near zero and say little about near-term revenue.

Five questions to put to any vendor about its weights, SourceX included

SourceX works upstream of any weighting. It finds and qualifies businesses whose operational records suit AI developers, reviews their rights, writes the license, prepares and delivers the data, and pays the business. It has not published economically weighted datasets or results, and it shares company and commercial metrics with investors directly instead of in articles. SourceX is raising from accredited investors under Rule 506(c), and investor materials can be requested through its investors page, which is also the place to ask how these questions apply to it.

Put the same five questions to every agent-data company, and expect numbers with denominators in reply:

  • Which weight does each task carry, where does it come from, and when was it measured?
  • What is the effective number of tasks after weighting, and how wide is the interval on the headline result?
  • How does the headline result change under equal, capped and log weights?
  • What share of tasks, by count and by weight, has a verifiable outcome and a complete action trail?
  • Who can change the weights, and is every change versioned together with the results it affects?

Methods note: data, definitions and limits of the weighting analysis

Data: all_swelancer_tasks.csv in the openai/preparedness repository under project/swelancer, at commit b78dab1da774a6456b1ed9c48c9ff194ee3ff36d (committed 12 October 2026) and cloned on 27 October 2026. All 463 rows were used, and every row belongs to the Diamond set. Prices were read as listed; this article reports only shares and ratios, never currency amounts.

Definitions: tasks were sorted by price, with ties broken by question ID. A task's value share is its listed price over the sum of listed prices. Effective tasks equals the squared sum of weights over the sum of squared weights. The capped scheme replaces prices above the 90th percentile with that percentile, and the log scheme uses the natural logarithm of price. Agent A covers the cheapest 231 tasks and agent B the most expensive 69; both are illustrative constructions, not results from any model.

Not measured: whether any model actually solves high- or low-priced tasks more often, the value each fix created for the client, or the cost of errors. The concentration figures describe one public software benchmark, not enterprise workloads, and SourceX has published no economically weighted measurements of its own.

Frequently asked questions

What is an economically weighted agent dataset?

It is a set of agent tasks in which each task counts in proportion to an estimate of its economic importance, such as frequency, time, labor cost, market price or error cost, instead of counting once. The weights, their sources and the effective number of tasks after weighting should be published with it.

Is GDPval an economically weighted benchmark?

Its selection is. OpenAI chose sectors that each contribute more than 5% of U.S. GDP and, within them, the knowledge-work occupations that contribute most to total wages. That is different from weighting each task's contribution to a headline score, which a reader should check separately.

Why not weight agent tasks by wages alone?

Wages measure what labor costs, not what the work is worth or what an error would cost, and U.S. occupational wage statistics leave out the self-employed. Wage weights are a reasonable way to choose which work to include, but a score built on them should be shown beside frequency- and loss-based alternatives.

How many tasks does an economically weighted benchmark need?

More than its raw count suggests. Compute the effective number of tasks after weighting; on SWE-Lancer's public list, price weights turn 463 tasks into about 69 effective ones. Size the dataset so that intervals at the effective count are narrower than the differences you need to detect.

Does SourceX publish economically weighted datasets or results?

No. SourceX has not published weighted datasets, benchmark results or commercial metrics; company and commercial metrics go to investors directly. Investor materials can be requested at sourcex.si/investors, and any SourceX offering is limited to investors verified as accredited.

Sources

  • The public SWE-Lancer task file lists 463 Diamond tasks with a price each; every concentration and sensitivity figure here was computed from it. Source
  • SWE-Lancer builds tasks from real Upwork freelance software jobs, grades manager tasks against the original hiring manager's choice and reports value earned alongside tasks solved. Source
  • GDPval selected sectors above 5% of U.S. GDP and the five knowledge-work occupations contributing most to wages in each, for 1,320 tasks. Source
  • The Remote Labor Index evaluates whole freelance projects against human deliverables; the best agent automated 2.5% of them. Source
  • O*NET task ratings report frequency, importance and relevance with sample sizes, standard errors and a low-precision flag. Source
  • BLS OEWS estimates employment and wages for about 830 occupations of wage and salary workers, excluding the self-employed. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies