Deal economics
Training tokens vs verified tasks: the economics investors should track
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Track training tokens and verified tasks in separate ledgers and never convert one into the other. Tokens measure training volume and compute: Meta pretrained Llama 3 on over 15 trillion. Verified tasks measure checkable work, where value sits in the grader, the difficulty band and a shelf life that saturation shortens. An AI data company's economics depend on which unit its buyers pay in, so ask for demand evidence in that unit.
Value drivers
Training tokens
Key takeaways
- Tokens describe pretraining scale and compute spend; verified tasks describe post-training and evaluation work, and model developers switch units when they move from one to the other.
- A task's value comes from its tested grader and its difficulty band, neither of which has a token count, so a conversion factor between the units destroys information.
- Verified tasks expire: SWE-bench Verified went from hard to near saturation within about a year, and OpenAI stopped evaluating on it in 2026.
- Demand evidence counts only when it names a unit; track requests, paid evaluations, commitments and renewals per unit, by dated cohort and independent budget owner.
- Public filings disclose contract value, term and concentration, not tokens or tasks, so unit-level evidence has to come from the company and its buyers.
Should investors track training tokens or verified tasks?
Track both, in separate ledgers, and never convert one into the other. Training tokens measure how much text a model consumed and, indirectly, how much compute it used; verified tasks measure how many units of checkable work a model can be trained or scored on. An AI data company's economics depend on which unit its buyers actually pay in.
The definitions are exact. A training token is a fragment of text, a word or part of one, counted every time a model processes it during training, so a dataset trained for three epochs is processed three times over. A verified task is a single problem with defined inputs and success criteria plus a grader whose correctness has been checked, for instance by showing that a reference solution passes. Anthropic uses the same definition of a task in its January 2026 guide to agent evaluations.
The investor question is which unit follows the money. Pretraining is described in tokens; post-training and evaluation are increasingly described in tasks, environments and trajectories. A company that reports one unit while its buyers pay in the other is showing an investor the wrong dashboard. This article sets out a dual ledger, a demand ladder that names the unit at every rung, and a study design for the evidence nobody has published.
What a training token count tells you, and what it hides
A token count tells you the scale of training and the compute behind it; it says little about which data mattered. Meta's Llama 3 model card says the models were pretrained on over 15 trillion tokens from publicly available sources, and DeepSeek's V3 repository says that model was pretrained on 14.8 trillion tokens before supervised fine-tuning and reinforcement learning.
The same documents change units when they reach post-training. Llama 3's card describes its fine-tuning data as publicly available instruction datasets plus over 10 million human-annotated examples, an example count rather than a token count. DeepSeek reports 2.664 million H800 GPU hours for pretraining and 0.1 million GPU hours for all later training stages, a small slice of compute for the stages that turn a base model into the assistant people use.
Tokens are also how supervised fine-tuning is metered. OpenAI's fine-tuning data-preparation notebook estimates cost from billing tokens, with each example capped at 16,385 tokens, multiplied by the number of epochs, which the notebook's estimate targets at three. That makes a token count a good gauge of a buyer's compute bill and a poor gauge of what the data was worth to the buyer.
What makes a task verified, and why it is a separate unit
A task is verified when a program or an agreed expert check can score any attempt at it, and the check itself has been tested. Anthropic's guide asks for a reference solution that passes every grader and for tasks clear enough that two domain experts, working separately, would agree on whether an attempt passes or fails; it also lists token usage and cost per task among the metrics to track on a fixed bank of tasks, which puts tokens inside the task ledger as a cost line.
Post-training methods are built on that unit. DeepSeek's R1 repository describes R1-Zero, trained with large-scale reinforcement learning without supervised fine-tuning as a preliminary step. AI2's open-instruct repository documents reinforcement learning with verifiable rewards, or RLVR, for its Tulu 3 models, trained on a dataset that mixes GSM and MATH problems with instruction-following constraints, all with answers a verifier can check. OpenAI's reinforcement fine-tuning guide describes a programmable grader that scores every candidate response instead of training on fixed answers, and says the method needs experts who agree on the ideal output.
Builders count these datasets in tasks. The SWE-smith repository offers 52k software task instances, keeps only tasks that break at least one unit test, and reports that SWE-agent-LM-32B, trained on 5k agent trajectories, reached 40.2 percent pass@1 on SWE-bench Verified. GDPval, OpenAI's 2025 benchmark of knowledge work, is 1,320 tasks across 44 occupations. Neither leads with a token count, because length is not what makes the data useful.
Three reasons task counts cannot stand in for tokens
Converting tasks to tokens, or tokens to tasks, throws away the information an investor needs. Illustrative example: two tasks of about 2,000 tokens each, one checked by a passing test suite and one with no working check, have identical token counts and opposite value. Three properties of verified tasks have no token equivalent.
- Value sits in the grader. A task without a working check is only text; the reference solution, tests or rubric that make it gradable add almost no tokens but decide whether it can train or score anything.
- Only a difficulty band pays. Anthropic notes that capability evaluations should start at a low pass rate and that, with frontier models, a 0 percent pass rate across many trials usually means a broken task. Tasks a model always passes, or can never pass, give no learning or measurement signal.
- Tasks expire. Anthropic describes SWE-bench Verified scores rising from 30 percent to above 80 percent within a year, and OpenAI said in 2026 that it no longer evaluates on the set, citing flawed tests and signs of training contamination. METR's time-horizon analysis, which measures agents on tasks with known human completion times, reports horizons doubling about every 7 months. A task bank needs a refresh schedule that no token count reveals.
The dual ledger: what to track in each unit
Keep two ledgers side by side and reconcile them only through money and dates, never through an exchange rate. For a data company, the token ledger answers how much of a buyer's training volume it supplies; the task ledger answers how many checked units of work it delivers, at what pass-rate band, and how quickly they need replacing.
The table lists what each ledger should record and why a ratio between them misleads. The figure sets the two ledgers side by side.
| Question | Token ledger | Verified-task ledger | Why no ratio works |
|---|---|---|---|
| What is counted | Tokens processed in training, multiplied by epochs | Tasks with inputs, success criteria and a tested grader | The grader carries the value and has almost no token weight |
| Public scale reference | Llama 3: over 15 trillion pretraining tokens; DeepSeek-V3: 14.8 trillion | SWE-smith: 52k task instances; GDPval: 1,320 tasks | Counts sit many orders of magnitude apart, so ratios say nothing |
| Quality gate | Format checks and token limits before training | Reference solution passes; two experts give the same verdict | A clean token stream can still hold no gradable task |
| How buyers meter it | Fine-tuning cost estimated as billing tokens times epochs | Reinforcement fine-tuning built around a grader that scores each response | One meters volume consumed, the other checked outcomes |
| Shelf life | Set by rights, quality and whether the next run reuses it | Ends when models saturate the tasks or the tasks leak into training | Task banks need contracted refreshes; token counts hide this |
| Payment evidence to request | Contracts priced per token, per corpus or per term | Contracts priced per task, per environment or per refresh | Prices in one unit cannot be restated in the other |
Which demand evidence counts in each unit?
Demand evidence counts only when it names a unit, and the same five rungs apply to both ledgers: stated interest, a written request, a paid evaluation, a signed commitment and a renewal. Count each rung by dated cohort, meaning the quarter it first appeared in writing, and by independent budget owner.
Public filings rarely reach this level. Reddit's S-1 valued the data licensing arrangements it entered in January 2024 at $203.0 million in aggregate contract value, on terms of two to three years, and its FY2024 Form 10-K states that "to date, substantially all of the contract value associated with our licensing revenue is derived from two of our partners." Contract value, term and concentration are disclosed; tokens and tasks are not. Unit-level evidence has to come from the company and its buyers.
Use the matrix to see what each rung proves in each unit:
| Rung | Token-denominated evidence | Task-denominated evidence | Common misreading |
|---|---|---|---|
| Stated interest | A public call for large-scale training data | A published benchmark or task specification | Reading a public call as a budget |
| Written request | Request naming corpus size, language and license | Request naming task type, grader and pass-rate band | Counting requests that name no unit at all |
| Paid evaluation | Paid sample tested in a training ablation | Paid pilot task bank scored against a reference model | Treating a free sample as a paid test |
| Signed commitment | Contract priced per token, corpus or term | Contract priced per task, environment or refresh | Restating one unit's price in the other unit |
| Renewal | Repeat purchase for a new model generation | Scheduled refresh after tasks saturate or leak | Calling a one-off delivery recurring revenue |
Measuring unit economics with budget owners: a study design
The unit question can only be settled with buyer records, so the design should be written down before the first interview. SourceX has not published results from this design; it is published so readers can run it or ask a company for its equivalent.
- Question: for each data category, which unit do buyers request, test and pay in, and does the unit change between pretraining, post-training and evaluation budgets?
- Population: independent budget owners for pretraining data, post-training data and evaluation at model developers and enterprise AI teams, each counted once.
- Cohorts: requests grouped by the quarter in which they were first made in writing, with inbound and outbound sources kept apart.
- Variables: the unit named (tokens, records, examples, tasks, environments or trajectories), verification method, pass-rate band requested, price basis, contract term and refresh clause.
- Metrics: the share of each cohort that names a unit, the share reaching paid evaluation and signed commitment in each unit, and the time between rungs as a median with open cases censored rather than dropped.
- Denominators and uncertainty: budget owners interviewed per cohort as the denominator, counts reported with intervals, and no market-size estimate extrapolated from the sample.
Risks that would weaken the verified-task thesis
The case for watching verified tasks rests on assumptions that public evidence already strains. Name them before the upside.
- Synthetic tasks may be enough. The SWE-smith toolkit says it can create unlimited tasks for a GitHub repository; where a domain can be generated and checked by program, real-world task supply may command little.
- Graders can be gamed. Anthropic warns that tasks and graders must be built so that passing requires solving the problem rather than exploiting a loophole; a task bank that rewards shortcuts loses value fast.
- Saturation makes revenue lumpy. SWE-bench Verified went from hard to near saturation in about a year, so task income may depend on refresh contracts that buyers can delay.
- Tokens may remain the budget line. If buyers keep spending mainly on pretraining-style corpora, revenue follows token volume and a task ledger describes a small share of spend.
- Buyers are few. In Reddit's FY2024 10-K, a single pair of partners accounts for substantially all of its licensing contract value; any unit sold to a handful of buyers carries renewal and pricing risk.
- Method announcements are not spending. A release note that mentions reinforcement learning or a new benchmark commits no data budget.
Questions about units to put to any data company, SourceX included
Ask which unit the company counts, which unit its buyers pay in, and how it reconciles the two. A company that cannot answer usually reports whichever total sounds largest.
SourceX is one labeled application. Its public guidance for supplying companies is stated in records and years: buyers generally want consistent history going back typically two years or more and a record count in the tens of thousands or higher within one system, though a smaller set can qualify when the work is rare, and every deal is assessed separately. That is a third unit, closer to source records than to tokens or tasks, and an investor should ask how records map to whatever unit a buyer pays for.
SourceX has not published volumes, prices or commitments in any unit; company and commercial metrics are disclosed to investors directly instead. It is raising under Rule 506(c), from accredited investors, and those who request materials can test the company against the questions below.
- Which unit do your buyers pay in for each data category: tokens, records, examples, tasks or environments?
- For task-denominated deliveries, what share carry a tested grader or reference solution, and which pass-rate band do they target?
- How often does a delivered task bank need refreshing, and is the refresh written into the contract?
- How many independent budget owners reached paid evaluation and signed commitment in each unit, per quarterly cohort?
- Which of your reported totals mix units, and how were they combined?
- What happens to your task revenue if a synthetic generator can produce the same tasks?
Methods note: sources behind the dual ledger
The dual ledger, the unit-aware demand ladder and the study design were developed for this article on 2026-10-26 as editorial tools. Unit examples come from Meta's Llama 3 model card, the DeepSeek-V3 and DeepSeek-R1 repositories, AI2's open-instruct repository, the SWE-smith repository, OpenAI's fine-tuning data-preparation notebook and reinforcement fine-tuning guide, Anthropic's January 2026 evaluation guide, METR's time-horizon repository, OpenAI's GDPval release and its 2026 SWE-bench Verified post, and Reddit's S-1 and FY2024 Form 10-K. The two-task comparison is illustrative.
Not measured: prices per token or per task, how buyer budgets split between units, how often task banks are refreshed in practice, and any SourceX volume or revenue. Each needs buyer and supplier records that have not been published.
Frequently asked questions
Can verified tasks be converted into training tokens for comparison?
Not meaningfully. A task's value comes from its grader and its difficulty, neither of which has a token count, and two tasks of the same length can differ completely in usefulness. Compare the units through money and dates, such as spend per quarter in each, rather than through a conversion factor.
Why are post-training datasets counted in tasks or examples instead of tokens?
Because the check, not the volume, makes them useful. Llama 3's model card describes fine-tuning data as over 10 million human-annotated examples, and the SWE-smith repository counts 52k task instances that each break at least one unit test. The count tells a buyer how many graded units it receives.
What is a verifiable reward in AI training?
It is a reward a program can compute by checking the model's answer, for example matching a math result or running unit tests. AI2's open-instruct repository documents reinforcement learning with verifiable rewards for Tulu 3, and OpenAI's reinforcement fine-tuning guide describes graders that score every candidate response. The method works only where a correct answer can be checked.
Do verified task banks lose value over time?
Often, yes. Models can saturate a task bank or see it during training: Anthropic describes SWE-bench Verified rising from 30 percent to above 80 percent within a year, and OpenAI stopped evaluating on it in 2026. Ask how often a supplier's task banks need replacing and whether replacements are contracted.
Does SourceX report token or task volumes?
No. SourceX has published no volumes, prices or commitments in any unit, and its public guidance for supplying companies is stated in years of history and record counts. Investors can ask for materials at sourcex.si/investors, including how records map to the units buyers pay for.
Sources
- Llama 3 was pretrained on over 15 trillion tokens, and its fine-tuning data included over 10 million human-annotated examples. Source
- DeepSeek-V3 was pretrained on 14.8 trillion tokens using 2.664 million H800 GPU hours, with 0.1 million GPU hours for later stages. Source
- OpenAI's fine-tuning data-preparation notebook estimates cost as billing tokens, capped at 16,385 per example, times epochs. Source
- Anthropic's January 2026 guide defines tasks, asks for reference solutions and expert agreement, tracks token usage and cost per task, and describes saturation and loophole-resistant graders. Source
- DeepSeek-R1-Zero was trained with large-scale reinforcement learning without supervised fine-tuning as a preliminary step. Source
- AI2's open-instruct documents reinforcement learning with verifiable rewards for Tulu 3 on math and instruction-following constraint prompts. Source
- OpenAI's reinforcement fine-tuning uses a programmable grader that scores every candidate response and needs experts who agree on the ideal output. Source
- SWE-smith offers 52k task instances that break at least one unit test, says it can create unlimited tasks per repository, and reports a model trained on 5k trajectories at 40.2 percent pass@1 on SWE-bench Verified. Source
- GDPval comprises 1,320 tasks across 44 occupations. Source
- In 2026 OpenAI stopped evaluating on SWE-bench Verified after finding flawed tests and signs of training contamination. Source
- METR measures agents on tasks with known human completion times and reports time horizons doubling about every 7 months. Source
- Reddit's S-1 disclosed January 2024 data licensing arrangements with an aggregate contract value of $203.0 million and two-to-three-year terms. Source
- Reddit's FY2024 Form 10-K says substantially all of the contract value associated with its licensing revenue is derived from two of its partners. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.