AI data market
Real-world AI learning inputs market: a five-input map with falsifiers
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
The real-world AI learning inputs market is five markets, not one. Demonstrations, feedback, outcomes, environments and source documents teach different things, sell to different buyers and face different substitutes. Public evidence shows synthetic and distilled data already standing in for much demonstration and preference work, while recorded business outcomes and realistic environments remain hard to manufacture. Test each input on observed demand, rights-cleared supply and its cheapest credible substitute.
Value drivers
Real world AI learning
Key takeaways
- Splitting real-world learning inputs into five types stops a growing market from being averaged with a shrinking one.
- Demonstrations and preference labels face the strongest substitutes: over 98 percent of Nemotron-4 340B's alignment data was synthetic (NVIDIA, 2024).
- Recorded business outcomes are the input operating companies are best placed to supply, but they arrive slowly and must be linked to the decision.
- Environments are maintained products; the original tau-bench repository now warns that its tasks are outdated.
- A large stock is not demand: Reddit's FY2024 Form 10-K credits just two partners with substantially all of its licensing contract value.
What counts as a real-world AI learning input?
A real-world learning input is any record of actual activity that a model developer uses to teach or test a model, and it comes in five types that buyers purchase for different reasons. Grouping them all as real-world data hides the most important fact for an investor: some types are already being replaced by synthetic data, while others have no cheap substitute.
So the question is not whether AI needs evidence from the real world. It is which of these five inputs a buyer will keep paying an outside supplier for once synthetic data, distillation and in-house collection have taken their share, and the answer differs by type.
- Demonstrations: a person doing the task step by step so a model can imitate it, bought for supervised fine-tuning.
- Feedback: judgments about outputs, such as which of two answers is better or which step of a solution is wrong, bought to train reward models and tune preferences.
- Outcomes: what actually happened after a decision, such as a claim paid, a ticket reopened or a shipment delayed, bought as reward signals and as ground truth for evaluation.
- Environments: a working replica of a system, with tools, state and rules, in which an agent can act and be scored, bought for reinforcement learning and agent testing.
- Source documents: the text, images and records themselves, bought for pretraining and for retrieval at answer time.
What the public record shows each input doing inside model training
Every input type appears in published training pipelines, and the same publications show how much of each a developer needs and where substitutes have already arrived.
Demonstrations matter but saturate early. Meta's Llama 2 paper (2023) reports that the team stopped collecting supervised fine-tuning annotations after 27,540 of them, finding that tens of thousands of high-quality examples were enough. That is a purchase measured in tens of thousands of records, not billions.
Feedback comes in finer grains. The same paper describes annotators writing a prompt, picking the better of two sampled responses and rating how much better it was. OpenAI's PRM800K release goes further, with 800,000 step-level correctness labels on model-generated solutions to math problems, judging each reasoning step rather than only the final answer.
Outcomes can stand in for both when they can be checked automatically. DeepSeek's R1 paper (2025) describes training R1-Zero with rule-based rewards: math answers with deterministic results are verified directly, and code is run against predefined test cases. The authors say they avoided neural reward models because those may suffer from reward hacking in large-scale reinforcement learning.
Environments turn outcomes into something an agent can practice against. Sierra's tau-bench emulates conversations between a simulated user and an agent equipped with domain-specific API tools and policy guidelines. Its original repository now warns that its airline and retail tasks are outdated and points to a newer version with fixes, a reminder that environments are maintained products rather than one-off datasets.
Source documents are the oldest input and the largest stock. By Epoch AI's estimate, quality public human text, adjusted for repetition, forms an effective stock of about 300 trillion tokens, likely fully used between 2026 and 2032 on current trends. Real work products feed evaluation too: OpenAI's GDPval (2025) holds 1,320 tasks across 44 occupations, built from the work of professionals who average about 14 years of experience.
The falsifiable thesis map for real-world learning inputs
Each input gets one row: dated evidence of demand, the route by which rights-cleared supply is obtained, the cheapest credible substitute, and the observation that would prove a real-world premium wrong. A row without a falsifier would be a slogan, and every row here can fail.
Read the demand column as evidence that developers use the input, not that they buy it from outsiders at a rising price; public documents rarely separate the two. Status reflects public evidence on October 26, 2026 only.
The figure at the end of this section places each input on one scale, from easiest to commission or synthesize to hardest to obtain without real operations. The placements are editorial judgments drawn from the substitute column, not measurements, and they are the claim to test.
| Input and buyer purpose | Observed demand, dated | Rights-cleared supply route | Cheapest credible substitute | Falsifier |
|---|---|---|---|---|
| Demonstrations: supervised fine-tuning | Llama 2 (2023): fine-tuning collection stopped at 27,540 annotations | Commissioned from experts, with rights assigned at creation | Distillation: DeepSeek fine-tuned small models on 800,000 samples generated with R1 (2025) | Distilled or synthetic demonstrations pass the buyer's acceptance test on business tasks |
| Feedback: reward models and preference tuning | Llama 2 pairwise choices; PRM800K's 800,000 step labels | Commissioned labeling; real corrections sit in review and approval trails | Synthetic preferences: over 98% of Nemotron-4 340B alignment data was synthetic (2024) | Model-generated judgments match expert judgments on the buyer's own rubric |
| Outcomes: rewards and evaluation ground truth | DeepSeek-R1 (2025): rule-based rewards where results can be checked | Licensed from the company whose systems recorded the result | Automatic checkers in math and code; model graders elsewhere | Business tasks prove checkable by rules or graders without recorded outcomes |
| Environments: agent training and testing | tau-bench: simulated users, domain APIs and policies; tasks now flagged outdated | Built by developers, seeded with licensed workflows, policies and records | Environments generated from public documentation alone | Agents trained in generic environments pass real deployment acceptance tests |
| Source documents: pretraining and retrieval | Reddit: $203.0 million aggregate licensing contract value signed January 2024 | Licensed from platforms and archive holders | About 300 trillion tokens of public text (Epoch AI, 2024) | Paying buyers of licensed documents stay at one or two per seller |
Where operating businesses hold the input that is hardest to fake
Recorded outcomes, and the environments built from them, are where operating businesses hold an edge that commissioning and synthesis struggle to copy. A contractor can write a demonstration on request and a model can generate a preference pair, but nobody can commission what actually happened to ten thousand insurance claims after the adjuster decided.
The reason is structural. Rule-based rewards work where a checker exists, which is why DeepSeek leaned on math answers and compiler test cases. Most business decisions have no checker; the only verdict is the recorded result, arriving weeks or months later and often in a different system from the decision. That makes outcomes slow, noisy and costly to link, and also hard for anyone outside the company to reproduce.
Two caveats keep the verdict honest. Outcomes are valuable only when joined to the decision and its context, which few archives do on their own. And an edge in supply is not demand: a buyer still has to want that outcome for a task it is building, at a price that beats its alternatives.
Why rights clear differently for commissioned and operational inputs
Commissioned inputs arrive with rights attached; operational inputs have to earn them. When a developer pays experts to write demonstrations or label outputs, the contract assigns the work at creation. Outcomes, environment seeds and business documents were created for other purposes, so each carries the terms under which it was made: customer contracts, privacy notices, vendor terms and confidentiality duties.
Buyers say they want the second kind. OpenAI's data partnerships page is after large-scale datasets the public cannot already reach easily online, and it singles out material expressing human intention: long-form writing or conversations rather than disconnected snippets. That is exactly the material whose rights take longest to clear, because it sits inside companies and involves other people.
For outcomes the problem doubles. An outcome is useful only when joined to the decision that caused it, and the two often live in different systems under different terms: a dispatch decision in a field-service platform, the customer's complaint in email, the refund in accounting. Clearing one and not the other leaves a record that teaches nothing.
The strongest objection: a large stock of data with weak purchasing demand
The objection is right that stock is not demand, and the clearest public evidence comes from the best-known licensing seller. Reddit disclosed in its 2024 S-1 that data licensing arrangements signed in January 2024 carried an aggregate contract value of $203.0 million over terms of two to three years. Its FY2024 Form 10-K then reports that "to date, substantially all of the contract value associated with our licensing revenue is derived from two of our partners," and calls licensing for AI training a new business model without an established track record.
Scale the lesson down to business records. A company with a decade of tickets holds a large stock, and the roughly 300 trillion tokens of public text in Epoch AI's estimate show how much substitute material already exists for document-style inputs. Neither fact says anyone will pay for a particular archive; demand has to be shown buyer by buyer and input by input.
The objection is weaker for outcomes and environments, where the stock is small and specific rather than vast and generic, but it does not vanish. A market with two buyers is thin whatever the input.
How to test each row with evidence you can collect
Measure each input type on the same three questions, observed demand, rights-cleared supply and the substitute, with fixed denominators so companies can be compared. None of these measures is public for most private suppliers; they are what an investor should ask to see.
At a glance
How to test each row with evidence you can collect
Checklist · 5 items
- Observed demand: distinct budget owners who issued a dated written request naming the input type, the task and an acceptance test, per quarter, with the share of value from the largest two.
- Rights-cleared supply: of candidate records reviewed for each type, the share cleared for licensing and the median days to clearance, with open cases kept in the denominator rather than dropped.
- Substitute test: the buyer's acceptance score for licensed records against the cheapest commissioned or synthetic alternative, at equal review budget and on the same held-out cases.
- Repeat demand: whether the same budget owner requested the same input type again within twelve months, and what triggered it.
- Outcome linkage yield: the share of sampled decisions that can be joined to a recorded result and the context available at the time.
0 of 5 ticked. Tick each item as you go.
Five ways the real-world premium could disappear
Each row of the map can fail, and each failure has a public marker an investor can track. Name them before the upside.
- Synthetic and distilled data keep improving. NVIDIA reported in June 2024 that synthetic generation supplied over 98 percent of Nemotron-4 340B's alignment data, and DeepSeek fine-tuned small open models on 800,000 samples generated with R1. Each step narrows the market for human demonstrations and preference labels.
- Checkable rewards spread. DeepSeek trained R1-Zero through reinforcement learning without supervised fine-tuning as a first step, using rule-based rewards; every business task that gains an automatic checker needs fewer recorded outcomes.
- Buyer concentration persists. If the pattern in Reddit's FY2024 Form 10-K holds, with two partners behind substantially all of its licensing contract value, a supplier of any input facing a handful of buyers carries renewal and pricing risk.
- Experts are commissioned instead of archives licensed. GDPval's tasks were crafted and vetted by experienced professionals for the benchmark, showing that a buyer can obtain real-work evaluation material without licensing a company's records.
- Environments decay. The original tau-bench repository now warns that its tasks are outdated; environment value depends on upkeep, and a supplier that cannot maintain its seeds loses the row.
Diligence questions for any seller of real-world inputs, SourceX included
Ask which input type a company actually supplies and what evidence shows repeat demand for that type, using the map's definitions rather than the company's own.
SourceX is one place to apply the map. It helps a company turn records it selects into datasets licensed for AI and takes on the steps in between: sourcing and qualification, then rights review and the contract, then preparing, delivering and paying for the data. Delivery waits on the company: it approves the scope and terms, then signs, and only after that does any data change hands.
Demand, supply and commercial figures for SourceX are not public; investors get company and commercial metrics from it directly. It is raising from accredited investors in a Rule 506(c) offering, and anyone requesting materials can put the questions below to it, starting with how it defines a cleared record.
- Which of the five input types do your paid datasets fall into, and who classifies them?
- Over the past four quarters, how many distinct budget owners requested each type, and what portion of the value traces to the largest two?
- What share of reviewed records cleared rights for each type, and what is the median time to clear, with open cases counted?
- For outcome data, what share of decisions could be linked to a recorded result and its context?
- On which acceptance tests have your records beaten a commissioned or synthetic alternative at equal budget?
- Which public development, such as a new automatic checker or synthetic method, would remove demand for a type you supply?
Methods note: building the five-input thesis map
The five-input typology, the thesis map, the substitution placements and the measurement design are editorial frameworks built for this article on October 26, 2026. Evidence comes from Meta's Llama 2 paper, OpenAI's PRM800K repository, DeepSeek's R1 repository and paper, Sierra's tau-bench repository, Epoch AI's data-stock study, OpenAI's GDPval and data partnerships pages, NVIDIA's Nemotron-4 340B report, and Reddit's 2024 S-1 and FY2024 Form 10-K.
Not measured: prices or market size for any input type, how much each developer buys from outsiders rather than collecting in-house, rights clearance rates, and the outcome linkage yield of any company's archive, SourceX included. Placements on the substitution scale are judgments that anyone can re-score from the same sources.
Frequently asked questions
What are real-world AI learning inputs?
They are records of actual activity used to teach or test AI models. Five types matter commercially: demonstrations of a task, feedback on outputs, outcomes that record what happened after a decision, environments where agents can act, and source documents. Each is bought for a different purpose and faces a different substitute.
Which real-world inputs are hardest to replace with synthetic data?
On public evidence, recorded outcomes of business decisions and environments seeded from real workflows. Synthetic data already supplies most of the alignment data for some models and distillation can stand in for demonstrations, but nobody can generate what actually happened after a real decision without the records.
Is public training text running out?
Epoch AI puts the effective stock of quality public human text near 300 trillion tokens and, on current trends, expects it to be fully used between 2026 and 2032. That stock says little about demand for private business records, which are a different input with different buyers.
Why does a large stock of data not prove a market?
Because buyers pay for specific inputs that pass their tests, not for volume. Reddit holds a vast stock of content, yet its FY2024 Form 10-K places substantially all of its licensing contract value with two partners. Demand has to be shown buyer by buyer and input by input.
Does SourceX publish demand or supply metrics for these inputs?
No. Those figures are not public; SourceX reports company and commercial metrics to investors directly. An investor can request materials at sourcex.si/investors and use the five measures in this article as the request list. Any offering is limited to verified accredited investors.
Sources
- Meta's Llama 2 paper (2023) reports stopping supervised fine-tuning annotation after 27,540 annotations and describes preference collection in which annotators choose the better of two responses and rate the margin. Source
- OpenAI's PRM800K is a process supervision dataset with 800,000 step-level correctness labels for model-generated solutions to MATH problems. Source
- DeepSeek's R1 repository says DeepSeek-R1-Zero was trained via large-scale reinforcement learning without supervised fine-tuning as a preliminary step, and that R1's reasoning was distilled into smaller dense models. Source
- The DeepSeek-R1 paper describes rule-based accuracy rewards (deterministic math answers, compiler checks against test cases), avoiding neural reward models over reward hacking, and 800K samples generated with R1 to fine-tune small dense models. Source
- Sierra's tau-bench emulates conversations between a simulated user and an agent with domain-specific API tools and policy guidelines; the repository warns its airline and retail tasks are outdated and points to a newer version. Source
- Epoch AI estimates the effective stock of quality, repetition-adjusted public human text at about 300 trillion tokens, fully used between 2026 and 2032 on current trends. Source
- OpenAI's GDPval has 1,320 tasks across 44 occupations, based on real work products and built with professionals averaging about 14 years of experience; tasks were crafted and vetted by experienced professionals. Source
- OpenAI's data partnerships page seeks large-scale datasets not already easily accessible online, especially data that expresses human intention such as long-form writing or conversations. Source
- Reddit's 2024 S-1 discloses data licensing arrangements entered in January 2024 with an aggregate contract value of $203.0 million and terms of two to three years. Source
- Reddit's FY2024 Form 10-K says substantially all of the contract value associated with its licensing revenue is derived from two of its partners, and calls licensing for AI training a new business model without an established track record. Source
- NVIDIA's Nemotron-4 340B report (June 2024) says over 98% of the data used in its model alignment process was synthetically generated. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.