AI data market
Context-rich AI training data investment: 90,000 records, 2,294 tasks
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Context could matter more than another billion records because buyers pay for complete task episodes: the request, the state at the time, the actions, the outcome and a check. Few records survive that linking. SWE-bench began with about 90,000 pull requests and kept 2,294 tasks with an issue, a fix and tests. Back episode yield over volume only where demand, cleared rights and a failed substitute line up.
Value drivers
Context rich AI training
Key takeaways
- A complete task episode links five parts, trigger, state, actions, outcome and check; records that cannot be joined into episodes support far fewer uses.
- Linking is lossy: SWE-bench kept 2,294 of about 90,000 scraped pull requests, roughly 2.5 percent, and OpenAI later had developers screen a sample of those before releasing 500.
- Context measurably changes results: Anthropic cut top-20 retrieval failures from 5.7 to 2.9 percent by adding 50 to 100 tokens of context to each chunk.
- Volume hides repetition: one 61-word sentence appeared more than 60,000 times in C4, and 4.6 percent of C4 validation examples had a duplicate in training.
- The thesis fails if buyers can supply the context themselves, if rights review breaks the links, or if demand stays with a few buyers.
Why could context matter more than another billion records?
Because the uses buyers pay most for, training agents and testing them, need complete task episodes, and adding unlinked records does not create episodes. A billion rows that cannot be joined to a request, a state, an outcome and a check add volume without adding the thing a model learns a task from.
Define the two assets precisely. A complete task episode links five parts: the trigger (a customer request, an issue, an order), the state at the time (the account, the policy in force, the history), the actions taken, the outcome, and a check that says whether the outcome was right. A disconnected high-volume collection is a set of records, from one system or several, that cannot be reliably joined into those five parts, because keys are missing, histories were overwritten or outcomes live somewhere else.
For a context-rich AI training data investment, the evidence that would support the thesis is specific: buyers stating a preference for linked, intention-bearing data; a measurable loss of volume when records are linked into episodes, which makes complete episodes scarce; results that improve with context at equal volume; and substitutes that fail to close the gap. The sections below take each in turn, then name what would prove the thesis wrong.
What one public benchmark reveals about episode yield
Linking records into checkable episodes discards most of them, even in a tidy public source. SWE-bench, the software engineering benchmark Jimenez and colleagues published in 2023, is the clearest public measurement of that loss.
Its authors started from about 90,000 pull requests scraped from 12 popular open-source Python repositories and kept 2,294 task instances, each pairing a real GitHub issue with the merged fix and tests that fail before the fix and pass after it. By our arithmetic that is an episode yield of roughly 2.5 percent, or about 39 raw pull requests for every usable task. The project's collection code makes the tiers explicit: pull requests with an associated issue and a reference patch are flagged as usable for fine-tuning, while only those that also carry tests become candidates for evaluation.
A second filter followed. In August 2024 OpenAI had 93 software developers experienced in Python review 1,699 random samples from the SWE-bench test set, screening for issue descriptions too underspecified to define the problem and unit tests too narrow to accept valid fixes, and released a 500-sample subset called SWE-bench Verified. Even linked episodes, in other words, can lack the context a fair task needs.
Code repositories are unusually orderly: issues, commits and tests sit in one system with stable identifiers. Business archives spread the same five parts across a CRM, a ticketing tool, a billing system and email, so there is little reason to expect a higher yield there. The scarce asset is the joined episode, not the row.
Episodes and record piles compared, use by use
The difference shows up in what each asset can be used for, not in its size. The comparison holds the subject matter constant and changes only whether the records are linked into episodes.
| Question a buyer asks | Disconnected high-volume collection | Complete task episodes |
|---|---|---|
| What is counted? | Rows, tokens or files | Tasks with a trigger, state, actions, outcome and check |
| Can it train a multi-step task? | Only isolated steps, with no record of what came next | Yes: each step sits in sequence with its result |
| Can it build an evaluation? | Rarely, because there is no outcome or check to grade against | Yes, when the check is reliable, as with SWE-bench's fail-to-pass tests |
| How much of the volume is new information? | Unknown until deduplicated; repetition can be large | Bounded by distinct tasks, which are counted directly |
| What substitutes compete? | Public web text and synthetic text at very large scale | Buyer-built environments, commissioned experts and the buyer's own records |
| What does rights review touch? | Usually one system's terms | Every linked system's terms, so more review per unit |
Does adding context change model results, or only the story?
It changes results in at least one measured setting, and research on repetition shows why raw volume overstates information. Neither finding proves the investment case, but both point the same way.
Anthropic's September 2024 work on contextual retrieval found that splitting documents into chunks often strips the context needed to find the right passage. Prepending 50 to 100 tokens of chunk-specific context before indexing reduced the top-20 retrieval failure rate from 5.7 percent to 2.9 percent, a 49 percent cut, and to 1.9 percent, a 67 percent cut, when combined with reranking. The underlying text was identical; only its tie to the surrounding document changed.
On the volume side, Lee and colleagues reported in 2021 that standard language-model training sets hold many near-duplicates: one 61-word English sentence appeared more than 60,000 times in C4, and 4.6 percent of C4 validation examples had a duplicate in the training set. After deduplication, models emitted memorized text about ten times less often and reached the same or better accuracy in fewer training steps. Another billion records can be another billion copies.
Buyers say something similar in their own terms. OpenAI's data partnerships page says it is especially interested in data that expresses human intention, such as long-form writing or conversations rather than fragments, and in datasets not already easily accessible online. Intention lives in sequence and context; a fragment seldom carries it.
A context-rich AI training data investment thesis map
Five linked claims make up the thesis, and each is tested on the three axes that decide any data market: observed demand, supply whose rights can be cleared, and a substitute that might do the job more cheaply. The last column names the observation that would falsify the claim.
The map is built to be updated. Rows move as buyers publish requests, as archives are audited for episode yield and as substitutes improve, and a company pitching context-rich data should be able to say which row its own evidence moves, and in which direction.
| Claim | Demand evidence | Rights-cleared supply test | Credible substitute | Falsified if |
|---|---|---|---|---|
| Buyers pay for linked episodes over raw volume | OpenAI seeks data expressing human intention, not fragments, and not already online | Count episodes, not rows, in a sample the supplier may license | Commissioned experts writing tasks to order | Written requests specify row counts and accept unlinked records at the same price |
| Complete episodes are scarce even where records abound | SWE-bench kept 2,294 of about 90,000 pull requests | Measure the join rate across every system an episode needs | Simulated environments that generate episodes | Audited archives join at high rates with outcomes and checks intact |
| Context improves results at equal volume | Contextual retrieval cut top-20 retrieval failures by 49 percent | Pilot linked against unlinked records from the same licensed scope | Buyers adding context from their own documents at inference | A matched pilot shows more unlinked rows beat fewer episodes at the same budget |
| Episodes enable evaluation that volume cannot | Anthropic advises tasks two experts would grade the same way, with reference solutions | Confirm each episode carries a check the buyer accepts | Public benchmarks | Buyers evaluate only on public sets and decline private episodes |
| Rights survive the linking | No public demand signal; this is a supply constraint | Run rights review on the joined episode, not each table alone | Openly licensed public data | De-identification breaks the join keys, so cleared data stops being episodic |
Plenty of records, few buyers, and context the buyer already holds
The most serious counterargument is that volume is plentiful, purchasing is concentrated, and the context that matters most may already sit inside the buyer. Each part has evidence behind it.
Volume is plentiful. Epoch AI's 2024 study estimated the effective stock of quality-filtered, repetition-adjusted public human text at about 300 trillion tokens, with full use projected between 2026 and 2032 on current trends. Purchasing is concentrated: Reddit's Form 10-K for fiscal 2024 says that "substantially all of the contract value associated with our licensing revenue is derived from two of our partners." A large stock of data coexisting with very few paying buyers is the alternative explanation every context pitch has to beat.
The third part is the sharpest. Contextual retrieval works because a buyer adds context from its own documents. If the episodes a model needs are the buyer's own, the buyer can link them internally and pay no outside supplier. The thesis survives only for context the buyer cannot reach: other companies' operating episodes, decisions taken under different policies, outcomes the buyer never observed. Ask a supplier to show that its episodes come from outside the buyer's walls and could not be assembled from the buyer's own systems.
How to measure episode yield before backing a supplier
Measure the share of raw records that become complete, checkable episodes, by system and by link, with fixed denominators. The design can be run on any archive; SourceX has not published results from it, and the same definitions apply to every company in this market.
Join losses compound. If an episode needs four links and each joins 80 percent of the time independently, only about 41 percent of episodes survive, because 0.8 to the fourth power is 0.41. The numbers are illustrative, but the arithmetic explains why a supplier that quotes only its best join rate overstates its supply.
Report each stage with its count, the share lost and the reason, and keep records still under review in the denominator as open rather than dropping them. Run the steps in this order:
Checklist · 6 items
- Fix the episode schema: name the five parts and the field that proves each, for example a ticket ID for the trigger and a billing adjustment for the outcome.
- Draw the sample frame by system and date range before looking at any join rate, and record how records entered it.
- Measure the join rate for each link separately, then the share of triggers with every link intact.
- Check the check: confirm that the outcome field or test can grade an answer, and log ambiguous cases separately, as SWE-bench Verified's reviewers did.
- Deduplicate before counting, so repeated records inflate neither the numerator nor the denominator.
- Report episode yield as complete episodes divided by raw triggers, with an interval and the count of cases still open.
0 of 6 ticked. Tick each item as you go.
Signals that would sink the context thesis
Five developments would weaken the thesis, and each comes with a signal an investor can watch for.
- Buyers build context themselves. Retrieval over their own documents and longer context windows may cover most needs, leaving outside episodes a niche; watch for buyer requests that drop external episode data.
- Simulated environments get good enough. If generated episodes pass buyers' acceptance tests, the scarcity premium for real ones falls; watch for evaluations that accept synthetic tasks in place of real ones.
- Rights review breaks the links. De-identification that removes join keys or free-text references can turn an episode back into a pile of rows; watch post-review join rates.
- Demand stays with a few buyers. Reddit's filing shows how concentrated licensing revenue can be, which caps the pricing power of any supplier; watch the count of distinct buyers.
- Benchmarks are not businesses. SWE-bench measures code; yields and value in sales, claims or field service may differ, and no public study measures them yet.
Five numbers to demand from a context-rich data company
Ask for episode counts with denominators and for the evidence behind each row of the thesis map. The same requests apply to SourceX.
SourceX makes licensed AI datasets out of business records that companies choose to release, and it does the work around them: sourcing and qualification, rights review, contracting, preparing the data, delivering it and payment. It stocks nothing for AI buyers; each dataset is sourced from U.S. businesses after a request, and no data leaves the supplying company until it has approved the scope and terms and signed. Episode-yield and demand figures for SourceX are unpublished. The company shares company and commercial metrics with investors directly, and its capital raise, under Rule 506(c), is open to accredited investors, who can request materials through its investors page.
- Episode yield on the last archive assessed, with the sample frame and the count of cases still open.
- The join rate of the weakest link, and whether fixing it is technical or contractual.
- The number of written buyer requests that specified linked episodes rather than volume, and how many distinct buyers sent them.
- The share of delivered episodes a buyer could not have assembled from its own systems.
- The join rate after de-identification, compared with the rate before it.
Where the numbers come from and what they leave out
Sources were checked on 2026-10-26 on each publisher's own domain: the SWE-bench paper and repository, OpenAI's SWE-bench Verified announcement and data partnerships page, Anthropic's contextual retrieval and evaluation guides, Lee et al. (2021), Epoch AI (2024) and Reddit's FY2024 Form 10-K.
Our arithmetic: 2,294 divided by about 90,000 is roughly 2.5 percent, or about 39 pull requests per task. No yield is computed for SWE-bench Verified, because the material verified for this article does not state how many of the 1,699 reviewed samples passed before the 500 were released. The 80 percent join example is illustrative, not measured.
Not measured: episode yields in business archives, prices for linked against unlinked data, buyer counts, and any SourceX result. The thesis map and the five-part episode definition are editorial frameworks that a reader can apply and dispute.
Frequently asked questions
What is context-rich AI training data?
Data in which each example keeps the surrounding facts a model needs to learn a task: the request, the state at the time, the actions, the outcome and a way to check it. A single support message is a record; the ticket with its account history, replies, refund and reopen status is an episode.
Is more data still better for AI training?
For broad pretraining, scale still matters, and Epoch AI estimated about 300 trillion tokens of usable public text in 2024. For agents and evaluations, unlinked volume adds less, because repeated or context-free records cannot show a model how a task unfolds or whether the result was right.
How do you measure whether a dataset has enough context?
Count complete episodes rather than rows. Define the parts an episode needs, measure how often records join across each link, confirm the outcome can grade an answer, deduplicate, and report complete episodes as a share of raw triggers with the open cases shown.
Why is SWE-bench relevant to business data investors?
It is a rare public record of how many raw records survive linking into checkable tasks: about 90,000 pull requests yielded 2,294 tasks. Business archives spread episodes across more systems, so the benchmark is a useful reference point, not a measurement of any business.
Does SourceX publish episode yield figures?
No. SourceX has not published episode-yield, demand or commercial figures and shares company and commercial metrics with investors directly. Readers evaluating SourceX as a company can request investor materials at sourcex.si/investors, where any offering is limited to verified accredited investors.
Sources
- Jimenez et al., SWE-bench (2023): about 90,000 pull requests scraped from 12 popular open-source Python repositories were filtered to 2,294 task instances pairing a GitHub issue with its fix and fail-to-pass tests. Source
- The SWE-bench repository lists 2,294 test task instances and a 500-instance Verified split, and its collection code flags instances with an issue and gold patch as usable for fine-tuning and only those with tests as evaluation candidates. Source
- OpenAI (August 2024) had 93 software developers experienced in Python review 1,699 random SWE-bench test samples for underspecified issues and overly narrow tests, and released a 500-sample verified subset. Source
- Anthropic's contextual retrieval (September 2024): adding 50 to 100 tokens of chunk-specific context cut the top-20 retrieval failure rate from 5.7 to 2.9 percent (49 percent) and to 1.9 percent (67 percent) with reranking. Source
- Lee et al. (2021): a 61-word sentence repeats over 60,000 times in C4; 4.6 percent of C4 validation examples have a training duplicate; deduplicated models emit memorized text ten times less often and need fewer training steps. Source
- OpenAI's data partnerships page seeks large-scale datasets not already easily accessible online and is particularly interested in data that expresses human intention, such as long-form writing or conversations rather than fragments. Source
- Epoch AI (2024) estimates the effective stock of quality, repetition-adjusted public human text at about 300 trillion tokens, fully used between 2026 and 2032 on current trends. Source
- Reddit's Form 10-K for fiscal 2024 says substantially all of the contract value associated with its licensing revenue is derived from two of its partners. Source
- Anthropic's guide to agent evaluations says a good task is one where two domain experts would independently reach the same pass or fail verdict, and recommends a reference solution for each task. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.