Skip to content

AI data market

Context-rich AI training data investment: 90,000 records, 2,294 tasks

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Context could matter more than another billion records because buyers pay for complete task episodes: the request, the state at the time, the actions, the outcome and a check. Few records survive that linking. SWE-bench began with about 90,000 pull requests and kept 2,294 tasks with an issue, a fix and tests. Back episode yield over volume only where demand, cleared rights and a failed substitute line up.

Value drivers

Context rich AI training

This page covers context-rich AI training data investment: 90,000 records, 2,294 tasks.Illustrative

Key takeaways

  • A complete task episode links five parts, trigger, state, actions, outcome and check; records that cannot be joined into episodes support far fewer uses.
  • Linking is lossy: SWE-bench kept 2,294 of about 90,000 scraped pull requests, roughly 2.5 percent, and OpenAI later had developers screen a sample of those before releasing 500.
  • Context measurably changes results: Anthropic cut top-20 retrieval failures from 5.7 to 2.9 percent by adding 50 to 100 tokens of context to each chunk.
  • Volume hides repetition: one 61-word sentence appeared more than 60,000 times in C4, and 4.6 percent of C4 validation examples had a duplicate in training.
  • The thesis fails if buyers can supply the context themselves, if rights review breaks the links, or if demand stays with a few buyers.

Why could context matter more than another billion records?

Because the uses buyers pay most for, training agents and testing them, need complete task episodes, and adding unlinked records does not create episodes. A billion rows that cannot be joined to a request, a state, an outcome and a check add volume without adding the thing a model learns a task from.

Define the two assets precisely. A complete task episode links five parts: the trigger (a customer request, an issue, an order), the state at the time (the account, the policy in force, the history), the actions taken, the outcome, and a check that says whether the outcome was right. A disconnected high-volume collection is a set of records, from one system or several, that cannot be reliably joined into those five parts, because keys are missing, histories were overwritten or outcomes live somewhere else.

For a context-rich AI training data investment, the evidence that would support the thesis is specific: buyers stating a preference for linked, intention-bearing data; a measurable loss of volume when records are linked into episodes, which makes complete episodes scarce; results that improve with context at equal volume; and substitutes that fail to close the gap. The sections below take each in turn, then name what would prove the thesis wrong.

What one public benchmark reveals about episode yield

Linking records into checkable episodes discards most of them, even in a tidy public source. SWE-bench, the software engineering benchmark Jimenez and colleagues published in 2023, is the clearest public measurement of that loss.

Its authors started from about 90,000 pull requests scraped from 12 popular open-source Python repositories and kept 2,294 task instances, each pairing a real GitHub issue with the merged fix and tests that fail before the fix and pass after it. By our arithmetic that is an episode yield of roughly 2.5 percent, or about 39 raw pull requests for every usable task. The project's collection code makes the tiers explicit: pull requests with an associated issue and a reference patch are flagged as usable for fine-tuning, while only those that also carry tests become candidates for evaluation.

A second filter followed. In August 2024 OpenAI had 93 software developers experienced in Python review 1,699 random samples from the SWE-bench test set, screening for issue descriptions too underspecified to define the problem and unit tests too narrow to accept valid fixes, and released a 500-sample subset called SWE-bench Verified. Even linked episodes, in other words, can lack the context a fair task needs.

Code repositories are unusually orderly: issues, commits and tests sit in one system with stable identifiers. Business archives spread the same five parts across a CRM, a ticketing tool, a billing system and email, so there is little reason to expect a higher yield there. The scarce asset is the joined episode, not the row.

Horizontal bar chart on a linear scale from zero: about 90,000 pull requests scraped, 2,294 tasks with an issue, a fix and tests, and a 500-sample verified subset drawn after expert review of 1,699 samples.
From raw records to checkable tasks in SWE-bench. Source: Jimenez et al., 2023; OpenAI, 2024. Counts as published.

Episodes and record piles compared, use by use

The difference shows up in what each asset can be used for, not in its size. The comparison holds the subject matter constant and changes only whether the records are linked into episodes.

Episodes and record piles compared, use by use
Question a buyer asksDisconnected high-volume collectionComplete task episodes
What is counted?Rows, tokens or filesTasks with a trigger, state, actions, outcome and check
Can it train a multi-step task?Only isolated steps, with no record of what came nextYes: each step sits in sequence with its result
Can it build an evaluation?Rarely, because there is no outcome or check to grade againstYes, when the check is reliable, as with SWE-bench's fail-to-pass tests
How much of the volume is new information?Unknown until deduplicated; repetition can be largeBounded by distinct tasks, which are counted directly
What substitutes compete?Public web text and synthetic text at very large scaleBuyer-built environments, commissioned experts and the buyer's own records
What does rights review touch?Usually one system's termsEvery linked system's terms, so more review per unit

Does adding context change model results, or only the story?

It changes results in at least one measured setting, and research on repetition shows why raw volume overstates information. Neither finding proves the investment case, but both point the same way.

Anthropic's September 2024 work on contextual retrieval found that splitting documents into chunks often strips the context needed to find the right passage. Prepending 50 to 100 tokens of chunk-specific context before indexing reduced the top-20 retrieval failure rate from 5.7 percent to 2.9 percent, a 49 percent cut, and to 1.9 percent, a 67 percent cut, when combined with reranking. The underlying text was identical; only its tie to the surrounding document changed.

On the volume side, Lee and colleagues reported in 2021 that standard language-model training sets hold many near-duplicates: one 61-word English sentence appeared more than 60,000 times in C4, and 4.6 percent of C4 validation examples had a duplicate in the training set. After deduplication, models emitted memorized text about ten times less often and reached the same or better accuracy in fewer training steps. Another billion records can be another billion copies.

Buyers say something similar in their own terms. OpenAI's data partnerships page says it is especially interested in data that expresses human intention, such as long-form writing or conversations rather than fragments, and in datasets not already easily accessible online. Intention lives in sequence and context; a fragment seldom carries it.

A context-rich AI training data investment thesis map

Five linked claims make up the thesis, and each is tested on the three axes that decide any data market: observed demand, supply whose rights can be cleared, and a substitute that might do the job more cheaply. The last column names the observation that would falsify the claim.

The map is built to be updated. Rows move as buyers publish requests, as archives are audited for episode yield and as substitutes improve, and a company pitching context-rich data should be able to say which row its own evidence moves, and in which direction.

A context-rich AI training data investment thesis map
ClaimDemand evidenceRights-cleared supply testCredible substituteFalsified if
Buyers pay for linked episodes over raw volumeOpenAI seeks data expressing human intention, not fragments, and not already onlineCount episodes, not rows, in a sample the supplier may licenseCommissioned experts writing tasks to orderWritten requests specify row counts and accept unlinked records at the same price
Complete episodes are scarce even where records aboundSWE-bench kept 2,294 of about 90,000 pull requestsMeasure the join rate across every system an episode needsSimulated environments that generate episodesAudited archives join at high rates with outcomes and checks intact
Context improves results at equal volumeContextual retrieval cut top-20 retrieval failures by 49 percentPilot linked against unlinked records from the same licensed scopeBuyers adding context from their own documents at inferenceA matched pilot shows more unlinked rows beat fewer episodes at the same budget
Episodes enable evaluation that volume cannotAnthropic advises tasks two experts would grade the same way, with reference solutionsConfirm each episode carries a check the buyer acceptsPublic benchmarksBuyers evaluate only on public sets and decline private episodes
Rights survive the linkingNo public demand signal; this is a supply constraintRun rights review on the joined episode, not each table aloneOpenly licensed public dataDe-identification breaks the join keys, so cleared data stops being episodic

Plenty of records, few buyers, and context the buyer already holds

The most serious counterargument is that volume is plentiful, purchasing is concentrated, and the context that matters most may already sit inside the buyer. Each part has evidence behind it.

Volume is plentiful. Epoch AI's 2024 study estimated the effective stock of quality-filtered, repetition-adjusted public human text at about 300 trillion tokens, with full use projected between 2026 and 2032 on current trends. Purchasing is concentrated: Reddit's Form 10-K for fiscal 2024 says that "substantially all of the contract value associated with our licensing revenue is derived from two of our partners." A large stock of data coexisting with very few paying buyers is the alternative explanation every context pitch has to beat.

The third part is the sharpest. Contextual retrieval works because a buyer adds context from its own documents. If the episodes a model needs are the buyer's own, the buyer can link them internally and pay no outside supplier. The thesis survives only for context the buyer cannot reach: other companies' operating episodes, decisions taken under different policies, outcomes the buyer never observed. Ask a supplier to show that its episodes come from outside the buyer's walls and could not be assembled from the buyer's own systems.

How to measure episode yield before backing a supplier

Measure the share of raw records that become complete, checkable episodes, by system and by link, with fixed denominators. The design can be run on any archive; SourceX has not published results from it, and the same definitions apply to every company in this market.

Join losses compound. If an episode needs four links and each joins 80 percent of the time independently, only about 41 percent of episodes survive, because 0.8 to the fourth power is 0.41. The numbers are illustrative, but the arithmetic explains why a supplier that quotes only its best join rate overstates its supply.

Report each stage with its count, the share lost and the reason, and keep records still under review in the denominator as open rather than dropping them. Run the steps in this order:

Checklist · 6 items

  1. Fix the episode schema: name the five parts and the field that proves each, for example a ticket ID for the trigger and a billing adjustment for the outcome.
  2. Draw the sample frame by system and date range before looking at any join rate, and record how records entered it.
  3. Measure the join rate for each link separately, then the share of triggers with every link intact.
  4. Check the check: confirm that the outcome field or test can grade an answer, and log ambiguous cases separately, as SWE-bench Verified's reviewers did.
  5. Deduplicate before counting, so repeated records inflate neither the numerator nor the denominator.
  6. Report episode yield as complete episodes divided by raw triggers, with an interval and the count of cases still open.

0 of 6 ticked. Tick each item as you go.

Signals that would sink the context thesis

Five developments would weaken the thesis, and each comes with a signal an investor can watch for.

  • Buyers build context themselves. Retrieval over their own documents and longer context windows may cover most needs, leaving outside episodes a niche; watch for buyer requests that drop external episode data.
  • Simulated environments get good enough. If generated episodes pass buyers' acceptance tests, the scarcity premium for real ones falls; watch for evaluations that accept synthetic tasks in place of real ones.
  • Rights review breaks the links. De-identification that removes join keys or free-text references can turn an episode back into a pile of rows; watch post-review join rates.
  • Demand stays with a few buyers. Reddit's filing shows how concentrated licensing revenue can be, which caps the pricing power of any supplier; watch the count of distinct buyers.
  • Benchmarks are not businesses. SWE-bench measures code; yields and value in sales, claims or field service may differ, and no public study measures them yet.

Five numbers to demand from a context-rich data company

Ask for episode counts with denominators and for the evidence behind each row of the thesis map. The same requests apply to SourceX.

SourceX makes licensed AI datasets out of business records that companies choose to release, and it does the work around them: sourcing and qualification, rights review, contracting, preparing the data, delivering it and payment. It stocks nothing for AI buyers; each dataset is sourced from U.S. businesses after a request, and no data leaves the supplying company until it has approved the scope and terms and signed. Episode-yield and demand figures for SourceX are unpublished. The company shares company and commercial metrics with investors directly, and its capital raise, under Rule 506(c), is open to accredited investors, who can request materials through its investors page.

  • Episode yield on the last archive assessed, with the sample frame and the count of cases still open.
  • The join rate of the weakest link, and whether fixing it is technical or contractual.
  • The number of written buyer requests that specified linked episodes rather than volume, and how many distinct buyers sent them.
  • The share of delivered episodes a buyer could not have assembled from its own systems.
  • The join rate after de-identification, compared with the rate before it.

Where the numbers come from and what they leave out

Sources were checked on 2026-10-26 on each publisher's own domain: the SWE-bench paper and repository, OpenAI's SWE-bench Verified announcement and data partnerships page, Anthropic's contextual retrieval and evaluation guides, Lee et al. (2021), Epoch AI (2024) and Reddit's FY2024 Form 10-K.

Our arithmetic: 2,294 divided by about 90,000 is roughly 2.5 percent, or about 39 pull requests per task. No yield is computed for SWE-bench Verified, because the material verified for this article does not state how many of the 1,699 reviewed samples passed before the 500 were released. The 80 percent join example is illustrative, not measured.

Not measured: episode yields in business archives, prices for linked against unlinked data, buyer counts, and any SourceX result. The thesis map and the five-part episode definition are editorial frameworks that a reader can apply and dispute.

Frequently asked questions

What is context-rich AI training data?

Data in which each example keeps the surrounding facts a model needs to learn a task: the request, the state at the time, the actions, the outcome and a way to check it. A single support message is a record; the ticket with its account history, replies, refund and reopen status is an episode.

Is more data still better for AI training?

For broad pretraining, scale still matters, and Epoch AI estimated about 300 trillion tokens of usable public text in 2024. For agents and evaluations, unlinked volume adds less, because repeated or context-free records cannot show a model how a task unfolds or whether the result was right.

How do you measure whether a dataset has enough context?

Count complete episodes rather than rows. Define the parts an episode needs, measure how often records join across each link, confirm the outcome can grade an answer, deduplicate, and report complete episodes as a share of raw triggers with the open cases shown.

Why is SWE-bench relevant to business data investors?

It is a rare public record of how many raw records survive linking into checkable tasks: about 90,000 pull requests yielded 2,294 tasks. Business archives spread episodes across more systems, so the benchmark is a useful reference point, not a measurement of any business.

Does SourceX publish episode yield figures?

No. SourceX has not published episode-yield, demand or commercial figures and shares company and commercial metrics with investors directly. Readers evaluating SourceX as a company can request investor materials at sourcex.si/investors, where any offering is limited to verified accredited investors.

Sources

  • Jimenez et al., SWE-bench (2023): about 90,000 pull requests scraped from 12 popular open-source Python repositories were filtered to 2,294 task instances pairing a GitHub issue with its fix and fail-to-pass tests. Source
  • The SWE-bench repository lists 2,294 test task instances and a 500-instance Verified split, and its collection code flags instances with an issue and gold patch as usable for fine-tuning and only those with tests as evaluation candidates. Source
  • OpenAI (August 2024) had 93 software developers experienced in Python review 1,699 random SWE-bench test samples for underspecified issues and overly narrow tests, and released a 500-sample verified subset. Source
  • Anthropic's contextual retrieval (September 2024): adding 50 to 100 tokens of chunk-specific context cut the top-20 retrieval failure rate from 5.7 to 2.9 percent (49 percent) and to 1.9 percent (67 percent) with reranking. Source
  • Lee et al. (2021): a 61-word sentence repeats over 60,000 times in C4; 4.6 percent of C4 validation examples have a training duplicate; deduplicated models emit memorized text ten times less often and need fewer training steps. Source
  • OpenAI's data partnerships page seeks large-scale datasets not already easily accessible online and is particularly interested in data that expresses human intention, such as long-form writing or conversations rather than fragments. Source
  • Epoch AI (2024) estimates the effective stock of quality, repetition-adjusted public human text at about 300 trillion tokens, fully used between 2026 and 2032 on current trends. Source
  • Reddit's Form 10-K for fiscal 2024 says substantially all of the contract value associated with its licensing revenue is derived from two of its partners. Source
  • Anthropic's guide to agent evaluations says a good task is one where two domain experts would independently reach the same pass or fail verdict, and recommends a reference solution for each task. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies