AI data market
Usable AI data scarcity: five filters between abundance and shortage
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Usable AI data scarcity and data abundance coexist because they are counted at different filters. Stored bytes are plentiful: Common Crawl's open archive alone exceeds 10 petabytes. Scarcity appears after four more filters: rights to use the records, fit to a task a buyer pays for, quality after filtering, and evaluation coverage with defined success criteria. The investable question is which filter binds, for which buyer, and what purchase evidence shows it.
Value drivers
Usable AI data scarcity
Key takeaways
- Abundance and scarcity are measured at different filters: bytes, rights, task fit, quality and evaluation coverage, each with its own unit.
- Public web text is abundant in bytes and runs short at task fit and evaluation coverage; private business records pass task fit more often and run short at rights.
- Evaluation coverage looks scarce on both sides: GDPval spans 44 occupations against 867 in the 2018 occupational classification, and fresh test items expose overfitting.
- Weak demand can mimic scarcity; Reddit's 2024 annual report attributes substantially all licensing contract value to two partners.
- Ask any data company which filter it clears, how it counts, and whether the buyer's acceptance test, not the seller, defines usable.
Scarce or abundant? It depends which filter you count at
Both claims are true because they count different things. Measured in stored bytes, data is abundant and growing; measured as records that may be used, fit a paid task, survive quality filtering and come with a way to score the result, usable AI data is scarce for many tasks.
The confusion misleads investors in both directions. A pitch that cites the volume of data in the world proves nothing about usable supply, and a pitch that cites a data shortage proves nothing about demand for one particular dataset. Common Crawl has kept an open repository of web crawl data since 2008 that now exceeds 10 petabytes, publishing crawls about once a month that each typically contain more than two billion pages, yet most of that text cannot show a workflow buyer how a claim was decided or a fault was fixed.
The useful question is narrower: for a named buyer task, which filter removes most of the candidate data, and is that filter tightening or loosening? This article separates five filters, maps a falsifiable thesis for each with dated evidence, tests the strongest alternative explanation and lists what would prove the scarcity thesis wrong.
Five filters between stored bytes and a usable dataset
The five filters are bytes, rights, task fit, quality and evaluation coverage, and each has its own unit; mixing units is how abundance and scarcity get argued past each other. Count at the filter that matches the buyer's decision, and report what was lost at each step rather than only what survived.
The filters apply in this order, each counted in its own unit:
- Bytes: data stored anywhere, counted in volume. Abundance here is real and says nothing about use.
- Rights: records that someone with authority may let a model developer use for a stated purpose, counted as records covered by documents.
- Task fit: records that show the request, steps, decision and outcome of a task a buyer will pay to improve, counted as complete examples per task.
- Quality: examples that survive deduplication, filtering and consistency checks, counted after the filters rather than before.
- Evaluation coverage: tasks with defined inputs and success criteria that a model has not already seen, counted per task cell.
The five-filter thesis map, with evidence for and against
Scarcity binds at different filters for public text and private records, and the map shows where. Public web text clears the byte filter easily and loses most at task fit and evaluation coverage; private business records clear task fit more often and lose most at rights. Evaluation coverage is scarce on both sides, which makes it the strongest candidate for a durable shortage.
The reads in the last column are editorial judgments as of October 26, 2026, not measurements. Each row states the evidence that would disprove it:
| Filter | Scarcity thesis | Evidence now, dated | Disproved if | Read: public text / private records |
|---|---|---|---|---|
| Bytes | Volume is not the limit for any major text task | Common Crawl: more than 10 petabytes collected since 2008 | A buyer task fails for lack of stored data rather than permission or fit | Abundant / Abundant |
| Rights | Permission removes a large share of candidate data | Data Provenance audit (2024): license omission over 70 percent and errors over 50 percent on popular hosting sites | Audits find license metadata complete and accurate for most datasets | Mixed / Scarce |
| Task fit | Little open data records the decisions and outcomes of paid work | OpenAI's data partnerships page seeks data not easily accessible online | Open sources pass workflow buyers' acceptance tests at scale | Scarce / Mixed |
| Quality | Filtering keeps only part of the raw pool | DCLM (2024): model-based filtering was key to a strong set from a 240 trillion token pool; Epoch (2024): about 300 trillion effective tokens | Unfiltered pools train models as well as filtered sets | Mixed / Mixed |
| Evaluation coverage | Unseen tasks with success criteria are the scarcest layer | GDPval (2025): 44 occupations; 2018 SOC: 867. GSM1k (2024): drops up to 8 percent on fresh items | Fresh, scored tasks exist for most occupations buyers target | Scarce / Scarce |
Why a petabyte archive can license as zero
Storage and permission are independent: an archive can be enormous and still yield nothing a model developer may use. The volume test and the rights test answer different questions, and the second is usually the one that fails.
Public datasets show how often rights information is missing or wrong even where it should be easiest to find. After tracing more than 1,800 text datasets, the Data Provenance Initiative reported in Nature Machine Intelligence in 2024 that widely used hosting sites left the license off over 70 percent of the time and showed the wrong one over 50 percent of the time; the correct licenses were often more restrictive than what the sites displayed. A buyer that relies on a hosting label inherits that error.
Private records start from a different place. They are rarely mislabeled because they are rarely labeled at all: every outside use needs someone with authority to approve it under the company's contracts and customer notices. That is why the rights filter reads Mixed for public text and Scarce for private records in the map, and why a large private archive is a supply claim only after its rights evidence is counted.
Task fit and quality shrink public text faster than volume suggests
Most public text fails a workflow buyer's task-fit test before quality filtering starts, and quality filtering then narrows what remains to the part that actually improves training. That is how a large token count and a shortage of usable examples can describe the same corpus.
Quality filters are where raw abundance becomes measured scarcity. The DataComp-LM project assembled a pool of about 240 trillion tokens from Common Crawl and found model-based filtering central to building a strong training set; a 7-billion-parameter model trained on 2.6 trillion tokens of its filtered DCLM-Baseline set reached 64 percent five-shot accuracy on MMLU. Epoch AI's 2024 estimate of the effective stock of public human text, about 300 trillion tokens, is already adjusted for quality and repetition, and Epoch projects that stock fully used between 2026 and 2032 on current trends.
Task fit is the filter public text cannot pass for workflow capabilities. OpenAI's data partnerships page seeks large-scale data that is hard for the public to find online today, and it turns away sensitive personal information and anything belonging to a third party. A ticket thread with its resolution, a claim file with its decision or a code review with its outcome is the kind of example a workflow buyer wants, and such records live in company systems rather than on the open web.
Evaluation coverage: the shortage that outlasts the others
Evaluation coverage is the filter most likely to stay scarce, because a usable test needs defined inputs, a success criterion and items the model has not already seen, and business records rarely arrive with all three. Anthropic's engineering guidance on agent evaluations treats each task as one test with fixed inputs and a stated bar for success, and because outputs vary it recommends several trials per task.
Coverage is thin even in strong public benchmarks. GDPval, which OpenAI published in 2025, spans 44 occupations chosen from the nine sectors that each contribute more than 5 percent of U.S. GDP, while the 2018 Standard Occupational Classification lists 867 detailed occupations. GDPval was designed around knowledge work in leading sectors, not to cover every job, which is the point: most occupations have no public, expert-scored task set.
Freshness is the other half. When researchers built GSM1k, new grade-school math problems written to mirror the established GSM8k benchmark, some model families scored up to 8 percent lower on the fresh items, while many frontier models showed little sign of overfitting. A test that has leaked into training data loses its value as evidence, so evaluation supply has to be renewed, not assembled once.
Scarcity or a thin market? Purchase evidence decides
Weak demand can mimic scarcity, and the two predict different evidence. If usable records were truly scarce, buyers would compete for them through more buyers, firmer terms for sellers and repeat purchases; if demand is thin, usable records sit unsold while a few buyers set the terms.
Public filings show demand that is real but concentrated. Reddit's annual report for 2024 reports that, so far, two partners have provided substantially all of the contract value associated with its licensing revenue, and it cautions that those deals might lapse at renewal or be renewed on less favorable terms. Two buyers can sustain a market for one large licensor; on their own they cannot prove a broad shortage.
The scarcity thesis survives only if purchase evidence moves with it: more distinct buyers issuing written requests for whatever clears the scarce filter, terms that improve for sellers who clear it, and repeat orders after acceptance. If usable, rights-cleared, scored records exist and still do not sell, the binding constraint is demand, and no amount of supply work will fix it.
What could make the usable-scarcity thesis wrong
Five developments would weaken the thesis or move the binding filter, and each can be watched.
- Synthetic data and data efficiency. Epoch AI's 2024 study names synthetic data, other modalities and data-efficiency gains as likely routes past the public-text limit; if they work for workflow tasks, task-fit scarcity eases.
- Concentrated demand. Reddit's 2024 annual report attributes substantially all licensing contract value to two partners; a market with few buyers caps what scarcity is worth.
- Better provenance. If license metadata on public datasets becomes complete and accurate, the rights filter loosens for public data, though not for private records.
- Buyers write their own tests. Developers may turn their own logs into evaluation tasks, making evaluation scarcity an internal cost rather than a purchase.
- Sellers define usable. When the company selling data also sets the filters, scarcity claims inflate; the buyer's written acceptance test is the only definition that counts.
Questions that test any usable-data claim, SourceX included
Ask which filter a company claims to clear, how it counts, and who defined usable. A credible answer names the unit at each filter and the share of candidate records lost there.
SourceX's model reflects the same distinction. Companies that license selected business records to AI developers use it for the whole chain, from sourcing and qualification through rights review and contracting to preparation, delivery and payment. Buyer datasets are never pre-built; each is sourced from U.S. businesses once requested, so usability is set by a buyer's task rather than by a stock count. Filter-level yields and buyer metrics for SourceX are unpublished; it provides company and commercial metrics to investors directly. Its raise, under Rule 506(c), is for accredited investors, who can request materials through its investors page.
Five questions to put to any data company:
- At which filter does your supply most often fail, and what share of candidate records is lost there?
- Who defines usable: your own checklist or the buyer's written acceptance test?
- What share of records offered is covered by documents showing authority to license?
- How many evaluation tasks with defined success criteria have you delivered, and how do you keep them out of training data?
- How many distinct buyers issued written requests in the past year, and how many bought again?
Methods note: sources and limits of the five-filter map
The five filters, the thesis map and the reads in its last column are editorial frameworks assembled on October 26, 2026. Evidence comes from primary sources checked on their publishers' own domains: Common Crawl's about page, the Data Provenance Initiative's audit, the DataComp-LM paper, Epoch AI's 2024 data-stock study, OpenAI's data partnerships and GDPval pages, the Bureau of Labor Statistics occupational classification, the GSM1k paper, Anthropic's guidance on agent evaluations and Reddit's annual report for 2024.
Not measured: the share of private business records lost at each filter, which no public source reports; the reads in the map are judgments, not scores; and the comparison of 44 GDPval occupations with 867 classified occupations counts categories, not tasks or economic weight. No SourceX supply, buyer or acceptance data was used.
Frequently asked questions
Has AI run out of training data?
Not in bytes. Epoch AI estimated in 2024 that the effective stock of quality, repetition-adjusted public human text is about 300 trillion tokens and projected it fully used between 2026 and 2032 on current trends. What runs short first is usable data for specific tasks: records with permission, a fit to the task and a way to score the result.
If data is so abundant, why do AI developers pay for it?
Because they pay for what passes the later filters, not for volume. OpenAI's data partnerships page, for example, asks for datasets that are not easily accessible online. Records that show real decisions and outcomes, with authority to use them, are scarce even when stored data is not.
What is the difference between data abundance and usable data?
Abundance counts stored data. Usable data counts records a developer may use for a stated purpose, that fit a paid task, that survive quality filtering and that can be scored. A corpus can be large at the first count and small at the last.
Is evaluation data scarcer than training data?
For many business tasks, yes. A usable evaluation task needs defined inputs, a success criterion and items models have not seen, and leaked tests lose their value; GSM1k found drops of up to 8 percent for some model families on fresh items. Evaluation sets therefore have to be renewed.
Does SourceX keep a stock of datasets to sell?
No. It builds nothing in advance: each buyer dataset is sourced from U.S. businesses after the buyer uses the request form to set out scope, volume, language, format, timing and licensing needs. Supply and buyer metrics for SourceX are not published.
Sources
- Common Crawl maintains an open repository of web crawl data collected since 2008, totaling more than 10 petabytes, with roughly monthly crawls of more than two billion pages each. Source
- The Data Provenance Initiative's audit of more than 1,800 text datasets (Nature Machine Intelligence, 2024) found license omission above 70 percent and error rates above 50 percent on widely used dataset hosting sites. Source
- DataComp-LM (2024) built a pool of about 240 trillion tokens from Common Crawl and found model-based filtering key; a 7B model trained on 2.6 trillion tokens of DCLM-Baseline reached 64 percent five-shot MMLU. Source
- Epoch AI (2024) estimates the effective stock of quality, repetition-adjusted public human text at about 300 trillion tokens, fully used between 2026 and 2032 on current trends, and names synthetic data, other modalities and data efficiency as routes beyond it. Source
- OpenAI's data partnerships page seeks large-scale datasets not already easily accessible online and is not seeking sensitive or personal information or information that belongs to a third party. Source
- Anthropic's guidance on agent evaluations defines a task as a single test with defined inputs and success criteria and recommends multiple trials per task because outputs vary. Source
- OpenAI's GDPval (2025) spans 1,320 tasks across 44 occupations chosen from the nine sectors that each contribute more than 5 percent of U.S. GDP. Source
- The 2018 Standard Occupational Classification contains 867 detailed occupations. Source
- GSM1k, a fresh benchmark mirroring GSM8k, found accuracy drops of up to 8 percent for some model families, while many frontier models showed minimal signs of overfitting. Source
- Reddit's annual report for 2024 says that, to date, substantially all of the contract value associated with its licensing revenue is derived from two of its partners, and that the arrangements may not be renewed or may be renewed on less favorable terms. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.