AI data market
Proprietary dataset replacement analysis: three routes a buyer tries first
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A proprietary dataset is only as defensible as its cheapest working substitute. Run three replacement tests in the order a buyer would: a public dataset with a usable license, synthetic records from a current model, and a direct license from the original record holders. The first route that passes the buyer's own acceptance test caps what the dataset can command; if all three fail, the moat claim survives for now.
Value drivers
Proprietary dataset
Key takeaways
- The cheapest route that passes the buyer's acceptance test, not the data company's own valuation, sets the ceiling on a proprietary dataset's price.
- Public data usually fails on legal scope before volume: an audit of more than 1,800 datasets found license information missing for more than two-thirds on popular hosting sites.
- Synthetic data carries real weight, with NVIDIA reporting over 98 percent synthetic alignment data for Nemotron-4 340B, but it imitates what a generator knows and loses rare cases when used carelessly.
- Direct sourcing is the route investors skip: large sources license to AI developers themselves, so an intermediary's dataset is defensible mainly where holders are too many or too small to reach one by one.
- The strongest objection is seed-and-synthesize: a small licensed sample expanded with a model may replace a large proprietary set, so test that blend as a fourth arm.
What does a proprietary dataset replacement analysis decide?
It decides how much of a dataset's value survives once a buyer looks for a substitute. Every proprietary dataset competes with three replacements, a public dataset, synthetic records and a direct license from whoever holds the original records, and the cheapest one that passes the buyer's own acceptance test sets the most the dataset can command.
Call that the replacement ceiling. A dataset with a low ceiling may still sell, but at a price set by its substitute rather than by its uniqueness, and an investor who values it as a moat is paying for a premium buyers will not pay. A dataset whose three routes all fail has a high ceiling, at least until one of the routes improves.
The analysis needs four kinds of evidence for each route: technical utility on the buyer's task, legal scope for the buyer's use, replacement cost in time and steps, and repeat buyer behavior, meaning whether buyers who tried the substitute came back. The sections below take the routes in turn and then combine them in a scorecard with a disconfirming result for each.
Run the routes in the order a buyer pays for them
Test public data first, synthetic data second and direct sourcing third, because that is roughly the order of cost and effort for a buyer, and the first route that passes ends the search. A data company that has only been compared with the hardest route has been graded on an easy curve.
Each route needs a pass rule written before the test: the acceptance test the buyer would actually use, on a held-out set drawn from the buyer's own work, with the same model, budget and prompts for every arm. The rule should name a margin, so that a substitute within a stated distance of the proprietary arm counts as passing. Without a written margin, every comparison drifts toward the result its sponsor prefers.
Exclusivity belongs in a separate analysis. A license can stop one supplier from selling the same records to anyone else; it cannot stop any of the three routes on this page, which is why the replacement analysis comes first.
Test one: is there a public dataset that works and may be used?
A public replacement passes only when it does two things: it reaches the proprietary arm's score on the buyer's held-out task, and its license actually permits the buyer's use. Most public candidates fail the second condition before the first.
Volume is rarely the problem. Common Crawl has kept an open repository of web crawl data since 2008, more than 10 petabytes in total, and publishes crawls about once a month, each typically holding more than two billion web pages. What that volume seldom contains is the inside of a business: the ticket, the adjuster's note, the dispatch decision and what happened afterward.
Legal scope is where public data breaks. On popular hosting sites, more than two-thirds of datasets carried no license information and errors ran at 50 percent or more, according to the Data Provenance Initiative's 2024 audit of over 1,800 text datasets in Nature Machine Intelligence; the correct license was often more restrictive than the one displayed. The open web is also closing: the Consent in Crisis audit of about 14,000 web domains found that over 2023-2024, new restrictions left about 5 percent of all tokens in the C4 corpus, and 28 percent of its most actively maintained critical sources, fully restricted from AI use.
Public evaluation data needs the same care. When Scale AI researchers built GSM1k, a fresh set of grade-school math problems modeled on the public GSM8K benchmark, the current version of their paper reported accuracy drops of up to 8 percent for some models (the first version said 13 percent). A substitute that looks strong on a public benchmark may be scoring on material a model has already seen, so the held-out set has to come from the buyer's own work.
Test two: can a current model generate records that pass?
Sometimes, and more often each year, so the synthetic route has to be tested rather than dismissed. It passes when generated records match the proprietary arm on the buyer's held-out task, including the rare cases that usually carry the value.
Model developers already lean on generated data. By NVIDIA's account, over 98 percent of the data it used to align Nemotron-4 340B was generated synthetically. Meta's Llama 3.1 model card says the fine-tuning data included over 25 million synthetically generated examples, and that the model collection supports using its outputs to improve other models, including through synthetic data generation and distillation.
Two limits mark where the route fails. Shumailov and colleagues showed in Nature in 2024 that training on model-generated content without care makes the tails of the original distribution disappear, and they argue that real human-produced data matters where those tails matter. NVIDIA's own planning guide for synthetic data generation makes the second point: the seed file and prompts are where domain knowledge enters, and a run without that knowledge produces generic output drawn from the model's own training data. A generator can imitate the format of a claims file; it cannot know which claims were actually paid.
That turns the synthetic test into a test of outcomes. If the proprietary records carry ground truth the generator cannot know, such as results observed after a decision, the synthetic arm should lose on exactly those slices. If it does not lose there, the dataset was selling format, not facts.
Test three: would the buyer simply go to the record holders?
Often yes when the holders are few and large, and rarely when they are many and small. The direct route passes when a buyer could license comparable records from the original holders for less total effort than buying the assembled dataset.
AI developers already do this at the top of the market. OpenAI's data partnerships page invites organizations to share large-scale datasets that are not already easily accessible online, and it rules out sensitive or personal information and information that belongs to a third party. Its 2024 partnership with Reddit gave OpenAI access to Reddit's Data API through a contract with Reddit itself. Reddit's annual report on Form 10-K for fiscal 2024 shows how concentrated that demand can be: "substantially all of the contract value associated with our licensing revenue is derived from two of our partners."
The direct test is a count. List the holders behind the dataset, the share of records each contributes and the steps a buyer would repeat per holder: finding the company, checking its authority and customer terms, negotiating, preparing and accepting a delivery. If one or two holders supply most of the records, the buyer can go direct and the intermediary's ceiling is low. If the same coverage needs dozens of mid-sized companies, each with its own rights review, the coordination work is the moat, and it should be valued as an operating advantage rather than as ownership of data.
The replacement scorecard: four criteria across three routes
Score each route on the same four criteria and record the result that would disconfirm the moat. A route that passes on technical utility and legal scope at a lower replacement cost sets the ceiling, and repeat buyer behavior checks whether that ceiling holds in practice.
Fill every cell from evidence the company can produce, and mark a cell unknown rather than guessing. The figure draws the same logic as a decision path: the dataset reaches the defensible end only if all three routes fail.
| Criterion | Public alternative | Synthetic substitute | Direct sourcing |
|---|---|---|---|
| Technical utility | Best open dataset's score on the buyer's held-out task, against the written margin | Generated records' score, overall and on rare-case slices | Score of records licensed from the largest holders alone |
| Legal scope | License verified at the original source, not the hosting site, for the buyer's use | Generator terms allow the use, and seed data stays within its own license | Each holder can grant the rights under its customer terms and notices |
| Replacement cost | Time to filter, clean and review licenses | Time to prepare seeds, generate and validate | Holders to contact and steps per holder, in months |
| Repeat buyer behavior | Buyers who tested open data and still bought | Buyers who ran synthetic pilots and still renewed | Buyers who also hold direct deals and still renew |
| Disconfirming result | An openly licensed set reaches parity on the buyer's holdout | The synthetic arm matches on rare-case slices too | One or two holders supply most records and would deal direct |
The seed-and-synthesize objection
The strongest alternative explanation is that the dataset is useful but replaceable by a blend: a small licensed sample from one or two holders, expanded with a model into as many records as the buyer needs. If that works, the buyer pays for the seed, not the archive, and the replacement ceiling falls to the price of a sample.
The objection draws its force from the evidence above. NVIDIA's planning guide treats seed quality as the strongest lever on synthetic output, which means a buyer that can license good seeds can generate much of the rest, and Llama 3.1's model card explicitly supports using model outputs to build other models' data.
Test it with a fourth arm. License or set aside a stated fraction of the proprietary records, expand that sample synthetically to full size and compare it with the full dataset on the held-out task and its rare-case slices. If the blend matches, the value sits in a small core, and an investor should value that core and the access to it. If the blend loses where outcomes matter, the full record set is doing work a seed cannot.
What moves the replacement ceiling after the score
The scorecard is a snapshot, and four forces change it. Rescore after each major model release and whenever a large holder announces a direct deal.
- Generators improve. Each better model raises the synthetic route's score on routine slices; the Nature model-collapse result limits careless use of generated data, not careful use with real seeds.
- Public supply shifts both ways. New open releases raise the public route, while the restrictions documented in Consent in Crisis lower it, and both can happen in the same year.
- Holders learn their value. Once record holders see AI developers paying, more of them contract directly, as Reddit did with OpenAI, which lowers the cost of the direct route.
- Buyers concentrate. When two partners account for substantially all of Reddit's licensing contract value, a dataset's real market may be two or three buyers, and losing one moves the ceiling more than any substitute.
Asking SourceX and other data companies for replacement evidence
Ask every data company for the three route comparisons on a named buyer task, with the pass margin written down in advance, and ask SourceX for the same. A company that has never compared its records with a synthetic or public arm cannot yet say where its ceiling is.
SourceX works with companies that hold business records so they can license selected records to AI developers; it handles supplier sourcing and qualification, rights checks, contracts, preparation, delivery and payment. Delivery waits until the supplying company has approved scope and terms and signed, and from that point the agreement sets how far SourceX can use, sublicense and distribute the de-identified data. That structure is exactly what the direct-route test examines, since a buyer could in principle approach the same holders. Replacement tests and commercial metrics are unpublished; SourceX discloses company and commercial metrics to investors directly, and accredited investors can request materials through its investors page while it raises under Rule 506(c).
Questions to send with the request:
- Against which public dataset did you compare, and did you verify its license at the original source?
- What did a synthetic arm seeded with your own records score on rare-case slices?
- How many holders supply most of the records in your largest dataset, and could the buyer reach them directly?
- Which buyers ran a substitute pilot and still bought or renewed?
- What result would make you stop calling a dataset proprietary?
Scope of this analysis: sources, date and untested parts
This article's editorial frameworks (the replacement ceiling, the three-route order, the scorecard and the seed-and-synthesize arm) date from 2026-10-26 and lean on public sources checked that day: Common Crawl's about page, the Data Provenance Initiative audit, Consent in Crisis, the GSM1k paper, NVIDIA's Nemotron-4 340B page and its synthetic data planning guide, Meta's Llama 3.1 model card, Shumailov et al. in Nature, OpenAI's data partnerships page and Reddit partnership announcement, and Reddit's FY2024 Form 10-K.
Not measured: the replacement ceiling of any real dataset, the cost of any route in money, how often buyers choose each route, and SourceX's own datasets or deals. Recreation cost for a rights-cleared dataset is a separate analysis; this page counts steps and holders rather than pricing them.
Frequently asked questions
What is a replacement test for a dataset?
A replacement test compares a proprietary dataset with the best substitute a buyer could use instead, on the buyer's own held-out task, with a pass margin set in advance. The three substitutes worth testing are a public dataset with a usable license, synthetic records from a current model and a direct license from the original record holders.
Can synthetic data replace proprietary business data?
For routine formats and common cases, increasingly yes: NVIDIA reports that over 98 percent of Nemotron-4 340B's alignment data was synthetic. It struggles where value depends on rare cases or on outcomes the generator cannot know, and Shumailov et al. (Nature, 2024) showed that careless training on generated data erodes the tails of a distribution.
Why is public data often not a substitute even when it looks similar?
Because legal scope fails before utility does. The Data Provenance Initiative found license information missing for more than two-thirds of datasets on popular hosting sites, with correct licenses often more restrictive, and web restrictions rose sharply in 2023-2024. Verify the license at the original source before counting a public dataset as a substitute.
When can a buyer skip a data intermediary?
When a few large holders supply most of the records and will license directly, as Reddit did with OpenAI in 2024. When coverage requires many mid-sized holders, each needing its own rights review and contract, going direct costs the buyer more than buying the assembled dataset.
Has SourceX published replacement tests for its datasets?
Not yet. No replacement tests or commercial metrics from SourceX are public; the company shares company and commercial metrics with investors directly. A request through sourcex.si/investors can name a buyer task to test against, and any offering is limited to verified accredited investors.
Sources
- Common Crawl has maintained an open repository of web crawl data since 2008, totaling more than 10 petabytes, with monthly crawls typically over two billion pages. Source
- The Data Provenance Initiative's audit of more than 1,800 text datasets (Nature Machine Intelligence, 2024) found license information missing for more than two-thirds of datasets on popular hosting sites and error rates of 50 percent or more, with correct licenses often more restrictive. Source
- Consent in Crisis audited about 14,000 web domains and found that over 2023-2024 restrictions left about 5 percent of C4 tokens and 28 percent of its most actively maintained critical sources fully restricted. Source
- The GSM1k paper's current version reports accuracy drops of up to 8 percent on fresh grade-school math problems compared with GSM8K; the first version reported up to 13 percent. Source
- NVIDIA says over 98 percent of the data used in Nemotron-4 340B's alignment process was synthetically generated. Source
- Meta's Llama 3.1 model card says fine-tuning data included over 25 million synthetically generated examples and that the models' outputs may be used to improve other models through synthetic data generation and distillation. Source
- NVIDIA's synthetic data planning guide says seed files and prompts are where domain knowledge enters, that runs without it produce generic output, and that seed quality is the strongest lever. Source
- Shumailov et al. (Nature, 2024) showed that indiscriminate training on model-generated content makes the tails of the original distribution disappear. Source
- OpenAI's data partnerships page seeks large-scale datasets not already easily accessible online and excludes sensitive or personal information and information belonging to a third party. Source
- OpenAI's 2024 partnership with Reddit gave OpenAI access to Reddit's Data API through a contract with Reddit. Source
- Reddit's FY2024 Form 10-K states that substantially all of the contract value associated with its licensing revenue is derived from two of its partners. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.