Deal economics
Supplier scale data differentiation: a seven-line dilution scorecard
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
More suppliers make a dataset more defensible only when each one adds records that are new to the pool, consistent with it and hard to buy elsewhere. Otherwise volume grows while the defensible share shrinks, through overlap, uneven quality and expansion into commodity coverage. Investors should ask for marginal novelty per supplier, cross-supplier consistency and repeat buyer evidence, and test each claim against a stated way it could fail.
Checklist
Supplier scale data
Key takeaways
- Supplier count and record volume are inputs, not evidence of a moat; marginal novelty after deduplication is the number to ask for.
- Duplicated supply is cheap to substitute: Muennighoff and colleagues found up to four epochs of repeated data performed about as well as unique data in their tests.
- Uneven labeling across suppliers is a quality cost; an audit of ten benchmark test sets found label errors averaging at least 3.3 percent, enough to flip some model rankings.
- Expansion into well-covered segments adds volume a buyer can source elsewhere, while records no model has seen keep their evaluation value.
- A defensible multi-supplier dataset shows rising unique coverage, consistent definitions and repeat purchases from more than a few buyers.
Does adding suppliers make a dataset more defensible?
Only when each new supplier contributes records that are new to the pool, consistent with it and hard to obtain elsewhere. When suppliers repeat what the pool already holds, label the same events differently or extend coverage into segments anyone can buy, the dataset becomes larger and easier to replace at the same time.
That is the distinction supplier scale data differentiation turns on. A data company's pitch often counts suppliers, records and industries, because those numbers rise every quarter. Defensibility depends on a different number: the share of the pool a buyer could not rebuild cheaply from other suppliers, public data or its own records.
This framework is for an investor testing that claim. It names three dilution mechanisms, gives a seven-line scorecard with a test that could disprove each line, works an illustrative five-supplier example, and sets out the risks and the questions to bring to any multi-supplier data company, SourceX included.
Overlap: how much does each new supplier actually add?
Often less than its record count suggests, because suppliers in one industry use similar software, templates and processes, and their records repeat each other. The test is marginal novelty: the records a supplier adds that survive deduplication against everything already in the pool.
Public training corpora show how far repetition can go. Lee and colleagues found a single 61-word English sentence repeated more than 60,000 times in the C4 corpus, and overlap between training and validation data affecting more than 4 percent of the validation set in standard datasets. Models trained on their deduplicated data emitted memorized text ten times less often and needed fewer training steps to reach the same or better accuracy.
Repetition also caps what duplicates are worth. In experiments with up to 900 billion training tokens, Muennighoff and colleagues found that training on up to four epochs of repeated data changed loss negligibly compared with unique data, while with more repetition the value of added compute decayed toward zero. A buyer that already holds a record gains little from a second copy, and can reuse its own copy instead of paying for another.
Ask for marginal novelty per supplier in the order suppliers joined, measured with one deduplication rule throughout. A flat or falling curve means the supply base is getting wider, not deeper.
Inconsistent quality: why a bigger pool can be a noisier one
A bigger pool is noisier when suppliers define the same field differently, and the noise lands on labels and outcomes, where it costs most. One supplier's resolved is another's closed; one counts a return at the request, another at the refund.
Small label error rates move results. Northcutt and colleagues estimated label errors averaging at least 3.3 percent across the test sets of ten widely used benchmarks, and showed that on corrected ImageNet labels the smaller ResNet-18 outperforms ResNet-50 if the share of originally mislabeled test examples rises by just 6 percent. Pooling suppliers with different definitions builds that kind of disagreement into one dataset.
Selection can beat volume. Sorscher and colleagues showed in theory, and on image benchmarks in practice, that with a high-quality metric for pruning training examples, error can fall faster than the usual power law in dataset size. A smaller, well-selected set can therefore rival a larger unselected pool, which weakens any moat argument built on counts alone.
Ask for one consistency measure across suppliers: the share of records whose key labels map to a shared definition without manual exceptions, reported per supplier.
Commodity coverage: growth in segments a buyer can source elsewhere
Expansion dilutes defensibility when new suppliers add segments that are already well covered, because the buyer's alternative gets cheaper as the pool grows. The question for each new segment is what a buyer would spend to source it without this company.
Commodity sources improve once cleaned. The team behind the RefinedWeb dataset reported that properly filtered and deduplicated web data alone trained models that significantly outperformed publicly available models trained on The Pile, a curated corpus, in their evaluation. If cleaned public data can rival curated collections for general tasks, private supply has to earn its place on tasks public data does not cover.
Overlap with public material also erodes evaluation value. Zhang and colleagues built GSM1k, a fresh set of grade school math problems written to mirror the public GSM8k benchmark; some models scored up to 8% lower on the fresh set in the paper's current version, and several model families showed signs of systematic overfitting. Records no model has seen are what make an evaluation set worth buying, and supply that overlaps public sources loses that property.
Ask what share of the pool covers tasks, industries or exception types that public datasets and large vendors do not, and how that share moved as suppliers were added.
The supplier-scale moat scorecard, with a disconfirmation test per line
Score each line from evidence the company can show, and run its disconfirmation test before accepting the score. A line that survives its test supports the moat; a line the company cannot test is unproven, not neutral.
Use three marks rather than numbers: strong when the evidence passes its test, mixed when it passes for some suppliers or periods only, and unproven when the company cannot run the test. A moat case needs strong marks on marginal novelty and replacement cost at a minimum; the other five lines decide how durable it is.
Beside the three dilution mechanisms, the scorecard compares replacement cost, legal scope, technical utility and repeat buyer behavior. It uses the same definitions for every multi-supplier data business, so two companies can be compared line by line:
| Line | What to measure | Supports the moat when | Disconfirmation test | Evidence to request |
|---|---|---|---|---|
| Marginal novelty | Unique records each supplier adds after deduplication, in joining order | Later suppliers still add a material unique share | Deduplicate the last three suppliers against the rest; most of their records match | Per-supplier deduplication report with the matching rule |
| Cross-supplier consistency | Share of records whose key labels map to one definition without exceptions | The share holds steady as suppliers join | Re-label a random sample blind; disagreement clusters in newer suppliers | Label map, exception log and a blind re-label sample |
| Coverage scarcity | Share of the pool covering tasks or segments absent from public and vendor sources | The scarce share rises with supplier count | Search public datasets and vendor catalogs for each top segment; close substitutes exist | Segment list with the substitute search for each |
| Replacement cost | Time and effort for a buyer to rebuild comparable supply directly | Rebuilding needs many supplier relationships and rights reviews | A buyer sources a comparable slice from two or three companies within a quarter | Sourcing timelines and rights-review counts per supplier |
| Legal scope | Whether permitted uses and provenance records match across suppliers | Every supplier's records carry the same documented permitted uses | Trace ten records to their supplier agreements; terms differ or are missing | Agreement register mapped to record identifiers |
| Technical utility | Change in a buyer's holdout metric as suppliers are added | The metric improves with suppliers, not only with records | Train on the same volume drawn from fewer suppliers; the gap disappears | Volume-matched comparison on a sealed holdout |
| Repeat buyer behavior | Buyers who purchase again, and how concentrated contract value is | Repeat purchases come from more than a few buyers | Remove the top two buyers; most repeat activity disappears | Buyer-level repeat counts and a concentration table |
Illustrative: five suppliers, volume up, novelty down
Illustrative example; the suppliers and figures are invented. A data company adds five suppliers of service records in sequence and deduplicates each against the pool with one rule. Volume nearly quintuples, while the unique share of the pool falls to about half.
The fourth supplier is the largest and adds the fewest new records, because its software and customers overlap with suppliers two and three. The fifth is small but serves a niche the pool lacked, so it adds almost twice as many unique records as the fourth. A pitch that counts suppliers or records would rank them the other way round.
The scorecard's first line catches this; a record count does not. The figure plots the same five steps.
| Supplier, in joining order | Records contributed | New unique records | Pool records, cumulative | Unique share of pool |
|---|---|---|---|---|
| 1 | 100,000 | 100,000 (100%) | 100,000 | 100% |
| 2 | 80,000 | 64,000 (80%) | 180,000 | 91% |
| 3 | 120,000 | 36,000 (30%) | 300,000 | 67% |
| 4 | 150,000 | 15,000 (10%) | 450,000 | 48% |
| 5 | 40,000 | 28,000 (70%) | 490,000 | 50% |
Could a novel, consistent pool still be easy to replace?
Yes. Passing the dilution tests shows that a pool is not losing value as it grows; it does not show that a buyer cannot reach the same result another way. Useful data may still be cheaper to source directly from a few large companies, to generate synthetically, or to replace with a buyer's own records.
Scale also has a real upside the dilution case can miss. More suppliers bring more rare cases per year, more variation between companies for testing whether a model generalizes, and more chances that one supplier fills a gap. Those gains appear in the scorecard as rising scarcity and utility, so the same tests that detect dilution also credit genuine breadth.
The decision rule follows. Supplier scale supports a moat only when marginal novelty, consistency and scarcity hold up as suppliers join and replacement cost stays high. If any of those fails its disconfirmation test, the investment case rests on execution and relationships, which can be valuable but are a different thesis.
What could make the dilution thesis wrong
Four developments would weaken this framework, and each has a marker an investor can watch. Name them before the upside: a company unable to say which of the four would damage it most has not yet stress-tested its own moat.
- Buyers may pay for breadth regardless of overlap. If repeated data is nearly as useful as unique data for a buyer's task, as Muennighoff and colleagues found for up to four epochs in language model pretraining, uniqueness may earn less of a premium than this framework assumes.
- Cleaning tools may make consistency cheap. As deduplication and label-auditing methods improve, the cost of reconciling uneven suppliers falls, and consistency stops separating one pool from another.
- Demand may stay concentrated. Reddit's Form 10-K for fiscal year 2025 states that, to date, "substantially all of the contract value associated with our licensing revenue is derived from two of our partners". A multi-supplier pool selling to a few buyers carries renewal and pricing risk however distinct its records are.
- License records may not survive aggregation. When MIT Media Lab researchers audited more than 1,800 text datasets, they found license information frequently omitted or miscategorized on widely used dataset hosting sites, so a pool assembled from many suppliers can lose the provenance that makes it licensable.
Questions to ask a multi-supplier data company, SourceX included
Ask for the scorecard's evidence using your own definitions, and ask the same questions of every company in the space. SourceX is one application of the question. It helps companies make licensed AI datasets out of selected business records, managing the sourcing and vetting of suppliers, rights review, contracts, data preparation, delivery and payment, and no supplier's records are delivered until that company has approved the scope and terms and put its signature to the agreement.
Supplier, novelty and buyer metrics are not public, and SourceX supplies company and commercial metrics to investors directly. During its Rule 506(c) raise from accredited investors, materials can be requested through the investor page, and any line of the scorecard can be brought as a diligence question.
Checklist · 6 items
- What is marginal novelty per supplier, in joining order, under one stated deduplication rule?
- What share of records maps to a shared label definition without manual exceptions, by supplier?
- Which segments in the pool have no close public or vendor substitute, and how was that checked?
- How does a buyer's holdout metric change with more suppliers when record volume is held constant?
- Can every record be traced to a supplier agreement that states its permitted uses?
- How many buyers have purchased more than once, and what share of contract value comes from the top two?
0 of 6 ticked. Tick each item as you go.
Methods note: how the dilution scorecard was built
The three dilution mechanisms, the seven-line scorecard and the five-supplier example are editorial frameworks written for this article on 2026-10-26. Their evidence comes from machine learning research (Lee et al. 2021, Northcutt et al. 2021, Sorscher et al. 2022, Muennighoff et al. 2023, Penedo et al. 2023 and Zhang et al. 2024), the MIT Media Lab dataset licensing audit and Reddit's Form 10-K for fiscal year 2025.
Not measured: marginal novelty, consistency, scarcity or repeat purchasing at SourceX or any other company, and the price effect of uniqueness in private data deals. The research cited studies public corpora and benchmarks rather than licensed business records, so it indicates direction, not magnitude, for private supply. The example is illustrative and describes no real company.
Frequently asked questions
What makes a proprietary dataset defensible?
A dataset is defensible when a buyer cannot cheaply rebuild comparable records from other suppliers, public data or its own systems, and when the records' permitted uses are documented. Exclusivity in a contract helps only if the underlying records are hard to replace. Volume on its own is weak evidence.
Is a data moat stronger with more suppliers?
Only if each supplier adds records that are new to the pool, consistent with it and scarce elsewhere. Overlapping suppliers, uneven labeling and expansion into well-covered segments make a pool larger but easier to replace. Ask for marginal novelty per supplier rather than supplier counts.
How do you measure overlap between data suppliers?
Deduplicate each supplier's records against the existing pool with one fixed rule, exact matching for structured records and near-duplicate matching for text, and report the unique share each supplier adds in joining order. Keep the rule constant so the trend is comparable. Studies of public corpora, such as Lee and colleagues' work on C4, show how much repetition such checks can surface.
Why do label differences between suppliers matter to an AI buyer?
Labels and outcomes are what models learn from and evaluations score, so inconsistent definitions become noise in the most expensive place. Northcutt and colleagues found label errors averaging at least 3.3 percent across ten benchmark test sets, enough to change some model rankings. A multi-supplier pool needs a shared label map and an exception log.
Does SourceX publish supplier or buyer metrics?
No. Supplier, novelty and buyer metrics are unpublished; investors receive company and commercial metrics from SourceX directly. The company is raising capital from accredited investors under Rule 506(c), and sourcex.si/investors takes requests for materials.
Sources
- Lee et al. (2021) found a 61-word sentence repeated over 60,000 times in C4 and train-validation overlap affecting over 4 percent of validation sets; deduplicated training cut memorized output tenfold with fewer training steps. Source
- Muennighoff et al. (2023) found up to four epochs of repeated data changed loss negligibly versus unique data, with the value of added compute decaying toward zero beyond that, in runs up to 900 billion tokens. Source
- The RefinedWeb paper (Penedo et al., 2023) reports that properly filtered and deduplicated web data alone trained models that significantly outperformed public models trained on The Pile. Source
- The current version of the GSM1k paper (Zhang et al., 2024) reports accuracy drops of up to 8% versus GSM8k, with several model families showing systematic overfitting. Source
- Reddit's Form 10-K for fiscal year 2025 states that, to date, substantially all of the contract value associated with its licensing revenue is derived from two of its partners. Source
- The MIT Media Lab audit of more than 1,800 text datasets found license information frequently omitted or miscategorized on widely used dataset hosting sites. Source
- Northcutt et al. (2021) estimated label errors averaging at least 3.3 percent across ten benchmark test sets and showed small changes in mislabeled prevalence can flip model rankings. Source
- Sorscher et al. (2022) showed that with a high-quality data pruning metric, error can scale better than a power law in dataset size, in theory and on image benchmarks. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.