Skip to content

Deal economics

Data business moat scorecard: grade the evidence, not the pitch

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A data business moat scorecard should grade evidence, not adjectives. Score eight lines across four families: whether a buyer could get the same lift elsewhere, whether access to the records will last, whether the data lifts a held-out task, and whether buyers come back. Grade each line 0 to 3 by the strongest evidence that survived a test designed to disprove it, and let weak substitutability or access veto the verdict.

Value drivers

Data business moat scorecard

This page covers data business moat scorecard: grade the evidence, not the pitch.Illustrative

Key takeaways

  • A moat claim is only as strong as the evidence that survived an attempt to knock it down, so grade that evidence, not the claim.
  • Substitutability and access durability are gates: if either fails, the business may still be good, but its data is not what defends it.
  • Unknown scores zero, which keeps two companies comparable on one sheet instead of rewarding the vaguer pitch.
  • Synthetic data and buyers' own sourcing are the strongest alternative explanations; test against the best substitute, not the weakest.
  • Buyer concentration caps execution: Reddit's annual report for 2024 ties substantially all of its licensing contract value to two partners.

What should a data business moat scorecard measure?

It should measure whether a buyer or a rival could get the same value without this company, and how sure you can be of the answer. Four families cover that question for any business that sells or licenses data: substitutability, access durability, utility and execution.

Substitutability asks whether a buyer could get the same lift from public, synthetic, self-collected or competing data. Access durability asks whether the company's right and route to the records will survive a terms change, a supplier exit or a renewal. Utility asks whether the data improves a buyer's task on a test the seller never saw. Execution asks whether buyers come back and whether delivery gets cheaper with repetition.

Most moat discussions stop at the families and score them on conviction. This scorecard adds two things a fund partner can audit: an evidence grade for every line, and a disconfirmation test that names the result that would knock the line down. Together they put two data businesses on one sheet with the same rules.

Grade the evidence, not the claim: the 0-to-3 scale

The grade measures how much you know, not how strong the company says its advantage is, and no amount of management conviction can raise it. A line earns credit only for evidence that has already faced its disconfirmation test.

Unknown scores zero. The rule feels harsh, but it is what keeps the sheet comparable: a vague pitch cannot outscore a specific one, and a company that declines to share a cohort table earns the same grade as one whose cohort table failed.

Fix the comparison before scoring anyone. Use the same window, such as the trailing four quarters, the same units, accepted datasets and signed buyers, and the same named substitute for every company, and record who produced each piece of evidence and when. A grade earned on last year's cohort says little about this year's supply.

Every line takes the highest grade whose condition its evidence meets:

  • 0: unknown, asserted without documents, or failed its disconfirmation test.
  • 1: supported by company documents such as signed agreements or logs, but not measured against a fixed denominator.
  • 2: measured by the company with fixed denominators and dated cohorts, and the disconfirmation test was run and passed.
  • 3: confirmed by someone other than the company, such as a buyer's acceptance record, an independent evaluation or counsel's review of a random sample.

The scorecard: eight lines and the result that knocks each down

Score two lines per family. Each line pairs a pass condition with a disconfirmation test, the specific result that would drop it to zero or cap it, and the test runs before anything above a 1 is awarded.

Two families are gates. If any substitutability or access line scores 0, or neither line in one of those families reaches 2, record the verdict as no moat on current evidence. The business may still be worth backing for growth or service quality, but its data is not what defends it.

Above the gates, call the moat evidenced only when three of the four families hold a line at 3 and no line anywhere scores 0. Everything between is an unproven moat, and the useful output is the list of missing evidence with the cost of getting it. The cut-offs are editorial choices; change them if you like, but before scoring rather than after.

The eight lines, with the evidence that earns a 3:

The scorecard: eight lines and the result that knocks each down
LinePasses whenDisconfirmation testEvidence that earns a 3
S1 Best-substitute gapThe data beats the best named substitute on the buyer's held-out acceptance testRun the same recipe with the substitute; a gap inside run-to-run noise failsA buyer's or independent evaluator's report that names the substitute
S2 Direct-sourcing costA buyer or rival would need far more time or money to source equivalent records itselfA buyer obtains comparable records directly within one procurement cycleA buyer's written comparison of sourcing routes
A1 Rights scope and termSigned licenses cover AI training or evaluation for the stated term, with refresh rightsCounsel's random sample finds agreements that do not cover the claimed useCounsel's sample review with the sampling rule written first
A2 Route resilienceSupply survives the loss of the largest supplier, platform or channelRemove the largest single source from the ledger; most accepted volume disappearsThe investor's analyst reruns the ledger on raw records
U1 Held-out liftThe data improves a task on a test the seller never saw, graded by criteria two experts agree onRerun on a fresh, matched test set; the lift shrinks to noiseA buyer acceptance record or an independent replication
U2 Refresh valueNewer records add measurable lift over the last snapshot deliveredCompare refreshed and previous snapshots on one test; no difference failsA buyer renewal tied to a refresh, with its test results
E1 Repeat, diversified buyersBuyers reorder or renew, and no two buyers hold most contract valueIf two buyers hold most contract value, the line caps at 1A contract ledger reconciled to invoices and payments
E2 Delivery repeatabilityHours and rework per accepted dataset fall across signing cohortsPlot hours per accepted dataset by cohort; a flat or rising line failsTime records and rework logs sampled by the investor

Could useful data still be cheap to replace?

Yes, and that is the strongest alternative to any moat claim: data can be genuinely useful and still easy to substitute, or cheaper for the buyer to source itself. Lines S1 and S2 exist to test it, and they should be run against the best substitute, not a convenient one.

Synthetic data is the first candidate. In Textbooks Are All You Need (2023), Gunasekar and colleagues trained phi-1, a code model with 1.3 billion parameters, on about 6 billion tokens of filtered textbook-quality web data plus about 1 billion tokens of synthetic textbooks and exercises generated with GPT-3.5, and reported 50.6 percent pass@1 on HumanEval. Wherever a synthetic set closes the gap on a buyer's own test, S1 fails.

Synthetic substitution has limits that favor real records. Shumailov and colleagues, writing in Nature in 2024, showed that training on recursively generated data causes what they call model collapse in language models, variational autoencoders and Gaussian mixture models, and that the tails of the original distribution are lost. Rare cases and edge conditions are what operating records hold, so run S1 on the hard slice of the buyer's test, not on the average.

The buyer is the second candidate. OpenAI's data partnerships page invites organizations to share large-scale datasets that are not already easily accessible online to the public, including datasets kept private. A model developer that can source records directly does not need an intermediary, so S2 asks how long direct sourcing would take and grades the answer only from a buyer's own comparison.

Will access to the records survive a terms change?

Only if the rights are documented for the use being sold and the supply does not hang on one source. Lines A1 and A2 test those conditions separately, because a company can hold flawless contracts with one platform that changes its terms tomorrow.

Documentation fails more often than investors assume. Writing in Nature Machine Intelligence in 2024 about more than 1,800 text datasets, the Data Provenance Initiative reported that most datasets on popular hosting sites carried no license information, that error rates exceeded 50 percent, and that the correct licenses were often more restrictive than the listed ones. If public dataset licenses are that unreliable, a private seller's summary of its supplier rights deserves a sample review by counsel before A1 earns more than a 1.

Routes can close within a year. The Consent in Crisis audit of 14,000 web domains found that between 2023 and 2024, new restrictions left about 5 percent or more of all tokens in the C4 corpus, and 28 percent or more of its most actively maintained critical sources, fully restricted from use. Private supply is not web supply, but the lesson carries over: measure how much accepted volume depends on a single platform's goodwill.

Old terms cannot be repaired after the fact. The FTC's technology office warned in February 2024 that a company may act unfairly or deceptively if it adopts more permissive data practices, such as using data for AI training, and tells consumers only through a surreptitious, retroactive amendment to its terms or privacy policy. A data business cannot fix a weak A1 by rewriting history; the rights had to be right when the records were collected.

Utility and execution: what buyers do, not what sellers report

Grade utility on held-out tests and execution on buyer behavior, because both are easy to overstate from the inside. A seller's internal benchmark and a seller's revenue chart are the least informative versions of each.

Familiar tests flatter. When Zhang and colleagues built GSM1k, a fresh set of grade-school math problems designed to mirror the widely used GSM8k benchmark, they observed accuracy drops of up to 8 percent, with several model families showing evidence of systematic overfitting. A utility claim measured on a test the seller knows can shrink on a fresh one, which is why U1 requires a test the seller never saw.

Anthropic's engineering guidance on evaluating AI agents offers a usable bar for that test: a good task is one that two domain experts, judging separately, would both mark as a pass or both mark as a fail. If a buyer's acceptance criteria are too vague for two reviewers to agree, U1 cannot score above 1, however large the claimed lift.

Execution starts with who pays and whether they return. Reddit's Form 10-K for fiscal year 2024 says that, to date, "substantially all of the contract value associated with our licensing revenue is derived from two of our partners." Its quarterly report for the period ended March 31, 2026 put remaining performance obligations at $120.6 million, consisting primarily of long-term content licensing contracts. Long contracts support execution and two-partner concentration caps it, which is why E1 pairs renewal with a concentration test.

Delivery repeatability is the quiet half of execution. A data business whose hours per accepted dataset stay flat as cohorts accumulate is selling services with a data label; one whose hours fall is building leverage. E2 asks for time records by signing cohort, which most companies keep and few volunteer.

Illustrative: a pitch that scores 24 and evidence that scores 10

A deck that rates every line as strong scores 24, and the evidence behind the same company can score far lower once each test is run. The company below is invented, and its grades are examples chosen to show how the scale works.

Illustrative company Q licenses support and claims records from mid-sized software firms and claims a moat on all eight lines. After diligence, three lines reach 2, four reach 1, and route resilience scores 0 because one software platform's export route carries most of its accepted volume. The access gate fails, so the verdict is no moat on current evidence, whatever the other lines say.

The useful output is the evidence list, not the total. Company Q could lift S1 and U1 with an independent replication against a named substitute, but it can lift A2 only by signing suppliers on other platforms, which takes quarters, not weeks. The figure shows the gap line by line.

Illustrative: a pitch that scores 24 and evidence that scores 10
LinePitched gradeEvidence gradeWhat held it down
S1 Best-substitute gap31Internal benchmark only; no named substitute
S2 Direct-sourcing cost32Sourcing times measured, but no buyer comparison
A1 Rights scope and term32Agreements reviewed by the company, not sampled by counsel
A2 Route resilience30One platform's export route carries most accepted volume
U1 Held-out lift31Lift shown only on a test the seller built
U2 Refresh value31Refresh comparison not yet run
E1 Repeat, diversified buyers31Two buyers hold most contract value
E2 Delivery repeatability32Hours fall across three cohorts in time records
A dumbbell chart of eight moat lines for illustrative company Q, each pitched at grade 3 and graded lower on evidence, with route resilience at 0 failing the access gate; totals 24 pitched and 10 evidenced.
Pitched versus evidenced grades for an invented company. Illustrative.

Five ways this scorecard can mislead an investor

The scorecard can mislead in five ways, and an investor should name them in the memo before the score. Each comes with a public example where one exists.

  • Selected cohorts flatter grades. A company that chooses which datasets to show can pass U1 and E2 on its best work, so insist on the full cohort, rejected candidates included.
  • Model progress moves the substitute. Synthetic and public data improve with each model generation, as phi-1's results with synthetic textbooks showed in 2023, so an S1 grade expires and needs a date.
  • Concentration can sit behind a good grade. Reddit's annual report for 2024 ties substantially all of its licensing contract value to two partners, and a company can show strong renewals while depending on a handful of buyers.
  • Rules change what access is worth. The FTC's 2024 warning about retroactive terms changes shows that regulators judge how rights were obtained, not only whether a contract exists, so A1 needs counsel rather than a checkbox.
  • A moat is not a price. The scorecard says whether data defends a business; it says nothing about valuation, growth or whether the moat is already priced in.

Diligence requests that move a line from 1 to 3, SourceX included

Ask for the evidence that would raise a grade, in the same form from every company you compare. The same requests apply to SourceX, which helps companies turn selected business records into licensed datasets for AI; its role covers qualifying and sourcing those companies, reviewing rights, contracting, preparing and delivering the data, and payment.

SourceX is a labeled application here, not a scored example: it has not published grades, cohort tables or buyer metrics for this scorecard, and nothing in this article reports them. An A1 review would start at the signature, since the supplying company must approve the scope and terms and sign the agreement before any of its data is delivered. Investors receive company and commercial metrics directly from SourceX, and the raise, under Rule 506(c), is limited to accredited investors, who can request materials and bring any request below.

  • For S1: which substitute did your most recent buyer evaluation name, and who ran it?
  • For A1: may counsel review a random sample of supplier agreements, with the sampling rule written down first?
  • For A2: how much accepted volume would remain if your largest supplier or source platform left?
  • For U1: can a buyer share an acceptance record from a test you did not build?
  • For E1: how much contract value sits with your two largest buyers, and how many buyers renewed?
  • For E2: what were hours per accepted dataset by signing cohort over the last four quarters?

Methods note: sources behind the eight lines, and the blind spots

The four families, the eight lines, the 0-to-3 evidence scale, the gate rule and illustrative company Q are editorial constructs built for this article on 2026-10-26. The evidence comes from the Data Provenance Initiative's licensing audit (2024), the Consent in Crisis audit (2024), the FTC technology office's February 2024 post, Reddit's annual report for 2024 and quarterly report for March 31, 2026, Textbooks Are All You Need (2023), Shumailov and colleagues in Nature (2024), the GSM1k paper, OpenAI's data partnerships page and Anthropic's guidance on agent evaluations.

Not measured: any real company's grades, SourceX's included; how often each disconfirmation test fails in practice; the right weights between families; and any link between a score and investment returns. The gate and verdict cut-offs are judgments, published so a reader can change them before scoring.

Frequently asked questions

What makes a data business defensible?

A data business is defensible when buyers cannot get the same result elsewhere at similar cost, its access to the records survives terms changes and supplier exits, the data improves buyers' tasks on fresh tests, and buyers come back. Volume and exclusivity help only when they serve one of those four conditions.

How do you test whether proprietary data is really a moat?

Write down, for each claim, the result that would disprove it, then run that test before giving credit. Compare the data with the best named substitute on a held-out test, sample the supplier agreements, remove the largest source from the ledger and check buyer concentration. Unknown answers score zero.

Can synthetic data erase a data moat?

It can for tasks where synthetic data matches real records on the buyer's test: phi-1 reached 50.6 percent pass@1 on HumanEval in 2023 with synthetic textbooks in its training mix. It is weaker for rare cases, because training on recursively generated data loses the tails of the original distribution. Test on the hard slice.

Why does buyer concentration matter in a moat assessment?

A data business with few buyers depends on renewals it does not control. Reddit's annual report for 2024 says substantially all of its licensing contract value comes from two partners. A strong data asset with concentrated demand still carries renewal and pricing risk that the moat does not remove.

Does SourceX publish moat scores for itself?

No. SourceX has not published grades, cohort tables or buyer metrics for this scorecard. It shares company and commercial metrics with investors directly through sourcex.si/investors, and any offering is limited to verified accredited investors.

Sources

  • Data Provenance Initiative audit (Nature Machine Intelligence, 2024) of more than 1,800 text datasets: license information missing for most datasets on popular hosting sites, error rates above 50 percent, and correct licenses often more restrictive than those listed. Source
  • Consent in Crisis (2024): audit of 14,000 web domains; within 2023-2024, about 5 percent or more of C4 tokens and 28 percent or more of its most actively maintained critical sources became fully restricted. Source
  • FTC Office of Technology (February 2024): adopting more permissive data practices, such as AI training, disclosed only through a surreptitious, retroactive change to terms or a privacy policy may be unfair or deceptive. Source
  • Reddit Form 10-K for fiscal year 2024: to date, substantially all of the contract value associated with its licensing revenue is derived from two of its partners. Source
  • Reddit quarterly report for the period ended March 31, 2026: remaining performance obligations of $120.6 million, consisting primarily of long-term content licensing contracts. Source
  • Shumailov et al., Nature (2024): training on recursively generated data causes model collapse in LLMs, VAEs and Gaussian mixture models, with tails of the original distribution lost. Source
  • Textbooks Are All You Need (2023): phi-1, a 1.3B-parameter model trained on 6B tokens of textbook-quality web data and 1B tokens of GPT-3.5-generated synthetic textbooks and exercises, reached 50.6 percent pass@1 on HumanEval. Source
  • GSM1k (arXiv 2405.00332, current version): evaluating leading LLMs on a fresh benchmark mirroring GSM8k showed accuracy drops of up to 8 percent, with several model families showing systematic overfitting. Source
  • OpenAI data partnerships: seeks large-scale datasets not already easily accessible online to the public, with an option for private datasets. Source
  • Anthropic, Demystifying evals for AI agents: a good task is one where two domain experts would independently reach the same pass/fail verdict. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies