Skip to content

Deal economics

Dataset qualification economic threshold: when to decline, gate by gate

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A dataset qualification economic threshold is the point where the expected value of the next step, counting only costs still ahead, falls below zero or below what the same team would earn on a better candidate. Work it backward from acceptance, gate by gate. In this article's illustrative model, a candidate with 34 percent acceptance odds is worth opening, because cheap early checks let the team stop before preparation.

Value drivers

Dataset qualification

This page covers dataset qualification economic threshold: when to decline, gate by gate.Illustrative

Key takeaways

  • Decline on forward-looking value: cost already sunk in a candidate never argues for continuing.
  • The naive test, acceptance odds against total cost over price, rejects candidates a staged view would open: 34 percent odds against a 46 percent cost ratio, yet plus 10.1 points of expected value.
  • The bar falls as evidence arrives: in the model, buyer acceptance odds of about 60 percent are needed at intake but only 38 percent once rights and fit are cleared.
  • When the team is full, rank candidates by expected value per point of expected cost; a positive candidate can still be one to defer.
  • Order gates by cost per failure caught; a records-free fit check before rights review lifts the illustrative candidate from 10.1 to 11.3 points.

When does declining a dataset make economic sense?

Decline when the expected value of the next step, counting only the costs still ahead, is negative, or is positive but smaller than what the same people would earn on the next candidate in line. That is the dataset qualification economic threshold, and it moves at every gate, because each check changes both the odds of acceptance and the cost that remains.

Two habits hide the threshold. Counting sunk cost makes a candidate that has already absorbed a rights review and a fit test feel too expensive to drop. Judging every candidate at intake by total cost over price makes thin-odds candidates look hopeless even when cheap early checks would sort them. An investor should expect a data licensing company to avoid both habits and to show its declines as evidence of how it does so.

This article builds a decline waterfall that values one illustrative candidate backward from acceptance, runs six variants that end in a no, a deferral or a reorder, adds the hurdle a full team should apply, and maps the evidence that moves each gate's odds. Values are illustrative index points, and none describes SourceX.

Why total cost over price sets the wrong bar

Because a candidate dataset is not one bet but a series of small ones, each of which can be stopped. The naive test compares the odds of final acceptance with total remaining cost divided by price; the right test values each step with the option to stop after it.

Take the illustrative candidate: 100 points of net revenue if accepted; steps costing 6 points for rights review, 4 for a fit test against the buyer's criteria, 30 for preparation and 6 for delivery and review support; and pass rates of 70, 60, 95 and 85 percent. Its odds of ending accepted are about 34 percent, below the 46 percent ratio of total cost to price, so the naive test says decline. Valued step by step, its expected cost is only 23.8 points, because most failures surface before preparation, and its expected value is plus 10.1 points. Opening it is the better call.

Sunk cost drops out at every gate. When the candidate reaches preparation, 10 points are already spent; the decision weighs the plus 45.1 points of expected value still ahead against zero, not against the 10 points that are gone. A company that justifies continuing by what it has already spent will carry weak datasets into its most expensive stage.

The decline waterfall: one candidate valued backward from acceptance

Start from an accepted dataset and work back one gate at a time: multiply by the gate's pass rate, subtract the gate's cost, and the running figure is the value of continuing from that point. Wherever the running figure would turn negative, that gate is where the candidate should be declined.

For the illustrative candidate every gate stays positive: 79.0 points before delivery, 45.1 before preparation, 23.0 before the fit test and 10.1 at intake. The last column shows how low each pass rate could fall, with the others held, before stopping would be the better choice.

The bar drops as evidence accumulates. At intake the buyer's acceptance odds would need to be about 60 percent for this candidate to pay; once rights and fit are cleared, 38 percent is enough, and at delivery 6 percent covers the remaining cost. A late decline therefore means the decisive evidence arrived late, which is a process question worth asking, not bad luck.

The decline waterfall: one candidate valued backward from acceptance
Gate, read from acceptance back to intakePass rateCost of the step, pointsValue of continuing, pointsLowest pass rate that still pays
Accepted and paidNot applicableNone100.0Not applicable
Delivery and buyer review85%679.06%
Preparation95%3045.138%
Fit test against the buyer's criteria60%423.09%
Rights review, the first spend after intake70%610.126%
A decision tree for one illustrative candidate dataset: rights review, fit test, preparation and buyer review, each with its pass rate, cost and the value of continuing, from 10.1 points at intake to 79.0 before delivery, and a stop branch showing the lowest pass rate that still pays at each gate.
The decline waterfall as a decision tree. Illustrative: pass rates, costs and values are index points, not measured data.

Six candidates the model turns away, and what would reopen each

Change one input from the baseline and the same arithmetic yields a decline, a deferral or a reorder. Each case is illustrative and isolates one common reason a candidate fails; the value shown is at the decision point where the no should land.

Two patterns stand out. Candidates A, D and F should be stopped or reshaped at intake, before any review spend, because the input that sinks them is visible on day one: no buyer task, a heavy privacy workload or thin history. Candidate C is the expensive kind, a no that arrives after 10 points are spent; it is still the right call, because continuing would commit 35.7 points of expected cost to earn an expected 28.5.

Six candidates the model turns away, and what would reopen each
Candidate (Illustrative)Input changed from baselineValue at the decision point, pointsDecisionWhat would reopen it
ANo named buyer task: acceptance odds 40%-6.0 at intake; even the fit test does not payDecline at intakeA written request naming a task and an acceptance test
BDoubtful rights: rights pass 30%+0.9 at intakeOpen only with spare capacity; otherwise deferA narrower scope that leaves out the doubtful records
CEvaluation records overlap public material: acceptance odds fall to 30% at the fit test-7.2 before preparation, with 10 already spentDecline before preparationFresh records that pass an overlap check
DHeavy de-identification doubles preparation to 60-2.5 at intakeDecline or reprice; breaks even near 107 points of revenueA higher price or a cheaper de-identification route
EBuyer expected to keep 70% of records: revenue 70About 0.0 at intakeA coin flip; decline when the team is fullA written acceptance test that lifts the share kept
FThin history: fit test pass 25%-0.9 in the standard order; +2.4 with the fit test firstReorder, then openA records-free fit check run before rights review

A full team needs a higher bar: capacity and gate order

When the team has slack, any candidate with positive expected value is worth opening. When it is full, every candidate competes for the same hours, and the threshold rises to what the next-best candidate would earn with them.

A workable yardstick is expected value per point of expected cost. The baseline candidate earns about 0.43 points for each point it is expected to cost (10.1 over 23.8); candidate B earns about 0.07 (0.9 over 13.6). If the queue holds candidates yielding 0.30, B should wait even though its expected value is positive, and a company that opens it anyway is spending scarce review hours below their best use.

Deferral is not decline. A deferred candidate keeps its record, its reason code and the evidence that would reopen it, so it can return when capacity frees up or a buyer request raises its odds. Investors should see deferrals reported apart from declines: a pile of deferrals is unserved supply, while a pile of declines is screened-out supply.

Gate order matters as much as capacity. A gate earns an early slot when its cost per failure caught is low: rights review costs 6 points and stops a candidate 30 percent of the time, about 20 points per failure caught, while the fit test costs 4 and stops 40 percent, about 10. Running the fit test first, where it can be done without moving records, lifts the baseline from 10.1 to 11.3 points and turns candidate F from minus 0.9 into plus 2.4.

Which evidence moves a gate's odds, and what it costs to collect

Each gate's pass rate should come from evidence gathered on that candidate, and the cheapest evidence concerns rights and buyer fit. The table pairs each gate with the evidence that moves its odds, the public finding that explains why, and the low-cost form of the check.

Evidence has a price of its own, so a check belongs in the waterfall too: it is worth running early when it changes the decision often enough to repay its cost on every candidate it touches.

Which evidence moves a gate's odds, and what it costs to collect
GateEvidence that moves the oddsPublic finding behind itLow-cost form of the check
Demand, at intakeA written buyer request that names a taskOpenAI's data partnerships page wants large datasets the public cannot already easily reach online, and turns away sensitive or personal information and third-party informationMatch the candidate to a named request before any review
Rights: licensesContracts and grants read in full, not license labelsThe Data Provenance Initiative audit (2023) found license omission of 70 percent or more and error rates of 50 percent or more on widely used dataset hosting sitesCollect the documents before touching any records
Rights: noticesPrivacy notices in force when each record period was collectedThe FTC's Office of Technology warned in February 2024 that adopting more permissive data practices through a surreptitious, retroactive terms change may be unfair or deceptiveDate-match each record period to its notice
Fit testThe buyer's criteria written as tasks with defined inputs and success criteriaAnthropic's guide to agent evals defines a task that way and runs several trials per task because outputs varyRun the criteria on a manifest or a small authorized sample
Fit test: evaluation dataAn overlap check against public benchmarksIn the GSM1k study, new problems matched to GSM8k cut leading models' accuracy by as much as 8 percent (arXiv 2405.00332)Compare a sample with public sets before preparation
PreparationThe de-identification route the records needHIPAA de-identification uses either a documented expert determination or the safe harbor's removal of listed identifiers (45 CFR 164.514)Price the route before committing to preparation

Could healthy deal margins hide a company that says yes too often?

Yes, and it is the strongest alternative to reading a company's declines as discipline. A company that opens weak candidates can still report good margins on the datasets it delivers, because the losses sit in candidates that failed late, in rework charged to overhead and in cash collected months after acceptance.

Three checks separate the two stories. First, look at declines by gate: a disciplined company declines mostly at intake and rights review, and each late decline carries a reason that could not have been known earlier. Second, restate delivered-dataset margins with failed candidates, rework and overhead charged to them; if the margin collapses, the yes rate was too high. Third, audit a sample of declines, since a strict rule can also turn away good supply; a re-review that overturns many of them means the bar is set too high.

The decline rate on its own settles nothing. A high rate can reflect discipline or a narrow buyer base, and a low rate can reflect strong sourcing or a team that cannot say no. The gate and the reason are what carry the information.

What could make a strict decline rule the wrong rule

A threshold is only as good as its pass rates, and several conditions can push a well-run company to decline the wrong candidates or accept too many. Each comes with a dated public source where one exists.

  • Pass rates are estimates. A company with a few dozen past candidates per gate cannot pin its odds down precisely, and a threshold computed from optimistic rates lets too many candidates through; ask for the counts behind each rate.
  • Demand is concentrated. Reddit's Form 10-K for fiscal 2025 says that to date "substantially all of the contract value associated with our licensing revenue is derived from two of our partners." When a few buyers set acceptance, odds for whole categories move together, and a rule tuned to one buyer misprices the rest.
  • Budgets can shift. Sequoia's David Cahn put the gap between yearly AI infrastructure spending and the revenue needed to justify it at $125 billion or more in September 2023, and wrote in June 2024 that it was heading toward $500 billion. If buyers cut spending, acceptance odds fall across the board and yesterday's threshold admits too much.
  • False noes are invisible. A declined supplier rarely returns, and the loss never shows in a margin; without a re-review of sampled declines, a company cannot tell discipline from excess caution.
  • Strategic value sits outside the model. A first dataset in a new category can teach a team how to prepare the next ones, which a per-candidate model does not credit; a company that overrides its rule for that reason should say so and track the result.

Decline questions for a data licensing company, SourceX included

Ask for the declines, not just the deals. The gate and reason behind each no show whether a company prices its threshold or improvises it. Put the same questions to SourceX.

SourceX is one application of the threshold. Because SourceX handles qualification and sourcing, rights review, contracting, preparation and delivery of the data, and payment, every gate in the waterfall sits inside one operator. The opening gate is a short company assessment, and no data uploads are required for it, so a cheap check comes before any records move. On volume, its general guide is that buyers want a single system holding tens of thousands of records or more, with consistent history usually running two years or longer; a smaller set can still qualify when the work behind it is rare, and each case is assessed deal by deal. Datasets for buyers are sourced on request rather than held in stock, so the demand gate starts from a buyer's stated scope.

SourceX has not published decline rates, gate pass rates or margins, and this article presents none; it shares company and commercial metrics with investors directly. Its raise under Rule 506(c) is limited to accredited investors, who can request materials through the investors page. Questions to bring:

  • At which gate did each declined candidate stop last year, and with what reason code?
  • What pass rate do you assume at each gate, and how many past candidates sit behind each one?
  • How much had been spent on each late decline, and why could its reason not be found earlier?
  • When the team is full, how do you rank candidates, and how many positive candidates did you defer?
  • Who re-reviews a sample of declines, and how many were overturned?
  • Which candidate did you open against your own rule, and what happened to it?

Methods note: assumptions behind the decline waterfall

The decline waterfall, the candidate variants, the capacity yardstick and the evidence map are an editorial model built for this article on 2026-10-26. Every input is illustrative: net revenue of an accepted dataset indexed to 100 points; step costs of 6 points for rights review, 4 for the fit test, 30 for preparation and 6 for delivery and review support; pass rates of 70, 60, 95 and 85 percent; and a 0.30 yield for the next-best candidate. Values come from backward induction with the option to stop after each gate, and each variant changes one input.

The model treats gate outcomes as independent and ignores intake screening cost, timing, refunds and price changes. Public inputs come from OpenAI's data partnerships page, the Data Provenance Initiative audit (2023), the FTC Office of Technology (2024), Anthropic's guide to agent evals, the GSM1k paper (2024), 45 CFR 164.514, Reddit's Form 10-K for fiscal 2025 and Sequoia essays from 2023 and 2024. Not measured: real pass rates, decline rates, deferrals or overturned declines at SourceX or any other company.

Frequently asked questions

What is a dataset qualification threshold?

It is the point at which the next step on a candidate dataset is expected to lose money, or to earn less than the same work would on a better candidate. It is computed from the odds of passing each remaining gate, the cost still to come and the revenue on acceptance, never from cost already spent.

Should a data company decline datasets that would be profitable?

Sometimes. When the team is at capacity, a positive candidate takes hours from a better one, so it should wait if its expected value per point of expected cost falls below the next candidate's. Deferring keeps the record and the evidence that would reopen it.

Why is a late decline more expensive than an early one?

Because the cost already spent cannot be recovered. In this article's illustrative model, a no before rights review costs nothing, while a no before preparation has already used 10 points on review and testing. Late declines are sometimes unavoidable, but each should carry a reason that could not have been found earlier.

Is a high decline rate a good sign in a data licensing company?

Not on its own. A high rate can reflect discipline or a narrow buyer base, and a low rate can reflect strong sourcing or weak screening. Ask for declines by gate and reason, and for an independent re-review of a sample of them.

Does SourceX publish how many datasets it declines?

No. SourceX has not published decline rates, pass rates or margins, and it shares company and commercial metrics with investors directly. Materials are available on request at sourcex.si/investors, and any offering is open only to verified accredited investors.

Sources

  • OpenAI's data partnerships page seeks large-scale datasets that are not already easily accessible online and says it is not seeking datasets with sensitive or personal information, or information that belongs to a third party. Source
  • The Data Provenance Initiative's audit of more than 1,800 text datasets found license omission of 70 percent or more and error rates of 50 percent or more on widely used dataset hosting sites. Source
  • The FTC's Office of Technology wrote in February 2024 that a company adopting more permissive data practices, such as using data for AI training, and informing consumers only through a surreptitious, retroactive amendment to its terms or privacy policy may be acting unfairly or deceptively. Source
  • Anthropic's engineering guide to agent evals defines a task as a single test with defined inputs and success criteria and runs multiple trials per task because outputs vary. Source
  • The GSM1k paper (arXiv 2405.00332, current version) reports accuracy drops of up to 8 percent for leading models on new problems built to mirror GSM8k, with several model families showing systematic overfitting. Source
  • 45 CFR 164.514 sets two de-identification routes for protected health information: an expert determination with documented methods and results, or the safe harbor's removal of listed identifiers with no actual knowledge that the rest could identify a person. Source
  • Reddit's Form 10-K for fiscal 2025 states that, to date, substantially all of the contract value associated with its licensing revenue is derived from two of its partners. Source
  • In September 2023, Sequoia's David Cahn estimated a gap of $125 billion or more between yearly AI infrastructure spending and the revenue needed to justify it. Source
  • In June 2024, David Cahn wrote that the $125 billion gap was heading toward $500 billion. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies