Deal economics
Dataset qualification economic threshold: when to decline, gate by gate
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A dataset qualification economic threshold is the point where the expected value of the next step, counting only costs still ahead, falls below zero or below what the same team would earn on a better candidate. Work it backward from acceptance, gate by gate. In this article's illustrative model, a candidate with 34 percent acceptance odds is worth opening, because cheap early checks let the team stop before preparation.
Value drivers
Dataset qualification
Key takeaways
- Decline on forward-looking value: cost already sunk in a candidate never argues for continuing.
- The naive test, acceptance odds against total cost over price, rejects candidates a staged view would open: 34 percent odds against a 46 percent cost ratio, yet plus 10.1 points of expected value.
- The bar falls as evidence arrives: in the model, buyer acceptance odds of about 60 percent are needed at intake but only 38 percent once rights and fit are cleared.
- When the team is full, rank candidates by expected value per point of expected cost; a positive candidate can still be one to defer.
- Order gates by cost per failure caught; a records-free fit check before rights review lifts the illustrative candidate from 10.1 to 11.3 points.
When does declining a dataset make economic sense?
Decline when the expected value of the next step, counting only the costs still ahead, is negative, or is positive but smaller than what the same people would earn on the next candidate in line. That is the dataset qualification economic threshold, and it moves at every gate, because each check changes both the odds of acceptance and the cost that remains.
Two habits hide the threshold. Counting sunk cost makes a candidate that has already absorbed a rights review and a fit test feel too expensive to drop. Judging every candidate at intake by total cost over price makes thin-odds candidates look hopeless even when cheap early checks would sort them. An investor should expect a data licensing company to avoid both habits and to show its declines as evidence of how it does so.
This article builds a decline waterfall that values one illustrative candidate backward from acceptance, runs six variants that end in a no, a deferral or a reorder, adds the hurdle a full team should apply, and maps the evidence that moves each gate's odds. Values are illustrative index points, and none describes SourceX.
Why total cost over price sets the wrong bar
Because a candidate dataset is not one bet but a series of small ones, each of which can be stopped. The naive test compares the odds of final acceptance with total remaining cost divided by price; the right test values each step with the option to stop after it.
Take the illustrative candidate: 100 points of net revenue if accepted; steps costing 6 points for rights review, 4 for a fit test against the buyer's criteria, 30 for preparation and 6 for delivery and review support; and pass rates of 70, 60, 95 and 85 percent. Its odds of ending accepted are about 34 percent, below the 46 percent ratio of total cost to price, so the naive test says decline. Valued step by step, its expected cost is only 23.8 points, because most failures surface before preparation, and its expected value is plus 10.1 points. Opening it is the better call.
Sunk cost drops out at every gate. When the candidate reaches preparation, 10 points are already spent; the decision weighs the plus 45.1 points of expected value still ahead against zero, not against the 10 points that are gone. A company that justifies continuing by what it has already spent will carry weak datasets into its most expensive stage.
The decline waterfall: one candidate valued backward from acceptance
Start from an accepted dataset and work back one gate at a time: multiply by the gate's pass rate, subtract the gate's cost, and the running figure is the value of continuing from that point. Wherever the running figure would turn negative, that gate is where the candidate should be declined.
For the illustrative candidate every gate stays positive: 79.0 points before delivery, 45.1 before preparation, 23.0 before the fit test and 10.1 at intake. The last column shows how low each pass rate could fall, with the others held, before stopping would be the better choice.
The bar drops as evidence accumulates. At intake the buyer's acceptance odds would need to be about 60 percent for this candidate to pay; once rights and fit are cleared, 38 percent is enough, and at delivery 6 percent covers the remaining cost. A late decline therefore means the decisive evidence arrived late, which is a process question worth asking, not bad luck.
| Gate, read from acceptance back to intake | Pass rate | Cost of the step, points | Value of continuing, points | Lowest pass rate that still pays |
|---|---|---|---|---|
| Accepted and paid | Not applicable | None | 100.0 | Not applicable |
| Delivery and buyer review | 85% | 6 | 79.0 | 6% |
| Preparation | 95% | 30 | 45.1 | 38% |
| Fit test against the buyer's criteria | 60% | 4 | 23.0 | 9% |
| Rights review, the first spend after intake | 70% | 6 | 10.1 | 26% |
Six candidates the model turns away, and what would reopen each
Change one input from the baseline and the same arithmetic yields a decline, a deferral or a reorder. Each case is illustrative and isolates one common reason a candidate fails; the value shown is at the decision point where the no should land.
Two patterns stand out. Candidates A, D and F should be stopped or reshaped at intake, before any review spend, because the input that sinks them is visible on day one: no buyer task, a heavy privacy workload or thin history. Candidate C is the expensive kind, a no that arrives after 10 points are spent; it is still the right call, because continuing would commit 35.7 points of expected cost to earn an expected 28.5.
| Candidate (Illustrative) | Input changed from baseline | Value at the decision point, points | Decision | What would reopen it |
|---|---|---|---|---|
| A | No named buyer task: acceptance odds 40% | -6.0 at intake; even the fit test does not pay | Decline at intake | A written request naming a task and an acceptance test |
| B | Doubtful rights: rights pass 30% | +0.9 at intake | Open only with spare capacity; otherwise defer | A narrower scope that leaves out the doubtful records |
| C | Evaluation records overlap public material: acceptance odds fall to 30% at the fit test | -7.2 before preparation, with 10 already spent | Decline before preparation | Fresh records that pass an overlap check |
| D | Heavy de-identification doubles preparation to 60 | -2.5 at intake | Decline or reprice; breaks even near 107 points of revenue | A higher price or a cheaper de-identification route |
| E | Buyer expected to keep 70% of records: revenue 70 | About 0.0 at intake | A coin flip; decline when the team is full | A written acceptance test that lifts the share kept |
| F | Thin history: fit test pass 25% | -0.9 in the standard order; +2.4 with the fit test first | Reorder, then open | A records-free fit check run before rights review |
A full team needs a higher bar: capacity and gate order
When the team has slack, any candidate with positive expected value is worth opening. When it is full, every candidate competes for the same hours, and the threshold rises to what the next-best candidate would earn with them.
A workable yardstick is expected value per point of expected cost. The baseline candidate earns about 0.43 points for each point it is expected to cost (10.1 over 23.8); candidate B earns about 0.07 (0.9 over 13.6). If the queue holds candidates yielding 0.30, B should wait even though its expected value is positive, and a company that opens it anyway is spending scarce review hours below their best use.
Deferral is not decline. A deferred candidate keeps its record, its reason code and the evidence that would reopen it, so it can return when capacity frees up or a buyer request raises its odds. Investors should see deferrals reported apart from declines: a pile of deferrals is unserved supply, while a pile of declines is screened-out supply.
Gate order matters as much as capacity. A gate earns an early slot when its cost per failure caught is low: rights review costs 6 points and stops a candidate 30 percent of the time, about 20 points per failure caught, while the fit test costs 4 and stops 40 percent, about 10. Running the fit test first, where it can be done without moving records, lifts the baseline from 10.1 to 11.3 points and turns candidate F from minus 0.9 into plus 2.4.
Which evidence moves a gate's odds, and what it costs to collect
Each gate's pass rate should come from evidence gathered on that candidate, and the cheapest evidence concerns rights and buyer fit. The table pairs each gate with the evidence that moves its odds, the public finding that explains why, and the low-cost form of the check.
Evidence has a price of its own, so a check belongs in the waterfall too: it is worth running early when it changes the decision often enough to repay its cost on every candidate it touches.
| Gate | Evidence that moves the odds | Public finding behind it | Low-cost form of the check |
|---|---|---|---|
| Demand, at intake | A written buyer request that names a task | OpenAI's data partnerships page wants large datasets the public cannot already easily reach online, and turns away sensitive or personal information and third-party information | Match the candidate to a named request before any review |
| Rights: licenses | Contracts and grants read in full, not license labels | The Data Provenance Initiative audit (2023) found license omission of 70 percent or more and error rates of 50 percent or more on widely used dataset hosting sites | Collect the documents before touching any records |
| Rights: notices | Privacy notices in force when each record period was collected | The FTC's Office of Technology warned in February 2024 that adopting more permissive data practices through a surreptitious, retroactive terms change may be unfair or deceptive | Date-match each record period to its notice |
| Fit test | The buyer's criteria written as tasks with defined inputs and success criteria | Anthropic's guide to agent evals defines a task that way and runs several trials per task because outputs vary | Run the criteria on a manifest or a small authorized sample |
| Fit test: evaluation data | An overlap check against public benchmarks | In the GSM1k study, new problems matched to GSM8k cut leading models' accuracy by as much as 8 percent (arXiv 2405.00332) | Compare a sample with public sets before preparation |
| Preparation | The de-identification route the records need | HIPAA de-identification uses either a documented expert determination or the safe harbor's removal of listed identifiers (45 CFR 164.514) | Price the route before committing to preparation |
Could healthy deal margins hide a company that says yes too often?
Yes, and it is the strongest alternative to reading a company's declines as discipline. A company that opens weak candidates can still report good margins on the datasets it delivers, because the losses sit in candidates that failed late, in rework charged to overhead and in cash collected months after acceptance.
Three checks separate the two stories. First, look at declines by gate: a disciplined company declines mostly at intake and rights review, and each late decline carries a reason that could not have been known earlier. Second, restate delivered-dataset margins with failed candidates, rework and overhead charged to them; if the margin collapses, the yes rate was too high. Third, audit a sample of declines, since a strict rule can also turn away good supply; a re-review that overturns many of them means the bar is set too high.
The decline rate on its own settles nothing. A high rate can reflect discipline or a narrow buyer base, and a low rate can reflect strong sourcing or a team that cannot say no. The gate and the reason are what carry the information.
What could make a strict decline rule the wrong rule
A threshold is only as good as its pass rates, and several conditions can push a well-run company to decline the wrong candidates or accept too many. Each comes with a dated public source where one exists.
- Pass rates are estimates. A company with a few dozen past candidates per gate cannot pin its odds down precisely, and a threshold computed from optimistic rates lets too many candidates through; ask for the counts behind each rate.
- Demand is concentrated. Reddit's Form 10-K for fiscal 2025 says that to date "substantially all of the contract value associated with our licensing revenue is derived from two of our partners." When a few buyers set acceptance, odds for whole categories move together, and a rule tuned to one buyer misprices the rest.
- Budgets can shift. Sequoia's David Cahn put the gap between yearly AI infrastructure spending and the revenue needed to justify it at $125 billion or more in September 2023, and wrote in June 2024 that it was heading toward $500 billion. If buyers cut spending, acceptance odds fall across the board and yesterday's threshold admits too much.
- False noes are invisible. A declined supplier rarely returns, and the loss never shows in a margin; without a re-review of sampled declines, a company cannot tell discipline from excess caution.
- Strategic value sits outside the model. A first dataset in a new category can teach a team how to prepare the next ones, which a per-candidate model does not credit; a company that overrides its rule for that reason should say so and track the result.
Decline questions for a data licensing company, SourceX included
Ask for the declines, not just the deals. The gate and reason behind each no show whether a company prices its threshold or improvises it. Put the same questions to SourceX.
SourceX is one application of the threshold. Because SourceX handles qualification and sourcing, rights review, contracting, preparation and delivery of the data, and payment, every gate in the waterfall sits inside one operator. The opening gate is a short company assessment, and no data uploads are required for it, so a cheap check comes before any records move. On volume, its general guide is that buyers want a single system holding tens of thousands of records or more, with consistent history usually running two years or longer; a smaller set can still qualify when the work behind it is rare, and each case is assessed deal by deal. Datasets for buyers are sourced on request rather than held in stock, so the demand gate starts from a buyer's stated scope.
SourceX has not published decline rates, gate pass rates or margins, and this article presents none; it shares company and commercial metrics with investors directly. Its raise under Rule 506(c) is limited to accredited investors, who can request materials through the investors page. Questions to bring:
- At which gate did each declined candidate stop last year, and with what reason code?
- What pass rate do you assume at each gate, and how many past candidates sit behind each one?
- How much had been spent on each late decline, and why could its reason not be found earlier?
- When the team is full, how do you rank candidates, and how many positive candidates did you defer?
- Who re-reviews a sample of declines, and how many were overturned?
- Which candidate did you open against your own rule, and what happened to it?
Methods note: assumptions behind the decline waterfall
The decline waterfall, the candidate variants, the capacity yardstick and the evidence map are an editorial model built for this article on 2026-10-26. Every input is illustrative: net revenue of an accepted dataset indexed to 100 points; step costs of 6 points for rights review, 4 for the fit test, 30 for preparation and 6 for delivery and review support; pass rates of 70, 60, 95 and 85 percent; and a 0.30 yield for the next-best candidate. Values come from backward induction with the option to stop after each gate, and each variant changes one input.
The model treats gate outcomes as independent and ignores intake screening cost, timing, refunds and price changes. Public inputs come from OpenAI's data partnerships page, the Data Provenance Initiative audit (2023), the FTC Office of Technology (2024), Anthropic's guide to agent evals, the GSM1k paper (2024), 45 CFR 164.514, Reddit's Form 10-K for fiscal 2025 and Sequoia essays from 2023 and 2024. Not measured: real pass rates, decline rates, deferrals or overturned declines at SourceX or any other company.
Frequently asked questions
What is a dataset qualification threshold?
It is the point at which the next step on a candidate dataset is expected to lose money, or to earn less than the same work would on a better candidate. It is computed from the odds of passing each remaining gate, the cost still to come and the revenue on acceptance, never from cost already spent.
Should a data company decline datasets that would be profitable?
Sometimes. When the team is at capacity, a positive candidate takes hours from a better one, so it should wait if its expected value per point of expected cost falls below the next candidate's. Deferring keeps the record and the evidence that would reopen it.
Why is a late decline more expensive than an early one?
Because the cost already spent cannot be recovered. In this article's illustrative model, a no before rights review costs nothing, while a no before preparation has already used 10 points on review and testing. Late declines are sometimes unavoidable, but each should carry a reason that could not have been found earlier.
Is a high decline rate a good sign in a data licensing company?
Not on its own. A high rate can reflect discipline or a narrow buyer base, and a low rate can reflect strong sourcing or weak screening. Ask for declines by gate and reason, and for an independent re-review of a sample of them.
Does SourceX publish how many datasets it declines?
No. SourceX has not published decline rates, pass rates or margins, and it shares company and commercial metrics with investors directly. Materials are available on request at sourcex.si/investors, and any offering is open only to verified accredited investors.
Sources
- OpenAI's data partnerships page seeks large-scale datasets that are not already easily accessible online and says it is not seeking datasets with sensitive or personal information, or information that belongs to a third party. Source
- The Data Provenance Initiative's audit of more than 1,800 text datasets found license omission of 70 percent or more and error rates of 50 percent or more on widely used dataset hosting sites. Source
- The FTC's Office of Technology wrote in February 2024 that a company adopting more permissive data practices, such as using data for AI training, and informing consumers only through a surreptitious, retroactive amendment to its terms or privacy policy may be acting unfairly or deceptively. Source
- Anthropic's engineering guide to agent evals defines a task as a single test with defined inputs and success criteria and runs multiple trials per task because outputs vary. Source
- The GSM1k paper (arXiv 2405.00332, current version) reports accuracy drops of up to 8 percent for leading models on new problems built to mirror GSM8k, with several model families showing systematic overfitting. Source
- 45 CFR 164.514 sets two de-identification routes for protected health information: an expert determination with documented methods and results, or the safe harbor's removal of listed identifiers with no actual knowledge that the rest could identify a person. Source
- Reddit's Form 10-K for fiscal 2025 states that, to date, substantially all of the contract value associated with its licensing revenue is derived from two of its partners. Source
- In September 2023, Sequoia's David Cahn estimated a gap of $125 billion or more between yearly AI infrastructure spending and the revenue needed to justify it. Source
- In June 2024, David Cahn wrote that the $125 billion gap was heading toward $500 billion. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.