Skip to content

Deal economics

Dataset no-deal decision efficiency: test the unfixable blockers first

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Dataset no-deal decision efficiency means reaching an unavoidable no at the lowest cost: before records move, before preparation starts, and with a decision record both sides can read. Run the cheap document tests first (task fit, rights and terms), because no data fix cures a failure there. Pay for samples only after those pass, leave money already spent out of the decision, and close with a reasoned notice and a data-handling step.

Checklist

Dataset no deal decision

This page covers dataset no-deal decision efficiency: test the unfixable blockers first.Illustrative

Key takeaways

  • An efficient no-deal is judged by how early and how cheaply the deciding evidence arrived, not by how quickly someone said no.
  • Task fit, rights and commercial terms can be tested from documents alone, and no cleaning, relabeling or re-export cures a failure in any of them.
  • Before rejecting a dataset on quality, run the acceptance test on known-good examples; a test that fails good cases needs fixing, not the data.
  • Money and effort already spent belong in the record, not in the decision; kill criteria signed at kickoff make that easier to hold to.
  • Close with a notice that names the failed gate and what would reopen the deal, and settle what happens to any records already shared.

What makes a no-deal decision efficient?

A no-deal decision is efficient when the evidence that settles it is gathered at the cheapest point it could have been, and the deal stops soon after that evidence exists. Speed alone is not the goal. A buyer who drops a supplier in a day because the first sample looked odd may have made a fast, wrong call; a buyer who learns in week six that the supplier never had the right to license the records made a slow, avoidable one.

Dataset no-deal decision efficiency can be measured on any portfolio of deals with four numbers. Time to no counts calendar days from first contact to the recorded decision. Effort after the deciding evidence counts the hours both sides spent once a disqualifying fact was already sitting in someone's files. Pre-delivery share is the fraction of no-deals reached before any records changed hands. Reversal rate, the share of no-deals later reopened and accepted, checks the opposite error: a team that stops too eagerly shows up there.

Each measure needs a fixed denominator (all deals that passed a first fit call in a stated period) and a rule for deals still open on the measurement date, which stay in the count rather than being dropped. SourceX has not published these measures for its own deals. They are offered as a method, followed below by the test order that keeps them low, the counter-check that prevents a premature no, and the closing steps that stop a no-deal from turning into a records problem.

Why do dataset deals keep running after the answer is clear?

Because the work already done feels like a reason to continue. Hal Arkes and Catherine Blumer described the sunk cost effect in Organizational Behavior and Human Decision Processes in 1985 as a greater tendency to continue an endeavor once money, effort or time has been invested, and a dataset deal offers that temptation at every step: an NDA signed, a schema reviewed, a sample prepared, a test harness built.

Roles add pressure. The delivery owner on the supplier side is often judged on accepted datasets, and the evaluation owner on the buyer side has already told a model team that data is coming. Neither wants to be the one who ends it. The familiar result is a second sample after the first failed for a reason no sample can fix, or a third round of redaction when the real objection was the license scope.

Gary Klein's premortem, described in Harvard Business Review in September 2007, is a cheap counterweight. Before work starts, the team imagines the project has already failed and writes down why. Klein cites research finding that this kind of prospective hindsight increases the ability to correctly identify reasons for future outcomes by 30 percent. For a dataset deal, the list that comes out of the exercise is the first draft of the kill criteria.

The no-deal decision tree: cheapest disproof first

Order the tests by what they cost to run and by whether a failure can be cured, and stop at the first failure nobody can cure. The first three gates need only documents and a few hours of reading; a failure there ends the deal for that scope, because no data work changes what a contract permits, what either side will sign or what a task needs. The last three gates cost more, and a failure there opens a narrower question: fix, narrow or stop.

The order is an editorial framework built from cost and curability, not a measured benchmark, and the cost column names the kind of work involved rather than hours from a study. Teams with a different cost structure can reorder within the two groups but should not move a sample test ahead of the document tests. The figure draws the same order as a staircase, with the exit each step offers.

Run the gates in this order and record the answer to each in the no-deal record described later:

The no-deal decision tree: cheapest disproof first
GateQuestionEvidence that settles itCost to testIf the answer is no
1. Task fitCan these records show what the buyer's written task and acceptance test ask for?The request, the acceptance test, a field list and a plain description of ten real recordsReading timeStop unless the buyer rewrites the task; data work cannot add what was never recorded
2. Rights to the useMay the holder license this use of these records?Customer contracts, privacy notices, platform terms and the signer's authorityDocument reviewStop for that scope, or narrow to records the documents clearly cover
3. Terms in rangeDo price basis, use, term, exclusivity and deletion fall inside both sides' written limits?Each side's walk-away limits and one counterproposalOne negotiation roundStop after one counter fails; data quality cannot close a terms gap
4. Enough distinct casesDo enough distinct cases exist, counted without an export?Counts by year, type and outcome from system reports or metadataA report runNarrow the scope or stop; near-duplicates used as padding fail later
5. Test validityDoes the acceptance test pass examples everyone agrees are good?Results on a small set of known-good and known-bad casesA test runFix the test before judging any data against it
6. Sample resultDoes a small sample pass the agreed test, and is any defect curable within the remaining budget?Per-criterion results, defect counts and a timed repair estimateSample preparationNarrow, repair once against a deadline, or stop
A six-step staircase of gates that rises with the cost to test: task fit, rights and terms on the low steps exit to no deal, then case counts, test validity and the sample result exit to narrow, fix the test or stop.
Cheapest disproof first. Illustrative order based on cost and curability, not measured costs.

Which blockers can no data fix cure?

Rights, commercial terms and a genuine mismatch between the task and the records cannot be cured with data work, so they deserve the earliest and cheapest tests. Cleaning, relabeling, re-exporting or redacting changes what a dataset looks like. It does not change what the holder may license, what either side will accept commercially, or what the source system captured in the first place.

Rights are the gate most often checked too late, partly because license status looks settled when it is not. Across more than 1,800 text datasets, the Data Provenance Initiative (Nature Machine Intelligence, 2024, published by the MIT Media Lab) found that widely used hosting sites failed to state a license over 70 percent of the time and stated the wrong one over 50 percent of the time, and the licenses its researchers traced were often more restrictive than the ones the repositories listed. Private business records usually start with less documentation than that. If the rights review runs after a sample is built, the sample cost is spent before the cheapest question is answered.

Task mismatch is the second quiet killer. A support archive that records each reply but never whether the issue was resolved cannot teach a model which reply worked, however carefully it is cleaned. Terms are the third: when the buyer needs exclusivity the supplier will not grant, or a use the supplier's customer contracts forbid, the dataset's quality stops mattering. In each case the honest outcome is a quick no for that scope, with a note on what would have to change for a different answer.

Could the acceptance test be wrong instead of the data?

Yes, and a no-deal built on a broken test is the most expensive kind, because it throws away data that would have worked. Before rejecting a dataset on quality grounds, run the acceptance test on a handful of examples both sides agree are good and a handful both agree are bad. If good examples fail, the instrument is the defect.

Benchmark builders have documented this failure in public. When OpenAI introduced SWE-bench Verified, it described a campaign in which professional software developers screened the original SWE-bench test samples, because some unit tests were overly specific or unrelated to the issue and could reject correct solutions, and many issue descriptions were underspecified. The screened subset kept 500 samples. Anthropic's engineering guide to agent evaluations makes the same point from the other direction: with frontier models, a 0 percent pass rate across many trials is most often a sign of a broken task, not an incapable agent.

A dataset acceptance test fails in the same ways. A rubric that demands a field the request never mentioned, a grader that penalizes a valid date format, or a threshold set before anyone measured a known-good baseline will reject good data. Gate five exists so that a quality failure is a finding about the records, not about the yardstick.

How do you keep sunk cost out of the stop decision?

Write the stop conditions before the work starts, and at every gate decide only on what the remaining work would cost and what it would produce. Money and hours already spent go into the record, where they help the next deal, and stay out of the decision, where they distort this one.

A forward-only review asks three questions at each gate: which gates remain, what evidence each will produce, and what the remaining work will cost in hours and calendar days. If the remaining gates cannot pass, or cannot be known to pass at a cost either side will pay, the deal stops. Time boxes make this mechanical: each gate gets a date by which it resolves, and after that date the default answer is no.

Kill criteria that both sides sign at kickoff, drafted from a premortem, usually include these:

  • The rights review cannot confirm the intended use for the core scope by an agreed date.
  • The buyer cannot state the task and acceptance test in writing before a sample is requested.
  • Price basis, exclusivity or permitted use remain outside either side's written limits after one counterproposal.
  • Distinct cases in the core scope fall below the minimum the buyer named in its request.
  • A repaired sample fails the same criterion a second time.
  • A gate misses its date without a written reason and a new date both sides accept.

What should a no-deal notice say?

It should name the gate that failed, the evidence behind that finding, whether the problem is curable and what would reopen the conversation, and it should disclose nothing that belongs to a third party. A clear notice keeps the relationship usable, lets the other side fix what is fixable before the next deal, and closes the file.

Public procurement has a tested model. Under FAR 15.506, a postaward debriefing of an unsuccessful offeror must include at least the government's evaluation of significant weaknesses or deficiencies in the proposal, the overall evaluated cost or price, and a summary of the rationale for award, and it may not include point-by-point comparisons with other offerors' proposals. Dataset deals have no such rule, but the shape travels well: explain your own findings in full, and never reveal another supplier's data, another buyer's results or anyone's prices.

A no-deal notice built on that shape contains:

  • The failed gate and the date it was tested.
  • The finding in one or two sentences, with the evidence it rests on: a contract clause, a test result or a count.
  • Whether the finding is curable by data work, by renegotiation or not at all.
  • What would reopen the deal, stated as a condition rather than a promise.
  • What happens to records or samples already exchanged, and by when.
  • A named contact for questions, and no comparison with other parties.

Records already shared, and the record of the decision

Data received for an evaluation that has ended should be returned or deleted as the agreement requires, with the deletion recorded, while the decision record, which holds no records, should be kept. Teams often do the reverse: they keep the sample in case and lose the notes that explain why the deal stopped.

Where personal data is involved, European law points the same way. Article 5 of the GDPR requires personal data to be limited to what is necessary for its purpose and kept in identifiable form no longer than necessary, and Recital 39 asks controllers to set time limits for erasure or periodic review. Once a deal has stopped, the evaluation purpose that justified holding a sample has usually ended too; counsel confirms what applies to the specific records and agreement. For the deletion itself, NIST published SP 800-88 Revision 2, Guidelines for Media Sanitization, in September 2025; it treats sanitization as an organization-wide program and includes a decision flow for choosing a method based on factors such as the sensitivity of the information and whether the media will be reused.

The no-deal record holds what the next team needs and nothing it should not have. Write it within days of the decision, let both deal owners read it, and keep it with the deal file:

Records already shared, and the record of the decision
FieldWhat to recordIllustrative entry
Deal referenceAn internal code, the request version and the dates of first contact and decisionDeal 114, request version 3, decided on day 41
Failed gateThe first gate that failed and the gate order usedGate 2, rights to the use
Deciding evidenceThe document or result that settled it, cited by location rather than copiedCustomer agreement template, confidentiality clause, section 9
Date the evidence existedWhen the deciding fact first appeared in anyone's filesDay 6, in the supplier's contract inventory
Effort after the evidenceHours both sides spent after that date, by task30 hours on sample preparation and testing
CurabilityData work, renegotiation, or not curable for this scopeNot curable for this scope; possible for contracts signed after a template change
Records exchanged and dispositionWhat was shared, where it was held, and the return or deletion step with its date200 de-identified tickets, deleted from the evaluation store on day 44
Reopen conditionThe condition that would justify a new reviewNew customer agreements that permit de-identified licensing

Putting kill criteria into the data request

The cheapest place to make a no-deal decision is the request itself. A buyer who writes down the task, the acceptance test, the minimum number of distinct cases and the uses it needs licensed lets gates one to four run before anyone prepares a record, and gives every supplier the same chance to say no early.

That is how requests to SourceX are meant to work. Nothing is pulled from inventory: SourceX looks for each dataset among U.S. businesses after a buyer asks, using a request form that records the buyer's licensing needs, timing, format, language, volume and scope. On the supply side, no data reaches SourceX until the company has agreed to the scope and terms and put its signature on the agreement, so a rights or terms failure surfaces before records move. A request is not a promise of a match; an acceptance test written in advance is the buyer's best protection against both a slow no and a premature one.

Frequently asked questions

When should a buyer walk away from a dataset deal?

Walk away at the first failure no data work can cure: the records cannot show what the task needs, the holder cannot license the use, or the terms stay outside one side's limits after a counterproposal. A failed sample is different. Check that the acceptance test passes known-good cases, then decide whether one timed repair is worth its cost.

Is a fast no-deal a failed project?

Not if the deciding evidence arrived early and cheaply. A no reached from documents before any records move costs a few hours and frees both sides to pursue better matches, while a yes forced through a mismatch costs far more later. Track the effort spent after the deciding evidence existed; that number shows whether the no was efficient.

What should a dataset rejection notice include?

The gate that failed, the evidence behind the finding, whether it is curable, what would reopen the deal and what happens to any shared records. Leave out comparisons with other suppliers and anyone else's data or prices, the same line U.S. federal debriefings under FAR 15.506 draw for unsuccessful offerors.

Should a buyer delete sample data after a no-deal?

Usually the agreement decides, and it should say so explicitly. Where personal data is involved, GDPR Article 5 limits retention to what the purpose requires, and an ended evaluation rarely still needs the sample. Keep the decision record, which cites evidence by location instead of copying records, and have counsel confirm the specific obligations.

How long should a dataset evaluation run before a decision?

Set a date for each gate before work starts rather than one overall deadline. Document gates such as task fit, rights and terms can usually resolve from paperwork, while sample gates take longer. When a gate misses its date without a written reason both sides accept, treat the default answer as no.

Sources

  • Arkes and Blumer's 1985 paper The psychology of sunk cost (Organizational Behavior and Human Decision Processes, vol. 35, issue 1) describes a greater tendency to continue an endeavor once money, effort or time has been invested. Source
  • Gary Klein's September 2007 HBR article describes the premortem, in which a team imagines a project has failed and lists reasons, citing research that prospective hindsight increases the ability to correctly identify reasons for future outcomes by 30 percent. Source
  • The Data Provenance Initiative audited more than 1,800 text datasets (Nature Machine Intelligence, 2024) and found license omission above 70 percent and error rates above 50 percent on widely used hosting sites, with traced licenses often more restrictive than listed. Source
  • OpenAI's SWE-bench Verified announcement describes professional developers screening SWE-bench samples because some unit tests were overly specific or unrelated and could reject correct solutions, and many issue descriptions were underspecified; the verified subset has 500 samples. Source
  • Anthropic's guide to agent evaluations says that with frontier models a 0 percent pass rate across many trials is most often a signal of a broken task, not an incapable agent. Source
  • FAR 15.506 requires a postaward debriefing to include the government's evaluation of significant weaknesses or deficiencies, the overall evaluated cost or price and a summary of the award rationale, and bars point-by-point comparisons with other offerors' proposals. Source
  • GDPR Article 5(1)(c) and (e) require personal data to be limited to what is necessary and kept in identifiable form no longer than necessary; Recital 39 asks controllers to set time limits for erasure or periodic review. Source
  • NIST published SP 800-88 Revision 2, Guidelines for Media Sanitization, on September 26, 2025, focusing on an organization-wide sanitization program and including a decision flow for choosing a sanitization method. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies