AI uses for records
Dataset quality vs buyer fit mismatch: three gates clean data fails
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A dataset quality buyer fit mismatch means records pass technical checks such as validity, completeness and uniqueness yet do not serve the buyer's task. Diagnose it with three ordered gates: signal (records hold the inputs, the action and an independent outcome), distribution (the mix matches where the model will run) and scope (rights, period and volume per slice cover the use). Only distribution gaps reliably yield to more data.
Checklist
Dataset quality
Key takeaways
- Quality checks compare records with a schema; fit compares them with a task, so a spotless quality report can coexist with a fit failure.
- Run three fit gates in order after quality: signal, distribution and scope, then check that the buyer's acceptance test measures the task.
- Signal and scope gaps cannot be cleaned away; they need a different task, different records or a renegotiated agreement.
- Distribution gaps are the one fit failure more data often fixes, through targeted sourcing or reweighting of missing slices.
- Log each rejection by gate with evidence so the next offer is screened before a sample leaves the supplier.
Why can a clean dataset still fail buyer fit?
A clean dataset fails buyer fit because quality checks compare records with a schema and with themselves, while fit compares them with a task. Valid formats, filled fields and unique keys say the data is well kept; they do not say it contains the decisions, outcomes, mix and permissions a particular model needs.
Data quality research drew this line decades ago. Wang and Strong's 1996 framework in the Journal of Management Information Systems, built from surveys of data consumers, separates intrinsic quality, whether data is good in its own right, from contextual quality, which is judged against the task at hand. A dataset can score well on the first and poorly on the second, and buyers pay for the second.
Regulation now uses the same split for some systems. Article 10 of the EU AI Act requires training, validation and testing data for high-risk AI systems to be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose. Error-free and complete are quality words; relevant and representative are fit words, and both sets are measured against a purpose. That is general information, not legal advice, and most commercial datasets sit outside that high-risk scope.
What a passing quality report proves, and what it leaves open
A passing quality report proves that the checks someone wrote came back clean. It cannot prove the checks asked the buyer's questions, because they were usually written before anyone knew what the buyer's model would do.
Validation tools make this explicit. Great Expectations, an open-source validation library, describes its expectations as unit tests for data: each one tests a declared rule, such as a column never being null or a value falling in a range. A suite written by the supplier encodes the supplier's assumptions, and a suite with no task in it cannot fail for task reasons.
The gap tends to surface late and far from its cause. Sambasivan and colleagues, interviewing 53 AI practitioners for a 2021 CHI paper, found data cascades, compounding downstream problems caused by data issues, with 92 percent prevalence, and described them as invisible and delayed. A fit failure discovered in a model evaluation weeks after delivery is that pattern in miniature.
Read a quality report with both columns in mind:
- It shows: formats are valid, required fields are present, keys are unique and linked records join.
- It shows: values fall in declared ranges and codes come from declared lists.
- It does not show: whether each record holds the decision and outcome the task needs.
- It does not show: whether the mix of customers, channels, years or languages matches deployment.
- It does not show: whether rights, period and volume per slice cover the intended use.
Gate one: do the records carry the task's signal?
The first fit gate asks whether each record contains the inputs a model would see, the action a person took and an outcome recorded independently of that action. If any of the three is missing, the dataset cannot teach or test the task, however clean it is.
Agent evaluation practice offers a useful standard for the outcome part. Anthropic's engineering guide to agent evals defines a task by its inputs and success criteria and judges the outcome by the final state of the environment: for a flight-booking agent, whether a reservation exists in the database matters more than the agent saying it booked one. Business records need the same discipline. A ticket marked resolved by the person who closed it is a claim; a ticket with no reopen in the following weeks, or a refund that cleared, is an outcome.
Test signal with a replay. Draw a seeded sample, hide everything recorded after the decision point and ask a reviewer who knows the work to reconstruct what was decided and why from what remains. Records that cannot be replayed, such as notes copied from a template or histories that keep only the final state, fail this gate. Cleaning cannot add a decision that was never written down; the only exits are a different task the records do support, or different records.
Gate two: does the record mix match where the model will run?
The second gate asks whether the mix of records, by customer type, channel, region, language, year and case difficulty, resembles the population the model will serve. A clean dataset drawn from the wrong mix teaches the wrong habits.
The cost of a mismatched mix is well documented. The WILDS benchmark, ten datasets built around real deployment shifts such as different hospitals and camera locations, found models performing substantially worse on shifted data than on data like their training set, with the gap persisting even under methods designed for shift. A support model trained on one region's tickets or a forecasting model trained on pre-migration years faces the same risk.
Test distribution with a slice table: list the slices the buyer's deployment depends on, the share of each in the buyer's expected traffic and the share in the dataset, and agree a tolerance before comparing. This is the one gate where more data often helps, through targeted sourcing of missing slices or reweighting, provided the missing slices exist somewhere the buyer can license.
Gate three: do rights, period and volume cover the intended use?
The third gate asks whether the permitted use, time window, geography and record count for each required slice cover what the buyer intends to do. A dataset can carry strong signal in the right mix and still fail because the agreement excludes model training, the period stops short or a critical slice holds too few records.
Scope problems are not technical, and they are common. When the Data Provenance Initiative traced the licenses of more than 1,800 text datasets for a 2024 paper in Nature Machine Intelligence, it found that popular hosting sites left the license out for over two-thirds of them and mislabeled it for over 50 percent, and the corrected terms were often stricter than the labels. No validation suite catches a license that says less than everyone assumed.
Volume belongs here too, judged per slice rather than in total. SourceX's general guide puts the usual buyer bar at a consistent history running typically two years or longer and a record count of tens of thousands or more, kept in one system, though smaller sets can qualify when the work is rare; each case is assessed individually. A scope gap goes back to the record owner and the agreement, because at SourceX the record owner approves scope and terms, then signs, before a single record is handed over.
The fit-gap decision tree: which exit applies?
Run the gates in order and stop at the first failure: technical quality, then signal, then distribution, then scope, then the alignment of the buyer's test with the task. The order matters because each later gate is meaningless if an earlier one fails, and because only the quality exit and, partly, the distribution exit can be closed with data work.
The decision tree below names the question at each gate, the evidence that answers it, the exit if it fails and whether cleaning or more data can close the gap:
| Gate | Question | Evidence to collect | Exit if it fails | Fixable by cleaning or more data? |
|---|---|---|---|---|
| 0. Quality | Do records pass validity, completeness, uniqueness and linkage checks? | Validator results by field and file | Repair or remove defects, then rerun | Yes, by cleaning |
| 1. Signal | Does each record hold inputs, the action taken and an independent outcome? | A replay of a seeded sample by a reviewer who knows the work | Re-scope to a task the records support, or decline | No |
| 2. Distribution | Do slice shares match the deployment profile within an agreed tolerance? | A slice table against the buyer's expected traffic | Source missing slices, reweight or narrow the claimed use | Partly, with more records |
| 3. Scope | Do rights, period, geography and volume per slice cover the use? | Approved scope, license terms and counts per slice | Renegotiate scope or terms, or narrow the use | No |
| 4. Test alignment | Does the written acceptance test measure the task? | Test version, the buyer's model evaluation and failing cases | Rewrite the test with the buyer before more data work | No |
Logging a fit rejection so the reason survives
Record every fit rejection against the gate it failed, with evidence, so the reason survives handoffs and the next offer is screened before it goes out. A rejection logged only as not a fit teaches nothing.
At a glance
Logging a fit rejection so the reason survives
Datasheets for Datasets, the documentation template proposed by Gebru and coauthors, asks creators to state a dataset's motivation, composition, collection process and recommended uses. A fit log is the buyer-facing mirror of that datasheet: it records which use the data was offered for, which gate it failed and what would have to change.
Capture these fields for each rejection:
- Gate code: Q for quality, S for signal, D for distribution, R for rights and scope, T for test alignment.
- The buyer's intended use, in one sentence, and the acceptance test version applied.
- The evidence: replay results, the slice table, the scope clause or the failing cases.
- Whether a technical fix was attempted, by whom, and what it changed.
- The decision: re-scope, source more, renegotiate, rewrite the test or decline.
- One line for the next offer: what a dataset must show before this buyer sees it.
Could the buyer's acceptance test be the real mismatch?
Sometimes the data fits and the test does not. Before accepting a fit rejection, check whether the written acceptance test actually measures the buyer's task, because a test built from generic quality rules can reject useful data or pass useless data.
The check is simple to run. Compare the acceptance test result with the buyer's own model evaluation on a small slice: if the evaluation improves while the acceptance test fails, or the test passes while the evaluation shows no gain, the test is misaligned. Rewriting it is cheaper than reworking a dataset to satisfy the wrong criteria.
Two other explanations deserve a hearing. The rejection may be commercial, a price or term dressed as a fit concern, which no data work addresses. Or the buyer may want something the supplier never had. OpenAI's data partnerships page, for example, describes interest in large datasets that reflect human intention and are not easily accessible online, and a buyer with that brief will decline tidy transactional tables however clean they are. Naming the real reason keeps both sides from spending weeks on the wrong fix.
Illustrative: tidy work orders rejected for a dispatch model
This example is illustrative, not a real deal. A field-service company offers three years of work orders, about 120,000 records, to a buyer building a dispatch model that recommends which technician to send. The quality report is spotless: valid schema, unique order IDs, timestamps in order and nearly every required field filled.
The fit gates tell a different story. Signal fails first: each order records the technician who went, but not who else was available, the travel times considered or why the dispatcher chose as they did, so the decision cannot be replayed. Distribution would have failed next, since most orders are residential while the buyer deploys in commercial accounts, though the review never needed to get that far.
The useful outcome is a re-scope rather than a rework. The same orders do hold inputs, actions and independent outcomes for a different task: estimating repair duration from the reported symptom, equipment type and site history, with the closed-out labor time as the outcome. Offered for that task and re-run through the gates, the dataset passes signal and is judged on distribution and scope like any other.
Turning a failed fit test into a sharper data request
The cheapest place to catch a fit gap is the request. A buyer who states the task, the deployment mix, the scope and the acceptance test up front lets a supplier run the gates before a sample exists.
At a glance
Turning a failed fit test into a sharper data request
Datasets through SourceX are found among U.S. businesses for each request rather than held in stock, and the request-data form asks for language, scope, format, volume, timing and licensing needs. The steps after that are SourceX's to run: finding and vetting a supplier, checking its rights, writing the contract, getting the data ready, then delivery and payment. Adding the intended use and an acceptance test to the request turns buyer fit from a late surprise into an early screen.
A fit-ready request includes:
- The task in one sentence, naming the inputs, the action and the outcome the model must learn or be tested on.
- The deployment profile: slices and expected shares, with an agreed tolerance.
- Scope: permitted use, period, geography and minimum records per critical slice.
- The acceptance test, versioned, plus the model evaluation it is meant to predict.
- What a rejection will include: gate code, evidence and failing cases.
Frequently asked questions
What are the dimensions of data quality, and do they cover buyer fit?
Common lists name accuracy, completeness, consistency, timeliness, uniqueness and validity. Those describe whether data is well kept. Wang and Strong's 1996 framework added contextual quality, judged against the task at hand, and that is where buyer fit lives: signal, mix and scope relative to a specific use.
Can collecting more data fix a buyer fit mismatch?
Only some of the time. More records help when the gap is distribution, meaning the right kind of records in the wrong mix, and the missing slices exist and can be licensed. More records do not help when each record lacks the decision or outcome the task needs, or when the agreement does not permit the use.
Who should define buyer fit, the supplier or the buyer?
The buyer defines the task, the deployment profile and the acceptance test, because only the buyer knows how the model will be used. The supplier documents what the records contain and how they were collected, so the buyer can judge fit before a sample is prepared.
Does the EU AI Act require training data to fit its intended purpose?
For high-risk AI systems, Article 10 requires training, validation and testing data to be relevant, sufficiently representative and, as far as possible, free of errors and complete in view of the intended purpose. Many commercial datasets fall outside that scope. This is general information; counsel should assess a specific system.
Is a fit rejection a sign that the supplier's data is bad?
No. A fit rejection says the records do not serve one buyer's task, not that they are poorly kept. The same records may pass every gate for a different task, which is why a rejection should record the intended use and the gate that failed.
Sources
- Wang and Strong's 1996 data quality framework, built from data consumer surveys, separates intrinsic quality from contextual quality judged against the task at hand. Source
- Article 10(3) of Regulation (EU) 2024/1689 requires training, validation and testing data for high-risk AI systems to be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose. Source
- Great Expectations describes expectations as expressive and extensible unit tests for data. Source
- Sambasivan et al. (CHI 2021) interviewed 53 AI practitioners and found data cascades, compounding downstream problems from data issues, with 92 percent prevalence, often invisible and delayed. Source
- Anthropic's engineering guide to agent evals defines a task by inputs and success criteria and judges the outcome by the final state of the environment, such as whether a reservation exists in the database. Source
- The WILDS benchmark of ten datasets reflecting real deployment shifts found substantially lower out-of-distribution than in-distribution performance, a gap that persisted with methods designed for shift. Source
- The Data Provenance Initiative audit of more than 1,800 text datasets (Nature Machine Intelligence, 2024) found license omission above two-thirds and error rates above 50 percent on hosting sites, with corrected licenses often more restrictive. Source
- Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process and recommended uses. Source
- OpenAI's data partnerships page describes interest in large datasets that reflect human intention and are not easily accessible online. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.