AI data market
AI data vendor technical due diligence: test a dataset like a dependency
By SourceX Editorial · Updated
10 sections
Short answer
Treat the dataset as a pinned dependency with a written interface. Before buying, agree the record unit and schema, require a version number and per-file checksums, and run your own validator on a sample drawn by an agreed method: integrity, schema, keys, missingness, value domains, coverage, duplicates and overlap with your evaluation set. Commit only after the sample passes in your pipeline and the supplier's change and continuity terms are written down.
Checklist
AI data vendor technical due
Key takeaways
- Write the data contract and schema before you judge any sample.
- Require a version scheme with defined meanings and per-file checksums, then pin every run to them.
- Run your own validator on the pilot sample and on every later delivery.
- Content can drift under an unchanged schema; diff records and field statistics between versions.
- The pilot should prove fit in your pipeline and a full acquisition cost, not just sample quality.
- Validators cannot confirm rights or de-identification; those go to legal and privacy review.
How do you validate a third-party dataset as an engineering dependency?
The same way you would vet a library before it goes into production: fix what you are depending on, write down the interface you expect, test a real copy against that interface in your own environment, and decide in advance what happens when the upstream changes. For a dataset, that means a version identifier and file checksums, a schema contract with keys and allowed values, an automated validator that runs on every delivery, and a pilot that proves the data works in your pipeline before the larger purchase.
AI data vendor technical due diligence often stops at questionnaires and a hand-picked sample. Those tell you what the supplier believes about its data. Dependency-style testing tells you what the files actually contain, whether the next delivery will look the same, and how much engineering work sits between the delivery and a usable training or evaluation set.
Where the software dependency analogy holds and where it breaks
It holds for identity, integrity and interfaces. A dataset can carry a version number with defined meaning, per-file hashes and a declared structure, and a consumer can pin to them. The Croissant metadata format from MLCommons recommends semantic MAJOR.MINOR.PATCH versions for datasets: a patch changes serialization or packaging only, a minor version can add fields or records while old data stays retrievable as it was, and a major version means existing data was edited, removed or reshuffled across splits. The same format lets each file declare a sha256 checksum and strongly recommends recording them for versioned datasets.
It breaks in three places. First, you cannot rebuild a dataset from source the way you rebuild a package, so a broken delivery is only fixed by the supplier. Second, the content can drift while the schema stays identical, which a type check will not catch. Third, a dataset carries permissions that code usually does not: the license term, permitted uses and any record-level restrictions are part of the dependency, and engineering checks cannot confirm them. Plan for all three explicitly.
Write the interface before you ask for a sample
Write a one-page data contract first, because a sample judged without one gets judged on impressions. The contract names the unit of record (a ticket, a work order, a claim), the fields and their types, which fields together identify a record, which values mean missing, and which fields link to other tables.
Use an existing schema language rather than prose. The Frictionless Table Schema standard, for example, defines a primaryKey whose fields cannot be null, uniqueKeys modeled on SQL unique constraints, and a missingValues list that says which strings are treated as missing. Whatever format you choose, it should be machine-readable so the validator can read it directly.
Ask for documentation that answers the questions your engineers will otherwise ask later. Datasheets for Datasets proposes questions grouped by stage of the dataset lifecycle, covering motivation, composition, collection, preprocessing and labeling, uses, distribution and maintenance; the Hugging Face dataset card puts license, language and size in a YAML metadata block at the top of the README. Either structure works; an empty maintenance section is a finding in itself.
What should the technical diligence checklist and sample validator include?
The checklist covers what you need to see before the pilot; the validator in the next section covers what you test on every file. Each row names the evidence and the condition that counts as a pass, so a reviewer can sign it off without a meeting.
| Area | Evidence to request | Pass condition | Owner |
|---|---|---|---|
| Identity and version | Dataset name, version number and changelog | Version scheme defined, with what each level of change means | Data engineering |
| Integrity | Manifest of files with sizes, record counts and checksums | Every delivered file matches the manifest | Data engineering |
| Schema contract | Machine-readable schema with keys, types and missing-value codes | Validator reads it without manual edits | Data engineering |
| Semantics | Data dictionary with field definitions and code lists | Every code in the sample appears in the dictionary | Research or domain lead |
| Coverage | Counts by period, segment and source system | Each slice your task needs is present at the agreed minimum | Research lead |
| Sampling method | Written description of how the sample was drawn | Draw is random or stratified as agreed, not curated | Research lead |
| Overlap and contamination | Hashes or a method to check against your evaluation data | Overlap measured and below the agreed limit | Evaluation lead |
| Rights metadata | License, permitted uses, term and record-level restrictions | Recorded in the manifest and reviewed by counsel | Legal |
| Change policy | Notice period and format for schema or content changes | Breaking changes require advance notice and a new major version | Procurement |
| Continuity and exit | Re-delivery, correction and end-of-term handling | Written procedure for corrections and for deletion or retention at term end | Procurement and legal |
The sample validator: ten checks to run on every delivery
Run the validator in your own environment against the schema contract, on the pilot sample and then on every later delivery. Keep the rule, the measured value and the result in a report stored beside the files, so the history of each version is auditable. Thresholds are yours to set per task; the point is that they are written down before the data arrives.
At a glance
The sample validator: ten checks to run on every delivery
Do not skip the last two checks because the schema passed. Duplicates inflate the volume you pay to integrate, and overlap with your evaluation data quietly inflates every score you report afterward. Lee and colleagues found many near-duplicate examples in common language modeling datasets and reported that train-test overlap affected over 4 percent of the validation set of standard datasets, enough to distort evaluation. A purchased dataset can overlap your private test set for the same reason public ones do: the same documents circulate between sources.
| Check | Rule | What to compute | If it fails |
|---|---|---|---|
| V1 Manifest and checksums | Each file hash matches the manifest | Recomputed hash per file | Reject the delivery and ask for a re-send |
| V2 Record counts | Counts match the manifest per file and per table | Row or object count | Reject; a silent truncation is common |
| V3 Schema conformance | Fields, types and formats match the contract | Per-field type error count | Reject or quarantine failing records by rule |
| V4 Key uniqueness | Primary and unique keys hold | Duplicate key count | Reject; ask how ids are generated |
| V5 Missingness | Missing rate per field within the agreed band | Share of declared missing codes and nulls | Flag; renegotiate if a required field is sparse |
| V6 Value domains | Codes, ranges and units within the dictionary | Out-of-domain value count per field | Flag for semantic review |
| V7 Referential links | Foreign keys resolve to delivered tables | Orphan record count | Reject if orphans break the task unit |
| V8 Period and segment coverage | Each required slice meets its minimum | Counts by period and segment | Flag; adjust scope or volume |
| V9 Duplicates | Exact and near duplicates within the agreed limit | Exact hash and near-duplicate rates | Flag; ask for de-duplicated counts |
| V10 Evaluation overlap | No overlap with your held-out evaluation set beyond the limit | Matches against evaluation hashes | Remove overlapping records before any training |
Pin, diff and upgrade: version control for a licensed dataset
Pin every training run to an exact dataset version and hash, and treat each new delivery as an upgrade that has to pass review. Tools built for this already exist: DVC, for example, records a hash for each tracked file or directory, and for data imported from another repository it records the source and a rev_lock commit hash, which only changes when someone explicitly updates the import. The general practice matters more than the tool: a run can always name the exact bytes it used.
Map the supplier's version numbers to your review effort. Semantic Versioning reserves the major number for incompatible changes, and the Croissant guidance applies the same idea to data. A patch delivery needs the integrity checks; a minor delivery needs the full validator and a check that old records are unchanged; a major delivery needs the validator, a fresh evaluation run and sign-off from whoever owns the downstream model. Diff the record ids and the per-field statistics between versions, not only the schema.
What the pilot has to prove before you sign for full volume
The pilot has one job: show that the data works in your stack and moves your own metric, at a cost you can predict for full volume. Agree its exit criteria before the sample is drawn.
Record effort in hours and runs, not impressions: how long mapping took, how many records were quarantined, how many validator runs a delivery needed before it passed. Those measurements scale to the full volume far more reliably than the license line alone, and they are what makes two suppliers comparable.
Checklist · 6 items
- The sample passes the validator, or fails only on checks with an agreed remedy.
- The data loads through your real pipeline, and you have measured the engineering hours needed to clean, map and validate it.
- A model trained or evaluated with the sample shows a measurable change against a baseline on held-out data the supplier never saw.
- The supplier has explained how the full delivery will be drawn, and the sample's coverage and duplicate rates are expected to hold at volume.
- The acquisition cost model is complete: license, integration and cleaning time, validation runs, storage, legal and privacy review, and the cost of handling corrections and re-deliveries.
- Rights metadata, the change policy and the exit procedure are written and agreed.
0 of 6 ticked. Tick each item as you go.
Worked example: a support-ticket dataset that changed under the same schema
This example is illustrative; the supplier, counts and rates are invented to show the method. An AI team licenses customer support tickets to train a triage model. The pilot sample of 20,000 tickets, version 1.2.0, passes all ten validator checks, and the evaluation run on the team's private held-out set shows an improvement over baseline. The team pins training to the version and its manifest hashes.
Three months later the supplier ships version 1.3.0, labeled a minor update. The schema check passes; the field names and types are identical. The validator flags two things anyway. V6 finds a new priority code that is not in the dictionary, and V8 shows that one product line's share of tickets has dropped sharply. The supplier explains that it merged two help desks and remapped priorities. Because existing records were edited, the team treats the delivery as a major change under its own policy, asks for an updated dictionary and a release note, reruns its evaluation, and retrains only after the regression suite holds. The training runs that used 1.2.0 remain reproducible because they are pinned to its hashes.
When dependency-grade testing is more than you need, and what it cannot cover
A small, one-time evaluation set delivered once and frozen does not need an upgrade process; integrity, schema, overlap and a careful read of the records are enough. Sensitive data that cannot leave the supplier changes where the validator runs, not whether it runs: agree that it executes in the supplier's environment or a controlled workspace and returns only the report.
No validator answers whether the supplier had the right to license the records, whether the de-identification is adequate, or whether permitted uses match your plans. Those questions go to legal and privacy reviewers with the contract and provenance evidence in hand, and a clean validation report should never be read as an answer to them.
Requesting business records built to a contract
Records from operating businesses are often a better fit for applied models than scraped text, but they arrive with the dependency problems above: legacy codes, merged systems and gaps by period. Writing the interface first and testing a sample against it is how a buyer keeps those problems from reaching training.
When the dependency is a dataset sourced through SourceX, the data request is the first draft of the interface: scope, volume, language, format, timing and licensing needs, which SourceX uses to find U.S. businesses to supply it rather than shipping something off a shelf. Deliveries too large to host stay in the seller's storage or arrive on encrypted drives, so decide early where the validator will run.
Frequently asked questions
Is a training dataset really a software dependency?
Functionally, yes: your model's behavior depends on it, it can change upstream, and a bad version can break a release. It differs in that you cannot rebuild it from source and it carries license terms, so the dependency process needs a legal review step that code dependencies often skip.
What is a sample validator for a dataset?
It is a script or job that reads the agreed schema and runs fixed checks on delivered files: checksums, record counts, types, keys, missingness, value domains, links, coverage, duplicates and overlap with your evaluation data. It produces a report with a pass or fail for each rule and keeps that report with the version.
Should the buyer or the supplier run the validator?
Both, ideally. The supplier can run it before shipping to catch obvious errors, but the buyer should run its own copy in its own environment, because the purpose is independent verification. When data cannot leave the supplier, agree on a controlled environment where the buyer's validator runs and only the report comes back.
How should a buyer handle a dataset that is updated every month?
Pin each training run to one version and its hashes, and treat every monthly delivery as an upgrade that must pass the validator before anyone uses it. Agree in the contract how breaking changes are announced and versioned, and diff record ids and field statistics between deliveries, not just the schema.
Which diligence items need legal rather than engineering review?
License scope, permitted uses, term, record-level restrictions, provenance and the adequacy of de-identification. Engineering checks can confirm that rights metadata is present and consistent, but not that the rights exist, so counsel should review the specific contract and evidence.
Sources
- The Croissant 1.1 specification recommends semantic versioning for datasets, defines what patch, minor and major changes mean for data, and strongly recommends sha256 checksums for files in versioned datasets. Source
- The Table Schema standard defines primaryKey, uniqueKeys and missingValues for tabular data contracts. Source
- Datasheets for Datasets proposes documentation questions grouped by dataset lifecycle stage, from motivation to maintenance. Source
- Hugging Face documents the dataset card as the repository README with a YAML metadata section for license, language and size. Source
- DVC documents how .dvc files record hashes for tracked data and, for imports, a rev_lock commit hash of the source repository. Source
- Lee and colleagues report many near-duplicate examples in language modeling datasets and train-test overlap affecting over 4 percent of validation sets of standard datasets. Source
- Semantic Versioning 2.0.0 reserves major version increments for incompatible changes. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.