Skip to content

AI data market

AI data vendor technical due diligence: test a dataset like a dependency

By SourceX Editorial · Updated

10 sections

Short answer

Treat the dataset as a pinned dependency with a written interface. Before buying, agree the record unit and schema, require a version number and per-file checksums, and run your own validator on a sample drawn by an agreed method: integrity, schema, keys, missingness, value domains, coverage, duplicates and overlap with your evaluation set. Commit only after the sample passes in your pipeline and the supplier's change and continuity terms are written down.

Checklist

AI data vendor technical due

This page covers AI data vendor technical due diligence: test a dataset like a dependency.Illustrative

Key takeaways

  • Write the data contract and schema before you judge any sample.
  • Require a version scheme with defined meanings and per-file checksums, then pin every run to them.
  • Run your own validator on the pilot sample and on every later delivery.
  • Content can drift under an unchanged schema; diff records and field statistics between versions.
  • The pilot should prove fit in your pipeline and a full acquisition cost, not just sample quality.
  • Validators cannot confirm rights or de-identification; those go to legal and privacy review.

How do you validate a third-party dataset as an engineering dependency?

The same way you would vet a library before it goes into production: fix what you are depending on, write down the interface you expect, test a real copy against that interface in your own environment, and decide in advance what happens when the upstream changes. For a dataset, that means a version identifier and file checksums, a schema contract with keys and allowed values, an automated validator that runs on every delivery, and a pilot that proves the data works in your pipeline before the larger purchase.

AI data vendor technical due diligence often stops at questionnaires and a hand-picked sample. Those tell you what the supplier believes about its data. Dependency-style testing tells you what the files actually contain, whether the next delivery will look the same, and how much engineering work sits between the delivery and a usable training or evaluation set.

Where the software dependency analogy holds and where it breaks

It holds for identity, integrity and interfaces. A dataset can carry a version number with defined meaning, per-file hashes and a declared structure, and a consumer can pin to them. The Croissant metadata format from MLCommons recommends semantic MAJOR.MINOR.PATCH versions for datasets: a patch changes serialization or packaging only, a minor version can add fields or records while old data stays retrievable as it was, and a major version means existing data was edited, removed or reshuffled across splits. The same format lets each file declare a sha256 checksum and strongly recommends recording them for versioned datasets.

It breaks in three places. First, you cannot rebuild a dataset from source the way you rebuild a package, so a broken delivery is only fixed by the supplier. Second, the content can drift while the schema stays identical, which a type check will not catch. Third, a dataset carries permissions that code usually does not: the license term, permitted uses and any record-level restrictions are part of the dependency, and engineering checks cannot confirm them. Plan for all three explicitly.

Write the interface before you ask for a sample

Write a one-page data contract first, because a sample judged without one gets judged on impressions. The contract names the unit of record (a ticket, a work order, a claim), the fields and their types, which fields together identify a record, which values mean missing, and which fields link to other tables.

Use an existing schema language rather than prose. The Frictionless Table Schema standard, for example, defines a primaryKey whose fields cannot be null, uniqueKeys modeled on SQL unique constraints, and a missingValues list that says which strings are treated as missing. Whatever format you choose, it should be machine-readable so the validator can read it directly.

Ask for documentation that answers the questions your engineers will otherwise ask later. Datasheets for Datasets proposes questions grouped by stage of the dataset lifecycle, covering motivation, composition, collection, preprocessing and labeling, uses, distribution and maintenance; the Hugging Face dataset card puts license, language and size in a YAML metadata block at the top of the README. Either structure works; an empty maintenance section is a finding in itself.

What should the technical diligence checklist and sample validator include?

The checklist covers what you need to see before the pilot; the validator in the next section covers what you test on every file. Each row names the evidence and the condition that counts as a pass, so a reviewer can sign it off without a meeting.

What should the technical diligence checklist and sample validator include?
AreaEvidence to requestPass conditionOwner
Identity and versionDataset name, version number and changelogVersion scheme defined, with what each level of change meansData engineering
IntegrityManifest of files with sizes, record counts and checksumsEvery delivered file matches the manifestData engineering
Schema contractMachine-readable schema with keys, types and missing-value codesValidator reads it without manual editsData engineering
SemanticsData dictionary with field definitions and code listsEvery code in the sample appears in the dictionaryResearch or domain lead
CoverageCounts by period, segment and source systemEach slice your task needs is present at the agreed minimumResearch lead
Sampling methodWritten description of how the sample was drawnDraw is random or stratified as agreed, not curatedResearch lead
Overlap and contaminationHashes or a method to check against your evaluation dataOverlap measured and below the agreed limitEvaluation lead
Rights metadataLicense, permitted uses, term and record-level restrictionsRecorded in the manifest and reviewed by counselLegal
Change policyNotice period and format for schema or content changesBreaking changes require advance notice and a new major versionProcurement
Continuity and exitRe-delivery, correction and end-of-term handlingWritten procedure for corrections and for deletion or retention at term endProcurement and legal

The sample validator: ten checks to run on every delivery

Run the validator in your own environment against the schema contract, on the pilot sample and then on every later delivery. Keep the rule, the measured value and the result in a report stored beside the files, so the history of each version is auditable. Thresholds are yours to set per task; the point is that they are written down before the data arrives.

At a glance

The sample validator: ten checks to run on every delivery

Do not skip the last two checks because the schema passed. Duplicates inflate the volume you pay to integrate, and overlap with your evaluation data quietly inflates every score you report afterward. Lee and colleagues found many near-duplicate examples in common language modeling datasets and reported that train-test overlap affected over 4 percent of the validation set of standard datasets, enough to distort evaluation. A purchased dataset can overlap your private test set for the same reason public ones do: the same documents circulate between sources.

The sample validator: ten checks to run on every delivery
CheckRuleWhat to computeIf it fails
V1 Manifest and checksumsEach file hash matches the manifestRecomputed hash per fileReject the delivery and ask for a re-send
V2 Record countsCounts match the manifest per file and per tableRow or object countReject; a silent truncation is common
V3 Schema conformanceFields, types and formats match the contractPer-field type error countReject or quarantine failing records by rule
V4 Key uniquenessPrimary and unique keys holdDuplicate key countReject; ask how ids are generated
V5 MissingnessMissing rate per field within the agreed bandShare of declared missing codes and nullsFlag; renegotiate if a required field is sparse
V6 Value domainsCodes, ranges and units within the dictionaryOut-of-domain value count per fieldFlag for semantic review
V7 Referential linksForeign keys resolve to delivered tablesOrphan record countReject if orphans break the task unit
V8 Period and segment coverageEach required slice meets its minimumCounts by period and segmentFlag; adjust scope or volume
V9 DuplicatesExact and near duplicates within the agreed limitExact hash and near-duplicate ratesFlag; ask for de-duplicated counts
V10 Evaluation overlapNo overlap with your held-out evaluation set beyond the limitMatches against evaluation hashesRemove overlapping records before any training

Pin, diff and upgrade: version control for a licensed dataset

Pin every training run to an exact dataset version and hash, and treat each new delivery as an upgrade that has to pass review. Tools built for this already exist: DVC, for example, records a hash for each tracked file or directory, and for data imported from another repository it records the source and a rev_lock commit hash, which only changes when someone explicitly updates the import. The general practice matters more than the tool: a run can always name the exact bytes it used.

Map the supplier's version numbers to your review effort. Semantic Versioning reserves the major number for incompatible changes, and the Croissant guidance applies the same idea to data. A patch delivery needs the integrity checks; a minor delivery needs the full validator and a check that old records are unchanged; a major delivery needs the validator, a fresh evaluation run and sign-off from whoever owns the downstream model. Diff the record ids and the per-field statistics between versions, not only the schema.

What the pilot has to prove before you sign for full volume

The pilot has one job: show that the data works in your stack and moves your own metric, at a cost you can predict for full volume. Agree its exit criteria before the sample is drawn.

Record effort in hours and runs, not impressions: how long mapping took, how many records were quarantined, how many validator runs a delivery needed before it passed. Those measurements scale to the full volume far more reliably than the license line alone, and they are what makes two suppliers comparable.

Checklist · 6 items

  1. The sample passes the validator, or fails only on checks with an agreed remedy.
  2. The data loads through your real pipeline, and you have measured the engineering hours needed to clean, map and validate it.
  3. A model trained or evaluated with the sample shows a measurable change against a baseline on held-out data the supplier never saw.
  4. The supplier has explained how the full delivery will be drawn, and the sample's coverage and duplicate rates are expected to hold at volume.
  5. The acquisition cost model is complete: license, integration and cleaning time, validation runs, storage, legal and privacy review, and the cost of handling corrections and re-deliveries.
  6. Rights metadata, the change policy and the exit procedure are written and agreed.

0 of 6 ticked. Tick each item as you go.

Worked example: a support-ticket dataset that changed under the same schema

This example is illustrative; the supplier, counts and rates are invented to show the method. An AI team licenses customer support tickets to train a triage model. The pilot sample of 20,000 tickets, version 1.2.0, passes all ten validator checks, and the evaluation run on the team's private held-out set shows an improvement over baseline. The team pins training to the version and its manifest hashes.

Three months later the supplier ships version 1.3.0, labeled a minor update. The schema check passes; the field names and types are identical. The validator flags two things anyway. V6 finds a new priority code that is not in the dictionary, and V8 shows that one product line's share of tickets has dropped sharply. The supplier explains that it merged two help desks and remapped priorities. Because existing records were edited, the team treats the delivery as a major change under its own policy, asks for an updated dictionary and a release note, reruns its evaluation, and retrains only after the regression suite holds. The training runs that used 1.2.0 remain reproducible because they are pinned to its hashes.

When dependency-grade testing is more than you need, and what it cannot cover

A small, one-time evaluation set delivered once and frozen does not need an upgrade process; integrity, schema, overlap and a careful read of the records are enough. Sensitive data that cannot leave the supplier changes where the validator runs, not whether it runs: agree that it executes in the supplier's environment or a controlled workspace and returns only the report.

No validator answers whether the supplier had the right to license the records, whether the de-identification is adequate, or whether permitted uses match your plans. Those questions go to legal and privacy reviewers with the contract and provenance evidence in hand, and a clean validation report should never be read as an answer to them.

Requesting business records built to a contract

Records from operating businesses are often a better fit for applied models than scraped text, but they arrive with the dependency problems above: legacy codes, merged systems and gaps by period. Writing the interface first and testing a sample against it is how a buyer keeps those problems from reaching training.

When the dependency is a dataset sourced through SourceX, the data request is the first draft of the interface: scope, volume, language, format, timing and licensing needs, which SourceX uses to find U.S. businesses to supply it rather than shipping something off a shelf. Deliveries too large to host stay in the seller's storage or arrive on encrypted drives, so decide early where the validator will run.

Frequently asked questions

Is a training dataset really a software dependency?

Functionally, yes: your model's behavior depends on it, it can change upstream, and a bad version can break a release. It differs in that you cannot rebuild it from source and it carries license terms, so the dependency process needs a legal review step that code dependencies often skip.

What is a sample validator for a dataset?

It is a script or job that reads the agreed schema and runs fixed checks on delivered files: checksums, record counts, types, keys, missingness, value domains, links, coverage, duplicates and overlap with your evaluation data. It produces a report with a pass or fail for each rule and keeps that report with the version.

Should the buyer or the supplier run the validator?

Both, ideally. The supplier can run it before shipping to catch obvious errors, but the buyer should run its own copy in its own environment, because the purpose is independent verification. When data cannot leave the supplier, agree on a controlled environment where the buyer's validator runs and only the report comes back.

How should a buyer handle a dataset that is updated every month?

Pin each training run to one version and its hashes, and treat every monthly delivery as an upgrade that must pass the validator before anyone uses it. Agree in the contract how breaking changes are announced and versioned, and diff record ids and field statistics between deliveries, not just the schema.

Which diligence items need legal rather than engineering review?

License scope, permitted uses, term, record-level restrictions, provenance and the adequacy of de-identification. Engineering checks can confirm that rights metadata is present and consistent, but not that the rights exist, so counsel should review the specific contract and evidence.

Sources

  • The Croissant 1.1 specification recommends semantic versioning for datasets, defines what patch, minor and major changes mean for data, and strongly recommends sha256 checksums for files in versioned datasets. Source
  • The Table Schema standard defines primaryKey, uniqueKeys and missingValues for tabular data contracts. Source
  • Datasheets for Datasets proposes documentation questions grouped by dataset lifecycle stage, from motivation to maintenance. Source
  • Hugging Face documents the dataset card as the repository README with a YAML metadata section for license, language and size. Source
  • DVC documents how .dvc files record hashes for tracked data and, for imports, a rev_lock commit hash of the source repository. Source
  • Lee and colleagues report many near-duplicate examples in language modeling datasets and train-test overlap affecting over 4 percent of validation sets of standard datasets. Source
  • Semantic Versioning 2.0.0 reserves major version increments for incompatible changes. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies