Skip to content

Systems and records

Dataset portability for buyer acceptance: how to test it before integration

By SourceX Editorial · Updated

9 sections

Short answer

Test portability by rebuilding the delivered package in a fresh workspace that has no access to the supplier's systems. The package passes when it loads from relative paths alone, its declared schema, keys and splits hold, its checksums verify, and rerunning the supplier's own validator in that workspace reproduces the counts and reports the supplier shipped. Anything that needs a supplier login, a missing script or an undocumented code is a portability failure.

Checklist

Dataset portability buyer

This page covers dataset portability for buyer acceptance: how to test it before integration.Illustrative

Key takeaways

  • A portability test asks whether a stranger holding only the package can load, interpret and rebuild it; byte integrity and label quality are separate checks.
  • Run the test in a clean workspace with pinned tools, no network route to the supplier and the package's own documentation as the only instructions.
  • Replay means rerunning the supplier's shipped validator or build step and comparing its outputs to the reports the supplier delivered, with tolerances written down in advance.
  • Most failures trace to a short list: absolute or mixed paths, unversioned schemas, timestamps without zone semantics, keys regenerated per export and undeclared splits.
  • Decide before testing which failures send the package back and which only need a documented workaround, and tie fixes to a version bump with a defined meaning.

How do you test dataset portability before the buyer integrates it?

Copy the delivered package into a workspace that has never seen it, cut every route back to the supplier, and try to rebuild the dataset using only what is in the package. If the import, the schema checks, the key checks and a rerun of the supplier's own validation all succeed and match what the supplier reported, the package is portable. If any step needs a supplier credential, a file that is not in the package or an explanation by email, it is not.

The test is worth running before integration because integration hides portability faults. Once an engineer has patched a path, guessed a timestamp zone and remapped a few identifiers inside the buyer's pipeline, the package appears to work, but the fixes live in one person's notebook. The next delivery, the next team or an auditor will hit the same gaps. A clean-room import surfaces them while the supplier can still fix them at the source.

The output is a short acceptance record: the workspace description, each step's result, the evidence files and a pass, conditional pass or return decision. That record is what the buyer's acceptance clause can point to.

What is a portability test, and what does it leave to other checks?

A portability test checks that the package is self-sufficient: it can be located, loaded, typed, joined, split and validated by someone outside the supplier's environment. It sits between two other checks that buyers often run and sometimes confuse with it.

Integrity verification asks whether every byte arrived. Packaging formats make that precise. The BagIt specification, RFC 8493, treats a bag as complete only when every payload file is listed in every payload manifest, and as valid only when it is complete and every checksum verifies. A package can pass that test and still be unusable, because checksums say nothing about whether a path resolves or a code is documented.

Quality acceptance asks whether the records are accurate, consistent and fit for the use case: label audits, residual personal-data sampling, coverage against the specification. That work assumes the package already loads correctly. Running portability first keeps quality reviewers from spending their time on import problems.

Rights and permissions are a separate question again. A package that rebuilds perfectly in a clean room tells you nothing about whether the supplier may license the records or whether personal data was handled lawfully. Keep that review in its own track.

Which package properties break first when a dataset leaves the supplier?

The same handful of properties fail again and again, and published packaging standards already say how each should behave. The table maps each failure to the rule it breaks and the check that catches it in a fresh workspace. The rules cited come from the Data Package standard, the Croissant 1.1 specification, the Parquet format and the Hugging Face dataset configuration docs.

Which package properties break first when a dataset leaves the supplier?
PropertyHow it breaks outside the supplierWhat the standard expectsCheck in the fresh workspace
File locationsPaths point to a supplier bucket, a mounted drive or an absolute directory on the exporter's laptop.The Data Package security guidance treats local paths as relative to the descriptor file and disallows absolute paths; a resource path array may not mix URLs and relative paths.Resolve every path relative to the descriptor with network access off; any absolute path or failed lookup fails the step.
Schema versionField definitions live in a wiki or an email thread, so the buyer cannot tell which revision the files follow.Croissant datasets declare conformsTo with a versioned URI and carry a version for the content.Confirm the descriptor names its schema and content versions and that they match the delivery note.
Row identityPrimary keys are regenerated or re-hashed on each export, so joins across deliveries silently break.Table Schema defines a primary key as fields that uniquely identify each row and cannot be null; Croissant RecordSets can declare a key.Test uniqueness and non-null on declared keys, and compare a sample of keys with the previous delivery if one exists.
Cross-table referencesChild rows reference parents that were filtered out of the extract.Table Schema foreign keys and Croissant field references model SQL foreign keys between resources.Count orphaned child rows for every declared reference; the expected count is zero unless documented.
TimestampsLocal times are exported without a zone, so event order shifts when the buyer's tools assume UTC.Parquet's TIMESTAMP type carries an isAdjustedToUTC flag that separates instant semantics from local semantics.Read the logical type of every timestamp column and confirm the documented zone rule matches it.
Missing valuesBlanks, zeros and placeholder strings all mean unknown in different columns.Table Schema lets a schema declare missingValues, the strings to be treated as missing.Profile each column for undeclared sentinel values and compare against the declared list.
SplitsTrain, validation and test membership exists only in the supplier's training scripts.Hugging Face dataset repositories can map files to named splits in a configs block, and Croissant can describe splits as records.Rebuild each split from the package's own declaration and compare row counts to the delivery note.
File hashesHashes are listed for some files, or computed before a final re-export.Data Resource hash defaults to MD5 with other algorithms named by prefix; Croissant strongly recommends sha256 for every FileObject in a versioned dataset.Recompute every listed hash and list files present on disk but absent from the manifest.

How should the fresh workspace be set up so the result means something?

The workspace should be as ignorant as a future integrator: a new container or virtual machine, a read-only copy of the package, pinned versions of the reading tools, and no network route to the supplier's storage, warehouse or code repositories. If the workspace can quietly reach a supplier bucket, a broken path may resolve by accident and the test passes for the wrong reason.

Give the tester only the package. The README, data card or Croissant file inside it is the instruction set. If the tester needs to ask the supplier a question to proceed, log the question verbatim; each one is a documentation defect even if the answer is easy.

Record the environment as evidence: operating system image, tool names and versions, the package's version string and the time of the run. Without that record, a later disagreement about whether the package loaded becomes a dispute about whose machine was configured differently.

  • Fresh container or VM, built from a recorded image.
  • Read-only copy of the package, verified against its manifest before anything else runs.
  • Outbound network off, or limited to public package indexes for installing pinned tools.
  • No supplier credentials, tokens, VPN profiles or shared drives.
  • A tester who did not take part in scoping the purchase, so tacit knowledge does not fill gaps.

The fresh-workspace import and replay acceptance guide

Run the steps in order and stop at the first hard failure; later steps depend on earlier ones. Each row names the evidence to keep so that the acceptance record can be assembled from files rather than recollection. Owners are suggestions for a typical buyer team and can be reassigned.

The fresh-workspace import and replay acceptance guide
StepActionPass conditionEvidence to keepOwner
1. IntegrityVerify every file against the package manifest before opening any data.All listed hashes match and no unlisted payload files exist.Manifest verification log with counts of files checked, matched and unexpected.Data engineer
2. DescriptorValidate the package descriptor (for example datapackage.json or a Croissant JSON-LD file) with its reference validator.Validator exits cleanly at the declared spec version.Validator output and the validator's version.Data engineer
3. Path resolutionResolve every resource path relative to the descriptor with the network off.Every path resolves inside the package; no absolute paths or private URLs.List of resolved paths and any failures.Data engineer
4. Typed loadLoad each resource with the declared types and missing-value rules.No type coercion errors; undeclared sentinel values below the agreed limit.Per-column type report and sentinel profile.Data engineer
5. Keys and referencesTest declared primary keys for uniqueness and non-null, and count orphans for each reference.Zero duplicates and zero orphans, or exceptions listed in the documentation.Key and orphan counts per resource.Integration owner
6. SemanticsCheck that every field used downstream has a definition, unit, zone rule or code list in the package.No downstream field relies on knowledge held outside the package.Field coverage list with gaps marked.Integration owner
7. Splits and countsRebuild each declared split and total from the package alone.Counts match the delivery note exactly.Count table: declared versus rebuilt.Data engineer
8. ReplayRun the supplier's shipped validator or build step inside the workspace.It runs without supplier access and its outputs match the shipped reports within written tolerances.Replay outputs, diff against shipped reports, tolerance table.Data engineer
9. Questions logCollect every question the tester needed answered to proceed.No question was required, or each one is fixed in the documentation.Questions log with the supplier's fixes.Integration owner
10. DecisionClassify the result as pass, conditional pass or return, and file the record.Decision matches the rules agreed before testing.Signed acceptance record with links to steps 1 to 9.Buyer acceptance lead

What does replay mean, and what has to match exactly?

Replay means running, in the buyer's clean workspace, the same validation or build steps the supplier ran before shipping, and comparing the outputs. The supplier delivers its validator, its expected outputs and the command that produced them; the buyer reruns the command and diffs the results. If the supplier cannot ship a runnable validator, ask for its expected outputs as plain reports and rebuild the checks from the documentation instead.

Write the comparison rules before the run. Counts, key sets, split membership and hash lists should match exactly, because any difference there means the package is not the one the supplier tested. Floating-point summaries such as means or quantiles can differ in the last digits across libraries, so state a tolerance. Row order should not matter unless the documentation says it does.

A replay that needs the supplier's warehouse, a private package, an internal reference table or an environment variable pointing to a supplier service has failed, even if the outputs would have matched. That dependency is exactly what portability is meant to remove.

If you use Croissant, its reference library includes a command that validates a JSON-LD descriptor, which makes step 2 repeatable. For Data Package descriptors, use the validator that matches the declared version of the standard.

Worked example (illustrative): a field service work-order package

This example is illustrative; the counts are invented to show the procedure and do not describe a real delivery. A buyer receives a package of field service records: a descriptor, work orders, line items, technician reference rows and a folder of job photos, with a delivery note stating row counts per resource and per split.

Steps 1 and 2 pass. Step 3 fails on one resource: the photo index uses absolute paths to the supplier's object storage, so 100 percent of photo links fail with the network off. Step 4 passes with a note: the scheduled_start column is a local timestamp with isAdjustedToUTC set to false, and the documentation says local time but not which zone. Step 5 finds that about 3 percent of line items reference work orders absent from the extract; the supplier's filter removed cancelled jobs but kept their lines.

Step 8, replay, exposes a further issue. The supplier's validator imports an internal helper package that is not in the delivery, so it cannot run. The buyer rebuilds counts from the documentation and they match, which shows the data is intact but the validator is not portable.

The decision is return with three fixes: relative photo paths, a stated zone per site, and either removal of orphaned lines or a documented reason to keep them. The validator dependency is a conditional item: the supplier vendors the helper into the package. Because the fixes change paths and remove rows rather than only repackaging, the corrected delivery should carry a new major version under Croissant's versioning guidance, not a patch.

Which result should send the package back, and which only needs a note?

Return the package when a failure would force the buyer to hold knowledge outside it: unresolvable paths, missing schemas, broken keys, undeclared splits or a validator that needs supplier systems. Accept with a note when the package is self-sufficient but a definition is thin or a tolerance needs agreeing. Agree this split in writing before testing so the decision is not negotiated after the fact.

Some packages need a narrower test rather than a weaker one. Croissant describes live datasets whose files keep changing and recommends no checksum on files still expected to update; for those, test the frozen files fully and test the live ones on structure only, with a cut-off date recorded. Multi-terabyte packages that cannot be copied into a test workspace can be tested on a manifest-selected sample of shards plus a full manifest verification where the data sits.

Version fixes so that later deliveries are interpretable. Croissant's guidance is a useful default: a patch changes serialization only, a minor version adds data without altering what was there, and a major version changes or removes existing records. A fix that rewrites keys is a major change even if it feels like a correction.

Bring in a specialist when the portability problem is really a rights or privacy problem in disguise, for example when photos cannot be shipped because they show customers, or when identifiers cannot be stabilized without re-identifying people. Those decisions belong to counsel and the privacy owner, not to the acceptance engineer.

Where a sourcing partner fits in a portability check

Because SourceX gathers each buyer's dataset from U.S. businesses after the request arrives, portability requirements can be written into that request next to scope, volume, format and timing instead of being negotiated after delivery. The supplying company approves the scope and terms and signs before anything ships, and that agreement is the point at which the package format and the replay outputs the buyer expects get fixed.

Large deliveries stay in the seller's own storage or travel on encrypted drives instead of being hosted by SourceX, which makes the sample-shard variant of this test the usual route for very large packages. Buyers who want to set out portability and acceptance expectations at the start can use the request form below.

Frequently asked questions

Can a buyer validate and rebuild the package without the supplier's internal systems?

It should be able to, and that is the point of the test. A portable package carries its descriptor, schemas, code lists, split definitions and a runnable validator or its expected outputs. If any step needs a supplier login, a private library or an internal lookup table, record it as a failure and ask for the missing piece to be shipped inside the package.

Is checksum verification the same as a portability test?

No. Checksums prove the bytes you received match the bytes the supplier listed. Portability asks whether those bytes can be found, typed, joined and interpreted without outside help. Run integrity first because nothing else is meaningful if files are missing, then run the portability steps.

Which file format is most portable for tabular training data?

Formats that carry types with the data travel better than plain text. Parquet stores logical types, including whether a timestamp is adjusted to UTC, while CSV relies entirely on the accompanying schema. Whatever the format, the package still needs a descriptor that names paths, keys and versions.

Should the supplier ship its validation code?

Ask for it. A shipped validator lets the buyer replay the supplier's checks and compare outputs, which is stronger evidence than a summary report. If the code cannot be shared, ask for the expected outputs and enough documentation to rebuild the checks independently.

What version change should a portability fix trigger?

It depends on what the fix touches. Repackaging the same records is a patch-level change under Croissant's guidance, while removing rows or rewriting identifiers changes existing data and should be a major version. Recording the right level tells downstream users whether earlier results are still comparable.

Sources

  • The Data Package standard requires each resource to declare where its data lives, forbids mixing URLs and relative paths in one path array, and defines a hash property that defaults to MD5 with other algorithms named by prefix. Source
  • The Data Package security guidance says local file references are relative to the descriptor file and absolute paths are disallowed. Source
  • Table Schema defines primary keys as non-null fields that uniquely identify rows, foreign keys modeled on SQL, and a missingValues list of strings treated as missing. Source
  • Croissant 1.1 recommends MAJOR.MINOR.PATCH dataset versions with defined meanings, strongly recommends sha256 checksums for versioned files, advises no checksum on live files still changing, and supports record keys and field references. Source
  • The mlcroissant reference library provides a validate command that checks a Croissant JSON-LD file. Source
  • Parquet's TIMESTAMP logical type carries an isAdjustedToUTC flag that distinguishes instant semantics from local semantics. Source
  • RFC 8493 defines a complete BagIt bag as one where every payload file appears in every payload manifest, and a valid bag as a complete bag whose checksums all verify. Source
  • Hugging Face dataset repositories can declare which files belong to which split in a configs block in the README YAML. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies