Skip to content

Systems and records

Dataset artifact checksum schema: verify every file your records reference

By SourceX Editorial · Updated

9 sections

Short answer

To verify every artifact a dataset references, list each file once with a stable ID, relative path, media type, byte size and algorithm-prefixed SHA-256 digest, and repeat that digest inside every record that points to it. A validator then checks that references resolve, sizes and digests match, redacted copies link to their originals, and no orphan or withheld file is referenced.

Checklist

Dataset artifact checksum

This page covers dataset artifact checksum schema: verify every file your records reference.Illustrative

Key takeaways

  • A delivery hash list proves files are intact; only a digest inside each record reference proves the record still points at the bytes it was labeled against.
  • Each artifact entry needs a stable ID, relative path, media type, size, algorithm-prefixed digest, role and, for redacted copies, a derived_from link.
  • Run checks in order: header, grammar, uniqueness, path safety, resolution, pinning, size, digest, coverage, lineage.
  • Ship a failing fixture that breaks one rule per defect so the buyer can prove the validator rejects what it should.
  • A portable package validates on an isolated machine; remote artifacts carry URL, size and digest in a fetch list.

How do you verify the integrity of every artifact a dataset references?

Give every referenced artifact one manifest entry with a stable ID, a relative path, a media type, a byte size and an algorithm-prefixed SHA-256 digest, and make each record carry both the artifact ID and that digest. A validator then proves three things: every reference resolves, every listed artifact is present with the declared size and digest, and no unreferenced artifact slips in unexplained.

That is the whole method in one paragraph. The rest of this page turns it into a schema you can hand to a supplier, a worked package built from support tickets with attachments, the checks a buyer's validator should run, and the failure messages that tell you which side of the link broke.

The scope here is narrower than general delivery verification. A delivery manifest proves the files you received are the files that were sent. An artifact manifest proves that the screenshots, PDFs, audio clips and attachments your records point to are the exact bytes the records were labeled against, which is the property a training or evaluation run actually depends on.

Why a file list with hashes still lets broken references through

A file-level manifest answers whether each delivered file is intact; it does not answer whether each row still points at the right file. Records that reference artifacts break in ways a hash list never sees, because the hash list has no idea which row expects which bytes.

Three failure types cover most of what goes wrong. A dangling reference is a record that names an artifact the package does not contain, often because an attachment export failed silently. An orphan is an artifact no record references, which usually means a join key changed or a record was filtered out after the files were copied. A swapped artifact is the subtle one: the reference resolves and the file hashes correctly against the manifest, but the file is a different version than the one the record was labeled against, for example an unredacted screenshot replaced by its redacted copy after annotation.

Archive packaging standards already separate these concerns. BagIt, published as RFC 8493, calls a bag complete when every payload file is listed in a manifest and valid only when every checksum in every manifest verifies, and it requires each payload manifest to list every payload file exactly once. That gives you completeness and validity at file level. The artifact manifest adds the third property, referential integrity between records and files, by putting the digest inside the reference as well as inside the file list.

What should the checksum and artifact validation manifest include?

The manifest needs one entry per artifact with an identity, a location, a size, a digest and a role, plus a short package header that pins the versions. The field set below borrows from three specifications that already solve parts of this problem: the OCI content descriptor, the Git LFS pointer file and the Croissant FileObject.

Keep the manifest as JSON Lines, one artifact per line, so it streams for packages with millions of files and diffs cleanly between releases. The package header lives in a separate small JSON file that also carries the digest of the artifact manifest itself, because a file cannot contain its own hash.

What should the checksum and artifact validation manifest include?
FieldLevelRuleWhy it exists
schema_versionPackage headerVersion of this manifest schema, for example 1.0.0Lets the validator refuse a manifest written to rules it does not know
content_versionPackage headerImmutable label for this release of the records and artifactsA re-delivery and a new release must never share a label
artifacts_manifest_digestPackage headersha256 digest of the artifacts JSON Lines fileProves the artifact list itself was not edited after sign-off
artifact_idArtifactStable, unique, never reused across releasesRecords point to it; a reused ID silently rebinds old labels to new bytes
pathArtifactRelative POSIX path from the package root, no leading slash, no ..Package stays portable and cannot write outside its folder
media_typeArtifactRegistered media type such as image/png or application/pdfTells the loader how to parse the bytes and lets the validator sniff for mismatches
sizeArtifactExact byte length, non-negative integerCheap pre-check that catches truncation before hashing
digestArtifactalgorithm:hex, lowercase, for example sha256 plus 64 hex charactersThe content identity every record reference must repeat
role and derived_fromArtifactRole such as attachment, screenshot or audio; derived_from names the source artifact_id for redacted or transcoded copiesSeparates originals from derivatives so labels attach to the right version
referenced_by_countArtifactNumber of record references the supplier expects to resolve hereLets the validator flag orphans and unexpected fan-out without a second export

Put the digest in the reference, not only in the manifest. A record that stores just a path or a URL will happily resolve to whatever file is there today; a record that stores an ID plus the expected digest fails loudly when the bytes change.

This is the pattern content-addressed systems settled on. The OCI image specification makes digest and size required properties of every descriptor, says retrieved content should be verified against the digest when it comes from an untrusted source, and says content whose length does not match the size should not be trusted. Its digest grammar is the algorithm, a colon, then the encoded hash, and for sha256 the encoded part must be 64 lowercase hex characters, with uppercase explicitly disallowed.

Git LFS reaches the same design from the other direction: the repository stores a small pointer file whose oid line is the hash method and the lowercase hex hash, plus a size in bytes, while the large file lives elsewhere. Browsers apply the idea to scripts. W3C Subresource Integrity lets a page declare the expected hash of a fetched resource, requires user agents to support SHA-256, SHA-384 and SHA-512, and allows several hashes on one resource so an algorithm can be retired later without breaking old pages.

For a dataset, the reference object inside each record therefore needs two fields, artifact_id and digest, and optionally the role the record expects. Avoid storing a bare URL as the reference. If the artifact must be fetched remotely, keep the URL in the manifest entry alongside its digest, so moving the file changes one manifest line instead of every record.

A worked package: support tickets with attachments (illustrative)

The package below is invented to show structure. It describes no real company or dataset: an illustrative help-desk export with 3 tickets and 4 artifacts, one of which is a redacted copy of a screenshot. Digests are shortened here for readability; a real manifest carries the full 64 hex characters.

At a glance

A worked package: support tickets with attachments (illustrative)

Read the lines as the files would appear in the package root.

  • package.json: schema_version 1.0.0, content_version 2026-10-r1, artifacts_manifest_digest sha256:7c1e...(64 hex), record_files tickets.jsonl, artifacts_file artifacts.jsonl
  • artifacts.jsonl line 1: artifact_id att-0001, path artifacts/att-0001.pdf, media_type application/pdf, size 48213, digest sha256:a3f9...(64 hex), role attachment, referenced_by_count 1
  • artifacts.jsonl line 2: artifact_id scr-0002, path artifacts/scr-0002.png, media_type image/png, size 302117, digest sha256:51bd...(64 hex), role screenshot_original, withheld true, referenced_by_count 0
  • artifacts.jsonl line 3: artifact_id scr-0002r, path artifacts/scr-0002r.png, media_type image/png, size 298540, digest sha256:e08c...(64 hex), role screenshot_redacted, derived_from scr-0002, referenced_by_count 2
  • artifacts.jsonl line 4: artifact_id aud-0003, path artifacts/aud-0003.ogg, media_type audio/ogg, size 1180442, digest sha256:9d42...(64 hex), role call_audio, referenced_by_count 1
  • tickets.jsonl ticket T-101: attachments list with one reference, artifact_id att-0001 and digest sha256:a3f9...
  • tickets.jsonl ticket T-102: screenshots list with one reference, artifact_id scr-0002r and digest sha256:e08c...
  • tickets.jsonl ticket T-103: one reference to scr-0002r (the same screenshot reused in a follow-up ticket) and one to aud-0003

Which validator checks run, and in what order?

Run cheap structural checks first, then resolution, then byte checks, so a broken package fails in seconds instead of after hashing terabytes. Every check below needs only the package itself and a standard SHA-256 tool.

The order matters for diagnosis as much as speed. If reference resolution fails, hashing is pointless; if sizes disagree, the digest will too, and the size message is easier for a supplier to act on.

Which validator checks run, and in what order?
CheckRuleCatchesValidator message
V1 headerpackage.json parses, schema_version is supported, artifacts_manifest_digest matches the artifacts fileEdited or truncated manifestheader: artifacts.jsonl digest mismatch
V2 grammarEvery digest is algorithm:hex; sha256 values are 64 lowercase hex charactersUppercase or unprefixed hashes, MD5 passed off as SHA-256artifact scr-0002r: digest not sha256 grammar
V3 uniquenessartifact_id and path unique; no two paths differ only by caseCollisions that overwrite on case-insensitive file systemspath collision: Artifacts/a.png vs artifacts/a.png
V4 path safetyPaths relative, no leading slash, no .. segment, inside package rootPath traversal and absolute paths from the supplier's machineartifact att-0001: path escapes package root
V5 resolutionEvery record reference names an existing artifact_idDangling references from failed attachment exportsT-104 attachments[0]: unknown artifact_id att-0009
V6 pinningDigest in each reference equals the manifest digest for that artifact_idSwapped or re-redacted files after labelingT-102 screenshots[0]: reference digest differs from manifest
V7 presence and sizeFile exists at path and its byte length equals sizeMissing files and truncationaud-0003: size 1048576, expected 1180442
V8 digestRecomputed digest equals manifest digestCorruption and silent replacementatt-0001: sha256 mismatch
V9 coverageReference count per artifact equals referenced_by_count; zero only if withheld or declared standaloneOrphans and unexpected reusescr-0002: 1 reference to a withheld original
V10 lineagederived_from points to a listed artifact; withheld originals are never referenced by recordsUnredacted originals leaking into training recordsT-105 references withheld original scr-0002

What does a failing package look like to the validator?

A useful failing example breaks one rule at a time so each message can be tied to one fix. Ship it next to the valid package, labeled as a test fixture, so the buyer can prove the validator rejects what it should before the first real delivery arrives.

The illustrative failing package below changes the worked example in four places. Each change should trip exactly one check, and a validator that passes any of them is not finished.

  • Re-redaction after labeling: scr-0002r was re-exported with a wider redaction box, so its bytes and manifest digest changed, but ticket T-102 still carries the old digest. V6 fails on T-102 with a pinning error, which tells the buyer the labels on that ticket were made against different pixels.
  • Silent attachment export failure: ticket T-104 references att-0009, which has no manifest entry and no file. V5 fails with an unknown artifact_id; the supplier needs to re-run the export, not re-hash anything.
  • Truncated transfer: aud-0003 arrived at exactly 1,048,576 bytes instead of the declared size. V7 fails on size before any hashing starts, which points to the transfer rather than the source system.
  • Original leaked into records: ticket T-105 references scr-0002, the withheld unredacted original. V10 fails, and the package should be quarantined until the supplier confirms whether any other record does the same.
  • What should pass: the reuse of scr-0002r by both T-102 and T-103 is legitimate, because referenced_by_count declares 2. Fan-out is not an error when the manifest says it is expected.

Can the buyer check and rebuild the package on an isolated machine?

Yes, if the package is self-describing: relative paths, digests in both the manifest and the references, a published schema for the manifest, and no step that needs the supplier's login, database or ticketing system. The buyer should be able to unpack, validate and load the package on an isolated machine.

Remote artifacts are the usual exception, and BagIt's fetch mechanism shows how to handle them cleanly. RFC 8493 lets a bag travel with holes, listing files to be fetched later, and a bag with holes cannot be checked for completeness until those files are in place. The dataset equivalent is a fetch list where each line carries the URL, the size and the expected digest, so a fetched file is held to the same digest as one that shipped inside the package.

Tabular metadata standards already reserve a place for the digest. The Frictionless Data Resource specification has a bytes property and a hash property that defaults to MD5 but accepts other algorithms when the value is prefixed with the algorithm name, such as sha1 followed by a colon. Croissant lets every FileObject carry a sha256 checksum and strongly recommends recording checksums for all used files in versioned datasets. If your buyer already loads Croissant or Data Package metadata, map artifact_id, digest and size into those fields rather than inventing a parallel format, and keep the record-level reference digest as the extra field neither standard requires.

Rebuilding is the stricter test. Ask whether a second engineer, given only the package and the validator, gets the same artifact set, the same reference map and the same pass or fail result. If the answer depends on a call to the supplier, the package is not yet portable.

Putting artifact integrity rules into a licensed data request

The cheapest moment to get artifact integrity is before export, so state the reference format, the digest algorithm and the failing-fixture requirement in the data request itself. Asking for digests after the files have been copied usually means hashing whatever arrived, which proves nothing about what was labeled.

An artifact manifest requirement belongs in the format line of a SourceX data request, next to scope, volume, language, timing and licensing needs; the records are then sourced from U.S. businesses for that request rather than pulled from an inventory. The supplying business signs off on scope and terms before SourceX receives any file, and that sign-off is the point to agree that redacted derivatives, not originals, are what records reference. For very large artifact sets, deliveries stay in the seller's own storage or ship on encrypted drives, so the digest-bearing manifest is what lets the buyer check bytes it did not transfer over a network.

Frequently asked questions

What should the checksum and artifact validation manifest include?

At minimum, a package header with the schema version, the content version and the digest of the artifact manifest, and one entry per artifact with a stable ID, relative path, media type, byte size, algorithm-prefixed digest and role. Add derived_from for redacted or transcoded copies and an expected reference count so orphans and leaks can be detected.

Is a SHA-256 checksum per file enough to verify referenced artifacts?

No. A per-file checksum proves the file matches the manifest, not that the record pointing at it was labeled against those bytes. Repeat the digest inside each record's reference and check that the two agree; that catches swapped and re-redacted artifacts that a file list misses.

Should records reference artifacts by path, URL or ID?

Reference by a stable artifact ID plus the expected digest. Paths and URLs belong in the manifest entry, where moving a file changes one line. A bare URL in a record resolves to whatever is hosted today, so it cannot prove the content is unchanged.

How do you handle redacted copies of screenshots or documents?

Treat each redacted copy as its own artifact with its own digest and a derived_from link to the original. Mark the original as withheld if it is not delivered, and make the validator fail any record that references a withheld original.

Can MD5 hashes from a supplier's export be reused?

Use them as a transport check if that is all the source system produces, but recompute SHA-256 on receipt and make that the reference digest. Label the algorithm in the digest string so a 32-character value is never mistaken for a SHA-256.

Sources

  • The OCI image specification makes digest and size required on every content descriptor, defines the algorithm:encoded digest grammar and requires 64 lowercase hex characters for sha256. Source
  • Git LFS pointer files record an oid made of the hash method and lowercase hex hash (sha256 only) plus the size in bytes. Source
  • Croissant lets each FileObject carry a sha256 checksum and strongly recommends checksums for all used files in versioned datasets. Source
  • The Frictionless Data Resource specification defines bytes and hash properties, with MD5 as the default and other algorithms named by a prefix. Source
  • W3C Subresource Integrity requires support for SHA-256, SHA-384 and SHA-512 and allows several hashes per resource for algorithm agility. Source
  • RFC 8493 (BagIt) defines complete and valid bags, requires each payload manifest to list every payload file exactly once, and allows fetched files that make a bag checkable only once present. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies