Skip to content

Systems and records

A dataset artifact manifest standard that ties every file to its task

By SourceX Editorial · Updated

10 sections

Short answer

A task-level manifest should list, for every task, each associated file by relative path with its role, whether a model may see it, its media type, size and SHA-256 digest, plus package-level schema and content versions. That lets a buyer validate the package alone. This dataset artifact manifest standard adds that task index to Croissant or Frictionless, with valid and failing examples.

System export

Dataset artifact manifest

This page covers A dataset artifact manifest standard that ties every file to its task.Illustrative

Key takeaways

  • A delivery manifest proves bytes arrived; a task manifest proves which files belong together and what each one is for.
  • Croissant, Frictionless, RO-Crate and BagIt cover discovery and integrity but not task membership or model visibility.
  • Give every artifact one role and one visibility, and keep graders and reference outputs evaluator_only.
  • A buyer should be able to run every validation check with only the package and the published schema.
  • Version data and schema separately, and never reuse a task ID.

What should a task-level manifest say about each file?

For every task, the manifest should list each file that belongs to it by relative path, say what role the file plays, say whether a model may see it, and give its media type, size and SHA-256 digest. Package-level fields then state the schema version, the content version and the license reference, so a buyer can check the whole delivery without asking the supplier a single question.

That is the difference between a delivery manifest and a task manifest. A delivery manifest proves that every byte arrived; a dataset artifact manifest standard at task level proves what the bytes mean together. A support ticket, its two screenshots, the knowledge-base article that solved it and the rubric used to grade a model's answer are five files and one task. If the package only lists five files, the buyer has to rebuild the grouping from filenames, and that is where leakage and silent omissions start.

The decision a buyer faces is therefore not which file format to accept but which questions the package must answer on its own: which files make up task T, which of them a model may read, which only a grader may read, and whether each one is byte-for-byte the file the supplier intended. A manifest that answers those four questions is enough; one that cannot answer any of them leaves the buyer reverse-engineering a folder tree.

What do existing packaging standards cover, and where do they stop?

Each established format solves part of the problem, and none of them, on its own, declares task membership and model visibility for every file. The practical answer is to keep a standard descriptor for discovery and integrity, and add a small task index that points into it.

What do existing packaging standards cover, and where do they stop?
FormatWhat it records about filesTask groupingWhat is missing for task data
MLCommons Croissant 1.1FileObject with contentUrl, encodingFormat and sha256; FileSet with includes and excludes globs; containedIn for files inside archivesRecords are described through RecordSets, not per-task file listsNo standard property for a file's role in a task or whether a model may see it
Frictionless Data PackageResources with a unique name, a path, bytes and a hash (MD5 by default, other algorithms by prefix)Resources are tables or files, grouped only by the packageTask membership and roles must be added as custom properties
RO-Crate 1.2File entities linked from the root through hasPart; a relative @id must resolve to a file present in the crateNested Dataset entities can model foldersRoles need an extra vocabulary; visibility is not modeled
BagIt (RFC 8493)Payload manifests named manifest-algorithm.txt that map file paths to checksumsNone; the payload is opaque octetsProves completeness only
SWE-bench task repositoryOne directory per instance with task.yaml, tests.json, problem_statement.md, gold.patch, test.patch, eval.sh and a DockerfileDirectory per task, checked by a dataset check commandA convention for one benchmark, not a general manifest
Hugging Face dataset configsYAML in the README maps data_files to splits and subsetsBy split, not by taskNo per-task artifact roles

Which fields does the task artifact manifest specify?

The specification has two levels: a package header and one entry per task, each holding a list of artifacts. Field names below are a proposal written for this article, version 0.1; keep them stable once you ship, because buyers write loaders against them.

Which fields does the task artifact manifest specify?
FieldLevelRequiredRule
manifest_schema_versionPackageYesVersion of this specification the file follows, for example 0.1; a loader refuses versions it does not know
content_versionPackageYesVersion of the data itself, MAJOR.MINOR.PATCH, changed whenever any task or artifact changes
package_id and license_refPackageYesStable package identifier and a pointer to the governing agreement or license file inside the package
tasks[].task_idTaskYesUnique, never reused, independent of file names
tasks[].split and task_typeTaskYesSplit from a declared list; task type from a declared list
artifacts[].pathArtifactYesPOSIX path relative to the manifest; no leading slash, no parent segments, no hidden folders
artifacts[].roleArtifactYesOne value from the role vocabulary in the next section
artifacts[].visibilityArtifactYesmodel_input, model_context or evaluator_only
artifacts[].media_type, bytes, sha256ArtifactYesIANA media type, exact size in bytes and a 64-character lowercase hex SHA-256 digest
artifacts[].derived_fromArtifactWhen derivedPath of the artifact it was produced from, such as a redacted screenshot pointing at its source

Which roles may an artifact play, and which may the model see?

Give every artifact exactly one role and one visibility, and make the validator enforce the pairing. Roles describe what the file is for; visibility describes who may read it during training or evaluation. Keeping them separate is what stops a grading rubric from slipping into a prompt.

At a glance

Which roles may an artifact play, and which may the model see

  • instruction: the task statement a model receives, such as the customer's ticket text. Visibility model_input.
  • context: material the model may consult, such as the attached screenshot or the account's plan tier. Visibility model_input or model_context.
  • reference_output: the accepted answer or action, such as the agent's final reply. Visibility evaluator_only for evaluation sets; model_input only in supervised training sets that say so in task_type.
  • grader: a rubric, test file or script that scores an answer. Always evaluator_only.
  • environment: anything needed to rerun the task, such as a container definition or a seeded database snapshot. Usually evaluator_only.
  • label: a human judgment about the task, such as a resolution category. Visibility set per task_type.
  • documentation: per-task notes, redaction logs or provenance records. Never model_input.

What does a valid manifest for two support-ticket tasks look like (illustrative)?

The example below is invented to show the structure; it is not drawn from any real company's records. Package header: manifest_schema_version 0.1, content_version 1.0.0, package_id example-support-tasks, license_ref LICENSE/agreement-reference.txt. Two tasks, both in the eval split with task_type support_reply.

Every artifact path sits under tasks/ with the task_id as its folder, which is a convenience; the manifest, not the folder, is the source of truth for membership. Digests are shown shortened here for reading; the real file carries all 64 characters.

What does a valid manifest for two support-ticket tasks look like (illustrative)?
task_idpathrolevisibilitymedia_type
T-000184tasks/T-000184/ticket.jsoninstructionmodel_inputapplication/json
T-000184tasks/T-000184/screenshot-1-redacted.pngcontextmodel_inputimage/png
T-000184tasks/T-000184/kb-article-212.mdcontextmodel_contexttext/markdown
T-000184tasks/T-000184/final-reply.jsonreference_outputevaluator_onlyapplication/json
T-000184tasks/T-000184/rubric.jsongraderevaluator_onlyapplication/json
T-000185tasks/T-000185/ticket.jsoninstructionmodel_inputapplication/json
T-000185tasks/T-000185/final-reply.jsonreference_outputevaluator_onlyapplication/json
T-000185tasks/T-000185/rubric.jsongraderevaluator_onlyapplication/json

Can a buyer check the package with nothing but the package itself?

Yes, if every check uses only the package itself: the manifest, the files and the published schema. The SWE-bench project makes the same point for its task repositories, where a dataset check confirms that every task has its files, its metadata and a registered split before anything is built or published. A buyer-side validator for this specification runs these nine checks.

At a glance

Can a buyer check the package with nothing but the package itself

  • Schema: the manifest parses, and manifest_schema_version is one the validator knows.
  • Paths: every path is relative and contains no parent segment, absolute prefix or hidden folder, the same restriction the Frictionless glossary places on local paths for security reasons.
  • Existence: every listed path exists in the package.
  • Integrity: every file's size and SHA-256 digest match the manifest; SHA-256 is specified in NIST FIPS 180-4.
  • Coverage: every file in the package is listed, either under exactly one task or in a package-level files list; unlisted files fail.
  • Uniqueness: task_id values are unique, and no path appears in two tasks unless it is declared shared at package level.
  • Completeness by type: each task has the roles its task_type requires, for example one instruction and one reference_output or grader for support_reply.
  • Leakage: no evaluator_only artifact is referenced from, or byte-identical to, a model_input artifact in the same task.
  • Lineage: every derived_from points to a listed artifact, or to a source the documentation says was withheld.

Which defects should a failing example contain, and which check catches each?

A useful specification ships with a failing example as well as a valid one, so a buyer can prove their validator actually rejects bad packages. The invented manifest below contains six defects; each should fail a named check.

Which defects should a failing example contain, and which check catches each?
Defect (illustrative)ExampleCheck that catches itFix
Parent path../exports/ticket.jsonPathsCopy the file into the package and point to it relatively
Digest mismatchScreenshot re-encoded after the manifest was writtenIntegrityRegenerate the manifest last, after all transformations
Orphan filetasks/T-000186/notes.txt on disk, not in the manifestCoverageList it with a role or remove it
Duplicate task_idTwo tasks both called T-000184 after a mergeUniquenessAssign a new ID; never renumber shipped tasks
Missing graderTask T-000185 with no rubric or reference_outputCompleteness by typeAdd the artifact or change the task_type
Leaked answerfinal-reply text pasted inside ticket.jsonLeakageRemove it from the instruction and regenerate digests

How should the manifest change when tasks are added, fixed or withdrawn?

Change content_version on every data change, change manifest_schema_version only when the specification changes, and never reuse a task_id. Croissant draws the same line: it recommends MAJOR.MINOR.PATCH versions following Semantic Versioning 2.0.0 for dataset content, keeps a separate sdVersion for the metadata, and lets each file carry a sha256 checksum so a version can be pinned.

A workable rule for task packages: a patch release fixes artifacts without changing any task's meaning (a re-redacted screenshot); a minor release adds tasks; a major release removes tasks, changes roles or alters reference outputs, because any of those can change a model's measured score. Withdrawn tasks stay in the manifest as tombstones with a reason, so a buyer comparing two versions can see exactly what left and why.

Keep a changelog file inside the package and list it with role documentation. It costs a few lines per release and saves the buyer a diff of two large manifests.

How do you attach the task index to Croissant or Frictionless without forking them?

Keep the standard descriptor exactly as its specification defines it, and ship the task index as a separate file that refers to the descriptor's own identifiers. That way generic tools keep working, and only loaders that understand tasks need to read the extra file.

With Croissant, each artifact in the task index can carry the @id of the FileObject that describes it, so the checksum and media type live in one place and the task index adds only task_id, role and visibility. Croissant IDs must be unique within a dataset, which makes them safe join keys. With a Frictionless Data Package, resource names are already required to be unique within the package, so the task index can point at resource names in the same way. With RO-Crate, the task index can itself be listed as a File entity, so the crate's hasPart list still accounts for every file.

Two rules keep the pair consistent. Generate both files from one source in the same build step, never by hand. And make the validator compare them: every artifact in the task index must resolve to an entry in the descriptor, with the same path and digest, and every descriptor entry must appear in the task index or in the package-level files list.

How should a buyer request a task-indexed package?

If you are buying records sourced from real businesses, ask for the task index as part of the delivery specification, not after the files arrive. SourceX finds records from U.S. businesses for each request instead of keeping finished datasets on hand, so what a buyer enters on the request form (scope, volume, format, timing and licensing) shapes the preparation; naming a task-level manifest there puts it in scope from the start.

For very large deliveries, the manifest matters even more: SourceX does not host multi-terabyte datasets, which stay in the seller's own storage or ship on encrypted drives, so the buyer's validator is the main way to confirm that what arrived is complete and grouped correctly.

Frequently asked questions

Is there an official standard for task-level dataset manifests?

Not one that covers task membership and model visibility for every file. Croissant, Frictionless Data Package, RO-Crate and BagIt cover discovery, structure and integrity; a small task index on top of one of them fills the gap.

Should the manifest be JSON, YAML or CSV?

JSON or JSON Lines is the safest default because nested artifact lists fit naturally and most loaders parse it without extra libraries. A CSV export of the artifact table is a useful companion for review, but the JSON file should be the one the validator reads.

Why SHA-256 rather than MD5?

Both detect accidental corruption; SHA-256 is one of the secure hash algorithms specified in NIST FIPS 180-4, which makes it the safer default to name in a specification. Frictionless defaults to MD5 but accepts other algorithms by prefix, and Croissant uses sha256.

Can one file belong to several tasks?

Yes, if it is declared once at package level as shared and each task references it. Silent duplication across task folders makes coverage and leakage checks unreliable.

How do I keep grading files away from the model during training?

Mark them evaluator_only and have the data loader read visibility, not folder names. The validator's leakage check then confirms no evaluator_only content is repeated inside a model_input artifact.

Sources

  • Croissant 1.1 describes files as FileObjects with contentUrl, encodingFormat and sha256, file collections as FileSets with includes and excludes patterns, requires IDs to be unique within a dataset, and recommends semantic versioning for dataset content with a separate metadata version. Source
  • A Frictionless Data Resource must have a name unique in the package and a path or inline data, and may carry bytes and a hash that is MD5 unless another algorithm is prefixed. Source
  • Frictionless local paths must be relative siblings or children of the descriptor; absolute paths, parent paths and hidden folders must not be used. Source
  • RO-Crate 1.2 File entities must be linked from the root data entity through hasPart, and a relative @id means the file must be present at that path. Source
  • BagIt (RFC 8493) payload manifests, named manifest-algorithm.txt, map payload file paths to checksums. Source
  • SWE-bench task repositories hold one directory per instance with task.yaml, tests.json, problem_statement.md, gold.patch, test.patch, eval.sh and a Dockerfile, and a dataset check confirms every task has its files, metadata and a registered split. Source
  • Hugging Face dataset repositories can map data files to splits and subsets with a configs block in the README YAML. Source
  • NIST FIPS 180-4 specifies the SHA family of hash algorithms, including SHA-256. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies