Skip to content

Rights and contracts

Dataset machine-readable license tags: link each tag to the agreement

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

A machine-readable license tag should be a pointer, not a summary of the deal. Each tag carries the agreement's stable ID, its version and amendments, a hash of the signed text and the clause that defines the tag's meaning. The agreement, in turn, defines every tag value in a schedule and says which source wins when they disagree. A validator then checks both directions before any training run.

License terms

Dataset machine readable

This page covers dataset machine-readable license tags: link each tag to the agreement.Illustrative

Key takeaways

  • A license tag is a pointer to a signed agreement, not a summary of it.
  • Each tag needs the agreement ID, version, a hash of the executed text and the defining clause.
  • The agreement should define every tag value in a schedule and state that it controls.
  • Treat an unresolved or conflicting tag as a stop, the default ODRL conflict rule.
  • Version content, metadata and agreement separately; amendments rarely change the bytes.
  • A clean tag says nothing about ownership or privacy permission; those need their own review.

Link each tag to one identified, versioned agreement and to the clause that defines the tag, and make the agreement say that it controls. A tag such as eval_only or LicenseRef-DLA-0042 is useful only if a pipeline can resolve it to the signed text that gives it meaning, and only if a person reading that text reaches the same conclusion the pipeline did.

In practice the link has four parts. An identifier for the agreement that never changes, a version that changes with every amendment, a cryptographic hash of the executed document so nobody can swap the text quietly, and a clause or schedule reference for each tag value. The metadata travels with the files; the agreement stays with the parties, and the hash proves the two refer to the same text.

The reverse link matters as much. The agreement should contain a tag vocabulary schedule listing every value the metadata may use, what each one permits or forbids, and a precedence sentence stating that the signed agreement governs where the metadata is wrong or silent. Without that schedule, a tag is a label one engineer chose, and the team that inherits the dataset has no way to test whether the label was ever right.

What can each common metadata field actually say about a license?

Every widely used dataset metadata format has somewhere to put a license, but each field was designed for a different job, and none of them was designed to carry a private commercial agreement. The table shows what each field can express and where it stops, based on the current specification text for each format.

What can each common metadata field actually say about a license?
FieldWhat it holdsUseful for a private agreementLimit to plan around
SPDX license expressionLicense IDs from the SPDX list, user-defined LicenseRef- values, and AND, OR, WITH operatorsLicenseRef-<agreement-id> names a custom license without pretending it is openThe ID is a name only; the terms must live in a referenced document
SPDX 3 declared and concluded licenseTwo relationships: what was found in the artifact, and what the SPDX data creator concluded governs itSeparates what the supplier asserted from what the buyer's review acceptedWhen the two differ, the explanation is a free-text comment
Croissant licenseA URL or a schema.org CreativeWork, and more than one value is allowedA CreativeWork can name the agreement and carry its identifierThe spec recommends known license URLs such as the SPDX list, which fit open data better than private deals
Hugging Face card licenseA license keyword in the card's YAML metadata, or other with a license_name and a LICENSE fileother plus a named license file keeps the tag honest on a private repoA single field per repository, so subsets with different terms need separate repos or extra metadata
DCAT license and rightslicense links a legal document; rights links a broader rights statementrights can point to a statement that supplements the licenseDCAT advises against dataset-level terms that differ from the distribution's terms
ODRL Agreement policyA policy with a required uid, one assigner, one assignee, permissions, prohibitions and dutiesExpresses permitted and prohibited actions in a form software can evaluateIt encodes rules; it does not replace the signed contract or its definitions

When the tag and the signed contract disagree, which one wins?

The signed agreement should win, and both documents should say so. A metadata file is easy to edit, copy and template; a signed agreement is the record both parties approved. If a pipeline lets the tag override the contract, a careless copy of last year's metadata can silently widen what a model team believes it may do.

Formal policy languages already make a choice here that is worth copying. The ODRL information model lets a policy declare a conflict strategy in which permissions win, prohibitions win, or the whole policy becomes void, and when no strategy is stated the default is that the entire policy is void if a conflict is detected. For licensed training data, the void default is the safer reading of a disagreement: stop, do not train, and ask the people who signed.

SPDX 3 offers a second pattern. It distinguishes a declared license, which is what was found in or asserted about the artifact, from a concluded license, which is what the SPDX data creator decided actually governs it, and it asks for a written explanation in a comment when the two differ. A buyer can use the same split: keep the supplier's tag as declared, record the legal reviewer's reading as concluded, and require a note whenever they diverge.

  • Precedence sentence in the agreement: the signed agreement and its schedules control; metadata is a convenience copy.
  • Pipeline rule: any tag that cannot be resolved to a clause in the current agreement version blocks use of the affected records.
  • Review rule: a disagreement between declared and concluded terms is logged with the reviewer's name and a short reason, not fixed by editing the tag.

The metadata to agreement mapping example

The mapping below is the asset: each machine-readable field, an illustrative value, the part of the agreement it must resolve to, and the check a validator runs. Every value is illustrative and invented to show structure; it does not describe a real agreement, supplier or dataset. Field names follow no single standard on purpose, so a team can place them in Croissant, SPDX, a dataset card or its own manifest.

The metadata to agreement mapping example
Metadata fieldIllustrative valueResolves to in the agreementValidator check
licenseLicenseRef-DLA-0042Cover page agreement number and the precedence clauseValue starts with LicenseRef- and matches agreement_uid; never an open-license ID
agreement_uidurn:example:dla:0042Agreement number on the signature pageSame value in every file of the package
agreement_versionv3, amendments A1 and A2Amendment log in the final scheduleVersion equals the latest countersigned amendment on the buyer's register
agreement_sha256Hash of the executed PDFThe executed copy held by both partiesBuyer recomputes the hash on its own countersigned copy and gets a match
permitted_usefine_tune, evaluationTag vocabulary schedule, one row per valueEvery value appears in the schedule for this version
prohibited_useredistribution, re_identificationRestrictions clauseNo value appears in both permitted_use and prohibited_use
scope_selectorsplit = holdout; records with region = EUDataset description scheduleSelector matches at least one file in the manifest and no file outside it
term_endIllustrative date in the agreementTerm and termination clauseTraining jobs started after this date are blocked
deletion_dutyOn termination, delete copies within the agreed periodReturn or destruction clauseA deletion owner and ticket queue exist before ingestion
content_version2.1.0 with sha256 per fileDelivery schedule listing the files deliveredEvery file hash in the manifest matches the delivered bytes

Worked case: a holdout split that the metadata said was training data

This case is illustrative. A buyer's data engineering team receives a second delivery of support conversations under a data license. The package carries a manifest, file hashes and a license block with agreement_uid, agreement_version v2, and permitted_use fine_tune, evaluation for the whole dataset.

Starting state: the buyer's contract register shows that amendment A2, countersigned two weeks earlier, moved a holdout split to evaluation only. The legal team updated the register; nobody updated the metadata template the supplier reused.

Linked evidence: the validator compares agreement_version in the package (v2) with the register (v3). It then resolves each permitted_use value against the v3 tag vocabulary schedule and finds that the scope_selector for the holdout split now maps to evaluation, not fine_tune.

Decision: the pipeline blocks fine-tuning on the holdout files and allows the rest of the delivery, because the precedence clause says the signed agreement controls and the conflict rule treats an unresolved tag as a stop. A ticket goes to the supplier asking for corrected metadata at the next patch version, and the reviewer records the concluded terms with a one-line reason.

Verified outcome: the check that matters is not the status field on the ticket. It is a query of the training job's input manifest showing zero holdout file hashes in any fine-tuning run since delivery, plus the corrected package whose agreement_version now reads v3.

Counterexample: a different supplier tags a private dataset CC-BY-4.0 because the field was copied from a public template. The tag parses, the hash checks pass, and nothing in the metadata looks broken. Only the mapping rule that private agreements must use LicenseRef- values, together with a lookup of the agreement number, catches it. Format validation alone would have passed it.

Checking the package offline with only the countersigned contract

Yes, if the package is self-describing and the agreement link is a hash rather than a live URL. A buyer should be able to check every file, every tag and the agreement reference using only the delivered package and its own countersigned copy of the contract.

Croissant gives part of the mechanism: each FileObject may carry a sha256 checksum, and the specification strongly recommends recording checksums for versioned datasets so a consumer can confirm the files match the description. The agreement hash extends the same idea to the legal side. Because the agreement text is confidential, the package should carry the hash and clause identifiers, never the text itself or a link into the supplier's document system.

  • Manifest with relative paths only; no presigned or internal URLs that expire or leak.
  • One sha256 per file, recomputed by the buyer on arrival.
  • agreement_uid, agreement_version and agreement_sha256 identical across every metadata file in the package.
  • A tag vocabulary version that matches the schedule in that agreement version.
  • A validator script or rule list the buyer can run offline, with its own version number recorded in the ingestion log.
  • A failure report listing each unresolved tag, the files it governs and the clause it should have pointed to.

How should tags change when the agreement is amended or the data changes?

Version the content, the metadata and the agreement separately, because each changes for different reasons. Croissant already separates the version of the dataset content from sdVersion, the version of the metadata itself, and recommends semantic versioning for content in which a major increment means existing records were edited, removed or moved across splits.

A license amendment rarely changes the bytes, so it should not bump the content version. It should bump the metadata version and the agreement version together, and the validator should refuse a package whose metadata predates the latest countersigned amendment.

How should tags change when the agreement is amended or the data changes?
ChangeContent versionMetadata versionAgreement version
Files re-serialized, same recordsPatchPatchUnchanged
New records added to a new splitMinorMinorUnchanged unless scope is defined by list
Records removed after a deletion requestMajorMajorUnchanged; log the deletion against the duty clause
Amendment narrows a split to evaluation onlyUnchangedMinor, new tagsNew amendment and new hash
Term extended with no other changeUnchangedPatch, new term_endNew amendment and new hash

Which tagging mistakes cause the most damage downstream?

The costly mistakes are the ones that pass format validation. A schema validator confirms a field exists and parses; it cannot tell whether the field describes the deal the parties signed.

  • An open-license identifier on private data, usually copied from a template. Require LicenseRef- values for any non-public agreement, which SPDX defines as user-defined license references.
  • A tag with no agreement version, so an old permission survives an amendment.
  • A vocabulary defined only in the metadata. If eval_only is not defined in a signed schedule, a reviewer cannot say what it permits.
  • Different terms at dataset and file level. DCAT advises against license information for a dataset that differs from its distributions because it can create legal conflicts; resolve scope with an explicit selector instead.
  • A rights statement treated as a license. In DCAT, license points to a legal document while rights covers broader statements such as attribution; mixing them hides which text governs.
  • Machine policy treated as the contract. An ODRL Agreement policy can encode permissions and prohibitions, but it carries only what someone chose to encode, so it must still resolve to the signed text.

What a license tag cannot tell you about ownership or privacy

A tag records what the parties agreed; it does not prove the licensor had the right to agree to it. Ownership of the records, contractual permission from the licensor's own customers, privacy permission for any personal data and the license granted to the buyer are four separate questions, and a clean tag answers only the last one. De-identification alone does not create a right to license data.

For that reason the mapping table should sit beside, not replace, a rights review that a reader can trace to evidence. This page is general information, not legal advice; counsel should review the specific agreement, records and jurisdictions before a team relies on any tag.

When a dataset is licensed through SourceX, the business that holds it approves the scope and terms and signs before delivery, and the de-identified data may then be used only as that agreement allows. A buyer who wants agreement-linked metadata packaged with the files can say so in the request form alongside scope, volume, format, timing and licensing needs; SourceX sources each such dataset from U.S. businesses when asked rather than holding inventory.

Frequently asked questions

What should a metadata to agreement mapping include?

It should list every machine-readable field, the clause or schedule it resolves to, and the check a validator runs. At minimum that means the agreement identifier, version, a hash of the executed text, a defined tag vocabulary and file checksums. Anything that cannot be traced to a clause should block use until someone resolves it.

Can I use an SPDX identifier for a private data license?

Use a user-defined LicenseRef- value rather than an identifier from the SPDX License List, because list identifiers name published licenses with known text. A LicenseRef- value is only a name, so pair it with the agreement identifier, version and hash. SPDX 3 also models licenses not on the list as custom licenses defined by the document's creator.

Is an ODRL policy the same as the license agreement?

No. ODRL can express permissions, prohibitions and duties between an assigner and an assignee in a form software can evaluate. It holds only what someone encoded, so the policy should carry the agreement's identifier and the signed text should state that it controls.

Should the agreement text ship inside the dataset package?

Usually not. The text is confidential and the package may be copied widely inside the buyer's organization. Ship the identifier, version and hash; each party checks the hash against its own executed copy.

Who should own the tag vocabulary?

The people who sign the agreement should approve it, and it should live in a schedule of that agreement. Data engineers can propose values, but a value that legal reviewers never defined cannot be enforced or audited later.

Sources

  • SPDX license expressions allow list identifiers, user-defined LicenseRef- references and AND, OR and WITH operators. Source
  • SPDX 3 distinguishes declared and concluded licenses and asks for a written explanation when they differ. Source
  • SPDX 3 models a license not on the SPDX License List as a custom license defined by the SPDX data creator. Source
  • Croissant's license property takes a URL or CreativeWork, per-file sha256 checksums are strongly recommended for versioned datasets, and sdVersion versions the metadata separately from content. Source
  • Hugging Face cards set license in YAML metadata; license: other needs a LICENSE file and a license_name. Source
  • DCAT defines license as a legal document and rights as a broader rights statement, and advises against dataset terms that differ from distribution terms. Source
  • An ODRL policy must have one uid; an Agreement must have one assigner and one assignee; the conflict strategy defaults to invalid. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies