Metadata Fields to Require With Licensed Text Corpora

Why document-level metadata beats dataset-level description

Dataset-level metadata tells you what a corpus is. Per-document metadata tells you what to do with each record. A dataset card's YAML block covers license, language and size for the whole repository and drives discovery on the Hub [6]. Croissant does the same thing in machine-readable JSON-LD for datasets, files and record structure [7].

That is not enough once a delivery lands in your curation pipeline. Open pretraining corpora carry fields on every record. FineWeb, for example, keeps the source URL, crawl dump, date and a language-identification score on each document, and its filtering and ablations depend on those fields [1]. The Common Pile records provenance and license by constituent source, so license decisions can be traced back to a named origin [2]. A licensed corpus should meet the same bar. If it doesn't, your team ends up rebuilding the metadata with classifiers, which is slower and less reliable than having the supplier export it from the system of record.

The core field set for licensed text

The core set has four groups: identity and provenance, rights, linguistic properties, and structure and processing. Ask for these as named columns in a sidecar file or as top-level keys in each JSONL record. Free-text notes are not a substitute.

Illustrative example: invented to show structure; it does not describe an available dataset.

Structural fields deserve particular attention. EDGAR-CORPUS splits each annual report into its numbered items and stores them as JSON, so a team can keep the MD&A and risk-factor sections and drop boilerplate exhibits [4]. A documentation feed that emits one JSON object per line, each with a source, page ID, URL, heading anchor, page version hash and section text, shows the same pattern for web documentation [5]. Licensed business text benefits most from this: knowledge-base articles, contracts and tickets all have internal structure that is worth keeping.

Rights and provenance fields: per record, not per contract

Record the license and its restrictions on every record. Filtering at training time can't read a PDF. Excluding NonCommercial or ShareAlike material is only possible when license metadata is recorded per source, and research on corpus building, including low-resource language corpora, treats licensing as a central obstacle [3]. The Common Pile follows the same practice, attaching license information to each constituent source rather than describing the whole corpus with one license [2].

Licensed enterprise corpora raise an extra issue: one supplier export can mix content the supplier owns with third-party material, such as quoted emails, attached vendor PDFs or pasted articles. Ask for a third_party_content flag or an attachment-origin field so those records can be held back. Our guide on third-party content inside licensed corpora covers the detection side. For the grant itself, see the rights grant you need for pre-training.

Provenance fields also feed downstream documentation. California's AB 2013 (Civil Code section 3111) requires developers of generative AI systems made available to Californians to post documentation about their training data, including a high-level summary of the datasets and their sources; as of October 2026, that documentation has been due since 1 January 2026 [11]. That is easier to write when every record already carries source_id, license_id and date fields. Responsible-AI documentation formats such as Croissant-RAI [8] and Data Cards [10] assume this kind of upstream record exists.

Linguistic fields: language, script, variety and translation status

Use full BCP 47 tags and record how each language label was produced. FineWeb's per-document language score shows that language ID is a probabilistic output to be thresholded, not a fact [1]. A tag such as sr-Latn or zh-Hant-TW keeps the script and regional distinctions that tokenizer and evaluation planning depend on. A bare sr or zh throws them away.

The native-versus-translated flag is the field suppliers leave out most often, and it matters for multilingual mixtures. Help centers and product documentation are often localized from an English master. If you train on those pages as native text, you overweight translationese and double-count content you already hold in English. Ask for is_translation, source_language and, where known, translation_method (human, mt, mt_post_edited). The multilingual data types comparison and the guide to licensing non-English text corpora go further on this.

Data Statements, developed for language datasets, add context fields such as speaker demographics, language variety and speech situation [9]. Most enterprise text won't support full demographics, and you shouldn't request personal attributes. Variety, register and the situation the text was written in (a live chat, an asynchronous ticket, a formal filing) are usually available, and they are useful.

Genre, domain and author role for mixture control

Mixture control works only if genre labels come from a controlled vocabulary and are assigned per document. A practical scheme has three levels: domain (support, engineering, legal, finance), genre (ticket thread, runbook, contract clause, earnings commentary) and register (formal, conversational, terse operational). Ask the supplier to publish the vocabulary with the delivery and to state whether labels come from system fields, rules or a classifier.

Author role separates text that looks similar but trains differently. In a support export, customer messages give natural problem descriptions, agent messages give resolutions, and system messages (macros, auto-acknowledgments, SLA notices) are boilerplate to down-weight or drop. Without the field, a deduplication pass will collapse the templates but leave near-template agent macros, which inflates a few phrasings. The same logic applies to engineering records, where bot comments from CI systems sit next to engineer-written reviews.

Processing and redaction fields that make deliveries auditable

Record every transformation applied before delivery, with versions. A minimal processing block names the extraction tool and version (for example, the PDF or HTML-to-text converter), the normalization applied (Unicode NFC, whitespace, boilerplate removal), the redaction method and version, and the hash computed after normalization. Without these fields you can't tell a supplier-side change from a real change in the data when a refresh arrives.

Redaction fields deserve care. Ask for the method, the entity types targeted, the placeholder format (for example [EMAIL_1] or a consistent pseudonym) and a per-record count of replacements. Records with zero replacements in a high-PII genre are a signal to spot-check. The PII redaction guide explains how to measure what redaction missed. SourceX removes or replaces personal details such as names, emails, phones and account numbers before delivery and records the method used; it checks a sample, but no method is perfect, so your own residual-risk check still matters. Buyers can request redacted operational text through SourceX and specify these fields up front.

Packaging the metadata: sidecars, JSONL and dataset-level wrappers

Put per-document fields either inline in JSONL records or in a Parquet sidecar keyed by doc_id, and wrap the whole delivery in a dataset-level description. Inline JSONL is simplest for text-only corpora. A sidecar is better when you also receive original files, which is covered in delivering original files with sidecar metadata. For format expectations, see what file formats AI buyers accept.

At the dataset level, a Croissant JSON-LD file [7] or a Hugging Face-style card [6] should declare the per-record schema, field meanings and controlled vocabularies, so your loader can validate the delivery automatically. Our guide to dataset cards for licensed enterprise data covers card content, and the sample manifest shows a manifest layout.

Illustrative example: invented to show structure; it does not describe an available dataset.

{"doc_id": "kb-000412-v3", "source_id": "src-07", "source_system": "confluence",
 "license_id": "contract:LIC-2026-014", "use_restrictions": ["no_redistribution"],
 "language": "en-US", "lang_confidence": 0.98, "lang_detector": "fasttext-lid.176",
 "created_at": "2021-03-14T09:22:00-05:00", "modified_at": "2025-11-02T16:40:00-05:00",
 "domain": "engineering", "genre": "runbook", "register": "terse_operational",
 "author_role": "engineer", "is_translation": false,
 "section_path": "incident_response/rollback", "content_sha256": "9f2c…",
 "redaction": {"method": "ner+regex", "version": "2.4", "entities": ["PERSON","EMAIL"], "count": 3},
 "token_count": 812, "tokenizer": "buyer-spec-v1", "text": "…"}

Acceptance checks to run on arrival

Validate metadata before you validate text. A short acceptance script catches most delivery problems in minutes:

• Every record has every required field, with no nulls in doc_id, license_id, language or created_at.
• doc_id is unique, and content_sha256 recomputes to the same value from the delivered text.
• Every value in genre, author_role and source_system appears in the published vocabulary.
• created_at precedes modified_at, and no dates fall after the export date or cluster on a single default value such as 1970-01-01.
• Language tags parse as valid BCP 47, and a re-run of language ID agrees with the delivered tag on a random sample.
• Records with restrictive use_restrictions values are counted and match what the license describes.
• Redaction counts are plausible by genre, and a manual sample shows no leftover emails or phone numbers.

Run an overlap check as well, because metadata can't tell you whether licensed text already appears in public crawls. See testing licensed data for overlap with public web corpora.

Sourcing licensed text with the metadata you need

SourceX sources operational text from US companies, including support and sales histories, engineering records, documents, and finance and legal workflows. Each dataset is rights-reviewed and delivered under a license that defines records, uses, term and delivery. Datasets are sourced on request, not held in stock, so after reviewing the text data hub and licensed text corpora for pre-training, describe your text corpus and required fields to SourceX.