Sourcing licensed data for foundation-model pre-training teams

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

The pre-training source portfolio: what each source adds and risks

A defensible pre-training mix treats each source as a separate supply line with its own volume role, rights posture and failure modes. The term itself is defined in the pretraining glossary entry. The question for a data lead is which line fills which gap in the mixture.

Web crawls remain the volume base, and published pipelines such as FineWeb show how much filtering, deduplication and quality classification a Common Crawl derivative needs before it is usable [3]. Open domain corpora, like Pile of Law's collection of court opinions, filings and statutes, show the open route for specialized text [5]. Check the dataset-level license and each underlying source's terms before commercial use, because an open release is not automatically cleared for commercial training. For a side-by-side of licensed, synthetic and scraped trade-offs, see licensed vs synthetic vs scraped training data.

Why licensed sources matter more as crawl access narrows

Licensed data matters more because the open web is closing to automated collection faster than most corpus roadmaps assumed. The Data Provenance Initiative's "Consent in Crisis" audit traced robots.txt and terms-of-service restrictions across the domains behind widely used training corpora and reported a rapid rise in restrictions over roughly a year [1]. Common Crawl says it honors robots.txt and avoids paywalled and login-gated pages [2]. That means newer snapshots can quietly lose the high-quality publisher domains a team relied on in earlier ones.

In the EU, Article 53(1)(c) of the AI Act requires providers of general-purpose AI models to keep a copyright policy that identifies and honors machine-readable rights reservations under Article 4(3) of the DSM Directive [6]. The GPAI Code of Practice copyright chapter turns that into concrete commitments for signatories [9]. The broader supply picture is covered in are AI companies running out of training data?.

Provenance records that feed EU and California disclosures

Every source in the mix should arrive with fields that let you fill regulator-facing summaries without reverse-engineering the corpus. As of October 2026, Article 53(1)(d) GPAI obligations have applied since 2 August 2025 [6]. The AI Office's template, published 24 July 2025, asks providers to describe data by source category, such as publicly available datasets, licensed private data and crawled data [7]. Crawled data calls for crawler details and the top 10% of domain names by content size [8].

California's AB 2013 required developers to post training-data documentation by January 1, 2026 [10]. The statute asks for a high-level summary that includes dataset sources or owners, the types of data points, whether the data includes copyrighted material, whether it was purchased or licensed, whether it contains personal information, and whether synthetic data was used [10]. The supplier-side view of the EU requirement is in what buyers need from suppliers for EU AI Act training data summaries.

Illustrative example: invented to show structure; it does not describe an available dataset.

source_manifest:
  source_id: lic-2026-0412
  source_category: licensed_private      # EU template category
  acquisition: licensed                  # AB 2013: purchased or licensed
  owner_type: "US regional insurer (supplier approved release)"
  content: "claims adjuster notes, policy correspondence"
  languages: [en]
  domain_tags: [insurance, claims]
  collection_period: "2019-01 to 2025-12"
  record_count_range: "1M-5M documents"
  personal_info_present_raw: true
  deidentification: {method: "NER + rule replacement", sample_checked: true}
  copyright_status: "supplier-owned business records"
  rights: {train_successor_models: true, retain_weights_post_term: true, open_weight_release: false}
  record_id_scheme: "sha256(source_id + doc_id)"
  dedup_vs_corpus: {method: "MinHash LSH", jaccard_threshold: 0.8, overlap_pct: null}

Where proprietary business records fit in the mix

Operational business records belong in mid-training, annealing and domain evaluation, not in the raw-volume base. Support transcripts, engineering tickets, claims notes, contracts and finance workflows carry professional vocabulary and multi-step reasoning that rarely appears on the public web. They also typically come in volumes measured in millions of documents, not trillions of tokens. That makes them most valuable late in training, when a smaller high-quality mixture shapes final capability.

Three uses are worth planning separately. First, mid-training and annealing mixtures, where domain-dense text is upweighted near the end of a run; the token-budget mechanics are covered in sourcing domain corpora for continued pre-training. Second, long-tail professional language that improves downstream post-training. Third, held-out domain evaluation, which should be carved off before any training use, as described for model evaluation teams.

Document-heavy sources are surveyed in enterprise document datasets for AI training. For corpus types and volume planning, see licensed text corpora for LLM pre-training.

Rights a pre-training license must grant

A pre-training license has to survive the model lifecycle, not just the first training run. The clauses that most often fail review are narrow ones that looked fine at signature.

• Successor models: the grant covers current and future model versions, distillations and continued pre-training, not one named model.
• Weights after term: trained weights and checkpoints can be kept and used after the license ends, with no deletion or retraining obligation.
• Open-weight release: if planned, the license permits releasing weights; see releasing open-weight models trained on licensed data.
• Removal mechanics: what happens when a supplier withdraws records: future runs only, or also existing checkpoints.
• Disclosure consent: the supplier agrees to being named or categorized in EU and AB 2013 summaries.

The full rights grant is broken down in the rights grant you need for pre-training data.

Ingestion requirements to set before the first delivery

State ingestion requirements in the request, because retrofitting them after delivery is expensive. Exact and near-duplicate text inflates memorization and train-test overlap, so deduplication against your existing corpus is a gate, not a nice-to-have [4]. Ask for overlap testing against public crawls too, since licensed text you pay for may already be in Common Crawl; the method is in testing licensed data novelty against pretraining corpora.

• Per-record stable IDs that map back to the supplier's source system, so removal requests can be executed.
• Evidence of PII handling: method, sample check results and residual-risk notes. The EDPB's Opinion 28/2024 treats a model's anonymity as something to demonstrate, not assume [11]. More on de-identification is in the privacy guide.
• Language, domain and document-type tags in a consistent schema, such as JSONL with UTF-8 text and a metadata object.
• Collection period and record-count range for disclosure fields.

Turning these into pipeline controls is covered for ML data engineering teams, and other buyer teams are indexed in the data sourcing by team hub.

How SourceX fits a pre-training data request

SourceX sources operational datasets from US companies on request and manages the commercial process, including licensing agreements and ongoing purchases. Categories include support and sales histories, engineering records, documents, finance and legal workflows, and new recordings of hands-on work. It does not source scraped web content, standalone contact lists or generic CCTV or photos, so it fits the proprietary-records line of a portfolio rather than the volume base. Pre-training teams can describe the data they need to SourceX; buyers describe the data, not the businesses.

Each dataset is rights-reviewed for ownership and consents. It is delivered under a license defining records, uses, term and delivery. Personal details are removed or replaced before delivery, the method is recorded and a sample is checked, though no method is perfect. Nothing is contracted until a supplier agrees, and a request does not guarantee a match.

Request proprietary pre-training data from US businesses

If your mid-training or domain mixture needs operational records that crawls cannot supply, describe the data, the rights you need and the documentation fields above. SourceX looks for US businesses that hold it, and every release is approved by the supplying company. Start a buyer request.