Turning Model Goals into Training Data Requirements

This page covers the internal derivation step. What you then send to suppliers is covered in our guide on how to write a data request for suppliers, and the wider buying sequence sits in the AI training data procurement hub.

Why model goals, not dataset categories, should drive the spec

A requirement is only useful if you can trace it back to a behavior the model needs and forward to a test that proves the data supports it. "Customer support transcripts" is a category; "multi-turn chat and email threads where an agent resolves a billing dispute, with the final resolution code" is a requirement. One dataset-purchasing guide puts the same point simply: articulate the purpose of the acquisition before you search [3].

Category-first specs fail in predictable ways. You license volume that duplicates what you already hold, miss the long-tail cases that caused the failures, and discover at acceptance that the field you needed (a resolution code, a timestamp, a final document version) was never in scope. Tracing each requirement to a goal also makes trade-offs visible when budget forces cuts.

Risk frameworks reinforce this discipline. The NIST AI Risk Management Framework asks teams to map the intended context of use before measuring and managing risk [5], and ISO/IEC 5259-2 defines data quality measures you can only set thresholds for once the purpose is stated [4].

Step 1: Write the model objective as behaviors, failure modes and eval targets

The objective statement should name what the model does, where it currently fails, and how you will measure improvement, in that order. Keep it to one page and get sign-off from the model owner before deriving any data requirement.

Use three lists:

• Target behaviors. Concrete tasks, for example "classify inbound support tickets into 40 product-defect codes" or "draft a variance explanation from a general ledger trial balance."
• Known failure modes. Pull these from error analysis, red-team logs or production incident reviews. Examples: confusing refund and chargeback intents, hallucinating clause numbers in contracts, failing on scanned PDFs with rotated pages.
• Eval targets. The metric, the held-out set and the threshold, such as macro-F1 on a frozen 2,000-item eval set, or task success rate on a defined agent benchmark.

If the failure modes list is long, the companion page on targeted data acquisition for model failures shows how to turn individual error clusters into narrow requests. This page stays at the program level: one objective, one specification.

Step 2: Derive coverage dimensions and build a coverage matrix

Coverage dimensions are the axes along which your deployment traffic varies, and each one becomes a requirement with a minimum per cell. Typical dimensions for operational data are record type, workflow stage, customer segment, channel, language, time range and edge-case class.

Build the matrix from your deployment distribution, not from what you guess suppliers hold. Sample production logs or the eval set, tag each item on every dimension, and compute the share per cell. Cells that matter for failure modes but are rare in production (escalations, legal holds, non-English threads) get an explicit floor rather than a proportional share.

Illustrative example: invented to show structure; it does not describe an available dataset.

Once you hold candidate data, the coverage gap analysis guide shows how to score it against the same matrix. Writing the matrix first means acceptance testing reuses work you have already done.

Step 3: Turn each dimension into a testable requirement

Every requirement should state a field or property, a threshold and the test that checks it at acceptance. If no one can say how a line will be verified on a sample, rewrite it or delete it.

Common requirement types for operational datasets:

• Field completeness. "resolution_code populated on ≥ 98% of closed tickets." Test: null rate per column on the delivered sample.
• Schema and format. Parquet or JSONL with a data dictionary listing name, type, description and example per field. Both formats carry typed columns or fields you can check on receipt.
• Label definitions. Written definitions for every class, the labeling source (system of record, human annotation, derived rule) and inter-annotator agreement if humans labeled.
• Temporal coverage. Created-at and closed-at timestamps in ISO 8601 with time zone, and a stated date range per record type.
• Linkage. Stable pseudonymous keys so a ticket, its messages and its outcome join without exposing identity.
• De-identification. Which identifiers are removed or replaced, and how. For health data, state whether you need HIPAA de-identified data or a limited data set, which still counts as protected health information and requires a data use agreement [8].
• Duplication and contamination. Near-duplicate rate below a threshold, and no overlap with your frozen eval set.

The ISO/IEC 5259-2 data quality measures give you shared vocabulary for completeness, accuracy, consistency and currentness, which helps when legal and engineering read the same spec [4]. For acceptance mechanics, the page on acceptance criteria for licensed training data shows how to turn these lines into a pass or fail protocol.

Step 4: Prioritize with Must, Should and Could, and cap the list

Prioritization keeps the spec short enough that suppliers can respond to it and flexible enough that partial matches still surface. One AI vendor RFP guide recommends 25 or fewer line items, each labeled Must, Should or Nice-to-have [2].

Use strict definitions. A Must is a requirement whose absence makes the data unusable for the stated objective: no resolution code means no supervised signal. A Should improves results measurably but has a workaround, such as deriving language from text. A Could is a convenience.

Calibrate the strictness. A data-preparation RFP guide warns that vague specifications draw vague bids, while overly rigid ones do not help you choose; it recommends giving suppliers project context [1]. In practice, a spec with 18 Musts can eliminate most operational sources, because real business systems rarely hold every field in the form you imagined.

Illustrative example: invented to show structure; it does not describe an available dataset.

Step 5: State rights and documentation needs alongside the data spec

Rights are requirements too, and they belong in the spec before sourcing starts, because a perfect dataset licensed for the wrong use is unusable. Write down whether you need training, fine-tuning, evaluation-only or retrieval use; whether derivative models can be commercialized; how long you may retain the data; and whether you need refresh deliveries.

Your own disclosure duties shape the documentation you need. California AB 2013 required developers of generative AI systems offered to Californians to post training-data documentation on or before 1 January 2026, and again for later releases or substantial modifications [9]. As of October 2026, providers placing general-purpose AI models on the EU market must, under AI Act Article 53(1)(d), publish a training-content summary using the AI Office template published 24 July 2025 [10]. If either applies, require supplier documentation that answers the questions in Datasheets for Datasets (motivation, composition, collection process, recommended uses) [7] or a Data Card covering sources, collection and annotation methods and intended use [6].

For how license terms map to these needs, see the AI training data licensing guide and, for reviewers, reviewing an AI data license as in-house counsel.

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

Step 6: Keep data requirements separate from supplier requirements

Data requirements describe the records; supplier requirements describe the organization delivering them, and mixing the two makes both harder to evaluate. The generic AI vendor RFP structure separates technical requirements from privacy, security and legal terms for the same reason [2].

Supplier requirements usually cover provenance documentation, security controls for transfer and storage, delivery capacity and cadence, and the ability to support a sample before contract. Score these with the data vendor evaluation scorecard and the supplier security review, not inside the data spec.

How requirements differ by training stage

The same objective produces different specs depending on whether the data feeds SFT, evaluation, agent training or RAG. The main differences are in volume, label rigor and freshness.

• SFT. Emphasizes outcome labels, diversity across the coverage matrix and consistent formatting of input and target.
• Evaluation. Smaller volume, stricter label definitions, documented provenance and a hard no-overlap rule with training data.
• Agent training. Needs full action sequences: tool calls, intermediate states, timestamps and final outcomes, not just final documents.
• RAG. Prioritizes document versioning, effective dates, access metadata and refresh cadence over labels.

The procurement-by-training-stage guide covers these differences in depth, and the LLM evaluation datasets hub covers eval-specific sourcing.

Requirements document checklist before going to market

A requirements document is ready when a reviewer outside the ML team can read it and say what will be accepted and why.

• Objective stated as behaviors, failure modes and eval targets, signed off by the model owner
• Coverage matrix with deployment shares and per-cell minimums
• 25 or fewer requirement lines, each with priority, acceptance test and trace
• Data dictionary for the fields you expect, with types and formats
• De-identification needs and any regulated categories (health, education, financial) flagged
• Rights needed: use types, derivative models, retention, refresh
• Documentation needed for your own disclosures
• Supplier requirements kept in a separate section
• Budget range and decision owner recorded internally

When the checklist is complete, convert it into an outbound request with the data request builder for AI teams or the AI training data RFP template.

How SourceX uses a requirements spec

A clear specification is what lets an intermediary search for a match. SourceX sources operational datasets from US companies on request, including support and sales histories, engineering records, documents, and finance and legal workflows; nothing is held in stock and a request does not guarantee a match. Buyers describe the data, not the businesses, and every release is approved by the supplying company. You can bring your requirements to SourceX once the internal spec is signed off.

Each dataset is rights-reviewed for ownership and consents and delivered under a license defining records, uses, term and delivery. Personal details such as names, emails, phones and account numbers are removed or replaced before delivery, the method is recorded and a sample is checked, though no method is perfect.

Bring your model's data requirements to SourceX

If your specification calls for operational records held by US businesses, SourceX can look for companies that hold them and manage the process from assessment of data and licensing permissions through agreement, transaction and ongoing management. Nothing is contracted until a supplier agrees, and terms are agreed per deal. Describe the data your model needs.