Skip to content

Privacy and preparation

Dataset documentation limitations: what a card must disclose before use

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

A dataset card should disclose every known condition under which the data will mislead a buyer: gaps in time, population and channels; fields whose meaning changed; labels that record a status rather than an outcome; effects of de-identification; measured error rates; and uses the data or the license does not support. Each limitation should be specific, quantified where possible and traceable to the records and release that show it.

Provenance trail

Dataset documentation

This page covers dataset documentation limitations: what a card must disclose before use.Illustrative

Key takeaways

  • Negative limitations tell a buyer what the data cannot support; generic caveats such as data may contain errors tell them nothing.
  • Templates such as the Hugging Face dataset card and the Croissant responsible-AI vocabulary give limitations a place but leave the content to the data owner.
  • A good limitation names the condition, its scope or size, the evidence that shows it and what the buyer should do about it.
  • Every statement should point to a query, record sample or release ID a reviewer can rerun, and change when the release changes.
  • Some limitations belong in the scope or the agreement rather than the card, for example when the license excludes a use or the gap defeats the buyer's task.

What limitations should a dataset card disclose before use?

Disclose anything you know that would change how a careful buyer uses, evaluates or trusts the data. In practice that means ten families of limitation: coverage in time, coverage of people or accounts, missing systems or channels, changes in field meaning, labels that are proxies, known error rates, transformations such as de-identification that remove information, out-of-scope tasks, license or rights limits on use, and how the data will or will not be maintained.

The test for each statement is whether a buyer could act on it. A limitation that names the condition, its size and the evidence lets the buyer exclude a period, reweight a segment or drop a use case. A limitation that only warns in general lets them do nothing and protects nobody.

What separates a useful limitation from a disclaimer?

A useful limitation is specific, bounded and checkable; a disclaimer is general, unbounded and unfalsifiable. Compare data may be incomplete with the sentence: tickets created by phone before March 2024 were not logged in the help desk, so phone issues are absent for the first 14 months of the dataset. Only the second tells a buyer which records to treat with care.

Public data publishers show what the plain form looks like. The Consumer Financial Protection Bureau describes its complaint database as not a statistical sample of consumers' experiences and says the complaints are not necessarily representative of all consumers' experiences with a product or company. That is a negative limitation: it says what inference the data does not support.

Limitations also need to reach beyond arithmetic. NIST's work on AI bias notes that efforts to address bias have focused mainly on computational factors such as dataset representativeness. A card that reports only class balance and missing-value rates can miss the limits a buyer cares about most, such as who made the decisions recorded in the labels and under which policy.

Where do dataset card templates put limitations, and what do they leave to you?

Common templates give limitations a heading but no content. The Hugging Face dataset card template has an Out-of-Scope Use section for misuse and uses the dataset will not work well for, and a Bias, Risks, and Limitations section meant to convey both technical and sociotechnical limitations, followed by recommendations.

The Croissant responsible-AI vocabulary from MLCommons makes the same idea machine-readable. Its dataLimitations property covers known limitations, data generalization limits related to distribution, quality or sources, and non-recommended uses. Related properties describe missing data, the collection timeframe and the release maintenance plan, which are where several of the limitations below should also be recorded.

Datasheets for Datasets asks the underlying questions directly: whether a sampled dataset is representative of the larger set and how that was checked, and whether there are tasks for which the dataset should not be used. None of these frameworks can tell you what your own records hide. That knowledge sits with the people who ran the systems, which is why the data owner should write the section rather than inherit a template default.

What does a limitations section look like with concrete examples?

Use the table as the structure of the section. Example statements are illustrative and should be replaced with your own figures and dates; the evidence column says what each statement must point to so a reviewer can rerun it.

What does a limitations section look like with concrete examples?
Limitation typeExample statement (illustrative)Evidence it points toWhat the buyer should do
Time coverageRecords run from January 2022 to June 2025; volume in 2022 is about a third of later years because one region migrated late.Monthly record counts by region from the release queryWeight or exclude early months for trend tasks
Population coverageAbout 40 percent of tickets come from 3 percent of accounts, mostly enterprise customers.Records-per-account profile and billing account totalsCap or reweight for customer-level tasks
Missing systems or channelsPhone and chat contacts are not included; only email and web form tickets are present.Channel field distribution and export configurationDo not infer channel mix or total demand
Field meaning changeThe priority field moved from a 3-level to a 4-level scale on a stated date; earlier values were not remapped.Schema history and value counts before and after the changeMap or split by period before training
Proxy labelsResolved means the agent closed the ticket, not that the customer confirmed a fix; about 1 in 10 resolved tickets was reopened.Status history joined to reopen eventsUse reopen-adjusted outcomes for success labels
Measured errorA reviewed sample of 400 records found category codes wrong in roughly 6 percent of cases.Audit sample IDs, reviewer instructions and results fileTreat category as noisy; budget for relabeling
Transformation effectsNames, emails and phone numbers were replaced with tokens and free-text dates shifted, so time-of-day patterns in text are unreliable.De-identification spec version and transformation logDo not use text timestamps for timing tasks
Out-of-scope tasksNot suitable for estimating customer satisfaction, because surveys were sent to closed tickets only.Survey trigger rule from the help desk configurationExclude satisfaction modeling
Rights and license limitsLicensed for model training and evaluation only; republication of record text is not permitted.Signed agreement clause reference and release IDKeep record text out of public outputs
MaintenanceThis is a fixed release with no scheduled updates; corrections will be issued as new versions.Release ID, version policy and change logPin the version used in each experiment

How do you tie each limitation to records and release evidence?

Give every limitation an identifier and three pointers: the release it applies to, the evidence artifact and the query or procedure that produced it. A reviewer should be able to pick any statement, open the artifact and reproduce the number without asking you.

Keep the evidence beside the data, not in an email thread. Monthly counts, channel distributions, schema history, audit samples and the de-identification spec version all belong in the release package or a referenced evidence folder with stable names. When a release changes, rerun the queries and update the statements; a limitation that describes the previous version is a new defect.

Separate what you observed from what you infer. The sentence about phone tickets is observed from the export configuration; a claim that phone customers have different problems would be an inference and should be labeled as one, or left out.

Order the section by impact rather than by template heading. Put first the limitations that would invalidate a likely use, such as a proxy label or a missing channel, and leave minor formatting quirks for the end or for the data dictionary.

  • Limitation ID and release ID, for example L-04 in release 2025-07-r2.
  • Statement as it appears in the card, with the number and date range.
  • Evidence artifact path and the query or script version that produced it.
  • Owner who can answer questions about the underlying system.
  • Recheck trigger, such as schema change, new source system or a new release.

How should an illustrative field service card handle a completed status that hides return visits?

All details are illustrative. A heating and cooling contractor prepares three years of work orders from its field service system for a buyer training a model to predict which jobs will need a second visit. The starting state is a draft card with one limitation: data may contain errors.

The data lead checks the status field against linked records. Work orders marked completed include jobs where the technician closed the order and a new order was opened for the same equipment within 30 days. Joining work orders by equipment ID shows that about 12 percent of completed orders were followed by a return visit, and in the first year return visits were often logged as notes on the original order instead of new orders.

The decision is to replace the generic line with two limitations. The first says that completed means closed by the technician, not fixed, and gives the measured return-visit share with the join rule. The second says that in the first year return visits were often recorded only as notes, so the outcome is undercounted for that period. Each points to the join query, the counts by year and the release ID.

The outcome check is simple: a second reviewer reruns the join on the release and gets the same counts, and the buyer confirms it will use the derived return-visit label rather than the status field. The counterexample is the original card. It would have passed a format review, and the buyer would have trained on a label that means something other than what its task needs.

Which vague statements should a reviewer send back?

A reviewer should return any limitation that cannot be checked or acted on. These are the patterns that come up most often in first drafts.

Rewriting one of these takes four questions. Which field, period, segment or channel does it affect? How big is it, measured or at least bounded? What artifact shows it? What should the buyer do differently because of it? If the owner cannot answer the second or third question, the honest version says so, for example: category accuracy has not been measured; a review of a random sample is planned for the next release. An admitted unknown is still more useful than a vague warning, because it tells the buyer to run that check themselves.

  • Data may contain errors or inconsistencies, with no rate, field or period.
  • Some records may be missing, without saying which systems, channels or dates.
  • The dataset may not be representative, without naming the reference population or the skew.
  • Labels reflect business judgment, without saying who decided, under which policy and whether policies changed.
  • Personal information has been removed, without the method, the fields affected and what that does to downstream use.
  • Use at your own risk, which transfers responsibility without transferring information.

When should a limitation change the scope instead of the card?

Change the scope when the limitation defeats the buyer's stated task. If the buyer wants to predict satisfaction and satisfaction was only measured on a biased subset, a well-written limitation does not rescue the use; narrowing or dropping it does. Write the decision down and keep the limitation in the card so later users of the same release see it.

Move a limitation into the agreement when it is about permission rather than data quality. A limit on republication or a prohibited use is a contract term first; the card should repeat it so engineers see it, but the signed agreement governs. Access permission, privacy permission, ownership and licensing rights are separate questions, and de-identification alone does not create a right to license.

For buyers building high-risk systems in the EU, Article 10 of the AI Act asks providers to examine training data for possible biases and to identify relevant data gaps or shortcomings and how they can be addressed. A specific limitations section is useful input to that work, but whether and how the rule applies is a question for the buyer's counsel on the specific facts.

Why draft limitations before a licensing review?

Writing the limitations section early is one of the cheapest ways to make a licensing review faster, because it answers the questions reviewers would otherwise send back one by one, and it surfaces scope problems before anyone negotiates terms.

Where a company licenses records through SourceX, the limitations it has already written feed the rights review, contract and data preparation SourceX runs, and they can narrow the scope the company approves and signs before any record is delivered. The process opens with a short company assessment that asks for no data upload.

Frequently asked questions

What is a negative limitation in dataset documentation?

It is a statement of what the data cannot support or where it will mislead: a gap, a skew, a changed field, a proxy label or an excluded use. It differs from a general caveat because it is specific enough to act on and points to evidence that shows it.

How specific should a dataset card limitation be?

Specific enough that a buyer could change what they do because of it. Name the field, period, segment or channel, give a measured size where you have one, and point to the query or audit that produced it.

Where does a limitation go in a Croissant or Hugging Face dataset card?

In a Hugging Face card it belongs under Bias, Risks, and Limitations, with unsuitable uses under Out-of-Scope Use. In Croissant metadata the responsible-AI extension provides a dataLimitations property, with related properties for missing data, collection timeframe and maintenance.

Should license restrictions appear in the dataset card?

Yes, as a short pointer, because engineers read the card more often than the agreement. The signed agreement remains the controlling document, so the card should quote or reference the clause rather than restate it in new words.

Do I need to update limitations for every release?

Yes. Rerun the evidence queries for each release and update the numbers and dates. A limitation copied unchanged from an earlier version can be wrong in either direction.

Sources

  • The Hugging Face dataset card template includes Out-of-Scope Use, Personal and Sensitive Information, and Bias, Risks, and Limitations sections. Source
  • Croissant RAI defines dataLimitations for known limitations, generalization limits and non-recommended uses, with related properties for missing data, collection timeframe and maintenance. Source
  • Datasheets for Datasets asks whether a sample is representative of the larger set and whether there are tasks the dataset should not be used for. Source
  • The CFPB states its complaint database is not a statistical sample of consumers' experiences and complaints are not necessarily representative. Source
  • EU AI Act Article 10(2) asks providers of high-risk systems to examine data for possible biases and identify relevant data gaps or shortcomings and how to address them. Source
  • NIST SP 1270 observes that efforts to address AI bias focus mainly on computational factors such as dataset representativeness. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies