Skip to content

Rights and contracts

Dataset acceptance with known limitations: a three-zone acceptance memo

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Accept a dataset with known limitations only through a written memo that fixes three zones. Zone one pins the approved scope: version, intended use, license clause and the acceptance test result in counts. Zone two lists each residual issue with its measured size, affected slice, owner and re-test trigger. Zone three names the uses and claims the acceptance does not cover. Rights and terms problems never qualify as limitations.

Checklist

Dataset acceptance known

This page covers dataset acceptance with known limitations: a three-zone acceptance memo.Illustrative

Key takeaways

  • Accept with limitations only when each issue is measured, leaves the acceptance test passing for the intended use and can be fenced.
  • Pin zone one to a version, a use, a test result in counts and the license clause of the signed agreement.
  • Give every residual issue in zone two a size with its denominator, a slice, an owner and a re-test trigger.
  • Write prohibited extrapolations as checkable boundaries on task, population, period, claim and combination.
  • State whether acceptance assumes a cure; silence can read as a waiver.

When can you accept a dataset with known limitations?

When every limitation is measured on the delivered version, does not make the acceptance test fail for the intended use, and can be fenced off in writing. If an issue fails any of those three conditions, the honest choices are a cure, a narrower scope or rejection, not acceptance with a note attached.

Accepting with limitations is common and often right. Real business records carry migrations, missing fields and uneven coverage, and a dataset that is excellent for one use can mislead in the use next to it. The risk is not the limitation itself; it is the limitation becoming invisible once the delivery is marked accepted, so that months later a model team trains on a slice nobody tested, or a slide claims a result the data never supported.

The evidence behind a sound decision is specific: the version identifier of what was tested, the acceptance test and its result with denominators, each residual issue measured on that version, and a named owner on each side. This article turns that evidence into a memo with three zones, approved scope, residual issues and prohibited extrapolations, plus a decision tree for choosing the outcome.

How an acceptance memo differs from a dataset card or a dispute record

A dataset card is the supplier's disclosure about the data; an acceptance memo is the buyer's decision about one version for one use, signed by the person accountable for it. The card says what is wrong with the data. The memo says what the organization agreed to do anyway, and where it agreed not to go.

Documentation standards supply the vocabulary but stop short of the decision. Datasheets for Datasets asks dataset creators whether there are tasks for which the dataset should not be used. Model Cards for Model Reporting places out-of-scope use cases inside its intended-use section. Hugging Face's dataset card template carries an Out-of-Scope Use section, described as covering misuse, malicious use and uses the dataset will not work well for, next to a section on bias, risks and limitations. None of them records who accepted a given version, against which test, or what happens when a limitation is later cured.

The memo also differs from an evidence packet and from a dispute record. The packet is what a supplier ships so the buyer can rerun checks; a dispute starts when the two sides disagree about a result. The memo comes after the evidence is in and both sides agree on what it shows, including the parts that fall short.

The decision tree: accept, fence, narrow, hold or reject

Run six questions in order on every residual issue, and let the most restrictive outcome across all issues decide the dataset. Rights and terms come first because no amount of fencing or cleaning repairs them.

The tree yields five outcomes. Accept as delivered when no residual issue touches the intended use. Accept with limitations when issues are measured and can be fenced. Accept a narrower scope when an issue is confined to a slice that can be excluded. Hold for cure when a fix is expected and the use can wait. Reject when the acceptance test fails for the intended use and no slice can be removed, or when the issue concerns permission or terms.

The questions, in the order they should be asked:

The decision tree: accept, fence, narrow, hold or reject
StepQuestionIf yesIf no
1. Rights and termsIs the issue about authority, consent, third-party content or a commercial term?Hold or reject; route it to counsel or the deal owner, never into the memoGo to step 2
2. MeasuredIs the issue measured on the delivered version, with a denominator?Go to step 3Measure it first; decide nothing on an estimate
3. TestDoes the issue make the acceptance test fail for the intended use?Go to step 4Go to step 5
4. SliceIs the failure confined to a slice that can be excluded with the test still passing?Accept a narrower scope and record the exclusionReject, or hold if a cure is expected
5. FenceCan the use, and the claims made from it, be written so the issue cannot mislead?Accept with limitations; write the fence in zone threeHold until the issue is cured or the use changes
6. CureIs a cure expected, with an owner and a date?Record the cure condition and a re-test triggerRecord the limitation as permanent

Zone one: the approved scope, pinned to a version

Zone one states exactly what was accepted: which version, for which use, against which test, with what result. Anything not named in zone one is not accepted, however similar it looks.

At a glance

Zone one: the approved scope, pinned to a version

The license line is there because labels travel badly. MIT Media Lab researchers in the Data Provenance Initiative checked more than 1,800 text datasets in 2024; on popular hosting sites, they found licenses omitted more than 70 percent of the time and misstated more than 50 percent of the time. A memo that cites the signed agreement itself does not inherit errors from a file header or a listing page.

Results belong in counts. A line reading 382 of 400 sampled records passed, with the sampling method named, tells a later reader how much evidence stands behind the decision; a bare 95.5 percent does not. Zone one needs five entries:

  • Version: the dataset identifier and version, and how to confirm a copy is that version, such as a file manifest with checksums.
  • Intended use: the task in one sentence, the model or evaluation it serves and the setting it will run in.
  • Acceptance test: the test version, the sample design and the result as counts.
  • License basis: the clause of the signed agreement that permits the use, quoted from the agreement.
  • Signatures: one accountable owner for the buyer and one for the supplier, with the date.

Zone two: residual issues, each with a size, an owner and a trigger

Zone two lists every known issue that survived acceptance, measured on the accepted version and tied to the slice it affects. An issue without a size is a rumor, and an issue without an owner never gets cured.

Write sizes as counts with denominators and say how they were found: 1,140 of 42,000 work orders missing an arrival time, all dated before a 2022 system migration, found by a full scan. Then state the issue's effect on the intended use in one sentence and its mitigation, which is usually an exclusion, a flag field or a restriction written into zone three.

NIST's AI Risk Management Framework asks organizations to document an AI system's knowledge limits and how its output may be used and overseen by people. Zone two is where those limits are first written down for the data underneath a system, before anyone builds on it.

Each row of the residual issue log doubles as a rejection reason entry, so one record serves this acceptance and later rejection analysis:

Zone two: residual issues, each with a size, an owner and a trigger
FieldWhat to recordIllustrative entryWhy it matters
Issue and reason codeA short name and a code from a fixed listR-12: arrival time blankLets issues be counted across deliveries
Measured sizeCount, denominator and method on the accepted version1,140 of 42,000 orders, full scanKeeps decisions off estimates
Affected sliceWhere the issue concentratesOrders dated before the 2022 migrationMakes an exclusion or a flag possible
Effect on intended useWhat the issue would distort if ignoredInflates arrival-time errors for early yearsTies the issue to the acceptance test
DispositionFenced, excluded, cure expected or permanentExcluded from arrival scoring; slice flaggedSeparates tolerated issues from fixed ones
Owner and re-test triggerWho acts, and the event that reopens the rowSupplier data lead; any new export or new useKeeps the memo current

Zone three: the extrapolations the acceptance does not cover

Zone three names the uses, populations, periods and claims the acceptance does not support, each written as a statement someone could check. It is the zone most memos skip and the one that prevents the most expensive mistakes.

At a glance

Zone three: the extrapolations the acceptance does not cover

An extrapolation is any step from the accepted scope to a wider one: another task, another region, years outside the window, a training use when only evaluation was tested, or a published performance claim that implies coverage the data lacks. A vague warning to use the data with caution fences nothing. A line forbidding first-visit fix rates for any region other than the two in the sample can be checked by anyone who reads the report that cites the data.

Machine-readable metadata can carry the fence alongside the files. The Croissant RAI vocabulary from MLCommons includes a dataLimitations property for known limitations, covering generalization limits and uses it does not recommend, and a dataUseCases property for recommended uses, so zone three can travel with a copy of the data as well as in the memo.

Write each prohibited extrapolation against one of five boundaries:

  • Task: uses other than the tested one, including swapping evaluation for training or the reverse.
  • Population: customers, sites, products or languages that are absent or thin in the accepted version.
  • Period: dates outside the accepted window, and any period on the wrong side of a known system change.
  • Claim: performance figures or public statements that would imply coverage the data lacks.
  • Combination: joining the dataset with other sources in ways the acceptance test never covered.
Three nested zones of an acceptance memo: approved scope at the center, residual issues around it, and prohibited extrapolations on the outside, with an illustrative arrow from a report claim stopped at the outer boundary.
The three zones of an acceptance memo, with counts from the illustrative work-order example.

Does accepting a known flaw give up the right to raise it later?

It can, which is why the memo should say whether acceptance assumed a cure. Under the Uniform Commercial Code's rules for sales of goods, acceptance made with knowledge of a non-conformity cannot be revoked because of it unless the acceptance rested on a reasonable assumption that the non-conformity would be seasonably cured.

Many data licenses are not sales of goods, and their own acceptance clauses usually govern, so the code is an analogy rather than the rule for a given deal. The drafting lesson carries over anyway: a memo that records a limitation as accepted, without saying a cure was expected, reads like a waiver. Counsel should review how the memo fits the agreement's acceptance, warranty and notice terms before anyone signs it.

Write one of two statements for each residual issue. Either the limitation is accepted as permanent and the use is fenced around it, or acceptance assumes a cure by a named date, failing which the buyer may reopen the issue under the agreement. Ambiguity between the two is where later arguments start.

Illustrative memo: work orders accepted for a dispatch evaluation

Illustrative example, not a real deal. A field-service company delivers 42,000 work orders from 2021 to 2024 for an evaluation set that scores how well a dispatch model predicts technician arrival times. The buyer's acceptance test samples 400 orders and checks that each carries the fields the scoring needs; 382 pass.

Three residual issues survive the tree. Arrival time is blank on 1,140 orders, all from before a 2022 migration; that slice is excluded from arrival scoring and flagged, so zone one records a narrower scope for that metric. Parts codes differ between the delivery's two regions on 3,600 orders; the buyer accepts them with a mapping table, the supplier owns a cure by a named date, and any parts-level claim is fenced to mapped codes. Weekend orders are thin, 1,900 in total; the memo accepts them but forbids weekend-specific performance claims.

Zone three adds two more lines. The data is accepted for evaluation, not training, matching the license basis cited in zone one. Results may not be described as covering regions beyond the two in the delivery. Each issue has an owner and a trigger, and only the parts mapping carries a cure condition; the other two limitations are recorded as permanent.

Re-test triggers, and turning limitations into the next request

Reopen the memo when the version, the use or the claim changes: a new export, a new task, a new region or a planned publication. Each moves the boundary zone three drew, and the earlier acceptance does not cover the new ground.

Limitations also make better requests. A buyer that has learned which slices fail can name them next time, with the measured tolerance it can live with, so the supplier can say early whether the source system can meet it.

If the records have to come from operating businesses, SourceX starts sourcing only once a buyer asks; nothing sits in inventory. The request-data form asks for volume, scope, format, language, licensing needs and timing. Delivery waits until the supplying company has approved scope and terms and signed, so a request that states its intended use, its acceptance test and the limitations it can tolerate lets that approval cover the right ground from the start.

Frequently asked questions

Can a buyer accept a dataset with known issues?

Yes, when each issue is measured on the delivered version, does not make the acceptance test fail for the intended use, and is fenced in writing. Issues about rights, consent or commercial terms should never be accepted as limitations; they go to counsel or the deal owner.

What should a dataset acceptance memo include?

The accepted version and intended use, the license clause, the acceptance test and its result as counts, each residual issue with its size, slice, owner and trigger, and the prohibited extrapolations. An accountable owner on each side should sign and date it.

What is a prohibited extrapolation?

A use or claim that steps beyond what was tested: another task, population or period, a training use when only evaluation was accepted, or a public claim implying coverage the data lacks. Write each one as a statement someone could check.

Does accepting a dataset with limitations waive later claims?

It may, depending on the agreement. By analogy with the Uniform Commercial Code's rule for goods, acceptance with knowledge of a defect is hard to undo unless a cure was reasonably assumed, so the memo should say whether acceptance assumes a cure. Counsel should review the specific terms.

How is an acceptance memo different from a dataset card?

A dataset card is the supplier's description of the data and its limits. The memo is the buyer's signed decision about one version for one use, including what the data will not be used for and when the decision reopens.

Sources

  • Datasheets for Datasets asks dataset creators whether there are tasks for which the dataset should not be used. Source
  • Model Cards for Model Reporting includes out-of-scope use cases in its intended-use section. Source
  • Hugging Face's dataset card template has an Out-of-Scope Use section covering misuse, malicious use and uses the dataset will not work well for, and a Bias, Risks, and Limitations section. Source
  • The Croissant RAI vocabulary (version 1.0) defines rai:dataLimitations for known limitations, including generalization limits and non-recommended uses, and rai:dataUseCases for recommended uses. Source
  • NIST AI RMF MAP 2.2: information about the AI system's knowledge limits and how its output may be utilized and overseen by humans is documented. Source
  • UCC 2-607(2): acceptance with knowledge of a non-conformity cannot be revoked because of it unless made on the reasonable assumption that it would be seasonably cured. Source
  • The Data Provenance Initiative audit of more than 1,800 text datasets (2024) found license omission rates above 70 percent and error rates above 50 percent on popular hosting sites. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies