What a Dataset Quality Report Should Contain: Measured Metrics, Methods and Known Defects

How a quality report differs from a dataset card

A dataset card describes what the data is; a quality report proves how good one shipped version of it is. Datasheets for datasets ask about motivation, composition, collection process and recommended uses [5], and Data Cards summarize upstream sources, collection and annotation methods and intended use [6]. Those answers rarely change between deliveries, so they belong in documentation such as our guide to dataset cards for licensed enterprise data and the datasheet for datasets glossary entry.

A quality report changes every time the supplier re-extracts, re-labels or appends data. It is the document your ML data engineer uses to accept or reject a delivery and the one procurement attaches to the acceptance record. If the two are merged, the measured numbers tend to drift out of date while the descriptive text stays fixed, and nobody can tell which version the figures refer to.

Anchoring the report structure to ISO/IEC 5259

ISO/IEC 5259-2 is the closest published standard for structuring a dataset quality report, because it defines a data quality model, a set of quality measures and guidance on reporting data quality for analytics and ML [1][2]. It builds on the older ISO/IEC 25012 data quality model and ISO 8000 [1]. Part 1 supplies the shared terminology [3], and Part 4 covers process approaches for training and evaluation data, including labeling for supervised learning [4].

You do not need a supplier to certify against the series, and the standards are paywalled, so many vendors will not have read them. The practical ask is narrower: name each reported characteristic using 5259 vocabulary (such as completeness, accuracy, consistency and currentness), state which measure was computed and give the formula. That makes reports from different suppliers comparable and gives your reviewers a neutral reference when a supplier invents its own metric names. For the definitions behind each metric, see training data quality metrics.

The ten sections a supplier report should include

A complete report has ten sections, and the first two identify exactly what was measured. Without a version identifier and a record count that reconciles to the delivery manifest, every later number is unanchored. Compare the counts against the file-level inventory described in the sample manifest.

1. Scope and version. Dataset name, delivery ID, extraction date range, source systems (for example Zendesk tickets, Salesforce Opportunity history, Jira issues), schema version and file hashes.
2. Record counts by stratum. Counts per stratum you specified in the request: product line, region, year, ticket channel, document type. Report the expected count next to the delivered count.
3. Field-level completeness. Null, empty-string and placeholder rates per field ("N/A", "unknown", "1900-01-01"), split by stratum where rates differ.
4. Field-level validity. Share of values passing type, format, range and enumeration checks: ISO 8601 timestamps, currency codes, status values against the source system's picklist, foreign keys that resolve.
5. Consistency checks. Cross-field rules, such as closed_at later than created_at, invoice totals equal to line-item sums, resolution codes present only on closed tickets.
6. Duplicate rate. Exact-duplicate rate by content hash and near-duplicate rate with the method and threshold (MinHash Jaccard 0.8 on 5-gram shingles, for example). See near-duplicate detection with MinHash and LSH.
7. Label audit results. For labeled fields: audited sample size, agreement statistic, estimated label error rate with a confidence interval, and confusion between the most-confused classes.
8. PII residue test. What de-identification was applied, which detectors were run afterward, how many records were reviewed by hand, and how many residual identifiers were found per 10,000 records.
9. Coverage and distribution notes. How the delivered distribution compares with the population it was drawn from, and where it is known to be thin.
10. Known defects and limitations. A dated list of issues the supplier is aware of but has not fixed, with affected record counts.

Method notes: why every number needs its provenance

A quality figure without its method cannot be checked, compared or relied on in an acceptance decision. A completeness rate of 98% means something different when computed on all 2.1 million records than on a 500-record sample drawn only from the most recent month. Require a method block for each metric.

The minimum method block has six fields: population or sample size, sampling frame and selection procedure (simple random, stratified, systematic), who measured it (supplier staff, a third-party annotation vendor, an automated check), the tool and version (for example Great Expectations suites, Deequ checks, a SQL script with its hash), the date run, and the threshold that defines pass or fail. Ask for the check code itself where the supplier will share it; re-running it on your side is the cheapest verification there is.

Treat a report that gives only dataset-level averages and no field-level statistics as a red flag. Buyer guides commonly recommend using structured review and acceptance workflows to identify incomplete or unusable submissions before a delivery is finalized [11]. You cannot apply those thresholds if the supplier reports one blended score.

Label audits and the label error rate

The label audit section should report an estimated error rate with a confidence interval, not an assurance that labels were reviewed. Research on popular benchmark test sets found label errors widespread enough to change which model appeared to perform best [8], so even well-known data is not exempt. Ask how the audit sample was drawn, who adjudicated disagreements and whether auditors saw the original label (which biases them toward agreement).

Model-assisted methods such as confident learning flag likely mislabeled examples from predicted probabilities [9], and suppliers increasingly report counts of flagged items. A flag count is not an error rate: ask what fraction of flagged items a human confirmed as wrong. For outcome fields taken from operational systems (a "resolved" status, a won/lost flag), the audit should test whether the field reflects what actually happened, as covered in verifying outcome labels in operational records. For the audit procedure itself, see how to audit annotation quality.

PII residue testing after de-identification

The report should state what personal data remained after de-identification, measured on a sample, rather than only what method was applied. Detectors such as Microsoft Presidio or cloud DLP services miss identifiers embedded in free text, attachments, file names and email signatures, which are common in support and sales histories. Ask for the residue rate per identifier type, the sample size and whether reviewers checked free-text fields specifically.

For health data, the report should name the HIPAA pathway: Safe Harbor removes 18 listed identifiers, while Expert Determination relies on a qualified expert's documented analysis [10]. Neither method eliminates re-identification risk, so the report should never claim zero residue. For the risk side of this question, see re-identification risk assessment for licensed datasets.

Coverage limits and the known-defects section

The known-limitations section is where a supplier states how the delivered data departs from the population you will deploy against. Researchers have recommended that dataset documentation record the known limits of coverage and distribution match and add explicit representativity questions [7]. In operational data, typical gaps are a single region, a period before a product launch, one customer tier, or tickets from one channel because the others ran on a different system.

A useful known-defects list is specific and dated: "Tickets from 3 to 17 March missing priority field due to a migration; 4,212 records affected" is useful, while "some fields may be incomplete" is not. Compare each listed gap with your own deployment distribution using a coverage gap analysis. An empty known-defects section on a large operational extract is itself a reason to ask more questions.

Illustrative report skeleton for a support-ticket delivery

The skeleton below shows the level of detail to request; adapt the fields to your modality.

Illustrative example: invented to show structure; it does not describe an available dataset.

quality_report:
  delivery_id: DLV-2026-07-B
  schema_version: tickets_v3
  extraction_window: 2023-01-01/2025-12-31
  files: {count: 48, manifest_sha256: "<hash>"}
  strata:
    - {key: "channel=email", expected: 410000, delivered: 408912}
    - {key: "channel=chat", expected: 220000, delivered: 219004}
  fields:
    - name: resolution_code
      completeness: {rate: 0.962, n: 627916, method: "full scan, SQL check v1.4"}
      validity: {rate: 0.991, rule: "value in source picklist (37 codes)"}
  duplicates:
    exact: {rate: 0.004, method: "SHA-256 of normalized body"}
    near: {rate: 0.031, method: "MinHash, 128 perms, Jaccard >= 0.8, 5-gram"}
  label_audit:
    field: intent_label
    sample: {n: 1200, frame: "stratified by channel and year"}
    auditors: "2 independent reviewers, blind to original label"
    error_rate: {estimate: 0.047, ci95: [0.036, 0.060]}
  pii_residue:
    method: "pattern + NER detection, then manual review"
    sample: {n: 2000, free_text_reviewed: true}
    residual_per_10k: {email: 1.5, phone: 0.5, person_name: 6.0}
  known_defects:
    - {id: KD-3, desc: "priority null for migration window", records: 4212}
  coverage_notes: "No tickets from enterprise tier; chat channel starts 2024-02."

Turning the report into acceptance criteria

The report becomes enforceable only when its fields map to thresholds written into the order or license. Agree, before delivery, which metrics are pass/fail, which are informational and what happens on failure: re-delivery, replacement of defective records or a price adjustment. Pair the report with your own re-measurement on a sample, using an acceptance sampling plan for dataset deliveries.

Counsel should check that the license warranties reference the report rather than generic "industry standard quality" language; see data warranties in AI training licenses. Keep each signed-off report with the delivery ID so later model audits can trace which version trained which model. More measurement topics sit in our training data quality assessment hub.

How SourceX handles quality evidence

SourceX sources operational datasets from US companies on request, including support and sales histories, engineering records, documents and finance and legal workflows. Diligence materials covering source, rights, preparation and allowed use are prepared per dataset, and personal details such as names, emails, phones and account numbers are removed or replaced before delivery, with the method recorded and a sample checked; no method is perfect. Buyers can describe the quality evidence they need as part of a request through the SourceX buyer page.

Request a dataset with documented quality evidence

SourceX sources operational datasets from US companies on request and delivers each under a license that defines the records, uses, term and delivery. Every dataset is rights-reviewed, and nothing is contracted until the supplying company agrees. Describe the data and the quality evidence you need at sourcex.si/buyers.