Skip to content

Privacy and preparation

Dataset quality issue triage: which defects to fix before licensing

By SourceX Editorial · Updated

10 sections

Short answer

Prioritize dataset defects by severity for the buyer's declared use, not by how many you find. Grade each defect class from blocking to cosmetic separately for fine-tuning, evaluation and retrieval, then give it one outcome: repair before delivery, remove and document the affected records, disclose with counts, or accept. Reconciliation gaps, early outcome labels and duplicates across splits usually come first.

Checklist

Dataset quality issue triage

This page covers dataset quality issue triage: which defects to fix before licensing.Illustrative

Key takeaways

  • Grade defects by their consequence for the declared use, then apply count thresholds; a single quality score hides the cases that matter.
  • Every defect gets one outcome: repair before delivery, remove and document, disclose with counts, or accept.
  • Reconciliation gaps, outcome labels recorded too early and duplicates across evaluation splits are usually blocking.
  • Each matrix row needs a rerunnable check that keeps the failing records as evidence.
  • Report sampled defect rates with sample size and an interval, and list unmeasurable defects as unmeasured.
  • When blocking defects concentrate in one slice that cannot be repaired, narrow the scope instead of disclosing it.

How do you prioritize dataset defects that must be repaired before licensing?

Rank every defect by two things at once: how badly it distorts what the records say, and whether it touches the use the buyer declared. A defect that changes labels, totals or time order inside the declared use is repaired before delivery. A defect confined to fields or records the use never reads can be excluded or disclosed instead of fixed.

That gives four possible outcomes, and each defect gets exactly one: repair before delivery, remove the affected records and document the removal, disclose it in the exception log with counts, or accept it as harmless for this use. The same defect can land in different outcomes for different uses, which is why a single quality score never settles the question.

Triage happens before any repair work starts. Teams that fix defects in the order they find them usually spend days on cosmetic issues while one reconciliation gap, the kind that makes a buyer reject the whole delivery, waits at the bottom of the list.

Why can a package that opens cleanly still be rejected?

Readability is the weakest test a dataset can pass. A CSV that parses, a JSON file that validates and attachments that open can still describe the wrong population, contradict the source system's totals or carry outcome labels written before the outcome was known.

The defects that change acceptance are the ones that break the link between the package and the business reality it claims to represent. Scope mismatches, where the export covers different dates, sites or record types than the description says, top that list. Row and case totals that do not reconcile to the system of record come next, because a buyer cannot tell what is missing.

Label problems matter more than they look. Northcutt, Athalye and Mueller estimated that at least 3.3 percent of test labels were wrong on average across ten widely used benchmark datasets, and showed that such errors can change which model looks best. A supplier's outcome field, such as a resolution code or claim status, carries the same risk when it is used as ground truth.

Duplicates are the other quiet failure. Lee and colleagues found that near-duplicate examples and train-test overlap in common language datasets inflate evaluation results, and that deduplication reduced overlap that affected more than 4 percent of some validation sets. In business records, duplicates usually come from re-opened tickets, merged accounts or a migration that copied history twice.

How should severity be defined so two reviewers grade the same defect alike?

Define severity by consequence for the declared use, never by how alarming the defect looks or how many rows it touches. Count still matters, but as a threshold attached to a severity, not as the severity itself.

Data testing tools already work this way. In dbt, a test returns a number of failures and can be configured with a severity of error or warn, plus error_if and warn_if conditions on that count, so the same uniqueness test can warn at a handful of duplicates and fail the build above a set number. A triage scheme for a licensing package should borrow that idea: the defect class fixes the severity ladder, and the count decides where on the ladder a given release sits.

How should severity be defined so two reviewers grade the same defect alike?
SeverityMeaning for the declared useTypical defectDefault outcome
S1 blockingRecords would teach or score the wrong answer, or the package misstates what it containsTotals that do not reconcile; outcome labels set before the outcome; wrong date rangeRepair before delivery or narrow the scope
S2 materialA known slice is unreliable, but the rest of the package is soundTimestamps in the wrong time zone for one site; a schema change mid-historyRepair if feasible, otherwise exclude the slice and document it
S3 limitingUsable, but the buyer must handle it in their pipelineFree-text fields with template boilerplate; optional fields sparsely filledDisclose with counts in the exception log
S4 cosmeticNo effect on what the records meanInconsistent capitalization of category names; trailing spacesAccept, or fix only if trivial

What should the severity and use-case triage matrix include?

The matrix crosses defect classes with the three uses business records are most often licensed for: fine-tuning, evaluation and retrieval. Each cell holds a severity from the ladder above. Read across a row to see how one defect moves between uses; read down a column to build the repair list for one buyer.

The rows follow the evidence checklist a buyer's acceptance owner will apply: scope, totals, timestamps, schema history, attachments, outcome labels and the exception log itself. Edit the cells for your own records, but keep the row set; dropping a row is how defects go unexamined.

What should the severity and use-case triage matrix include?
Defect classFine-tuningEvaluationRetrievalDefault action
Scope does not match the description (dates, sites, record types)S1S1S2Correct the description or re-extract; never ship a mismatch silently
Row or case totals do not reconcile to the source systemS2S1S2Reconcile and record the gap per period before delivery
Timestamps invalid, missing or in mixed time zonesS2S1 when tasks depend on order or durationS3Normalize to one zone and flag unrecoverable values
Schema changed during the history with no mappingS2S2S3Write a field mapping per version and tag each record's version
Attachments missing, corrupt or unlinked to their recordS3, or S1 if the task reads themS1 if the task reads themS2Relink by stable id or drop the field from the declared use
Outcome labels missing, overwritten or recorded too earlyS1S1S3Rebuild from history fields or exclude the affected records
Exact or near-duplicate recordsS2S1 across train and test splitsS3Deduplicate with a stated rule and keep the cluster ids
Exception log incomplete or missingS2S2S2Complete it before delivery; it is part of the package

Which validator finds each defect before the buyer does?

Every row in the matrix needs a check that can be rerun on the next release, with the failing records kept as evidence. Manual spot checks find defects once; scripted checks keep them found.

A machine-readable schema does much of the work. The Frictionless Table Schema specification lets a publisher declare a primary key, unique keys, foreign keys between tables and the strings that mean a value is missing, so a validator can test uniqueness and references instead of a reviewer eyeballing them. In dbt, the store_failures setting saves the records that failed a test to a table, which turns a failed check into a list a person can review.

  • Scope: compare minimum and maximum dates, site codes and record types in the export with the written description; fail on any value outside it.
  • Totals: count records and cases per month in the export and in a report run directly in the source system; any difference needs an explanation in writing.
  • Timestamps: reject impossible values, events closed before they opened and future dates; report the share of records with a stated time zone.
  • Schema history: list every field per release of the source system and map renamed or split fields; untagged records fail.
  • Attachments: confirm every attachment reference resolves to a file with a matching checksum and that every file belongs to a record.
  • Outcome labels: check that the label's timestamp is after the event it describes and that overwritten values are recoverable from the audit history.
  • Duplicates: run exact-match and near-duplicate checks within the package and across any split the buyer will use for evaluation.

In what order should repairs run once the grades are set?

Fix upstream defects before downstream ones, because a later repair often invalidates an earlier one. Scope comes first: there is no point deduplicating records that will be cut when the date range is corrected. Reconciliation comes second, since re-extracting a missing month can reintroduce duplicates or new schema versions.

Time and structure follow. Normalize timestamps and tag schema versions before touching labels, because label checks often depend on knowing when an event happened and which field definition applied at that time. Outcome labels are repaired after that, and deduplication runs last on the final set, including across any train and evaluation split the buyer has described.

Rerun the whole check list after each repair rather than only the check that failed. A single change to an extraction query can move counts in three other rows of the matrix, and the exception log has to describe the package that actually ships, not an earlier draft of it.

Which assumption moves the grades, and how should a sampled defect rate be reported?

The declared use is the assumption that moves the most cells. A ticket history licensed for retrieval can tolerate stale attachments that would make it useless as an evaluation set, so write the use down before grading anything and regrade if it changes.

The second assumption is that a sample represents the whole. Most label and timestamp checks are run on a sample, so report the defect rate with its sample size and an interval rather than a single figure, and say how the sample was drawn. A rate measured on the most recent year says little about records from before a system migration.

Say plainly what is unknown. When a defect cannot be measured, for example because the source system no longer keeps edit history for older records, mark it as unmeasured in the exception log instead of grading it as clean. A buyer can price in an honest unknown; a hidden one ends up as a rejected delivery.

Documentation frameworks make the same point from the buyer's side. The Datasheets for Datasets proposal by Gebru and colleagues asks dataset creators to describe composition, collection and preprocessing, including anything that might affect future uses. The triage matrix and exception log are the operational version of those answers.

Worked scenario (illustrative): a field-service work-order history

A field-service company prepares seven years of work orders from its service management system for a buyer who wants an evaluation set for a dispatch agent and, separately, a fine-tuning set. All counts and percentages below are illustrative.

Validation turns up five defects. Monthly work-order counts reconcile to the source system except for two months after a migration, which are short by about 6 percent. Completion timestamps for one region are stored in local time while the rest are in UTC. About 2 percent of jobs were re-opened and exported twice. Technician notes contain a pasted checklist template in roughly 40 percent of records. A resolution code field was redefined three years ago without a mapping.

Graded against the matrix, the reconciliation gap and the duplicates are S1 for evaluation, so they are repaired first: the two migration months are re-extracted and duplicates are collapsed with the cluster ids kept. The time-zone issue is S1 for evaluation because the agent is scored on response times, and it is fixed by normalizing to UTC. The resolution code change is S2; the team writes a mapping for codes that translate cleanly and excludes about 3 percent of records whose old codes have no equivalent. The checklist boilerplate is S3 and is disclosed with counts, since the buyer's pipeline can strip it.

The repair list is four items, not fourteen, and the exception log tells the buyer exactly which records were excluded and why.

When does triage stop being enough and the scope have to change?

Narrow the package when S1 defects concentrate in one period, site or source system and cannot be repaired from data that still exists. Shipping that slice with a disclosure invites rejection; dropping it and describing the narrower scope accurately does not.

Stop and re-extract when the fix would require editing records by hand at scale. Hand edits break the link to the source system that makes business records credible, and they cannot be reproduced on the next release.

Escalate outside the data team when a defect is really a rights or privacy question, for example records from a business unit that was sold, or free text that turns out to hold personal details nobody expected. Those go to the people who own rights review and de-identification, not into the severity ladder.

Putting the triage record in front of a licensing review

A finished matrix, the validator results and the exception log are what a buyer's acceptance owner asks for first, and they shorten every later conversation. SourceX licenses business data directly from companies, with data preparation and delivery among the steps it runs alongside rights review and contracting.

Scope is fixed before anything moves: the company approves the scope and terms and signs the agreement before any data is delivered, so a triage that narrows the package belongs before signature. Whether a company qualifies is answered by a brief assessment, and no records have to be uploaded for it.

Frequently asked questions

What is dataset quality issue triage?

It is the step where every known defect in a dataset is graded by severity for a declared use and assigned one outcome: repair, remove, disclose or accept. It happens before repair work starts, so effort goes to the defects that would change a buyer's acceptance decision.

Should every defect be fixed before a dataset is licensed?

No. Defects that change labels, totals or time order inside the declared use should be repaired, but defects confined to fields the use never reads can be disclosed or excluded. Fixing everything delays delivery and often introduces new errors through hand edits.

How is severity different from defect rate?

Severity describes the consequence of a defect for the declared use; the rate describes how many records it touches. A rare defect in outcome labels can be blocking for an evaluation set, while a common formatting defect may be cosmetic, so grade the class first and apply count thresholds second.

What belongs in the exception log after triage?

Each defect that ships or causes an exclusion: the check that found it, affected record counts, the severity for the declared use, the outcome chosen and the release version. Defects that could not be measured should be listed as unmeasured rather than omitted.

Can the same dataset pass for retrieval but fail for evaluation?

Yes. Evaluation depends on correct labels, reconciled totals and clean splits, while retrieval mainly needs the right documents to be findable and current. That is why the triage matrix grades each defect separately for each use instead of producing one overall score.

Sources

  • dbt tests return a count of failures and can be configured with severity error or warn and error_if and warn_if conditions on that count. Source
  • With store_failures enabled, dbt saves the records that failed a test to a table named after the test. Source
  • Table Schema lets a publisher declare primary keys, unique keys, foreign keys and the strings treated as missing values. Source
  • Northcutt, Athalye and Mueller estimate at least 3.3 percent of test labels are wrong on average across ten benchmark datasets and show errors can change which model looks best. Source
  • Lee and colleagues show near-duplicates and train-test overlap in language datasets inflate evaluation, with overlap affecting over 4 percent of some validation sets. Source
  • Datasheets for Datasets asks dataset creators to document motivation, composition, collection and preprocessing, including anything that may affect future uses. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies