Skip to content

Systems and records

Dataset acceptance issue severity: rate defects by what they break

By SourceX Editorial · Updated

10 sections

Short answer

Dataset defect severity should be set by what the defect does to the task the data supports, not by what kind of error it is. A typo in a free-text note that leaves meaning intact is cosmetic. A missing workflow step that breaks the sequence a model must learn, or changes the outcome an evaluation scores, is critical. Rate impact, scope and detectability; the level sets the repair and notice.

Checklist

Dataset acceptance issue

This page covers dataset acceptance issue severity: rate defects by what they break.Illustrative

Key takeaways

  • Severity measures task impact; defect type only says where to look.
  • A missing workflow step usually outranks any number of typos, because it changes what the record teaches.
  • The same defect can be minor for one use and critical for another, so rate it against the agreed use.
  • When two levels seem plausible, rate the higher one and revisit it once the scope is known.
  • Every issue needs an ID, links to affected source and derived records, a repair, a regression check and a recipient notice.

How should dataset issue severity distinguish a typo from a missing workflow step?

Rate the issue by the harm it does to the task, not by its type. A typo is a defect in a value; a missing workflow step is a defect in the structure of the case. The first usually leaves the meaning and the outcome intact, while the second can change what happened, in what order and with what result.

Take a field service job record. A misspelled part description in a technician note changes almost nothing for a model learning to plan jobs. A job record that silently lacks the step where parts were reserved teaches the model that technicians can be dispatched without parts, and an evaluation built on it will reward that plan. Calling both a data quality defect, at one severity, hides the difference.

The rubric below makes the distinction explicit. It defines four levels by a task impact test, adds a scope and detectability adjustment, and ties each level to a repair, a version change and a notice to anyone who has already received the data.

Why one severity label for every defect misleads an acceptance decision

A single label turns acceptance into a defect count, and counts are dominated by cheap, visible errors. Typos and formatting faults are easy to find in volume; a missing step only shows up when someone reconstructs the case. An acceptance report with hundreds of minor defects and three structural ones looks worse than one with three structural ones alone, which is the wrong way round.

Existing frameworks help with vocabulary but not with severity. ISO/IEC 25012 defines a data quality model for structured data with fifteen characteristics, such as accuracy, completeness and consistency, seen from an inherent and a system-dependent point of view. IEEE 1044-2009, which the IEEE standards site now lists as inactive, set out a uniform way to classify software anomalies with a core set of attributes. Both tell you what kind of defect you have. Neither tells you how much it matters for a particular training or evaluation task.

Translation quality offers a useful precedent. The MQM framework scores errors by type and by severity, and its commonly used scorecard weights neutral, minor, major and critical errors at 0, 1, 5 and 25. The principle carries over: a critical defect should outweigh many minor ones, and the weighting should be agreed before anyone starts counting.

The cost of a wrong rating is asymmetric. Rating a missing step as minor lets a structurally broken subset into training or evaluation, where it is expensive to find later. Rating a typo as critical only delays a delivery. That asymmetry is why the rubric leans towards the higher level when the evidence is incomplete.

The task impact severity rubric

Rubric version 1.0 (2026-10-18) is applied per issue, after the issue is reproduced and its scope is estimated. Rights and privacy defects, such as a record from an excluded customer or an unredacted identifier, sit outside the rubric: they stop delivery whatever their count and go to the rights or privacy owner.

The task impact severity rubric
LevelTask impact testTypical examplesAcceptance effectRequired action
S1 CosmeticNo change to meaning, sequence, label or joinTypo in a free-text note, inconsistent capitalization, trailing spacesAccept; log onlyBatch fix in a later patch release
S2 MinorChanges meaning of a non-label field in a few records; a careful reader recovers itTruncated note, wrong unit in a descriptive field, misspelled product nameAccept with issue listFix before next release; notify in release notes
S3 MajorChanges a feature the task uses, or a label, in a bounded subsetWrong status on some cases, swapped timestamps, duplicate events in one segmentConditional acceptance of unaffected subsetRepair, rerun checks, minor version, direct notice
S4 CriticalBreaks the sequence or outcome the task depends on, or affects the evaluation splitMissing workflow step, dropped segment, outcome joined to the wrong caseReject affected scope until curedRepair at source, regression suite, major version, notice and withdrawal of affected version
Out of rubricRights, consent or privacy condition not metExcluded customer records, unredacted personal dataStop deliveryEscalate to rights or privacy owner; recall if delivered

Five questions that set the level

Ask these in order and stop at the first yes that sets a level. They keep raters consistent and stop a defect's type from deciding its severity by default.

Two habits from incident response apply directly. PagerDuty's public incident response documentation advises treating an incident as the higher severity when unsure and reviewing the choice afterwards, and recommends that severity definitions be specific and metric driven. For datasets, that means rating up while the scope is unknown and writing each level's threshold as a share of records or tasks agreed with the recipient.

Write the answer to each question into the issue record, not just the resulting level. A later reviewer who disagrees can then see which answer they would change, which turns a severity argument into a factual one about scope, detectability or use.

  • Does the defect change the label, outcome or score the task is built on? If yes, S4 when it affects evaluation data or a sequence, otherwise S3.
  • Does it remove or reorder a step that the task needs to learn or check? If yes, S4.
  • Does it change a field the task uses as an input or join key? If yes, S3, or S4 when the join attaches records to the wrong case.
  • Can a careful user detect and correct it from the record itself? If yes, lower by one level, never below S2 for label or input fields.
  • Does it affect more than the agreed scope threshold of records or tasks? If yes, raise by one level, up to S4.

Why the same defect can be minor for one buyer and critical for another

Severity depends on the use the data is licensed for, so rate it against the use written into the specification, not in the abstract. A missing step is fatal for a process-mining or agent-planning dataset and may barely matter for a dataset of standalone customer messages.

When the use is not yet fixed, rate against the most demanding plausible use and say so in the issue record. Re-rate once the use is agreed, keeping both ratings.

Evaluation data deserves the strictest column. A defect in training data is diluted by everything around it, while a defect in an evaluation split changes the score directly and can reverse a comparison between two models. If a defect touches both splits, rate it against the evaluation use and repair that split first.

Why the same defect can be minor for one buyer and critical for another
DefectText model training useAgent or workflow training useEvaluation set use
Typo in a technician noteS1S1S2 if graders read the note
Missing parts reservation stepS2S4S4
Wrong close code on some jobsS2S3S4 if the close code is the gold label
Timestamps in mixed time zonesS1S3 because step order can invertS3
Duplicate dispatch eventsS1S3S3 or S4 if counts are scored

What should each issue record contain so a repair can be traced?

Each issue needs enough detail that someone who did not find it can confirm the repair and see who received the defective version. Keep it with the dataset, not in a chat thread or an email.

At a glance

What should each issue record contain so a repair can be traced

Most of these fields can be filled automatically if the issue starts as a saved query rather than a description. A query that selects affected source rows can be rerun after the repair to show the count dropped to zero, and joined to the derived files to list every task or example built from those rows. Teams that track issues in Jira or a similar tool can keep the record there, as long as the query, the version list and the recipient log travel with the dataset release.

  • Issue ID, date found, who found it and how
  • Severity, the use it was rated against, and any re-rating with the reason
  • Source references: system, table or object, and the record IDs or query that selects affected rows
  • Derived references: files, splits, task IDs and examples built from the affected rows
  • Scope estimate as a count and share of records and of tasks
  • Repair: what changed, at source or in the derived data, and by whom
  • Regression checks run after repair, with results, including a recount of the affected scope
  • Versions containing the defect and the version containing the fix
  • Recipients of each affected version and the date and form of the notice sent
  • Reconciliation: confirmation that counts before and after repair differ only by the intended change

Which released versions and recipients does a repair reach?

Every recipient of every version that contains the defect, which is why releases must be immutable and logged. Semantic Versioning states that once a version is released its contents must not be modified, and that any change ships as a new version. Applied to datasets, a fix never overwrites a delivered file; it produces a new version, and the release log shows who holds the old one.

Map severity to the version number so the change signals its weight. S1 and S2 fixes go in a patch release. An S3 repair that changes labels or inputs in a subset is a minor release with a direct notice to recipients. An S4 repair is a major release: the earlier version is withdrawn for the affected scope and recipients are told what to delete or replace.

Datasheets for Datasets asks dataset creators to say whether the data will be updated to correct errors, how often, by whom and how consumers will be told. Answer those questions before the first delivery. A recipient who already trained on a version with an S4 defect needs the affected record IDs, not just a new file.

Keep the recipient log even for internal users. Teams inside the same company train on snapshots too, and an S4 repair that reaches only external recipients leaves an internal model built on the defective version with nobody aware of it.

Worked example (illustrative): a field service work order dataset

This example is invented. A home services company prepares five years of work orders for a buyer building a job-planning agent. Acceptance review finds three issues, and all figures below are illustrative.

First, about 30 percent of technician notes contain spelling errors. Meaning is intact and nothing joins on the notes, so the issue is S1 and goes into the next patch. Second, about 6 percent of jobs that required parts show no parts reservation event, because a scheduling system migration did not carry the event over for one year. For a planning agent this is S4: the sequence the model must learn is broken, and part of the evaluation split is affected. Third, about 2 percent of jobs carry a close code that contradicts the invoice; this is S3 because close codes are an input, not the gold label.

The outcome is conditional acceptance. The migration year is withheld until the reservation events are restored from the old system's archive, then reissued as a major version with a recount showing the reservation gap closed. The close code fix ships as a minor version with the affected job IDs listed. The typo issue ships later as a patch.

Notice what the typo count would have done on its own. Roughly thirty percent of records with an error sounds like a failed delivery, while six percent sounds tolerable. Rated by task impact, the order reverses, and the acceptance decision follows the structural defect, not the noisy one.

When does a typo deserve a critical rating?

When it lands in a field that carries meaning the task depends on. A typo that turns a part number into another valid part number, flips a yes to a no in a decision field, shifts a decimal in a quantity or corrupts a case ID used to join records is a structural defect wearing a cosmetic disguise. Rate it by its effect, which is usually S3 or S4.

The rubric has limits. It assumes the use is known well enough to judge impact, which is not true for exploratory releases; rate those against the most demanding plausible use. It does not replace a sampling plan, because a level describes one issue, not the error rate of the whole delivery. And it is not a legal remedy schedule: what counts as rejection, cure or a credit is set by the agreement and should be reviewed by counsel against these levels before signing.

Agreeing severity before the first delivery

Severity disputes are cheaper before delivery than after. Attach the rubric, the defect-by-use matrix and the issue record template to the dataset specification, and agree the scope thresholds that raise a level.

When a company licenses operational records through SourceX, preparing and delivering the data sits with SourceX together with rights review and contracting. The signed agreement that fixes scope and terms comes before the first file moves, and it is the natural place to settle how defects found later will be rated, repaired and notified.

Frequently asked questions

What is the difference between defect severity and priority?

Severity describes how much a defect harms the task the data supports. Priority describes when the team will fix it, which also depends on deadlines and effort. A cosmetic defect in a file going out tomorrow can be high priority while staying low severity.

Should a large number of typos ever block acceptance?

Only if they cross into meaning, for example in identifiers, quantities or decision fields, or if they exceed a threshold agreed in the specification. Free-text spelling errors that leave meaning intact are usually logged and fixed in a later release. Agree the threshold before delivery so the count does not become a negotiation.

How do you find missing workflow steps in a dataset?

Reconstruct a sample of cases end to end and compare event sequences against the process the system enforces, then reconcile event counts by type and period against the source system. Gaps that cluster in one period or one site often trace back to a migration or integration change. Rate any confirmed gap with the rubric.

Who decides the severity level?

The person who reproduced the issue proposes a level, and a reviewer who knows the intended use confirms it. Where supplier and recipient disagree, record both ratings and the use each one assumed. Persistent disagreement usually means the specification does not describe the use precisely enough.

Do recipients need to be told about cosmetic fixes?

A line in the release notes is normally enough for cosmetic and minor fixes. Major and critical repairs need a direct notice naming the affected versions and record IDs. Keep the recipient log either way, because a later re-rating can turn a minor issue into a major one.

Sources

  • ISO/IEC 25012 defines a general data quality model for structured data, with fifteen characteristics viewed as inherent or system dependent. Source
  • IEEE 1044 provides a uniform approach to classifying software anomalies, with a core set of attributes for failures and defects. Source
  • The MQM scoring models commonly use four severity levels, neutral, minor, major and critical, with multipliers of 0, 1, 5 and 25. Source
  • PagerDuty's public incident response documentation advises treating an uncertain incident as the higher severity and recommends specific, metric-driven severity definitions. Source
  • Semantic Versioning defines major, minor and patch versions and states that released contents must not be modified; changes ship as a new version. Source
  • Datasheets for Datasets asks whether a dataset will be updated to correct labeling errors, how often, by whom and how updates will be communicated to consumers. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies