Skip to content

Systems and records

Dataset quality regression pipeline: keep a repaired defect from coming back

By SourceX Editorial · Updated

9 sections

Short answer

To keep a dataset repair from reintroducing an earlier defect, fix the defect at the pipeline stage that created it, then turn it into a permanent test pinned to the records that showed it. Run every such test on each rebuild, block the release when a repaired defect reappears, compare the new version with the last one, and log which recipients received which version.

Checklist

Dataset quality regression

This page covers dataset quality regression pipeline: keep a repaired defect from coming back.Illustrative

Key takeaways

  • A repair applied to a delivered file is undone by the next rebuild; fix the stage that produced the defect.
  • Every repaired defect becomes a named test with pinned record IDs, and the test outlives the ticket.
  • Repaired defects get blocking gates with zero tolerance on pinned records; broad quality metrics get warning thresholds.
  • Compare each release with the previous one, by file and by metric, so unexplained changes surface before delivery.
  • When a test fails because the source legitimately changed, record the decision and update the test rather than deleting it.

How do you prevent a dataset repair from reintroducing an earlier defect?

Make each repair permanent in three places: the code, the tests and the release record. Fix the defect where it entered, add a test that fails on the old data and passes on the repaired data, pin that test to the specific records that showed the defect, and run the whole collection of these tests before every release. A release that fails a repaired-defect test does not ship.

That collection of tests, run automatically on each build with gates that decide whether the build can be released, is a dataset quality regression pipeline. The idea is borrowed from software regression testing, where every fixed bug gets a test so it cannot quietly return. Datasets need it more than code does, because they are rebuilt from sources that change underneath them.

The question matters most for recurring deliveries. A one-time dataset can be fixed by hand. A dataset that is refreshed monthly from a ticketing system, a CRM or an ERP is rebuilt from scratch each time, and every manual fix that was not encoded in the pipeline disappears with the next run.

How do old defects find their way back?

Almost always through a route that bypasses the original fix. The table lists eight common routes in recurring data work and the test that catches each one. Use it as a checklist when you write the test for a new repair: ask which of these routes could undo it.

Most repairs are exposed to more than one route. A duplicate fix in extraction can be undone by a full re-extraction, by a restored branch and by a changed dedupe key, so its plan entry should name each route and confirm that at least one test runs after the stage each route touches. A test that only runs on the final packaged file catches all of them, but it tells you late and gives little hint of where the regression entered.

How do old defects find their way back?
Reintroduction routeHow it happensIllustrative exampleTest that catches it
Fix applied to output onlyThe delivered file was edited; the pipeline still produces the bad valueMis-encoded characters fixed in the export, not in the extraction stepPinned-record test run on a fresh build, not on the patched file
Full re-extractionA refresh pulls all history again and overwrites downstream correctionsMonthly rebuild restores merged duplicate ticketsDuplicate-key test on the full history after every rebuild
Restored or merged old branchAn older snapshot or code branch is merged back for an unrelated reasonA rollback to last quarter's filter logic re-includes internal test accountsExclusion test that lists the pinned test account IDs
Source resends old dataThe upstream system re-exports records with their original defectsVendor CSV feed resends a corrected batch in its original formValue test on pinned records plus a check on source batch IDs
Changed transformationA new or edited step reverses an earlier assumptionChanged dedupe key splits records that had been mergedCount and key tests on the pinned set before and after the step
Schema changeA renamed or retyped field is remapped, sometimes to the wrong columnStatus and resolution fields swapped after a source upgradeSchema test plus value-domain test on both fields
Manual hotfix not in codeSomeone fixed the data in the warehouse directlyA direct update removed personal data from notes in one table onlyPattern test across every table that carries the field
Dependency or library changeA parser, encoding or date library behaves differentlyTime zone parsing changes after a library upgradePinned timestamp values compared to their expected UTC values

Why the fix belongs in the pipeline, not in the delivered file

A fix in the delivered file is a fix to one version. A fix in the pipeline is a fix to every future version, and it can be tested. When a defect is found, trace it to the earliest stage that produced it, such as extraction, normalization, joining, de-identification or packaging, and repair that stage.

Then write the test at the same stage. dbt's documentation describes data tests as select queries that look for failing records: a uniqueness test selects duplicates, a not-null test selects nulls, and the test passes when zero failing rows come back. Any assertion that can be written as such a query can become a test. That form suits repaired defects well, because the defect itself is usually easy to describe as a query that should return nothing.

Keep the original defect reproducible. Save a small fixture of the records as they were before the repair, run the new test against it once, and confirm it fails. A regression test that has never failed has not proved it can catch anything.

Run the tests where they can be trusted to run. A regression suite that someone has to remember to start will be skipped in the week it matters most. Wire it into the build so that packaging a release is impossible without a passing run, and store the run's results with the release record so a reviewer can see which tests ran against which version.

The targeted regression and acceptance test plan

Write one plan entry per repaired defect, using these ten fields. The plan is the asset a release reviewer reads; the tests are its executable part.

At a glance

The targeted regression and acceptance test plan

Keep the plan next to the code, in the same repository, and review changes to it the way you review code changes. When a defect is reopened, update the existing entry rather than creating a second one, so the history of the defect, its tests and its notices stays in one place. Over time the plan becomes a record of how the dataset has been hardened, which is useful to new team members as well as to buyers.

The targeted regression and acceptance test plan
Plan fieldWhat it containsExamplePass rule
issue_id and severityDefect ticket ID and a severity level agreed with the data ownerDQ-214, severity 1 (wrong values in a required field)Severity set before the test is written
defect_signatureThe query or rule that returns the defective recordsTickets sharing source ID and created timeReturns rows on the pre-repair fixture
pinned_recordsIDs of the records that exhibited the defectA short list of ticket IDs from the issueEvery pinned ID still present, or its removal documented
root_cause_stageThe pipeline stage where the defect enteredExtraction: pagination re-read the last pageNamed stage matches where the repair was made
repair_referenceCode change or configuration that fixed itCommit or change request IDRepair is in version control, not a manual edit
regression_testsTests on pinned records and on the whole populationPinned duplicate check; global uniqueness of source IDZero failures on pinned records; population within threshold
acceptance_gateWhether a failure blocks release or warnsBlocking for pinned test; warning above a small ratio for populationGate type matches severity
version_comparisonExpected differences from the previous releaseFewer tickets after dedupe; no other file changesUnexpected file or metric changes explained before release
recipients_and_noticeWhich recipients received a version with the defect and the notice sentTwo recipients on versions 3 and 4; notice issued with the fixEvery affected recipient has a logged notice
retirement_ruleWhen the test may be changed or retired, and who approvesOnly if the source system is retired, approved by the data ownerNo test deleted without a recorded decision

Which checks block a release and which only warn?

Block on repaired defects, warn on broad drift. A defect that has been fixed once and found again is a process failure, so a single failing pinned record should stop the release. Population-level checks, such as the share of tickets with an empty resolution, move with the business and need thresholds rather than zero tolerance.

Test frameworks support this split directly. In dbt, a test's severity is error or warn, with error by default, and separate error_if and warn_if conditions that both default to any failing row. A repaired-defect test keeps the strict default; a population test can warn above a small count and error only above a larger one.

Keep the failing rows, not just the verdict. dbt's store_failures setting saves the records that failed a test into a table named after the test, by default in an audit schema. For a regression pipeline, that table is the evidence a reviewer needs to decide whether a failure is a real regression or a legitimate change in the source.

Which checks block a release and which only warn?
Check typeGateThresholdWho can override
Repaired defect on pinned recordsBlockingZero failuresNobody; fix or document a legitimate change first
Repaired defect across populationBlocking above thresholdAgreed count or ratio per defectData owner with a written reason
Privacy or de-identification patternBlockingZero matchesNobody; escalate to the privacy reviewer
Schema and value domainsBlockingNo unexpected fields or valuesData owner after a schema review
Distribution and volume driftWarningChange beyond the agreed band versus last releaseRelease reviewer, with an explanation in the notes

Comparing each release with the last one

Rule-based tests catch the defects you already know. A release-to-release comparison catches the ones you do not, by showing what changed and asking whether each change was expected.

Compare at three levels. Files: dvc diff, for example, lists files and directories that were added, modified, renamed or deleted between two commits of a versioned dataset, which is a quick way to see that a fix touched only what it should. Metrics: Deequ, a library for unit tests for data on Spark, can persist computed quality metrics in a metrics repository and run anomaly detection on those metrics over time, so a sudden jump in null counts after a rebuild is flagged. Schema: TensorFlow Data Validation generates a schema of expectations such as required values, ranges and vocabularies and reports anomalies such as missing features, out-of-range values and wrong types.

Write the expected differences into the plan before the build runs. If a dedupe repair should lower the ticket count and change nothing else, any other file or metric change is a question to answer before release, not after a buyer finds it.

Treat a missing comparison as a failed one. If the previous release cannot be loaded, because it was deleted, moved or never versioned, the pipeline cannot tell what changed, and the release should wait until a baseline is restored or a reviewer signs off on a fresh baseline with a written reason.

Worked example (illustrative): duplicate tickets return after a refresh

This example is illustrative and does not describe a real company. A support ticket dataset delivered monthly contained duplicate tickets because the extraction step re-read the last page of results when the API paginated. The team found the duplicates in version 3, removed them from the delivered file and sent version 3.1. Version 4 was rebuilt from the full history and the duplicates were back, since the extraction step had never been fixed.

With the plan, the defect gets issue ID DQ-214 and severity 1. The defect signature is a query for tickets sharing a source ID and creation time. Twelve ticket IDs from the original report are pinned, an illustrative count. The root cause stage is extraction, the repair is a pagination fix in the extractor, and the regression tests are a pinned duplicate check that blocks release and a global uniqueness test on source ID. The version comparison expects fewer rows and no other file changes. The recipients log shows which recipients received versions 3 and 4 and records the notice sent with version 5.

Now the exception. In version 6 the pinned test fails on one ticket. The stored failures show two records with the same source ID, but on inspection the source system had legitimately split one ticket into two after the customer raised a second issue, giving the new one a fresh creation time that happened to match. The correct response is to refine the defect signature to include the split marker, record the decision and who approved it, and keep the test. Deleting the test to get the release out would remove the only guard against the original defect.

The same plan entry also shows its limit. It guards against duplicates created by pagination, and against any route that brings back the twelve pinned tickets, but a different extraction bug that duplicates other tickets in another way would be caught only by the global uniqueness test and the release comparison. That is why every entry pairs a pinned test with a population test.

When the test plan is not enough

Regression tests catch known defects; they say nothing about new ones. Pair them with a sampled human review of each release and with the release-to-release comparison, and treat every new defect found that way as the start of a new plan entry.

Stop the release and escalate, rather than patch, in three situations. If a test finds personal data that de-identification should have removed, hold the release for the privacy reviewer and check every earlier version for the same pattern. If a repair would change which records are in scope, confirm the change against the agreement before shipping. And if a defect has come back more than once through different routes, the pipeline design is the problem, and a further test will not fix it.

Keep the plan proportionate. A dataset delivered once does not need ten fields per defect; a dataset refreshed for years does. The retirement rule matters as much as the tests: a test for a source system that no longer exists should be retired by a recorded decision, not left to fail and be ignored.

Showing a buyer that repairs stay repaired

Buyers of recurring datasets ask a simple question after any incident: how do we know it will not happen again? The plan answers it with evidence: the test, the failing rows it caught on the old data, its passing result on every later release, and the recipients who were notified.

Recurring releases with a test history are the kind of operational record SourceX helps a company license to AI developers, covering sourcing, rights review, contracting, preparation, delivery and payment. Because large deliveries remain in the seller's own storage or travel on encrypted drives, and SourceX does not host multi-terabyte datasets, the versioned releases and their regression results can stay with the team that maintains them. Checking whether the records qualify begins with a brief company assessment, and nothing is uploaded at that stage.

Frequently asked questions

What is a dataset quality regression pipeline?

It is the set of automated tests and release gates that runs on every build of a dataset and checks that previously repaired defects have not returned. Each repaired defect contributes at least one test pinned to the records that showed it, and a failing test blocks the release.

How is this different from ordinary data quality checks?

Ordinary checks describe what good data looks like in general, such as no nulls in a required field. Regression tests are tied to specific past defects and specific records, so they catch the return of a known problem even when general metrics look normal.

Should every data quality check block a release?

No. Block on repaired defects, privacy patterns and schema violations, where any failure means something is wrong. Use warnings with agreed thresholds for distribution and volume checks, which move with the business, and require an explanation in the release notes when they fire.

What if the source data legitimately changes and a test fails?

Inspect the stored failing rows, confirm the change is genuine, then refine the test and record the decision and approver. Do not delete the test, because it is still the only guard against the original defect returning through another route.

Which tools can run these tests?

Any tool that can express a check as a query or rule and fail a build will do. dbt data tests, Deequ checks on Spark and TensorFlow Data Validation schemas are common choices, and versioning tools such as DVC help compare one release with the last.

Sources

  • dbt data tests are select queries that return failing records, and a test passes when it returns zero failing rows. Source
  • dbt test severity is error or warn, error by default, with error_if and warn_if conditions that default to any failing row. Source
  • dbt's store_failures saves the records that failed a test in a table named after the test, by default in an audit schema. Source
  • Deequ defines unit tests for data on Apache Spark, persists computed metrics in a metrics repository and supports anomaly detection on quality metrics over time. Source
  • TensorFlow Data Validation generates a schema of expectations such as required values, ranges and vocabularies and detects anomalies such as missing features, out-of-range values and wrong types. Source
  • dvc diff shows DVC-tracked files and directories that were added, modified, renamed or deleted between two commits. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies