Systems and records
Dataset quality regression pipeline: keep a repaired defect from coming back
By SourceX Editorial · Updated
9 sections
Short answer
To keep a dataset repair from reintroducing an earlier defect, fix the defect at the pipeline stage that created it, then turn it into a permanent test pinned to the records that showed it. Run every such test on each rebuild, block the release when a repaired defect reappears, compare the new version with the last one, and log which recipients received which version.
Checklist
Dataset quality regression
Key takeaways
- A repair applied to a delivered file is undone by the next rebuild; fix the stage that produced the defect.
- Every repaired defect becomes a named test with pinned record IDs, and the test outlives the ticket.
- Repaired defects get blocking gates with zero tolerance on pinned records; broad quality metrics get warning thresholds.
- Compare each release with the previous one, by file and by metric, so unexplained changes surface before delivery.
- When a test fails because the source legitimately changed, record the decision and update the test rather than deleting it.
How do you prevent a dataset repair from reintroducing an earlier defect?
Make each repair permanent in three places: the code, the tests and the release record. Fix the defect where it entered, add a test that fails on the old data and passes on the repaired data, pin that test to the specific records that showed the defect, and run the whole collection of these tests before every release. A release that fails a repaired-defect test does not ship.
That collection of tests, run automatically on each build with gates that decide whether the build can be released, is a dataset quality regression pipeline. The idea is borrowed from software regression testing, where every fixed bug gets a test so it cannot quietly return. Datasets need it more than code does, because they are rebuilt from sources that change underneath them.
The question matters most for recurring deliveries. A one-time dataset can be fixed by hand. A dataset that is refreshed monthly from a ticketing system, a CRM or an ERP is rebuilt from scratch each time, and every manual fix that was not encoded in the pipeline disappears with the next run.
How do old defects find their way back?
Almost always through a route that bypasses the original fix. The table lists eight common routes in recurring data work and the test that catches each one. Use it as a checklist when you write the test for a new repair: ask which of these routes could undo it.
Most repairs are exposed to more than one route. A duplicate fix in extraction can be undone by a full re-extraction, by a restored branch and by a changed dedupe key, so its plan entry should name each route and confirm that at least one test runs after the stage each route touches. A test that only runs on the final packaged file catches all of them, but it tells you late and gives little hint of where the regression entered.
| Reintroduction route | How it happens | Illustrative example | Test that catches it |
|---|---|---|---|
| Fix applied to output only | The delivered file was edited; the pipeline still produces the bad value | Mis-encoded characters fixed in the export, not in the extraction step | Pinned-record test run on a fresh build, not on the patched file |
| Full re-extraction | A refresh pulls all history again and overwrites downstream corrections | Monthly rebuild restores merged duplicate tickets | Duplicate-key test on the full history after every rebuild |
| Restored or merged old branch | An older snapshot or code branch is merged back for an unrelated reason | A rollback to last quarter's filter logic re-includes internal test accounts | Exclusion test that lists the pinned test account IDs |
| Source resends old data | The upstream system re-exports records with their original defects | Vendor CSV feed resends a corrected batch in its original form | Value test on pinned records plus a check on source batch IDs |
| Changed transformation | A new or edited step reverses an earlier assumption | Changed dedupe key splits records that had been merged | Count and key tests on the pinned set before and after the step |
| Schema change | A renamed or retyped field is remapped, sometimes to the wrong column | Status and resolution fields swapped after a source upgrade | Schema test plus value-domain test on both fields |
| Manual hotfix not in code | Someone fixed the data in the warehouse directly | A direct update removed personal data from notes in one table only | Pattern test across every table that carries the field |
| Dependency or library change | A parser, encoding or date library behaves differently | Time zone parsing changes after a library upgrade | Pinned timestamp values compared to their expected UTC values |
Why the fix belongs in the pipeline, not in the delivered file
A fix in the delivered file is a fix to one version. A fix in the pipeline is a fix to every future version, and it can be tested. When a defect is found, trace it to the earliest stage that produced it, such as extraction, normalization, joining, de-identification or packaging, and repair that stage.
Then write the test at the same stage. dbt's documentation describes data tests as select queries that look for failing records: a uniqueness test selects duplicates, a not-null test selects nulls, and the test passes when zero failing rows come back. Any assertion that can be written as such a query can become a test. That form suits repaired defects well, because the defect itself is usually easy to describe as a query that should return nothing.
Keep the original defect reproducible. Save a small fixture of the records as they were before the repair, run the new test against it once, and confirm it fails. A regression test that has never failed has not proved it can catch anything.
Run the tests where they can be trusted to run. A regression suite that someone has to remember to start will be skipped in the week it matters most. Wire it into the build so that packaging a release is impossible without a passing run, and store the run's results with the release record so a reviewer can see which tests ran against which version.
The targeted regression and acceptance test plan
Write one plan entry per repaired defect, using these ten fields. The plan is the asset a release reviewer reads; the tests are its executable part.
At a glance
The targeted regression and acceptance test plan
Keep the plan next to the code, in the same repository, and review changes to it the way you review code changes. When a defect is reopened, update the existing entry rather than creating a second one, so the history of the defect, its tests and its notices stays in one place. Over time the plan becomes a record of how the dataset has been hardened, which is useful to new team members as well as to buyers.
| Plan field | What it contains | Example | Pass rule |
|---|---|---|---|
| issue_id and severity | Defect ticket ID and a severity level agreed with the data owner | DQ-214, severity 1 (wrong values in a required field) | Severity set before the test is written |
| defect_signature | The query or rule that returns the defective records | Tickets sharing source ID and created time | Returns rows on the pre-repair fixture |
| pinned_records | IDs of the records that exhibited the defect | A short list of ticket IDs from the issue | Every pinned ID still present, or its removal documented |
| root_cause_stage | The pipeline stage where the defect entered | Extraction: pagination re-read the last page | Named stage matches where the repair was made |
| repair_reference | Code change or configuration that fixed it | Commit or change request ID | Repair is in version control, not a manual edit |
| regression_tests | Tests on pinned records and on the whole population | Pinned duplicate check; global uniqueness of source ID | Zero failures on pinned records; population within threshold |
| acceptance_gate | Whether a failure blocks release or warns | Blocking for pinned test; warning above a small ratio for population | Gate type matches severity |
| version_comparison | Expected differences from the previous release | Fewer tickets after dedupe; no other file changes | Unexpected file or metric changes explained before release |
| recipients_and_notice | Which recipients received a version with the defect and the notice sent | Two recipients on versions 3 and 4; notice issued with the fix | Every affected recipient has a logged notice |
| retirement_rule | When the test may be changed or retired, and who approves | Only if the source system is retired, approved by the data owner | No test deleted without a recorded decision |
Which checks block a release and which only warn?
Block on repaired defects, warn on broad drift. A defect that has been fixed once and found again is a process failure, so a single failing pinned record should stop the release. Population-level checks, such as the share of tickets with an empty resolution, move with the business and need thresholds rather than zero tolerance.
Test frameworks support this split directly. In dbt, a test's severity is error or warn, with error by default, and separate error_if and warn_if conditions that both default to any failing row. A repaired-defect test keeps the strict default; a population test can warn above a small count and error only above a larger one.
Keep the failing rows, not just the verdict. dbt's store_failures setting saves the records that failed a test into a table named after the test, by default in an audit schema. For a regression pipeline, that table is the evidence a reviewer needs to decide whether a failure is a real regression or a legitimate change in the source.
| Check type | Gate | Threshold | Who can override |
|---|---|---|---|
| Repaired defect on pinned records | Blocking | Zero failures | Nobody; fix or document a legitimate change first |
| Repaired defect across population | Blocking above threshold | Agreed count or ratio per defect | Data owner with a written reason |
| Privacy or de-identification pattern | Blocking | Zero matches | Nobody; escalate to the privacy reviewer |
| Schema and value domains | Blocking | No unexpected fields or values | Data owner after a schema review |
| Distribution and volume drift | Warning | Change beyond the agreed band versus last release | Release reviewer, with an explanation in the notes |
Comparing each release with the last one
Rule-based tests catch the defects you already know. A release-to-release comparison catches the ones you do not, by showing what changed and asking whether each change was expected.
Compare at three levels. Files: dvc diff, for example, lists files and directories that were added, modified, renamed or deleted between two commits of a versioned dataset, which is a quick way to see that a fix touched only what it should. Metrics: Deequ, a library for unit tests for data on Spark, can persist computed quality metrics in a metrics repository and run anomaly detection on those metrics over time, so a sudden jump in null counts after a rebuild is flagged. Schema: TensorFlow Data Validation generates a schema of expectations such as required values, ranges and vocabularies and reports anomalies such as missing features, out-of-range values and wrong types.
Write the expected differences into the plan before the build runs. If a dedupe repair should lower the ticket count and change nothing else, any other file or metric change is a question to answer before release, not after a buyer finds it.
Treat a missing comparison as a failed one. If the previous release cannot be loaded, because it was deleted, moved or never versioned, the pipeline cannot tell what changed, and the release should wait until a baseline is restored or a reviewer signs off on a fresh baseline with a written reason.
Worked example (illustrative): duplicate tickets return after a refresh
This example is illustrative and does not describe a real company. A support ticket dataset delivered monthly contained duplicate tickets because the extraction step re-read the last page of results when the API paginated. The team found the duplicates in version 3, removed them from the delivered file and sent version 3.1. Version 4 was rebuilt from the full history and the duplicates were back, since the extraction step had never been fixed.
With the plan, the defect gets issue ID DQ-214 and severity 1. The defect signature is a query for tickets sharing a source ID and creation time. Twelve ticket IDs from the original report are pinned, an illustrative count. The root cause stage is extraction, the repair is a pagination fix in the extractor, and the regression tests are a pinned duplicate check that blocks release and a global uniqueness test on source ID. The version comparison expects fewer rows and no other file changes. The recipients log shows which recipients received versions 3 and 4 and records the notice sent with version 5.
Now the exception. In version 6 the pinned test fails on one ticket. The stored failures show two records with the same source ID, but on inspection the source system had legitimately split one ticket into two after the customer raised a second issue, giving the new one a fresh creation time that happened to match. The correct response is to refine the defect signature to include the split marker, record the decision and who approved it, and keep the test. Deleting the test to get the release out would remove the only guard against the original defect.
The same plan entry also shows its limit. It guards against duplicates created by pagination, and against any route that brings back the twelve pinned tickets, but a different extraction bug that duplicates other tickets in another way would be caught only by the global uniqueness test and the release comparison. That is why every entry pairs a pinned test with a population test.
When the test plan is not enough
Regression tests catch known defects; they say nothing about new ones. Pair them with a sampled human review of each release and with the release-to-release comparison, and treat every new defect found that way as the start of a new plan entry.
Stop the release and escalate, rather than patch, in three situations. If a test finds personal data that de-identification should have removed, hold the release for the privacy reviewer and check every earlier version for the same pattern. If a repair would change which records are in scope, confirm the change against the agreement before shipping. And if a defect has come back more than once through different routes, the pipeline design is the problem, and a further test will not fix it.
Keep the plan proportionate. A dataset delivered once does not need ten fields per defect; a dataset refreshed for years does. The retirement rule matters as much as the tests: a test for a source system that no longer exists should be retired by a recorded decision, not left to fail and be ignored.
Showing a buyer that repairs stay repaired
Buyers of recurring datasets ask a simple question after any incident: how do we know it will not happen again? The plan answers it with evidence: the test, the failing rows it caught on the old data, its passing result on every later release, and the recipients who were notified.
Recurring releases with a test history are the kind of operational record SourceX helps a company license to AI developers, covering sourcing, rights review, contracting, preparation, delivery and payment. Because large deliveries remain in the seller's own storage or travel on encrypted drives, and SourceX does not host multi-terabyte datasets, the versioned releases and their regression results can stay with the team that maintains them. Checking whether the records qualify begins with a brief company assessment, and nothing is uploaded at that stage.
Frequently asked questions
What is a dataset quality regression pipeline?
It is the set of automated tests and release gates that runs on every build of a dataset and checks that previously repaired defects have not returned. Each repaired defect contributes at least one test pinned to the records that showed it, and a failing test blocks the release.
How is this different from ordinary data quality checks?
Ordinary checks describe what good data looks like in general, such as no nulls in a required field. Regression tests are tied to specific past defects and specific records, so they catch the return of a known problem even when general metrics look normal.
Should every data quality check block a release?
No. Block on repaired defects, privacy patterns and schema violations, where any failure means something is wrong. Use warnings with agreed thresholds for distribution and volume checks, which move with the business, and require an explanation in the release notes when they fire.
What if the source data legitimately changes and a test fails?
Inspect the stored failing rows, confirm the change is genuine, then refine the test and record the decision and approver. Do not delete the test, because it is still the only guard against the original defect returning through another route.
Which tools can run these tests?
Any tool that can express a check as a query or rule and fail a build will do. dbt data tests, Deequ checks on Spark and TensorFlow Data Validation schemas are common choices, and versioning tools such as DVC help compare one release with the last.
Sources
- dbt data tests are select queries that return failing records, and a test passes when it returns zero failing rows. Source
- dbt test severity is error or warn, error by default, with error_if and warn_if conditions that default to any failing row. Source
- dbt's store_failures saves the records that failed a test in a table named after the test, by default in an audit schema. Source
- Deequ defines unit tests for data on Apache Spark, persists computed metrics in a metrics repository and supports anomaly detection on quality metrics over time. Source
- TensorFlow Data Validation generates a schema of expectations such as required values, ranges and vocabularies and detects anomalies such as missing features, out-of-range values and wrong types. Source
- dvc diff shows DVC-tracked files and directories that were added, modified, renamed or deleted between two commits. Source
See if your company qualifies
A short company assessment. No data uploads are needed.