Systems and records
Dataset issue record lineage: find every task one source error affects
By SourceX Editorial · Updated
10 sections
Short answer
To find every derived training task a reported source error affects, trace backward from the reported example to the faulty source record and field, decide whether the fault is one record or a rule, and query every source record in scope. Then trace forward through an example-level derivation table to each derived example that read the faulty field, each release that holds it and each recipient who received it.
Provenance trail
Dataset issue record lineage
Key takeaways
- Job and column lineage show which pipelines are involved, not which examples; keep an example-level derivation table.
- Write the defect scope as a query; a reported error often comes from a rule that touched many records.
- Keep only derived examples whose recorded inputs include the faulty field.
- Never edit a delivered release; ship a new version with a changelog and erratum entry.
- Close the issue only when every recipient of every affected release has confirmed the notice.
How do you find every derived training task affected by a reported source error?
Trace in two directions. First go backward from the example the buyer reported to the source record and the exact field that is wrong, and decide whether the defect is in that one record or in a rule that touched many. Then go forward from every defective source record through each derivation step to every derived example, in every release, and from each release to the recipients who received it.
Both walks depend on one thing being recorded at preparation time: a row-level link from each derived example to the source records and fields it was built from, plus the step and code version that built it. With that table, the impact of a reported error is a query. Without it, the impact is a guess, and the usual guess is to fix the one example and miss the other few hundred.
This page is for the business or data program that prepared and delivered the dataset. It covers how to scope, repair and notify after a reported error. Handling a supplier's withdrawal of records, and the tooling for job-level lineage, have their own guides linked below.
Why does pipeline lineage stop short of the affected examples?
Pipeline lineage tools record which job read which dataset, which is necessary but too coarse. The OpenLineage model, for example, describes jobs, runs and datasets, with facets attached; its column lineage facet maps output fields to the input fields used to compute them. That tells you a training file depends on the ticket export and that its answer column came from the resolution field. It does not say which training example came from which ticket.
Pipeline versioning has the same limit. DVC records each stage's dependencies and outputs in dvc.lock with a content hash, so a changed input invalidates the stages that depend on it and shows what must be rebuilt. It works at the file level. When one row in a source file is wrong, it tells you every output is suspect, which is true and not yet useful.
The missing layer is an example-level derivation table. The W3C PROV ontology gives a vocabulary for it: an entity wasDerivedFrom another, and wasRevisionOf marks a newer version carrying substantial content from an older one. A derivation table with one row per derived example per source record is that relation made concrete.
What should the issue to source and derivative impact map include?
Keep one issue record and attach the map to it. The fields below are what a reviewer needs to confirm that every affected example was found, repaired and reported. Values are illustrative.
| Field | What it holds | Illustrative value | Check before closing |
|---|---|---|---|
| issue_id and report | Who reported what, on which example and release | ISS-031, eval item E-0442, release 1.3 | Report reproduced on the named release |
| root source and field | Source record IDs and the exact field at fault | Ticket T-88213, resolution_code | Fault confirmed against the system of record |
| defect scope | Single record, rule or mapping, or upstream system error | Export mapping for code R2 | Scope query written down and rerunnable |
| defective source set | Every source record the scope query returns | 312 tickets (illustrative) | Count matches the scope query output |
| derivation edges | Derived example, source records, fields read, step, code version | E-0442 from T-88213, read resolution_code, step 4, v2.1 | Only examples that read the faulty field kept |
| affected releases | Releases containing any affected example | 1.2 and 1.3 | Each release manifest checked, not assumed |
| recipient and version log | Who received each affected release, when and by what route | Recipient A: 1.2 and 1.3; recipient B: 1.3 | Matches the delivery log |
| repair and new version | What changed and the release that carries it | Mapping fixed; release 1.4 | Superseded releases left unchanged |
| regression checks | Tests run on the repaired release | Hash compare of unaffected examples, counts | All passed and logged |
| notice and reconciliation | Change notice sent and each recipient's confirmation | Notice CN-07, 2 of 2 confirmed | Every recipient confirmed or escalated |
Which released examples and recipients does this source error reach?
Run the walk in a fixed order so two people get the same answer. Write each step's query into the issue record; a reviewer should be able to rerun it next month and get the same counts.
- Reproduce the report on the release the buyer named, using the example ID they gave.
- Trace backward from that example through the derivation table to its source records and the fields it read.
- Confirm the fault in the system of record, such as the help desk, ERP or CRM, not only in the export.
- Decide the scope: is this one bad record, or a mapping, transformation or upstream rule that affected a class of records? Write the scope as a query.
- Run the scope query to get the defective source set.
- Trace forward from that set through the derivation table, keeping only derived examples whose recorded inputs include the faulty field.
- Join the affected examples to each release manifest, then to the delivery log, to list recipients by version.
- Count affected examples per release and per task type, and record the counts on the issue.
When does one reported error mean a systemic problem?
Often. In prepared business data, a reported error frequently comes from a rule rather than a typo: a code mapping, a join on the wrong key, a time zone conversion, a de-identification pattern that missed a format. The first reported example is a sample of the class.
Research on public benchmarks shows how far errors spread when nobody looks. Northcutt, Athalye and Mueller estimated an average of at least 3.3 percent label errors across the test sets of 10 widely used datasets, and showed that corrected labels can change which model ranks higher. A single report is a cheap opportunity to find the rest before a buyer does.
Treat the scope step as the most important judgment in the process. If the fault is a mapping, the scope is every record with that code. If it is a data-entry error in one record, the scope is one record, but check whether the same person or form produced others.
How severe is it, and what does each severity require?
Severity decides speed and who must be told, so set it before the forward trace finishes. These tiers are a starting point; agree yours with recipients before the first delivery.
| Severity | Typical cause | Action | Notice |
|---|---|---|---|
| Privacy or rights | Identifier missed by de-identification, record outside licensed scope | Withdraw affected examples now; reissue | Immediate, with instructions to delete copies |
| Wrong answer in evaluation data | Reference answer or outcome label is incorrect | Patch release; mark affected items | Prompt, listing item IDs per release |
| Wrong label in training data | Mapping or labeling rule error | Patch release with corrected labels | Change notice with counts and IDs |
| Formatting or completeness | Truncated text, missing optional fields | Fix in the next scheduled release | Release notes |
| Disputed, not wrong | Experts disagree with the reported value | Annotate; do not overwrite | Reply to the reporter with the rationale |
How do you repair a release recipients have already pinned?
Never edit a delivered release in place. Recipients have trained or evaluated on it and need to reproduce what they did. Semantic Versioning states the rule plainly for software, and it carries over: once a version is released its contents must not be modified, and any change must ship as a new version.
Ship the repair as a new release with a changelog that names the issue, the affected example IDs and the counts per task type. The Datasheets for Datasets questionnaire asks whether a dataset has an erratum and where to find it; a running erratum page per dataset, updated with each issue, answers that question permanently.
Before release, run regression checks that compare the repaired build with the previous one. Every example outside the affected set should be byte-identical by hash. Counts per split and task type should change by exactly the number you expected. If anything else moved, the repair touched more than it should have.
Worked example (illustrative): one wrong reference answer in a support dataset
This case is invented to show the method; all counts are illustrative. A buyer reports that evaluation item E-0442 in release 1.3 of a customer support dataset expects the agent to issue a refund, while the ticket text plainly shows the refund was denied.
Backward trace: E-0442 was built from ticket T-88213 and read the resolution_code field. In the help desk, the ticket's code is R2, which means refund denied. The export mapping translated R2 as refund issued. Scope: every ticket with code R2. The scope query returns 312 tickets.
Forward trace: those 312 tickets feed 41 evaluation items, 280 response-writing examples and 312 classification labels. The derivation table shows the response-writing examples read only the conversation text, not resolution_code, so they are unaffected. Affected: 41 evaluation items and 312 labels, in releases 1.2 and 1.3. The delivery log shows recipient A received both releases and recipient B received 1.3.
Exception: 6 of the 312 tickets were corrected by hand in the export after an earlier review, so their labels are already right, and none of the 6 fed an evaluation item; the derivation table records the manual correction as a later revision. Final count: 347 affected examples. Release 1.4 fixes the mapping, the regression check confirms that every other example is unchanged, and the notice lists the 347 IDs by release. Both recipients confirm.
Where does the trace run out, and who decides what happens next?
The trace runs out where records stop. Releases prepared before example-level lineage was kept can only be scoped by re-deriving them from source, which is slow and sometimes impossible if the source has changed since. Say so in the notice rather than presenting a partial count as complete.
It also runs out at the recipient. What a buyer did with a release, such as models trained or evaluation scores published, is outside the supplier's records, and what each side must do about it is a contract question. A privacy or rights issue may carry legal notification duties; bring in counsel before the notice goes out, not after.
Why build traceable releases before the first delivery?
Most of this work is easy if the derivation table and delivery log exist before the first release and nearly impossible if they do not. SourceX runs data preparation, delivery and the contracting around them for businesses licensing operational records directly to SourceX, so traceability is part of preparing each release rather than an afterthought.
Before any data reaches SourceX, the business approves the scope and terms and signs the agreement; whether it qualifies at all is settled earlier, through a brief questionnaire that asks for no files.
Frequently asked questions
What is dataset issue record lineage?
It is the set of links that let a team trace a reported dataset error from the derived example back to its source record and field, and forward to every other derived example and release that used the same source. It sits below pipeline lineage, at the level of individual examples.
Is pipeline lineage from a tool like OpenLineage enough?
It is a good foundation but not enough on its own. Pipeline and column lineage show which jobs and fields feed an output; they do not show which example came from which source record. Add an example-level derivation table that records source IDs, fields read, step and code version.
Should we fix the reported example directly in the delivered files?
No. Recipients need the release they used to stay reproducible. Ship the repair as a new version with a changelog and an erratum entry, and leave the earlier release unchanged and clearly marked as superseded.
How do we know the repair did not break anything else?
Compare the repaired build with the previous release. Every example outside the affected set should be identical by hash, and counts per split and task type should change only by the expected number. Log the check on the issue record.
What if the reported value is disputed rather than wrong?
Treat it as a disagreement, not a defect. Record the reporter's view and the original expert rationale, annotate the example, and reply with the reasoning. Overwriting a defensible answer can remove exactly the ambiguity a buyer may need.
Sources
- The OpenLineage specification models lineage as jobs, runs and datasets, with metadata attached as facets. Source
- OpenLineage's column lineage facet maps output fields to the input fields used to compute them. Source
- DVC records each stage's dependencies and outputs with a content hash in dvc.lock, and changed dependencies invalidate the stages that use them. Source
- Semantic Versioning 2.0.0 says the contents of a released version must not be modified and any change must be released as a new version. Source
- W3C PROV-O defines wasDerivedFrom for derivation chains between entities and wasRevisionOf for a derived entity carrying substantial content from the original. Source
- Datasheets for Datasets asks dataset creators whether there is an erratum and, if so, for a link to it. Source
- Northcutt, Athalye and Mueller estimated an average of at least 3.3 percent label errors across 10 benchmark test sets and showed corrected labels can change model rankings. Source
See if your company qualifies
A short company assessment. No data uploads are needed.