Skip to content

Systems and records

Dataset issue record lineage: find every task one source error affects

By SourceX Editorial · Updated

10 sections

Short answer

To find every derived training task a reported source error affects, trace backward from the reported example to the faulty source record and field, decide whether the fault is one record or a rule, and query every source record in scope. Then trace forward through an example-level derivation table to each derived example that read the faulty field, each release that holds it and each recipient who received it.

Provenance trail

Dataset issue record lineage

This page covers dataset issue record lineage: find every task one source error affects.Illustrative

Key takeaways

  • Job and column lineage show which pipelines are involved, not which examples; keep an example-level derivation table.
  • Write the defect scope as a query; a reported error often comes from a rule that touched many records.
  • Keep only derived examples whose recorded inputs include the faulty field.
  • Never edit a delivered release; ship a new version with a changelog and erratum entry.
  • Close the issue only when every recipient of every affected release has confirmed the notice.

How do you find every derived training task affected by a reported source error?

Trace in two directions. First go backward from the example the buyer reported to the source record and the exact field that is wrong, and decide whether the defect is in that one record or in a rule that touched many. Then go forward from every defective source record through each derivation step to every derived example, in every release, and from each release to the recipients who received it.

Both walks depend on one thing being recorded at preparation time: a row-level link from each derived example to the source records and fields it was built from, plus the step and code version that built it. With that table, the impact of a reported error is a query. Without it, the impact is a guess, and the usual guess is to fix the one example and miss the other few hundred.

This page is for the business or data program that prepared and delivered the dataset. It covers how to scope, repair and notify after a reported error. Handling a supplier's withdrawal of records, and the tooling for job-level lineage, have their own guides linked below.

Why does pipeline lineage stop short of the affected examples?

Pipeline lineage tools record which job read which dataset, which is necessary but too coarse. The OpenLineage model, for example, describes jobs, runs and datasets, with facets attached; its column lineage facet maps output fields to the input fields used to compute them. That tells you a training file depends on the ticket export and that its answer column came from the resolution field. It does not say which training example came from which ticket.

Pipeline versioning has the same limit. DVC records each stage's dependencies and outputs in dvc.lock with a content hash, so a changed input invalidates the stages that depend on it and shows what must be rebuilt. It works at the file level. When one row in a source file is wrong, it tells you every output is suspect, which is true and not yet useful.

The missing layer is an example-level derivation table. The W3C PROV ontology gives a vocabulary for it: an entity wasDerivedFrom another, and wasRevisionOf marks a newer version carrying substantial content from an older one. A derivation table with one row per derived example per source record is that relation made concrete.

What should the issue to source and derivative impact map include?

Keep one issue record and attach the map to it. The fields below are what a reviewer needs to confirm that every affected example was found, repaired and reported. Values are illustrative.

What should the issue to source and derivative impact map include?
FieldWhat it holdsIllustrative valueCheck before closing
issue_id and reportWho reported what, on which example and releaseISS-031, eval item E-0442, release 1.3Report reproduced on the named release
root source and fieldSource record IDs and the exact field at faultTicket T-88213, resolution_codeFault confirmed against the system of record
defect scopeSingle record, rule or mapping, or upstream system errorExport mapping for code R2Scope query written down and rerunnable
defective source setEvery source record the scope query returns312 tickets (illustrative)Count matches the scope query output
derivation edgesDerived example, source records, fields read, step, code versionE-0442 from T-88213, read resolution_code, step 4, v2.1Only examples that read the faulty field kept
affected releasesReleases containing any affected example1.2 and 1.3Each release manifest checked, not assumed
recipient and version logWho received each affected release, when and by what routeRecipient A: 1.2 and 1.3; recipient B: 1.3Matches the delivery log
repair and new versionWhat changed and the release that carries itMapping fixed; release 1.4Superseded releases left unchanged
regression checksTests run on the repaired releaseHash compare of unaffected examples, countsAll passed and logged
notice and reconciliationChange notice sent and each recipient's confirmationNotice CN-07, 2 of 2 confirmedEvery recipient confirmed or escalated

Which released examples and recipients does this source error reach?

Run the walk in a fixed order so two people get the same answer. Write each step's query into the issue record; a reviewer should be able to rerun it next month and get the same counts.

  • Reproduce the report on the release the buyer named, using the example ID they gave.
  • Trace backward from that example through the derivation table to its source records and the fields it read.
  • Confirm the fault in the system of record, such as the help desk, ERP or CRM, not only in the export.
  • Decide the scope: is this one bad record, or a mapping, transformation or upstream rule that affected a class of records? Write the scope as a query.
  • Run the scope query to get the defective source set.
  • Trace forward from that set through the derivation table, keeping only derived examples whose recorded inputs include the faulty field.
  • Join the affected examples to each release manifest, then to the delivery log, to list recipients by version.
  • Count affected examples per release and per task type, and record the counts on the issue.

When does one reported error mean a systemic problem?

Often. In prepared business data, a reported error frequently comes from a rule rather than a typo: a code mapping, a join on the wrong key, a time zone conversion, a de-identification pattern that missed a format. The first reported example is a sample of the class.

Research on public benchmarks shows how far errors spread when nobody looks. Northcutt, Athalye and Mueller estimated an average of at least 3.3 percent label errors across the test sets of 10 widely used datasets, and showed that corrected labels can change which model ranks higher. A single report is a cheap opportunity to find the rest before a buyer does.

Treat the scope step as the most important judgment in the process. If the fault is a mapping, the scope is every record with that code. If it is a data-entry error in one record, the scope is one record, but check whether the same person or form produced others.

How severe is it, and what does each severity require?

Severity decides speed and who must be told, so set it before the forward trace finishes. These tiers are a starting point; agree yours with recipients before the first delivery.

How severe is it, and what does each severity require?
SeverityTypical causeActionNotice
Privacy or rightsIdentifier missed by de-identification, record outside licensed scopeWithdraw affected examples now; reissueImmediate, with instructions to delete copies
Wrong answer in evaluation dataReference answer or outcome label is incorrectPatch release; mark affected itemsPrompt, listing item IDs per release
Wrong label in training dataMapping or labeling rule errorPatch release with corrected labelsChange notice with counts and IDs
Formatting or completenessTruncated text, missing optional fieldsFix in the next scheduled releaseRelease notes
Disputed, not wrongExperts disagree with the reported valueAnnotate; do not overwriteReply to the reporter with the rationale

How do you repair a release recipients have already pinned?

Never edit a delivered release in place. Recipients have trained or evaluated on it and need to reproduce what they did. Semantic Versioning states the rule plainly for software, and it carries over: once a version is released its contents must not be modified, and any change must ship as a new version.

Ship the repair as a new release with a changelog that names the issue, the affected example IDs and the counts per task type. The Datasheets for Datasets questionnaire asks whether a dataset has an erratum and where to find it; a running erratum page per dataset, updated with each issue, answers that question permanently.

Before release, run regression checks that compare the repaired build with the previous one. Every example outside the affected set should be byte-identical by hash. Counts per split and task type should change by exactly the number you expected. If anything else moved, the repair touched more than it should have.

Worked example (illustrative): one wrong reference answer in a support dataset

This case is invented to show the method; all counts are illustrative. A buyer reports that evaluation item E-0442 in release 1.3 of a customer support dataset expects the agent to issue a refund, while the ticket text plainly shows the refund was denied.

Backward trace: E-0442 was built from ticket T-88213 and read the resolution_code field. In the help desk, the ticket's code is R2, which means refund denied. The export mapping translated R2 as refund issued. Scope: every ticket with code R2. The scope query returns 312 tickets.

Forward trace: those 312 tickets feed 41 evaluation items, 280 response-writing examples and 312 classification labels. The derivation table shows the response-writing examples read only the conversation text, not resolution_code, so they are unaffected. Affected: 41 evaluation items and 312 labels, in releases 1.2 and 1.3. The delivery log shows recipient A received both releases and recipient B received 1.3.

Exception: 6 of the 312 tickets were corrected by hand in the export after an earlier review, so their labels are already right, and none of the 6 fed an evaluation item; the derivation table records the manual correction as a later revision. Final count: 347 affected examples. Release 1.4 fixes the mapping, the regression check confirms that every other example is unchanged, and the notice lists the 347 IDs by release. Both recipients confirm.

Where does the trace run out, and who decides what happens next?

The trace runs out where records stop. Releases prepared before example-level lineage was kept can only be scoped by re-deriving them from source, which is slow and sometimes impossible if the source has changed since. Say so in the notice rather than presenting a partial count as complete.

It also runs out at the recipient. What a buyer did with a release, such as models trained or evaluation scores published, is outside the supplier's records, and what each side must do about it is a contract question. A privacy or rights issue may carry legal notification duties; bring in counsel before the notice goes out, not after.

Why build traceable releases before the first delivery?

Most of this work is easy if the derivation table and delivery log exist before the first release and nearly impossible if they do not. SourceX runs data preparation, delivery and the contracting around them for businesses licensing operational records directly to SourceX, so traceability is part of preparing each release rather than an afterthought.

Before any data reaches SourceX, the business approves the scope and terms and signs the agreement; whether it qualifies at all is settled earlier, through a brief questionnaire that asks for no files.

Frequently asked questions

What is dataset issue record lineage?

It is the set of links that let a team trace a reported dataset error from the derived example back to its source record and field, and forward to every other derived example and release that used the same source. It sits below pipeline lineage, at the level of individual examples.

Is pipeline lineage from a tool like OpenLineage enough?

It is a good foundation but not enough on its own. Pipeline and column lineage show which jobs and fields feed an output; they do not show which example came from which source record. Add an example-level derivation table that records source IDs, fields read, step and code version.

Should we fix the reported example directly in the delivered files?

No. Recipients need the release they used to stay reproducible. Ship the repair as a new version with a changelog and an erratum entry, and leave the earlier release unchanged and clearly marked as superseded.

How do we know the repair did not break anything else?

Compare the repaired build with the previous release. Every example outside the affected set should be identical by hash, and counts per split and task type should change only by the expected number. Log the check on the issue record.

What if the reported value is disputed rather than wrong?

Treat it as a disagreement, not a defect. Record the reporter's view and the original expert rationale, annotate the example, and reply with the reasoning. Overwriting a defensible answer can remove exactly the ambiguity a buyer may need.

Sources

  • The OpenLineage specification models lineage as jobs, runs and datasets, with metadata attached as facets. Source
  • OpenLineage's column lineage facet maps output fields to the input fields used to compute them. Source
  • DVC records each stage's dependencies and outputs with a content hash in dvc.lock, and changed dependencies invalidate the stages that use them. Source
  • Semantic Versioning 2.0.0 says the contents of a released version must not be modified and any change must be released as a new version. Source
  • W3C PROV-O defines wasDerivedFrom for derivation chains between entities and wasRevisionOf for a derived entity carrying substantial content from the original. Source
  • Datasheets for Datasets asks dataset creators whether there is an erratum and, if so, for a link to it. Source
  • Northcutt, Athalye and Mueller estimated an average of at least 3.3 percent label errors across 10 benchmark test sets and showed corrected labels can change model rankings. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies