Systems and records
Expert correction provenance dataset: what a senior reviewer's fix must carry
By SourceX Editorial · Updated
10 sections
Short answer
A senior reviewer's correction should carry the original value and the new one, pointers to the records the reviewer relied on as they stood at review time, the rule applied, a reason code with a one-line rationale, a confidence level, the alternative the reviewer rejected, and whether the fix applies beyond this item. Without that record, a correction is only a newer answer that nobody can audit or learn from.
Provenance trail
Expert correction provenance
Key takeaways
- A correction without its evidence is just a second opinion with a later timestamp.
- Record the reviewer's sources as they stood at review time, or hindsight will look like expertise.
- Reason codes must separate slips, misread evidence, misapplied rules, changed rules and genuine judgment calls, because each teaches a model something different.
- Model the correction as a revision of the original answer, attributed to the reviewer, so lineage survives export.
- Sample corrections for a second review: correctors make errors too, and the dataset should show how often.
What evidence should accompany a senior reviewer's correction?
Every correction should travel with six things: what the answer was, what it became, which records justify the change, which rule or standard was applied, why the reviewer chose this answer over the obvious alternative, and how sure the reviewer was. Two more fields make the record usable at scale: whether the fix applies to this item only or to a class of similar items, and who the reviewer is in role terms.
That is the core of an expert correction provenance dataset. It is a set of before-and-after pairs in which each pair can be traced to its supporting evidence and to the person who made the call. Businesses already produce corrections every day, in invoice coding review, claim file audits, support ticket quality checks and underwriting second looks. Most of them record only the new value.
The difference matters to anyone who later relies on the history. An auditor wants to know whether the correction was right. A model builder wants to know whether the original answer was a mistake the model should learn to avoid, or a defensible choice the reviewer simply preferred to make differently.
Why a fix button records the change but not the reason
Review tools are built to move work, not to explain it. Label Studio's documented review workflow is a good example: a reviewer can accept an annotation, correct a mostly right annotation and click Fix and Accept, or reject it, with options to close it without rework, return it to the original annotator, or pass the task to a different annotator. The documentation notes that this review workflow is part of the Enterprise edition.
Those states tell you that a fix happened and who made it. They do not tell you which part of the source the reviewer read, which guideline clause decided the matter, or whether the reviewer saw information the original annotator never had. Business systems behave the same way: field history in an ERP or CRM shows the old value, the new value, the user and the time, and leaves the reason in an email or in the reviewer's head.
The fix is cheap to add at the moment of review and nearly impossible to reconstruct later. A reviewer asked months afterward why they changed a cost center will give a plausible reason, not necessarily the actual one.
Is a correction a revision or a new answer?
Treat it as a revision, and say so in the data model. The W3C PROV ontology distinguishes a general derivation, where one entity is transformed, updated or constructed from another, from a revision, which revises an entity into a revised version. It also defines attribution, the ascribing of an entity to an agent. A correction is a revision of the original answer, attributed to the reviewer, that used specific evidence records.
Modelling it this way keeps lineage intact when the data is exported. The original answer is not deleted or overwritten in the dataset; it remains an entity with its own author, and the corrected answer points back to it. That is what lets a buyer build preference pairs, error labels or evaluation items from the same history without guessing which value came first.
It also helps to label what the reviewer was doing. The W3C Web Annotation vocabulary separates an assessment of a target, a request to change or edit it, and a question about it, and the data model says an annotation should have exactly one motivation. In correction data the same split is useful: a reviewer who changed the answer, one who graded it without changing it, and one who queried it are three different events and should not share a label.
The correction rationale and evidence annotation scheme
Use these ten fields for every correction. Fields marked as required in the rule column block the record from export if they are empty; the rest are strongly recommended.
Two design choices keep the scheme cheap to run. First, most fields are picked from lists or filled automatically: the item, the original value, the author role and the timestamps come from the system, so the reviewer types only the rationale and, when needed, the rejected alternative. Second, the scheme is attached to the review step that already exists, as a short panel on the same screen, rather than run as a separate documentation task afterward. Teams that bolt it on later tend to fill it in from memory, which brings back the hindsight problem the scheme is meant to remove.
Pilot it on one review queue for a few weeks before rolling it out. Count how often each field is left empty or filled with a default, and read a sample of rationales. Fields that are routinely skipped either need a better list of values or are not worth asking for in that queue.
| Field | What to record | Allowed values or example | Validation rule |
|---|---|---|---|
| correction_id and item_id | Stable ID for the correction and the work item it changes | COR-000123 on invoice line INV-88-2 | Required; item_id resolves to a record in the source system |
| original_value and original_author_role | The answer before review and the role of whoever produced it, human or model | Cost center 4100, entered by AP clerk | Required; value copied, never re-derived |
| corrected_value and reviewer_role | The answer after review and the reviewer's role and qualification level | Cost center 4300, senior AP reviewer, level 3 | Required; reviewer ID pseudonymized for export |
| action_type | What the reviewer did | changed, graded_only, queried, rejected_for_rework | Required; one value only |
| evidence_refs_as_of | Pointers to the records the reviewer relied on, with the version or timestamp each had at review time | PO 7731 line 2 as of review date; contract schedule B | Required for changed; each pointer resolves and has a timestamp |
| rule_applied | The guideline, policy clause or standard that decided the case, with its version | Coding policy section 4.2, version in force on review date | Required for changed unless reason code is slip |
| reason_code and rationale | One code from the taxonomy below and one or two sentences in the reviewer's words | rule_misapplied: PO line names project 4300 | Required; rationale under 60 words, no personal data |
| alternative_rejected | The other answer the reviewer considered and why it lost | 4100 would be right if the PO were overhead | Required when reason code is judgment_call |
| confidence | Reviewer's certainty on a fixed scale | certain, likely, judgment | Required; judgment forces a second review |
| scope_and_info_parity | Whether the fix applies to similar items, and whether the original author had the same information | scope: item; parity: same_info or new_info_after | Required; new_info_after excludes the pair from error training |
Which reason codes separate a mistake from a judgment call?
Reason codes exist so a buyer can filter corrections by what they teach. Eight codes cover most review work; add a domain code only if none of these fits and document it in the guideline.
The codes matter most at the boundaries. A reviewer choosing between evidence_missed and rule_changed is deciding whether the original author did anything wrong, so the guideline should give a short test for each pair of neighboring codes and one example of each. Recheck those boundaries whenever a code's share of corrections moves sharply from one month to the next, because a sudden shift usually means a new reviewer or a changed rule rather than a change in the work.
| Reason code | Meaning | Evidence the record must cite | Best training or evaluation use |
|---|---|---|---|
| slip | Typo, transposed digit, wrong pick from a list | The source value that was mistyped | Error detection; low value for reasoning |
| evidence_misread | The source said X and the author read Y | The exact field or passage misread | Grounding and extraction training |
| evidence_missed | A relevant record existed and was not consulted | The missed record, with proof it was available at the time | Retrieval and completeness checks |
| rule_misapplied | Right facts, wrong rule or wrong reading of the rule | Rule clause and version | Policy-following and preference pairs |
| rule_changed | The original was right under the old rule; the rule has since changed | Both rule versions and the effective date | Temporal evaluation; never as an author error |
| judgment_call | Both answers were defensible; the reviewer preferred one | The rejected alternative and the deciding consideration | Soft labels or multiple valid answers |
| upstream_data_wrong | The source record itself was wrong and has been fixed | The corrected source record and its fix date | Data quality signals; exclude from author error |
| policy_exception | The rule was set aside for this case by someone with authority | The approval and the approver's role | Exception handling, kept apart from rule following |
What would have changed the reviewer's mind?
Ask the reviewer to name it at the moment of correction, in one sentence. A reviewer who can say that a different PO line, a later contract amendment or a customer's written instruction would have flipped the answer has told you where the decision boundary sits. That sentence is often more useful to a model builder than the corrected value.
The question also exposes weak corrections. If the reviewer cannot name any evidence that would have changed the answer, the correction may rest on habit or preference rather than on the record, and the confidence field should say judgment. Those cases go to a second reviewer before they are counted as errors by the original author.
Keep the rejected alternative even when the reviewer is certain. Paired with the deciding consideration, it turns a single label into a contrast that shows why one answer beats another, which is the form preference and reward data need.
Worked example (illustrative): an invoice coded to the wrong cost center
This example is illustrative and does not describe a real company. An accounts payable clerk codes an invoice line to an overhead cost center. A senior reviewer changes it to a project cost center. In a plain field history, that is all anyone will ever see.
With the scheme, the record shows action_type changed, evidence_refs_as_of pointing to the purchase order line that names the project and the vendor contract schedule, rule_applied naming the coding policy clause that sends project purchases to the project, reason_code rule_misapplied, confidence certain, scope item, and info parity same_info. The rationale reads: the PO line names the project, so policy routes the cost there. That pair is a clean training example of a rule misapplied on available evidence.
Now the exception. In a second case the reviewer makes the same change, but the purchase order was amended to name the project two days after the clerk coded the line. The record says new_info_after and cites the amended PO with its timestamp. The clerk was right on the evidence available, so this pair must not be used to teach a model that the clerk erred. It belongs in a temporal or refresh evaluation instead. In an illustrative review sample of 50 corrections, even a handful of such cases would distort an error-rate estimate if they were not flagged.
How do you check the correctors?
Re-review a sample of corrections with a second senior reviewer who does not see the first reviewer's rationale until after deciding. Oversample judgment-level confidence and any reason code that is new to a reviewer. Record agreement per reason code, because a high overall agreement can hide one code that reviewers apply inconsistently.
Review tools can help direct that effort. Label Studio, for example, lets reviewers order tasks by agreement or by model confidence so that uncertain items are reviewed first. The same ordering logic works for corrections: review first the corrections that disagree with prior decisions on similar items.
Document the process the way dataset documentation standards expect. The Croissant responsible AI vocabulary includes a field for the annotation protocol and one for annotation analysis, which covers disagreement between annotations as a signal and how the final labels relate to individual responses. Datasheets for Datasets asks parallel questions in its collection and labeling sections. A correction dataset that cannot answer them is hard for a buyer to accept.
Feed what the second review finds back into the scheme. Corrections that the second reviewer reverses become their own records, with the first reviewer as the original author, so the dataset shows not only how often frontline answers are corrected but how often corrections hold.
When a correction record should stay out of a dataset
Some corrections are useful internally and wrong to share. Drop or hold back a record when the rationale reveals legal advice or a dispute, when the evidence pointers lead to records outside the agreed scope, or when the reviewer's free text names customers, employees or other people who should not appear in the export.
Correction histories can also read as performance records about the people who were corrected. Use role and level rather than names, pseudonymize author and reviewer IDs consistently, and agree internally that the exported data will not be used to evaluate individual staff. If that agreement cannot be made, narrow the export to corrections on machine output only.
Finally, stop and ask a domain specialist when the reason codes do not fit. A review process where most corrections end up as judgment calls may be measuring an unclear guideline rather than expert knowledge, and the guideline should be fixed before the history is packaged.
Getting a correction history ready for a licensing review
Being able to read your review history is not the same as having the right to license it. Before packaging, confirm who owns the underlying records and the review notes, what customer contracts and privacy commitments allow, and which fields need de-identification. De-identification on its own does not create a right to license.
A business with a review queue like this can have SourceX take it from sourcing and qualification through rights review, contracting, preparation, delivery and payment. Its correction history is delivered only after the company has signed off on scope and terms and signed the agreement, and finding out whether the company qualifies takes a short assessment with no data upload.
Frequently asked questions
What is an expert correction provenance dataset?
It is a set of before-and-after pairs in which each correction links to the original answer, the reviewer's role, the evidence the reviewer relied on as it stood at review time, the rule applied and a reason code. The provenance makes each correction auditable and lets a buyer decide what it can be used to teach.
Why record evidence as of the review time?
Records change after decisions are made. If you cite only the current version, a correction based on information that arrived later looks like a mistake by the original author. Timestamped pointers let you separate genuine errors from hindsight.
Do reviewers need to write long explanations?
No. A reason code plus one or two sentences is enough for most corrections, and the structured fields carry the rest. Longer notes increase the chance of personal data in free text and slow reviewers down, which leads to skipped fields.
Can corrections of model output use the same scheme?
Yes. Set the original author role to the model and version, and keep every other field. Corrections of model output are often easier to share because they do not describe staff performance, and they are directly useful as preference or error data.
How is this different from resolving annotator disagreement?
Disagreement resolution merges peers' answers into a label. A correction is hierarchical: a reviewer with authority changes someone else's answer and should justify it. The scheme keeps both answers and the justification instead of collapsing them into one value.
Sources
- Label Studio's Enterprise review workflow lets reviewers accept, fix and accept, or reject annotations with rework options, and order review by agreement or model confidence. Source
- PROV-O defines a revision as a derivation that revises an entity into a revised version, and attribution as ascribing an entity to an agent. Source
- The Web Annotation vocabulary distinguishes assessing, editing (requesting a change) and questioning as annotation motivations. Source
- The Web Annotation Data Model says there should be exactly one motivation for each annotation. Source
- The Croissant RAI vocabulary documents the annotation protocol and annotation analysis, including disagreement as signal and how final labels relate to individual responses. Source
- Datasheets for Datasets groups its questions by lifecycle stage, including collection process and preprocessing, cleaning and labeling. Source
See if your company qualifies
A short company assessment. No data uploads are needed.