Privacy and preparation
Expert knowledge update dataset: recording a changed expert view
By SourceX Editorial · Updated
10 sections
Short answer
Expert decision histories should keep each judgment as a dated version that is never overwritten. When new evidence changes an expert's view, record a new version linked to the earlier one, naming the trigger evidence and when it arrived, whether it was available before, the type of change, and likelihood and confidence before and after. Validate the timing and links so hindsight cannot pass as foresight.
Redaction
Expert knowledge update
Key takeaways
- Keep every expert judgment as a dated version; a revision links back to the view it replaces.
- Name the trigger evidence, when it arrived and whether it was available before the first call.
- Label the revision type and keep likelihood and confidence in basis as separate fields.
- Reaffirmed judgments after new evidence are data too, not empty rows.
- Validate time order and links, and keep records written after the outcome out of prospective training sets.
How should expert decision histories capture a changed view after new evidence?
As two linked versions, never one overwritten field. Keep the expert's first judgment exactly as it was made, with its date, reasoning and confidence, then record the revised judgment as a new version that points back to it and names the evidence that triggered the change, when that evidence arrived, what kind of change it caused and how confident the expert is now.
Most business systems do the opposite. A diagnosis field, a risk rating or a disposition code is simply edited, and the earlier view survives only in an audit log, if at all. The final answer is kept and the most instructive part, the moment the expert saw something that changed their mind, is lost.
The belief revision and evidence trigger annotation schema below fixes that with a small set of required fields, a validator that rejects records which cannot be trusted, and a worked example of a record that passes and one that fails.
Why is a changed mind worth recording at all?
Because the revision shows which evidence an expert treats as decisive and how far it moves them, which a final answer cannot show. A dataset of final answers teaches what experts concluded; a dataset of revisions teaches how expert judgment responds to new facts.
Research on forecasting points the same way. In a study published in Organizational Behavior and Human Decision Processes, Atanasov and colleagues analyzed more than 400,000 probabilistic predictions from a four-year geopolitical forecasting tournament and found that the most accurate forecasters made frequent, small updates. How people revise is evidence about the quality of their judgment, not noise around it.
Professional analytic standards treat revision as something to explain rather than hide. The U.S. Intelligence Community's analytic standards directive, ICD 203, lists explaining change to or consistency of analytic judgments among its tradecraft standards, and asks analysts to state both the likelihood of an outcome and their confidence in the basis for that judgment. Those two ideas map directly onto fields in the schema.
Where do changed expert views already sit in company records?
In any workflow where a first call is made before all the facts are in. The trigger evidence usually arrives as a separate record, so the revision can be rebuilt by joining the two in time order, provided the original judgment was not overwritten.
| Setting | First judgment | Typical trigger evidence | Where both usually live |
|---|---|---|---|
| Field service | Technician's initial diagnosis from symptoms | Part test result or a second site visit | Work order notes and parts returns in a service system |
| Commercial underwriting | Underwriter's referral decision on an application | Inspection report or loss runs received later | Underwriting workbench notes and document log |
| Credit and collections | Credit hold placed on a customer account | Trade references or a payment plan | ERP credit notes and account history |
| Quality control | Disposition of a held lot as reject or rework | Laboratory result or supplier certificate | Quality management records and lot history |
| Customer support escalation | Engineer's root-cause guess on a ticket | Logs or reproduction steps from the customer | Help desk ticket thread and linked bug report |
| Procurement | Buyer's supplier risk rating | Audit finding or delivery performance data | Supplier records and scorecards |
The belief revision and evidence trigger annotation schema
Each record describes one judgment version. A revision is a record whose prior_judgment_id points at an earlier one. Version 1.0 of the schema has the fields below; teams can add domain fields, but removing any of these makes the revision impossible to check.
At a glance
The belief revision and evidence trigger annotation schema
| Field | Allowed values | Purpose | Required when |
|---|---|---|---|
| judgment_id, case_id, version | Stable identifiers; integer version | Ties every version to one case in order | Always |
| prior_judgment_id | Id of the version this one revises | Builds the chain of views | Version above 1 |
| expert_id, qualification | Pseudonymous id; role and years in role | Shows whose judgment it is and why it counts | Always |
| judged_at, recorded_after_outcome | UTC timestamp; yes or no | Places the view in time and flags hindsight | Always |
| conclusion, alternative_considered | Domain code with short text; the rejected option and why | The call itself and the decision boundary | Always |
| likelihood, confidence_in_basis | Calibrated term or probability; low, moderate or high | How likely the call is, and how solid its evidence | Always |
| evidence_considered | List of record ids, each with the version or date seen | What the judgment rested on | Always |
| trigger_evidence, trigger_available_earlier | Record id, version and arrival time; yes, no or unknown | What prompted the revision, and whether it was new or missed | Version above 1 |
| revision_type | Reversal, strengthened, weakened, scope changed, reasoning only, reaffirmed | Names how the view moved | Version above 1 |
| rationale | Free text written at the time | The expert's own reason in their words | Always |
How should the kind of change and the confidence be labeled?
With a fixed revision type and two separate uncertainty fields, so that a change in confidence is not mistaken for a change in conclusion. An expert who keeps the same call but becomes much more certain has revised something real, and a dataset that only tracks conclusions will miss it.
The six revision types cover the moves that matter. Reversal means a different conclusion. Strengthened and weakened mean the same conclusion with more or less likelihood. Scope changed means the conclusion now applies to part of the case, for example two of five units. Reasoning only means the same call for a different reason, which is common when a first guess turns out right for the wrong cause. Reaffirmed records an expert who reviewed the new evidence and kept the view, which ICD 203 counts as worth explaining just as much as a change.
Likelihood and confidence in basis stay apart for the reason ICD 203 gives: how probable an outcome is and how good the evidence for that estimate is are different questions. Fix the likelihood vocabulary before annotation begins, with each term tied to a probability range, and use the same scale for every version so that strengthened and weakened can be computed rather than argued about.
How do the versions stay linked and traceable?
By borrowing relations that existing provenance and annotation standards already define, rather than inventing new ones. That makes the records easier to validate and to exchange with tools that already understand them.
The W3C PROV ontology defines wasDerivedFrom for an entity built from another, wasRevisionOf as the narrower case of a revised version of an earlier entity, and wasInvalidatedBy for an entity that ceased to be usable. In the schema, prior_judgment_id is a wasRevisionOf link, evidence_considered and trigger_evidence are derivation links, and a superseded judgment is invalidated by the revision activity, not deleted.
The W3C Web Annotation Data Model supplies the rest. It defines created and modified timestamps in UTC, an assessing motivation for annotations that evaluate a resource, and a Time State with a sourceDate for the time at which the annotated resource should be interpreted. That last idea is what evidence_considered needs: the record an expert saw, as it stood when they saw it, not as it reads today.
Which validator checks make a revision record trustworthy?
Checks that reject records whose timing, linkage or labels cannot be true. Several can be written directly in JSON Schema: the dependentRequired keyword makes properties required whenever another property is present, and if and then make a set of fields required only when a condition holds, such as a version above 1. The time-order checks need a short script.
- V1: a record with version above 1 must have prior_judgment_id, trigger_evidence, trigger_available_earlier and revision_type.
- V2: prior_judgment_id must point to a record with the same case_id and a version exactly one lower.
- V3: judged_at must be later than the prior version's judged_at.
- V4: trigger_evidence arrival time must fall between the two versions, unless trigger_available_earlier is yes.
- V5: revision_type must agree with the fields: reversal requires a different conclusion; strengthened or weakened requires the same conclusion and a moved likelihood; reaffirmed requires both unchanged.
- V6: every item in evidence_considered must name the version or date of the record seen.
- V7: recorded_after_outcome set to yes routes the record to a separate split and out of any training set that teaches prospective judgment.
- V8: rationale and alternative_considered must be non-empty and must not be copied from the prior version.
Illustrative worked record: a diagnosis revised after a part test
The case below is invented to show the schema; it is not drawn from any company's data. A commercial HVAC service manager first judges that a rooftop unit's repeated shutdowns come from a failing compressor. Two days later a bench test of the returned control board shows intermittent faults, and the manager revises the diagnosis.
The valid record is version 2 of the case. The failing record shows what the validator catches when the revision is entered carelessly, which is how most revisions are entered when no schema exists.
| Field | Valid version 2 | Failing version 2 | Rule broken |
|---|---|---|---|
| prior_judgment_id | Points to version 1 of the same case | Missing | V1 and V2 |
| judged_at | Two days after version 1 | Same timestamp as version 1 | V3 |
| conclusion | Control board fault | Control board fault | None |
| likelihood | Likely, moved from roughly even chance on this cause | Very likely, with no prior value to compare | V5 cannot be evaluated |
| trigger_evidence | Bench test record, version 1, arrived between the two judgments | Bench test record with no version or time | V4 and V6 |
| revision_type | Reversal | Strengthened | V5: the conclusion changed |
| recorded_after_outcome | No | Yes, entered after the repair succeeded | V7 |
What makes a revision record misleading, and who should review it?
Hindsight, incentives and missing qualifications. A revision typed up after the outcome is known tends to make the expert look more prescient than they were, which is why the schema flags it rather than trusting it. Incentives distort revisions too: an expert measured on fast closure may reaffirm too readily, and one reporting to a supervisor with a strong view may reverse to match it. Record who reviewed the case and any such pressures in the dataset documentation.
Documentation frameworks expect this. Datasheets for Datasets, by Gebru and colleagues, asks dataset creators to describe the mechanisms used to collect data and how the data was validated. For revision data, that means stating how judgments were captured, whether at the time or reconstructed, and who checked them.
Bring in a qualified domain expert to review the revision types and likelihood scale for the field, and an annotation research lead to run agreement checks on a sample. Where judgments touch people's credit, employment or similar decisions, the records may raise privacy or legal questions that need their own review before anyone licenses them.
When expert revision histories can become a licensed dataset
Companies that already keep work orders, underwriting notes, quality holds or escalation threads often hold years of first calls and revised calls side by side, which is uncommon in public data. Rebuilding them into this schema is preparation work; whether they can be licensed depends on rights in the records and the agreement, not on the schema.
SourceX helps businesses turn selected operational records, expert decision histories included, into licensed AI datasets, from qualification and rights review through preparation and delivery. Finding out whether a company qualifies takes a short assessment and no data upload, and no judgment record is delivered until the company has approved the scope and terms and signed.
Frequently asked questions
What is an expert knowledge update dataset?
It is a set of records in which experts' judgments are kept as dated versions, with each revision linked to the earlier view and to the evidence that changed it. It lets a model learn how expert judgment responds to new information, not only what the final answer was.
Why not just keep the expert's final answer?
The final answer hides which evidence mattered and how much it moved the expert. Two experts can reach the same final call through very different revisions, and that difference is often what a model most needs to learn.
Should a reaffirmed judgment be recorded if nothing changed?
Yes. An expert who reviews new evidence and keeps their view has made a decision, and recording it shows which evidence experts consider not decisive. Professional analytic standards treat explaining consistency as seriously as explaining change.
How do you stop hindsight from creeping into revision records?
Capture judgments at the time they are made, flag any record written after the outcome was known, and keep flagged records out of training sets meant to teach prospective judgment. Check timestamps against the arrival time of the trigger evidence.
Can we build this from existing business records?
Often, if the original judgment was preserved in notes, audit logs or version history and the trigger evidence is a separate dated record. Where fields were overwritten without history, the first view is lost and should not be reconstructed from memory.
Sources
- Atanasov and colleagues found, across more than 400,000 predictions in a four-year forecasting tournament, that the most accurate forecasters made frequent, small updates. Source
- ICD 203 lists explaining change to or consistency of analytic judgments as a tradecraft standard and asks for both likelihood and confidence in the basis of judgments. Source
- The W3C PROV ontology defines derivation, revision as a narrower kind of derivation, and invalidation of entities. Source
- The W3C Web Annotation Data Model defines created and modified timestamps, an assessing motivation and a Time State with sourceDate. Source
- The JSON Schema validation specification defines dependentRequired for properties required when another property is present. Source
- The JSON Schema core specification defines if, then and else for conditional validation. Source
- Datasheets for Datasets asks dataset creators to describe collection mechanisms and how the data was validated. Source
See if your company qualifies
A short company assessment. No data uploads are needed.