Skip to content

Privacy and preparation

Expert knowledge update dataset: recording a changed expert view

By SourceX Editorial · Updated

10 sections

Short answer

Expert decision histories should keep each judgment as a dated version that is never overwritten. When new evidence changes an expert's view, record a new version linked to the earlier one, naming the trigger evidence and when it arrived, whether it was available before, the type of change, and likelihood and confidence before and after. Validate the timing and links so hindsight cannot pass as foresight.

Redaction

Expert knowledge update

This page covers expert knowledge update dataset: recording a changed expert view.Illustrative

Key takeaways

  • Keep every expert judgment as a dated version; a revision links back to the view it replaces.
  • Name the trigger evidence, when it arrived and whether it was available before the first call.
  • Label the revision type and keep likelihood and confidence in basis as separate fields.
  • Reaffirmed judgments after new evidence are data too, not empty rows.
  • Validate time order and links, and keep records written after the outcome out of prospective training sets.

How should expert decision histories capture a changed view after new evidence?

As two linked versions, never one overwritten field. Keep the expert's first judgment exactly as it was made, with its date, reasoning and confidence, then record the revised judgment as a new version that points back to it and names the evidence that triggered the change, when that evidence arrived, what kind of change it caused and how confident the expert is now.

Most business systems do the opposite. A diagnosis field, a risk rating or a disposition code is simply edited, and the earlier view survives only in an audit log, if at all. The final answer is kept and the most instructive part, the moment the expert saw something that changed their mind, is lost.

The belief revision and evidence trigger annotation schema below fixes that with a small set of required fields, a validator that rejects records which cannot be trusted, and a worked example of a record that passes and one that fails.

Why is a changed mind worth recording at all?

Because the revision shows which evidence an expert treats as decisive and how far it moves them, which a final answer cannot show. A dataset of final answers teaches what experts concluded; a dataset of revisions teaches how expert judgment responds to new facts.

Research on forecasting points the same way. In a study published in Organizational Behavior and Human Decision Processes, Atanasov and colleagues analyzed more than 400,000 probabilistic predictions from a four-year geopolitical forecasting tournament and found that the most accurate forecasters made frequent, small updates. How people revise is evidence about the quality of their judgment, not noise around it.

Professional analytic standards treat revision as something to explain rather than hide. The U.S. Intelligence Community's analytic standards directive, ICD 203, lists explaining change to or consistency of analytic judgments among its tradecraft standards, and asks analysts to state both the likelihood of an outcome and their confidence in the basis for that judgment. Those two ideas map directly onto fields in the schema.

Where do changed expert views already sit in company records?

In any workflow where a first call is made before all the facts are in. The trigger evidence usually arrives as a separate record, so the revision can be rebuilt by joining the two in time order, provided the original judgment was not overwritten.

Where do changed expert views already sit in company records?
SettingFirst judgmentTypical trigger evidenceWhere both usually live
Field serviceTechnician's initial diagnosis from symptomsPart test result or a second site visitWork order notes and parts returns in a service system
Commercial underwritingUnderwriter's referral decision on an applicationInspection report or loss runs received laterUnderwriting workbench notes and document log
Credit and collectionsCredit hold placed on a customer accountTrade references or a payment planERP credit notes and account history
Quality controlDisposition of a held lot as reject or reworkLaboratory result or supplier certificateQuality management records and lot history
Customer support escalationEngineer's root-cause guess on a ticketLogs or reproduction steps from the customerHelp desk ticket thread and linked bug report
ProcurementBuyer's supplier risk ratingAudit finding or delivery performance dataSupplier records and scorecards

The belief revision and evidence trigger annotation schema

Each record describes one judgment version. A revision is a record whose prior_judgment_id points at an earlier one. Version 1.0 of the schema has the fields below; teams can add domain fields, but removing any of these makes the revision impossible to check.

At a glance

The belief revision and evidence trigger annotation schema

The belief revision and evidence trigger annotation schema
FieldAllowed valuesPurposeRequired when
judgment_id, case_id, versionStable identifiers; integer versionTies every version to one case in orderAlways
prior_judgment_idId of the version this one revisesBuilds the chain of viewsVersion above 1
expert_id, qualificationPseudonymous id; role and years in roleShows whose judgment it is and why it countsAlways
judged_at, recorded_after_outcomeUTC timestamp; yes or noPlaces the view in time and flags hindsightAlways
conclusion, alternative_consideredDomain code with short text; the rejected option and whyThe call itself and the decision boundaryAlways
likelihood, confidence_in_basisCalibrated term or probability; low, moderate or highHow likely the call is, and how solid its evidenceAlways
evidence_consideredList of record ids, each with the version or date seenWhat the judgment rested onAlways
trigger_evidence, trigger_available_earlierRecord id, version and arrival time; yes, no or unknownWhat prompted the revision, and whether it was new or missedVersion above 1
revision_typeReversal, strengthened, weakened, scope changed, reasoning only, reaffirmedNames how the view movedVersion above 1
rationaleFree text written at the timeThe expert's own reason in their wordsAlways

How should the kind of change and the confidence be labeled?

With a fixed revision type and two separate uncertainty fields, so that a change in confidence is not mistaken for a change in conclusion. An expert who keeps the same call but becomes much more certain has revised something real, and a dataset that only tracks conclusions will miss it.

The six revision types cover the moves that matter. Reversal means a different conclusion. Strengthened and weakened mean the same conclusion with more or less likelihood. Scope changed means the conclusion now applies to part of the case, for example two of five units. Reasoning only means the same call for a different reason, which is common when a first guess turns out right for the wrong cause. Reaffirmed records an expert who reviewed the new evidence and kept the view, which ICD 203 counts as worth explaining just as much as a change.

Likelihood and confidence in basis stay apart for the reason ICD 203 gives: how probable an outcome is and how good the evidence for that estimate is are different questions. Fix the likelihood vocabulary before annotation begins, with each term tied to a probability range, and use the same scale for every version so that strengthened and weakened can be computed rather than argued about.

How do the versions stay linked and traceable?

By borrowing relations that existing provenance and annotation standards already define, rather than inventing new ones. That makes the records easier to validate and to exchange with tools that already understand them.

The W3C PROV ontology defines wasDerivedFrom for an entity built from another, wasRevisionOf as the narrower case of a revised version of an earlier entity, and wasInvalidatedBy for an entity that ceased to be usable. In the schema, prior_judgment_id is a wasRevisionOf link, evidence_considered and trigger_evidence are derivation links, and a superseded judgment is invalidated by the revision activity, not deleted.

The W3C Web Annotation Data Model supplies the rest. It defines created and modified timestamps in UTC, an assessing motivation for annotations that evaluate a resource, and a Time State with a sourceDate for the time at which the annotated resource should be interpreted. That last idea is what evidence_considered needs: the record an expert saw, as it stood when they saw it, not as it reads today.

Which validator checks make a revision record trustworthy?

Checks that reject records whose timing, linkage or labels cannot be true. Several can be written directly in JSON Schema: the dependentRequired keyword makes properties required whenever another property is present, and if and then make a set of fields required only when a condition holds, such as a version above 1. The time-order checks need a short script.

  • V1: a record with version above 1 must have prior_judgment_id, trigger_evidence, trigger_available_earlier and revision_type.
  • V2: prior_judgment_id must point to a record with the same case_id and a version exactly one lower.
  • V3: judged_at must be later than the prior version's judged_at.
  • V4: trigger_evidence arrival time must fall between the two versions, unless trigger_available_earlier is yes.
  • V5: revision_type must agree with the fields: reversal requires a different conclusion; strengthened or weakened requires the same conclusion and a moved likelihood; reaffirmed requires both unchanged.
  • V6: every item in evidence_considered must name the version or date of the record seen.
  • V7: recorded_after_outcome set to yes routes the record to a separate split and out of any training set that teaches prospective judgment.
  • V8: rationale and alternative_considered must be non-empty and must not be copied from the prior version.

Illustrative worked record: a diagnosis revised after a part test

The case below is invented to show the schema; it is not drawn from any company's data. A commercial HVAC service manager first judges that a rooftop unit's repeated shutdowns come from a failing compressor. Two days later a bench test of the returned control board shows intermittent faults, and the manager revises the diagnosis.

The valid record is version 2 of the case. The failing record shows what the validator catches when the revision is entered carelessly, which is how most revisions are entered when no schema exists.

Illustrative worked record: a diagnosis revised after a part test
FieldValid version 2Failing version 2Rule broken
prior_judgment_idPoints to version 1 of the same caseMissingV1 and V2
judged_atTwo days after version 1Same timestamp as version 1V3
conclusionControl board faultControl board faultNone
likelihoodLikely, moved from roughly even chance on this causeVery likely, with no prior value to compareV5 cannot be evaluated
trigger_evidenceBench test record, version 1, arrived between the two judgmentsBench test record with no version or timeV4 and V6
revision_typeReversalStrengthenedV5: the conclusion changed
recorded_after_outcomeNoYes, entered after the repair succeededV7

What makes a revision record misleading, and who should review it?

Hindsight, incentives and missing qualifications. A revision typed up after the outcome is known tends to make the expert look more prescient than they were, which is why the schema flags it rather than trusting it. Incentives distort revisions too: an expert measured on fast closure may reaffirm too readily, and one reporting to a supervisor with a strong view may reverse to match it. Record who reviewed the case and any such pressures in the dataset documentation.

Documentation frameworks expect this. Datasheets for Datasets, by Gebru and colleagues, asks dataset creators to describe the mechanisms used to collect data and how the data was validated. For revision data, that means stating how judgments were captured, whether at the time or reconstructed, and who checked them.

Bring in a qualified domain expert to review the revision types and likelihood scale for the field, and an annotation research lead to run agreement checks on a sample. Where judgments touch people's credit, employment or similar decisions, the records may raise privacy or legal questions that need their own review before anyone licenses them.

When expert revision histories can become a licensed dataset

Companies that already keep work orders, underwriting notes, quality holds or escalation threads often hold years of first calls and revised calls side by side, which is uncommon in public data. Rebuilding them into this schema is preparation work; whether they can be licensed depends on rights in the records and the agreement, not on the schema.

SourceX helps businesses turn selected operational records, expert decision histories included, into licensed AI datasets, from qualification and rights review through preparation and delivery. Finding out whether a company qualifies takes a short assessment and no data upload, and no judgment record is delivered until the company has approved the scope and terms and signed.

Frequently asked questions

What is an expert knowledge update dataset?

It is a set of records in which experts' judgments are kept as dated versions, with each revision linked to the earlier view and to the evidence that changed it. It lets a model learn how expert judgment responds to new information, not only what the final answer was.

Why not just keep the expert's final answer?

The final answer hides which evidence mattered and how much it moved the expert. Two experts can reach the same final call through very different revisions, and that difference is often what a model most needs to learn.

Should a reaffirmed judgment be recorded if nothing changed?

Yes. An expert who reviews new evidence and keeps their view has made a decision, and recording it shows which evidence experts consider not decisive. Professional analytic standards treat explaining consistency as seriously as explaining change.

How do you stop hindsight from creeping into revision records?

Capture judgments at the time they are made, flag any record written after the outcome was known, and keep flagged records out of training sets meant to teach prospective judgment. Check timestamps against the arrival time of the trigger evidence.

Can we build this from existing business records?

Often, if the original judgment was preserved in notes, audit logs or version history and the trigger evidence is a separate dated record. Where fields were overwritten without history, the first view is lost and should not be reconstructed from memory.

Sources

  • Atanasov and colleagues found, across more than 400,000 predictions in a four-year forecasting tournament, that the most accurate forecasters made frequent, small updates. Source
  • ICD 203 lists explaining change to or consistency of analytic judgments as a tradecraft standard and asks for both likelihood and confidence in the basis of judgments. Source
  • The W3C PROV ontology defines derivation, revision as a narrower kind of derivation, and invalidation of entities. Source
  • The W3C Web Annotation Data Model defines created and modified timestamps, an assessing motivation and a Time State with sourceDate. Source
  • The JSON Schema validation specification defines dependentRequired for properties required when another property is present. Source
  • The JSON Schema core specification defines if, then and else for conditional validation. Source
  • Datasheets for Datasets asks dataset creators to describe collection mechanisms and how the data was validated. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies