Skip to content

Privacy and preparation

Expert judgment dataset pilot: how to capture verifiable expert decisions

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Pilot expert decision capture on one recurring decision type and a small, fixed set of cases. Settle participation, confidentiality and ownership first, then have qualified experts record each decision with the evidence they relied on, their reasoning, the alternatives they rejected and their confidence. Have a second expert decide an overlapping subset blind, and scale only if evidence is complete, decisions are reproducible from it, and disagreement and uncertainty are recorded rather than erased.

Checklist

Expert judgment dataset

This page covers expert judgment dataset pilot: how to capture verifiable expert decisions.Illustrative

Key takeaways

  • A verifiable expert decision record ties the call to the case evidence as it stood at the time, the reasoning, rejected alternatives and a stated confidence.
  • Start with one decision type that recurs often, has inputs already in a system of record and has a later outcome you can check.
  • Settle employee participation, client confidentiality and ownership of the records before the first case is captured, with counsel where law or contracts are involved.
  • Keep disagreement as data: record each expert's call separately and measure agreement and calibration instead of forcing one answer.
  • Decide to stop, narrow or scale from four measured signals: evidence completeness, reproducibility, agreement and calibration.

How do you pilot the capture of verifiable expert decisions?

Choose one decision your experts make repeatedly, pull a fixed set of real past or current cases, and ask a small group of qualified people to record each decision in a structured form: the call, the evidence they used, why, what else they considered and how sure they are. Then have a second expert make the same call independently on part of the set and compare. A few weeks and a few dozen cases are usually enough to see whether the method works.

The pilot answers a narrow question: can this team's judgment be captured in a form that someone else can check? It does not try to build a dataset large enough to license or to train anything. That comes later, and only if the pilot shows that records are complete, that a second expert reaching the same evidence tends to reach the same call, and that the cases where they differ are explained.

The output is a short brief with measured results and a stop, narrow or scale decision. The asset further down is the template for it.

What makes an expert decision verifiable rather than just recorded?

A decision is verifiable when another qualified person, given the same record, can see what the expert saw, follow the reasoning and judge whether the call was reasonable. Most business systems record only the outcome: the claim paid, the part replaced, the ticket escalated. The reasoning and the options that were rejected live in people's heads or in free-text notes that skip the important parts.

At a glance

What makes an expert decision verifiable rather than just recorded

That gap is why expert decisions are valuable to AI developers and why they are hard to capture. A record with only the final call teaches a model what was decided; a record with evidence, reasoning and alternatives teaches it how to decide and when the call is close.

The record needs nine fields. Keep it this small during the pilot; extra fields reduce completion.

  • Case reference: a stable identifier and a frozen snapshot of the inputs as they were at decision time.
  • Decision: the call made, from a fixed list where possible.
  • Evidence cited: pointers to the specific documents, readings or fields relied on.
  • Rationale: two to five sentences linking the evidence to the call.
  • Alternatives: other calls considered and the reason each was rejected.
  • Flip condition: what evidence would have changed the decision.
  • Confidence: a probability or a fixed scale stated before any outcome is known.
  • Expert profile: role, years in role and relevant qualification, kept pseudonymous.
  • Later outcome: what actually happened, filled in when it becomes known.

Which decision type should the pilot start with?

Start with a decision that happens often, whose inputs already sit in a system you control, and whose outcome shows up later. Frequency gives you enough cases quickly, existing inputs mean experts are not re-creating evidence from memory, and a later outcome lets you check calibration. The rows below are illustrative candidates from different kinds of businesses, not recommendations for any one company.

Which decision type should the pilot start with?
Candidate decision (illustrative)Where the inputs usually liveLater outcome signalPilot fit
Repair or replace a failed unitField service work orders, readings and parts historyRepeat failure or callback within a set periodStrong: frequent, documented, outcome observable
Escalate a support ticket to engineeringHelp desk ticket thread and account historyWhether engineering confirmed a defectStrong if ticket text is complete
Approve, adjust or deny a commercial claimClaims file, policy terms and adjuster notesReopened claim, dispute or recoveryGood, but client confidentiality review is heavier
Accept or reject a supplier quoteProcurement system, quote documents and specsDelivery performance against the quoteModerate: outcome arrives slowly
Approve a code change for releaseRepository, review comments and test resultsRevert or incident linked to the changeGood where review comments are substantive
Set a project estimateEstimating tool, scope documents and past jobsActual hours or cost against estimateModerate: many judgments bundled in one number

What must the consented scope review settle before the first case?

Four questions, each answered separately. Do the experts agree to take part, on terms they can refuse without consequence? May the case files be used this way under client and vendor contracts? What personal data sits in the cases and how will it be minimized? Who owns the decision records the pilot creates?

Participation is the subtle one. The experts are usually employees, and European regulators have said that the dependency in an employment relationship makes it unlikely an employee can refuse a request from the employer without fear of detriment. Where consent is used, GDPR Article 7 lets people withdraw it at any time, as easily as they gave it, so the pilot must be able to remove one person's records. Many companies rely on a different basis or on optional participation with a real alternative; counsel should decide which applies.

Client confidentiality often matters more than privacy. A claims file, an audit workpaper or an engineering ticket may be covered by a client contract that limits use. Settle that in the scope review, not after records exist. Ownership of the new records also needs a line in writing: the rationale an expert writes during the pilot is work product, and it should be clear who holds it before anyone discusses licensing.

Finally, look at incentives. Paying experts per case or rewarding agreement with a colleague can quietly change how they decide. Pay for time, and never tell a second rater what the first decided.

The consented scope review and utility pilot brief

Fill in one row per decision before the pilot starts and the last column when it ends. The defaults are illustrative sizes for a first pilot in a mid-sized team; adjust them to your case volume and the time experts can spare.

The consented scope review and utility pilot brief
Brief itemWhat to decideIllustrative pilot defaultEvidence to hold at close
Decision typeOne recurring call with existing inputs and a later outcomeOne decision type onlyWritten definition of the decision and its allowed outcomes
Case sampleWhich cases, from which period, chosen how60 cases sampled across the full range of difficultySampling rule and the list of case references
Expert poolWho qualifies and how qualification is shown3 to 5 experts with a defined minimum time in rolePseudonymous expert profiles
Capture formThe nine record fields and their allowed valuesStructured form, rationale capped at five sentencesForm version and field definitions
OverlapHow many cases get a blind second decisionOne third of cases decided twice, independentlyPaired decisions with both rationales
Disagreement handlingWhether and how disagreements are discussedKeep both calls; a short note on why they differDisagreement log with reasons
UncertaintyHow confidence is statedProbability that the call is right, stated before outcomes are knownConfidence values per decision
Utility testsWhich measures decide stop, narrow or scaleCompleteness, reproducibility, agreement, calibrationResults table with each measure
Participation and confidentialityBasis for expert participation, client contract limits, personal data handlingReviewed by counsel and the privacy owner before case oneSigned-off scope review and removal process
Stop rulesConditions that end or narrow the pilot earlyWritten before the first caseRecord of any rule triggered

How should disagreement and uncertainty be kept?

Keep each expert's call as its own record and treat disagreement as information. A survey in the Journal of Artificial Intelligence Research found that people disagree on the judgments used to train AI even on tasks thought to be objective, and that when there are many high-quality judgments per item, training on the spread of labels can outperform training on a single aggregated label. Forcing consensus in the pilot throws that signal away.

Measure agreement on the overlapping cases with a chance-corrected statistic. Cohen's kappa compares observed agreement between two raters with the agreement expected by chance, computed as observed minus expected, divided by one minus expected. Low agreement is not automatically a failure; read the disagreement log first. If differences trace to missing evidence, fix the capture. If they trace to genuinely different reasonable approaches, that is valuable and should be recorded as alternative paths.

Measure calibration once outcomes arrive. The Brier score is the mean squared difference between the stated probability and what happened; smaller is better, and because it is a strictly proper scoring rule, experts cannot improve their score by stating anything other than their honest belief. A team whose 90 percent calls come true far less often than that has a confidence problem worth knowing about before records are used to train a model.

What evidence would change the expert's decision?

Ask it directly, for every case. The flip condition field asks the expert to name the reading, document or fact that would have produced a different call, and whether another path would have been equally acceptable. It takes a minute and it is often the most useful line in the record.

Flip conditions turn a single decision into a small map of the decision boundary. They show which inputs carry weight and which are noise, and they give a reviewer a concrete way to check the reasoning: if the cited reading had been different, would the call have changed? When two experts disagree, comparing their flip conditions usually shows whether they weighed the same evidence differently or looked at different evidence.

Record equally valid paths as alternatives with a status, not as errors. In many fields two experienced people can take different routes to an acceptable result; a record that says so is more honest and more useful than one that pretends there was a single right answer.

How do you measure the pilot and decide to stop, narrow or scale?

Use four measures and set the thresholds before you start. Completeness is the share of records with every required field filled and at least one evidence pointer that resolves. Reproducibility is the share of overlapping cases where a second expert, given only the record's evidence, reaches the same call. Agreement is the chance-corrected statistic above. Calibration is the Brier score once outcomes exist.

Document the pilot as if an outsider will read it. Datasheets for Datasets proposes that every dataset be accompanied by documentation of its motivation, composition, collection process and recommended uses. The Data Cards extended template adds fields that matter for expert data, including the expertise of annotators, the number of annotators per example and the inter-rater adjudication policy, and Hugging Face dataset card metadata has an annotations_creators tag whose example values include expert-generated. Filling these in at pilot scale shows where your documentation is thin while it is cheap to fix.

Stop when the records cannot be made complete without reconstructing evidence from memory, when the scope review finds that client contracts forbid this use of the case files, or when experts cannot take part on terms they could refuse. More cases will not fix any of those.

Narrow when one sub-type of case reproduces well and another does not, or when completeness depends on a system only some sites use. Take the part that works and leave the rest for a second pilot.

Scale when completeness and reproducibility meet the thresholds you set, disagreements are explained, and calibration is either good or understood. Bring in a specialist reviewer when the decision type carries professional liability, such as engineering sign-off, financial advice or legal judgment, before the records travel beyond the team.

Worked example (illustrative): repair or replace calls at an equipment service company

All numbers in this example are invented to show the method. A commercial equipment service company picks the decision to repair or replace a failed compressor. Inputs are in its field service system: readings, unit age, parts history and technician notes. The outcome signal is a repeat failure within a set period.

Four senior technicians take part after a scope review with counsel; participation is optional and two others decline without consequence. Sixty past cases are sampled across unit types, and twenty are decided twice. In the first week only about 70 percent of records have a resolvable evidence pointer because technicians cite readings from memory; the form is changed to require a reading identifier, and completeness rises above 90 percent.

On the overlapping cases the paired technicians agree on most calls. The disagreement log shows that most splits involve units near a set age, where one technician weighs age heavily and another weighs recent readings. Both are recorded as valid alternatives with flip conditions. Calibration is checked as outcomes arrive; confidence above 0.9 turns out to be optimistic for older units.

The team decides to narrow and scale: keep the decision type, add a required age and readings block, and extend capture to two more regions.

From a pilot to a licensing conversation

A pilot shows whether your experts' decisions can be captured well; whether they can be licensed is a separate question about rights, volume and buyer fit. Buyer interest usually starts at two or more years of steady history and tens of thousands of records held in one system, though a smaller set of genuinely rare expert work can still qualify.

If the records look licensable, SourceX's first step is to assess the company briefly, without any uploaded records, and no expert decision record is delivered before your company has approved the scope and terms and signed the agreement. A finished pilot brief is a useful thing to have in hand for that assessment.

Frequently asked questions

How many cases does an expert judgment pilot need?

Enough to see patterns, not enough to train a model. A few dozen cases, with a third decided twice, usually shows whether records come out complete and whether a second expert can reproduce the call. If agreement or completeness is unclear, extend the same decision type before adding new ones.

Should experts discuss disagreements and agree on one answer?

Not before both calls are recorded. Keep each expert's decision and rationale separately, then add a short note on why they differ. Research on learning from disagreement shows that the spread of judgments is useful signal, and a forced consensus hides it.

How should experts state their confidence?

As a probability that their call is correct, given before the outcome is known. That lets you score calibration with a proper rule such as the Brier score when outcomes arrive. Fixed scales work too, but they need a written definition for each level.

Can employees be asked to take part in capturing their decisions?

They can be invited, but participation needs care. European regulators note that consent from employees is rarely freely given because of the employment relationship, and consent can be withdrawn at any time. Counsel should decide the basis, and the pilot should offer a real alternative to anyone who declines.

How is this different from collecting preference labels?

Preference labels ask raters to compare outputs a model produced. This pilot captures the business's own decisions on real cases, with the evidence and reasoning behind them. The two can be combined later, but they answer different questions.

Sources

  • scikit-learn documents Cohen's kappa as (p_o - p_e) / (1 - p_e), an inter-annotator agreement statistic, and the Brier score as the mean squared difference between predicted probability and actual outcome, a strictly proper scoring rule where smaller is better. Source
  • The JAIR survey Learning from Disagreement reports that humans disagree on judgments used to train AI and that training on soft labels can outperform aggregated labels when items have many high-quality annotations. Source
  • Datasheets for Datasets proposes that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses. Source
  • The Data Cards extended template asks for expertise of annotators, number of annotators per example and the inter-rater adjudication policy. Source
  • Hugging Face dataset card metadata includes an annotations_creators tag with example values crowdsourced, found, expert-generated and machine-generated. Source
  • GDPR Article 7(3) lets a data subject withdraw consent at any time and requires withdrawal to be as easy as giving consent. Source
  • EDPB consent guidelines say the employer-employee dependency makes it unlikely employees can refuse consent without fear of detriment. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies