Privacy and preparation
Expert tacit knowledge as evidence: how to capture it without inventing it
By SourceX Editorial · Updated
10 sections
Short answer
Anchor every expert statement to a specific past case and the records of that case, then label how it is known: seen in a record, recalled, inferred across cases or opinion. Record the alternative the expert rejected, what would change the call and how sure they are, and have a second reviewer check each line against the records. Where no record exists, mark the gap; never fill it.
Redaction
Expert tacit knowledge
Key takeaways
- An expert's account is a set of claims to check; it becomes evidence only when tied to a case and records someone else can open.
- Walk through specific past cases with their records open rather than asking for general rules.
- Classify each claim as observed, recalled, inferred, opinion or contradicted, and let the class decide how it may be used.
- Collect each expert's view privately before discussion, and keep both paths when records support both.
- Record what remains unknown as an open question; never fill a gap with a rule nobody gave and no record shows.
How do you turn tacit operating knowledge into reviewable evidence?
Treat what an expert tells you as a set of claims, attach each claim to a specific past case and the records that show what happened, and label how each claim is known. Reviewable evidence is a statement plus the material a second person can check it against, plus an honest note of what could not be checked. A polished narrative with no case behind it is not evidence, however experienced the author.
In practice that means five habits. Ask about real cases, not general rules. Pull the records for each case before the session, so the expert reacts to what the system shows. Write down the alternative the expert considered and why it lost. Ask what would have changed the decision, and how confident they are. Then give the worksheet to someone who was not in the room and ask them to confirm or reject each line against the records.
The decision rule underneath is simple: a statement can be used as observed fact only if a record supports it, as a hypothesis if records are silent, and not at all if records contradict it. Most of the value lies in the middle category, provided it stays labeled.
Why an expert's account is a claim to check, not yet evidence
Because memory and expertise both reshape what people report. Experts compress years of cases into rules that feel certain, forget the exceptions, and explain past decisions with reasons that fit the outcome they now know. None of that is dishonest. It is why structured elicitation exists.
Statisticians who study expert judgment say this plainly. The European Food Safety Authority's 2014 guidance on expert knowledge elicitation was written because unaided expert judgment of uncertain quantities, and particularly of the uncertainty around them, is often biased, and it sets out a process running from problem definition and expert selection through elicitation to documentation. Anthony O'Hagan's 2019 paper in The American Statistician, Expert Knowledge Elicitation: Subjective but Scientific, reviews the heuristics and biases that affect elicited judgments, including overconfidence, and the protocols designed to counter them.
Organizational research frames the same problem from the other side. Ikujiro Nonaka's 1994 paper in Organization Science describes organizational knowledge as created through a continuing dialogue between tacit and explicit knowledge. Writing tacit knowledge down is one step in that dialogue; checking what was written against what actually happened is the step that turns it into something a company, an auditor or a buyer can rely on.
Start each session from one real case and its records
Pick a specific case with an outcome you can verify, open its records together and walk through it in order. Experts describe their judgment far more accurately when they are reconstructing one real event than when they are asked for their general approach, and the records keep the reconstruction honest.
The Critical Decision Method, published by Gary Klein, Roberta Calderwood and Donald MacGregor in IEEE Transactions on Systems, Man, and Cybernetics in 1989, is built on this idea. It is a retrospective interview about a specific nonroutine incident, adapted from the critical incident technique, with probes aimed at the cues, perceptual discriminations and knowledge an experienced person used. It has been applied with fireground commanders, structural engineers, paramedics and programmers, among others.
For business records, choose cases from the system of record rather than from memory: a ticket that was escalated, a quote that was repriced, a job that needed a second visit. Bring the timeline, the notes and the outcome. Ask the expert what they noticed first, what they expected to happen, what they considered and rejected, and at what point they would have acted differently. Record their words as spoken; tidy the language later, never the substance.
The elicitation and source validation worksheet
Use one row per claim. A claim is the smallest statement that could be true or false on its own: a cue, a threshold, a sequence, an exception. The columns below are the asset; copy them into a spreadsheet and keep the order, because each one answers a question a reviewer will ask.
| Column | What goes in it | Who fills it | Reviewer question it answers |
|---|---|---|---|
| claim_id and case_id | A stable ID for the claim and the record ID of the case it came from | Facilitator | Which real event is this about? |
| statement | The claim in the expert's own words, then a one-line restatement they approve | Expert, then facilitator | Was the meaning changed when it was tidied? |
| cue | What the expert noticed that triggered the judgment | Expert | Could someone else observe the same signal? |
| record_evidence | Fields, notes or documents in the case that bear on the claim, by reference | Facilitator before the session | Can I open the support myself? |
| evidence_class | Observed, recalled, inferred, opinion or contradicted (defined below) | Second reviewer | How much weight can this carry? |
| rejected_alternative | What else the expert considered and why it lost | Expert | Was there another valid path? |
| decision_changer | What evidence would have made the expert choose differently | Expert | Which assumption is load-bearing? |
| confidence | Expert's own rating, with the range of cases it applies to | Expert | How sure, and for which cases? |
| reviewer_verdict | Confirmed, narrowed, rejected or unresolved, with a note | Second reviewer | Did someone independent check it? |
| open_question | What remains unknown and what record would settle it | Facilitator | Where must we not fill a gap? |
Label how each statement is known before anyone relies on it
Give each claim exactly one evidence class, assigned by the second reviewer rather than the expert, and let the class decide how the claim may be used. The classes stop a vivid recollection and a record-backed fact from looking alike once both are typed into the same document.
| Evidence class | Test | How it may be used | What to do next |
|---|---|---|---|
| Observed | A record in this case shows it directly | As fact for this case | Look for the same pattern in other cases |
| Recalled | The expert remembers it; the records are silent | As a hypothesis, clearly labeled | Find a case where records would show it |
| Inferred | The expert generalizes across cases they name | As a rule to test, not a rule to apply | Pull the named cases and count matches |
| Opinion | No case is offered; it is a belief about how things work | As context only | Ask for a case, or leave it out |
| Contradicted | A record shows something different | Not as stated | Discuss with the expert; keep both versions and the record reference |
Illustrative scenario: a dispatcher's rule for sending a senior technician
This scenario is invented to show the worksheet in use; it does not describe a real company. A heating and cooling service company's lead dispatcher decides which repeat calls get a senior technician. Asked for her rule, she says: anything that comes back within a month goes to a senior tech. The facilitator instead pulls three recent repeat calls from the field service system and walks through each.
In the first case she points to the earlier job's notes, which mention a part that was swapped without testing the circuit board, and says that phrase is what she watches for. The note exists, so the cue is observed for this case. In the second case she recalls that the customer had complained about noise for weeks before the first visit; nothing in the record mentions it, so that claim is recalled and goes on the open question list. In the third case, a callback within a month went to a junior technician; she explains that the equipment was under a manufacturer's warranty and the visit was only to swap a part. Her stated rule is now narrower than she said it was, and the exception is written down.
After the session a second reviewer pulls a sample of repeat calls from the same quarter, say an illustrative 40, and checks how many senior assignments the narrowed rule explains. Whatever share it explains is recorded as a count over that sample, not rounded into a claim about the business. The cases it does not explain become the agenda for the next session. At no point does anyone write a threshold, a cue or a reason the dispatcher did not give and the records do not show.
What to do when two experts disagree about the same case
Keep both paths and the evidence for each, then find out whether they apply to different conditions. Disagreement between experienced people is often information: two valid routes, a difference in the cases each has seen, or a cue one of them weighs and the other ignores. Forcing an early consensus throws that away.
Structured elicitation protocols handle this in ways a business can borrow. The IDEA protocol described by Victoria Hemming and colleagues in Methods in Ecology and Evolution in 2018, named for Investigate, Discuss, Estimate and Aggregate, has experts first make private individual judgments with their reasoning, then discuss, then revise privately before judgments are combined, a modified Delphi procedure. The Sheffield Elicitation Framework, SHELF, developed at the University of Sheffield, provides protocols, templates and guidance for eliciting judgments from a group of experts, and its companion R package implements methods such as pooling several experts' distributions.
For a worksheet the lesson is procedural. Collect each expert's account of the same case separately before any group discussion. Record where they differ in the rejected_alternative and decision_changer columns. If the records support both, keep both and write the condition that separates them. If the records support one, the other is marked contradicted for that case, not deleted.
Tracing every line back to its records and its author
Record, for every line, which records it was derived from and who said it, so anyone can rebuild the chain later. Without that link a worksheet drifts: lines get merged, reworded and copied into procedures until nobody can say whether a rule came from a case, a person or a guess.
The W3C PROV ontology gives a vocabulary for exactly this. It models entities, activities and agents; wasDerivedFrom links one entity to another it was built from, and wasAttributedTo links an entity to the agent responsible for it. You do not need to adopt PROV formally. Mirroring its two links, derived from these records and attributed to this person in this session, is enough to make a worksheet auditable.
Keep the session recording or notes as their own entity, dated, and point to them from each claim. When a claim is narrowed or rejected, keep the old version and link the new one to it. That history is often more useful to a reviewer than the final rule.
Where the worksheet stops and specialist review begins
Stop and bring in a specialist when a claim would be used for a safety, legal, medical or financial decision, when the records needed to check it contain personal or confidential information the reviewer is not cleared to see, or when the only support is a single person's opinion and the stakes are high. The worksheet documents judgment; it does not make a judgment safe to act on.
Watch incentives too. If experts are rewarded for the number of rules they produce, or feel their standing depends on sounding certain, the worksheet will fill with confident opinion. Reward open questions and narrowed claims as much as confirmed ones, and say so at the start of each session.
Finally, respect the people involved. Tell experts how their statements will be used, keep their names out of any dataset that leaves the company unless they agree, and remember that customer and employee details inside the case records need the same privacy review as any other record set.
When captured judgment becomes part of a licensed dataset
Companies that document expert judgment this way often find the records behind it, the escalated tickets, repeat jobs and repriced quotes, are valuable on their own as training and evaluation data. Licensing them is a separate decision from capturing them, and it depends on rights in the records, privacy review and the agreement, not on de-identification alone.
SourceX is one route for that step: it handles the rights review, contracting, preparation and delivery that turn a company's chosen records into a licensed AI dataset. Whether a business qualifies is answered by a short assessment rather than an upload, and the escalated tickets or repeat jobs behind a worksheet stay put until the company has approved scope and terms and signed.
Frequently asked questions
What should a structured elicitation and source validation worksheet include?
One row per claim, with the case it came from, the expert's own words, the cue they noticed, references to the records that bear on it, an evidence class, the alternative they rejected, what would change their decision, their confidence and an independent reviewer's verdict. Add an open question column so gaps are recorded instead of filled.
How do you stop a facilitator from putting words in the expert's mouth?
Record the expert's own wording first and only then write a restatement that the expert approves. Never add a threshold, cue or reason that the expert did not give and the records do not show. A second reviewer comparing the restatement with the original wording catches most drift.
Is tacit knowledge evidence if no record supports it?
It is a hypothesis, which is still useful. Label it as recalled or opinion, note which record would confirm it, and do not use it as fact until that record is found. Many valuable rules start this way and become evidence after a few cases are checked.
How many cases does one expert session need?
There is no fixed number; three to five well-documented cases are a practical starting point, because each claim they produce is tied to an event someone can check. Choose cases with different outcomes, including at least one where the expert's usual rule did not apply.
Who should be the second reviewer?
Someone with enough domain knowledge to read the records, who was not in the session and does not report to the expert. Their job is to confirm, narrow or reject each line against the records, not to judge whether the expert is good at the work.
Sources
- EFSA's 2014 guidance on expert knowledge elicitation notes that unaided expert judgment of uncertainty is often biased and sets out a documented elicitation process. Source
- O'Hagan's 2019 paper reviews the heuristics and biases, including overconfidence, that affect elicited expert judgments, and the leading protocols. Source
- Nonaka's 1994 paper describes organizational knowledge as created through dialogue between tacit and explicit knowledge. Source
- The Critical Decision Method elicits expert knowledge through probes about specific incidents, building on the critical incident technique. Source
- The IDEA protocol has experts investigate and estimate privately, discuss, then revise privately before aggregation. Source
- SHELF provides protocols, templates and guidance for eliciting judgments from a group of experts. Source
- The SHELF R package implements elicitation methods and can calculate a weighted linear pool across experts. Source
- The W3C PROV ontology defines entities, activities and agents, with wasDerivedFrom and wasAttributedTo relations. Source
See if your company qualifies
A short company assessment. No data uploads are needed.