Skip to content

Privacy and preparation

Expert decision rationale capture: how to record why the exception was made

By SourceX Editorial · Updated

10 sections

Short answer

To capture why an expert chose an exception, interview them about one specific recent case using only the record as it stood at decision time. Record the cues they noticed, the options they rejected, what fact would have changed their mind and how confident they were, then link each reason to the record that supports it. Keep the outcome sealed until this is done, and have a second expert review.

License terms

Expert decision rationale

This page covers expert decision rationale capture: how to record why the exception was made.Illustrative

Key takeaways

  • Ask about one real case, soon after the decision, not about general rules.
  • Show the expert only what the record held at decision time, and keep the outcome sealed until the rationale is written.
  • Give every reason its own row with a pointer to supporting evidence, and flag reasons that rest on knowledge no record holds.
  • Counterfactual triggers, the facts that would have changed the call, mark the boundary of an exception and are worth more than a longer explanation.
  • Keep disagreement between experts as data with a note of the crux instead of averaging it away.

How do you capture why an expert chose the exception?

Capture it case by case, soon after the decision, by walking the expert back through one specific exception and recording what they could see at the time, which cues they noticed, which options they weighed, why they rejected the rule's default, how sure they were, and what would have changed their mind. Then link every stated reason to the record that shows it, and have a second qualified person review the result before it is treated as data.

The method borrows from the critical decision method, an interview technique developed for decisions made under time pressure and changing conditions. Its published description adapts the critical incident technique and adds probes for the cues, distinctions and typicality judgments that separate experts from novices. The point for business records is the same: experts explain judgment far better when they retell a real case than when they are asked to write down a general rule.

Two things make the result usable for AI work rather than just interesting. The rationale must be tied to evidence that existed when the decision was made, and it must carry the expert's uncertainty and the alternatives they considered. Without both, a model learns to produce confident explanations that sound right and cannot be checked.

Why does asking an expert why, weeks later, produce the wrong answer?

Because the outcome leaks into the explanation. Once a credit hold release has been paid on time, or a waived inspection has led to a recall, the expert and the reviewer both judge the original choice differently. Psychologists call this outcome bias: a pre-registered replication of Baron and Hershey's 1988 experiment found that outcome information shifted how people rated the quality of the same decision, even among participants who said outcomes should not matter.

Time adds a second problem. After a few weeks the expert reconstructs a tidy story from the current state of the record, including notes and attachments added after the decision. The rationale then cites evidence that was not available when the choice was made, which teaches a model to reason from information it will not have in production.

The protocol below counters both: interview within a set window, show the expert only the record as it stood at decision time, and keep the outcome sealed until the rationale has been written and linked.

The decision interview protocol, pass by pass

This is the first half of the asset: a four-pass interview for one exception, version 1.0 dated 2026-10-18. Each pass has one job. Run it within ten working days of the decision where possible, record the session with consent, and keep the outcome hidden from both expert and interviewer until pass four is complete. Allow roughly thirty to forty-five minutes per case; the duration is a planning figure, not a measured result.

The decision interview protocol, pass by pass
PassWhat the interviewer asksWhat gets recordedTrap to avoid
1. Case and defaultWhat came in, and what would the standard rule or policy have done with it?case_id, decision_point, policy_default, decision_taken, decided_atStarting from the outcome instead of the request
2. Timeline at decision timeWalk me through what you looked at, in order, using only what was in the record then.Ordered list of evidence viewed, each with record ID and timestampShowing the expert notes or attachments added after decided_at
3. Cues and optionsWhat made this case different? What else could you have done, and why not that?Cues noticed, alternatives considered, reason each was rejectedAccepting a general rule of thumb instead of the specific cue in this case
4. What would change itWhat fact, if different, would have made you follow the default or pick another option? How sure were you?Counterfactual triggers, confidence on a fixed scale, known unknownsLetting the expert revise the decision now that time has passed

Give every reason its own row and point it at the exact record, field or passage that supports it, with a flag for whether that evidence was visible at decision time. A reason with no pointer is kept, but labeled as tacit, so a downstream user knows it cannot be checked.

Rationale research in machine learning uses a similar structure. Zaidan, Eisner and Piatko had annotators mark the words in a document that justified their label and used those annotator rationales to improve a text classifier. The ERASER benchmark formalizes evidence as a document ID with start and end token positions, and groups evidence into sets that are each sufficient to justify an answer. e-SNLI pairs each inference label with words that annotators highlighted, clicking each word individually, alongside natural language explanations.

Business decisions spread across systems, so the pointer is usually a record and field rather than a token span: an invoice line in NetSuite, a payment history in the ERP, a call note in Salesforce, a comment in a Jira ticket. The table below is the second half of the asset, the evidence linkage record.

How do you link each stated reason to the evidence behind it?
FieldWhat it holdsExample value (illustrative)Check before acceptance
reason_id and reason_textOne reason, in the expert's words, lightly edited for clarityR2: customer has paid every invoice within terms for six yearsOne claim per row; no compound reasons
evidence_refSystem, record ID and field or passageERP payment history, account A-0193, paid_date columnReference resolves to a real record in the delivered extract
visible_at_decisionWhether the evidence existed and was accessible when decidedYes, last updated before decided_atTimestamp of the evidence is earlier than decided_at
evidence_roleSupports, weakens, or was considered and set asideSupportsAt least one weakening or set-aside item on most exceptions
weightExpert's ranking of this reason among all reasons2 of 4Ranks are unique within the decision
tacit flagSet when the reason relies on knowledge not in any recordFalseTacit reasons have a short description of their basis

What evidence would change the expert's decision or make another path equally valid?

Ask for it directly and record the answer as counterfactual triggers. A useful trigger is specific and checkable: "if the customer had any invoice more than 30 days late in the past year, I would have kept the hold" is usable; "if things had looked riskier" is not. Triggers are what let a model learn the boundary of an exception rather than only its center.

Also record when another option would have been equally defensible. Experts often pick between two acceptable paths for reasons of workload, timing or relationship, and a dataset that presents only the chosen path teaches a model that the alternative was wrong. Architecture decision records offer a tested template for this: the MADR template asks for decision drivers, the options considered, the chosen option with its justification, consequences, and how the decision will be confirmed. Those headings translate well to operational exceptions.

If the expert cannot name any fact that would have changed the decision, treat that as a signal. Either the exception is effectively a standing rule that should be written down as policy, or the rationale is being reconstructed rather than recalled.

How should uncertainty and disagreement between experts be recorded?

Record them as data, not noise. Ask for confidence on a fixed scale at the time of the interview, and have a second expert review a sample of cases blind to the first expert's rationale and the outcome. Where they disagree, keep both rationales and a short note of the crux rather than forcing a single answer.

At a glance

How should uncertainty and disagreement between experts be recorded

Disagreement is most informative on exactly the cases buyers want: borderline exceptions where policy is silent. Averaging it away produces a cleaner-looking dataset that hides the cases a model most needs to handle carefully.

Calibration comes later and needs care. Once outcomes are unsealed, you can check whether high-confidence exceptions turned out better than low-confidence ones across many cases. Use that to describe the expert pool, not to relabel individual rationales, because a good decision can still meet a bad outcome.

  • confidence: one of four fixed levels, recorded before the outcome is shown.
  • second_review: reviewer role, agreement (agree, partly, disagree), crux note.
  • outcome: recorded separately with its own date, linked by case_id, unsealed after review.
  • calibration note: pool-level only, stating how many cases it covers.

Who counts as the expert, and what incentives distort the record?

The expert is the person who held the authority and made the call, described by role, tenure band and the decisions they are permitted to make, not by name. A rationale from someone who reviewed the case later is a review, not a decision record, and should be labeled that way.

Incentives shape what experts say. If exceptions are audited, experts will cite policy language defensively; if they are rewarded for speed, they may describe less checking than they did. Make the interview explicitly non-evaluative, pay for the time, and treat "I am not sure why" as an acceptable answer. Reviewers need the same protection: a second expert who expects to be judged on agreement will agree too often.

Record the incentive context briefly with each batch, for example whether the decisions were subject to a quarterly exception audit. Buyers use it to judge how much weight to put on stated reasons.

Worked example (illustrative): releasing a credit hold over policy

The case below is invented to show the protocol end to end; it does not describe a real company, customer or dataset. A distributor's policy holds any order that takes a business customer over its credit limit. A credit manager released one such hold.

  • Pass 1: case C-0412; policy default keep hold; decision release hold; decided by credit manager role, eight-year tenure band, authority to release holds within a set ratio of the limit.
  • Pass 2: evidence viewed in order: order value against limit in the ERP; six-year payment history; open dispute list (empty); a sales rep note saying the order is for a signed public-works contract.
  • Pass 3: cues: the overage came from one large seasonal order, not a trend; options considered: keep hold, release, release half and ship the rest on payment; split shipment rejected because the job site needed one delivery.
  • Pass 4: triggers: any invoice more than 30 days late in the past year, or an open dispute, would have kept the hold; confidence: high.
  • Evidence linkage: four reasons, three linked to ERP and CRM records visible before the decision; one tacit (the manager knew the customer's project manager from earlier jobs), flagged as tacit.
  • Review: a second credit manager agreed with the release but would have asked for a deposit; crux recorded as risk appetite on a first order of this size.

Where the protocol does not fit, and what to do instead

The protocol assumes a decision that one person owned, a record that shows what they saw, and an expert who is available and willing. When those do not hold, narrow the scope rather than forcing the template.

Group decisions made in a meeting need one interview per participant and a note of who had authority; a single combined rationale hides disagreement. Decisions resting mostly on phone calls or hallway conversation will be mostly tacit, which is still useful for training explanations but weak for grounding, so label the batch accordingly. Exceptions in regulated areas such as consumer credit, insurance claims or employment carry their own rules on reasons and records, and those cases should go to counsel or a compliance specialist before capture starts, not after.

Personal details in the evidence also need a plan. Customer names, account numbers and employee identities appear throughout decision records and should be removed or replaced before records leave the company; doing so does not by itself give the company a right to license them.

Turning captured rationale into licensable decision records

Exception rationale linked to evidence is among the more distinctive records a company can hold, because it shows judgment that is rarely written down anywhere else. A company that holds such records can license a selected set to AI developers through SourceX, which covers the rights review, contracting, preparation and delivery.

That conversation opens with a brief assessment of the company, and no files are uploaded for it. Nothing is delivered until the company has approved the scope and terms and signed the agreement, so capturing rationale for internal use and deciding later whether to license it are separate decisions.

Frequently asked questions

What is expert decision rationale capture?

It is the practice of recording, for a specific decision, why an expert chose what they did: the evidence they saw, the cues that mattered, the options they rejected and how confident they were. Done well, each reason points to a record that supports it. The result can train or evaluate systems that must explain or support similar decisions.

How soon after the decision should the interview happen?

As soon as practical, ideally within about two working weeks, while the expert still remembers what they looked at. Longer gaps invite reconstruction from the current record and the known outcome. If only old cases are available, show the expert the record as it stood at decision time and label the batch as retrospective.

Should the expert know the outcome during the interview?

No. Keep the outcome sealed until the rationale, evidence links and confidence are recorded. Outcome knowledge changes how people judge a decision even when they believe it should not, so revealing it first biases both the expert and any reviewer.

How many reviewers does each rationale need?

At least one qualified second reviewer on a sample of cases, and on every case that will be used for evaluation. The reviewer should not see the outcome or, ideally, the first expert's rationale before forming a view. Keep disagreements as recorded data with a note of the crux.

Is a free-text note in the system the same as a captured rationale?

No. A note written at the time is valuable evidence, but it is usually short, written for colleagues and missing alternatives, uncertainty and triggers. The interview uses the note as one input and adds the structure that makes the rationale checkable.

Sources

  • The critical decision method elicits expert knowledge for tasks under time pressure and changing conditions, extending the critical incident technique with probes for cues and judgments. Source
  • A pre-registered replication of Baron and Hershey's outcome bias experiment found outcome information shifted ratings of decision quality. Source
  • Zaidan, Eisner and Piatko (NAACL 2007) showed annotator rationales, the words marked as justifying a label, can improve text classification. Source
  • The ERASER benchmark represents evidence as document ID and token spans, grouped into evidence sets each sufficient to justify an answer. Source
  • e-SNLI annotators highlighted the words supporting each label, alongside natural language explanations. Source
  • The MADR decision record template asks for decision drivers, considered options, the chosen option and justification, consequences and confirmation. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies