AI uses for records
Expert rationale with unsupported claims: how to tie each reason to evidence
By SourceX Editorial · Updated
10 sections
Short answer
Link an expert explanation to evidence claim by claim. Split it into single assertions, attach the specific record behind each one, mark it supported, partly supported or unsupported, test whether the decision still stands without any unsupported claim, and record the expert's certainty on a fixed scale plus the evidence that would have changed the call. Unsupported but plausible steps become labeled judgment or are removed.
Checklist
Expert rationale unsupported
Key takeaways
- A convincing explanation and a correct one are different things; review the evidence, not the prose.
- Split each rationale into typed claims, because facts, rules, inferences and exclusions are verified differently.
- Only evidence that existed on the decision date can support the decision; later findings are hindsight.
- Record certainty on a fixed scale with numeric ranges and capture what evidence would have reversed the call.
- Keep legitimate disagreement as data instead of forcing a single gold rationale.
How should an expert explanation be linked to supporting evidence?
Link it claim by claim. Break the explanation into its separate assertions, attach to each one the specific record that supports it (a document ID, a field value, a log line, a photo with its capture time), mark each assertion as supported, partly supported or unsupported, and record how certain the expert was and what evidence would have changed the call. An explanation that reads well but has no record behind a key step is an opinion, and it should be labeled as one.
This matters most when the explanation will be reused: as a training target for a model that must explain its own decisions, as a reference answer in an evaluation set, or as part of a licensed dataset of expert judgment. In each of those uses the explanation teaches something. If an unsupported step sits inside a fluent paragraph, a model learns to produce fluent paragraphs with unsupported steps.
The checklist below is written for the people who hold these records: operations leaders, senior specialists and the reviewers who check their work. It assumes the explanations already exist in business systems, for example claim notes, underwriting referral memos, inspection reports or engineering change reasons, and asks what has to be added before they can be trusted as data.
Why a persuasive rationale is not yet a verified one
Because people judge explanations by how convincing they sound, and convincing is a different property from correct. Research on model interpretability draws the same line: Jacovi and Goldberg separate faithfulness, whether an explanation reflects the reasoning that actually produced a decision, from plausibility, how convincing it is to a human reader, and argue that faithfulness should be treated as a matter of degree rather than a yes or no property. The distinction transfers directly to human experts. A senior adjuster can write a confident, well-organized note whose stated reason was not the reason that drove the decision.
The research also shows how to test the link between a rationale and an outcome. The ERASER benchmark scores rationales on comprehensiveness, whether removing the cited evidence weakens the prediction, and sufficiency, whether the cited evidence alone is enough to reach it. Those are model-side metrics, but the questions are the right ones for a human rationale too: if you took away the evidence the expert cited, would the decision still stand on what remains, and is the cited evidence by itself enough to justify it?
Data collection practice offers a second lesson. When the e-SNLI dataset gathered free-text explanations from annotators, the collection asked them to highlight the words essential to the label and to build their explanation from those highlights, with automatic checks rejecting explanations that were too short or simply copied the input. The design ties the prose to marked evidence at the moment of writing, which is far cheaper than reconstructing the link later. Business records rarely have that structure, so the checklist has to add it after the fact.
Splitting an explanation into claims a reviewer can check
Split the explanation into single assertions and give each one a type, because each type is verified differently. A sentence such as the unit failed because the owner skipped two service intervals, which voids coverage under the plan holds three assertions: an observation about service history, a causal inference about the failure, and a rule about coverage. Each needs its own evidence, and any one of them can be the weak link.
Do the split in a separate column or file rather than editing the expert's words. The original note is evidence of what the expert wrote at the time; the claim list is the reviewer's analysis of it. Keeping both lets a later reader see exactly where the reviewer and the expert might disagree about what the note says.
| Claim type | Example assertion | What can support it | If no record supports it |
|---|---|---|---|
| Observed fact | The last recorded service visit was 26 months before the failure | A dated service record or system log with its record ID | Mark unsupported; ask whether the record exists elsewhere before keeping the claim |
| Measurement | Pressure read below the stated minimum at inspection | The instrument reading, unit, time and who took it | Keep only as the expert's recollection, labeled as such |
| Rule or policy | Skipped maintenance excludes the failure from coverage | The policy, contract clause or procedure version in force on the decision date | Treat the decision as unsupported until the governing text is found |
| Causal inference | The skipped service caused the failure | Teardown findings, failure analysis or a documented pattern in comparable cases | Label as expert judgment and record the uncertainty level |
| Exclusion of alternatives | A manufacturing defect was ruled out | The checks performed and their results | Record the alternative as open, not ruled out |
| Outcome or follow-up | The customer accepted the decision without appeal | Case closure record, appeal log or later correspondence | Leave the outcome field empty rather than inferring it |
The rationale verification and uncertainty review checklist
Use the checklist as a review pass on every rationale before it enters a dataset, and record the result of each check in its own field so that a buyer or an internal auditor can filter on it. The checks are ordered: there is no point calibrating the uncertainty of a claim whose evidence link is broken. A rationale passes when every claim that the decision depends on is supported, and every remaining unsupported claim is either labeled as judgment or removed.
If your team stores the links in a graph or metadata catalog, the W3C PROV-O vocabulary already has the relations you need: derivation, which it describes as the transformation of one entity into another, and attribution, which ties an entity to the agent responsible for it. Modeling each claim as an entity derived from a source record and attributed to the expert makes the evidence trail queryable rather than buried in a spreadsheet.
Checklist · 3 items
- Fields to add to each rationale record: rationale ID, case ID, expert role, decision date, claim ID, claim type, claim text, evidence record ID, evidence date, support status, load-bearing flag, uncertainty label, alternatives considered, contradiction notes, reviewer role, review date and checklist version.
- Support status values: supported, partly supported, unsupported, labeled judgment, removed.
- Release rule: a rationale ships as verified only when every load-bearing claim is supported or explicitly labeled as judgment with an uncertainty level.
0 of 3 ticked. Tick each item as you go.
| Check | Question the reviewer answers | Passes when | Action if it fails |
|---|---|---|---|
| 1. Claim split | Has the explanation been divided into single, typed assertions? | Each assertion has a claim ID and a type | Return to splitting; do not review a paragraph as one unit |
| 2. Evidence link | Does each claim point to a specific record, not a system or folder? | Record ID, field or page, and capture date are filled | Mark the claim unsupported and flag it for the expert |
| 3. Time check | Did the evidence exist when the decision was made? | Evidence date is on or before the decision date | Exclude later evidence from support; note it as hindsight |
| 4. Rule version | Is the cited policy the version in force on the decision date? | Version or effective date matches | Re-check the decision against the correct version |
| 5. Load-bearing test | Would the decision stand if this claim were removed? | Load-bearing claims are all supported | Downgrade the rationale to partial or exclude the case |
| 6. Sufficiency test | Do the cited records alone justify the decision? | A second reviewer reaches the same call from the evidence alone | Record the missing reasoning step as a gap |
| 7. Alternatives | Which other explanations were considered and why were they rejected? | At least the main alternative is named with its rejection evidence | Record open alternatives explicitly |
| 8. Uncertainty | How confident was the expert in each judgment claim? | A calibrated label from the fixed scale is attached | Ask the expert, or label as not stated |
| 9. Disagreement | Did any reviewer or later decision contradict this rationale? | Contradictions are logged with their source | Keep both views; do not silently pick one |
| 10. Version and sign-off | Who reviewed it, when, and against which checklist version? | Reviewer role, date and checklist version are recorded | The rationale is not release-ready |
How sure was the expert, in words a reader cannot misread?
Ask the expert to choose from a fixed scale with numeric ranges, not to describe confidence in their own words. Free-text hedges such as probably, likely or almost certainly mean different things to different readers, and a model trained on them inherits the vagueness. A published example of a fixed scale is the guidance note written for lead authors of the IPCC Fifth Assessment Report, which ties each likelihood term to a probability range: likely means 66 to 100 percent, and the note treats the boundaries as fuzzy. The same note advises giving the probability range directly when there is enough information to do so.
You do not need to adopt that scale, but you do need one that is written down, has numeric anchors and is used by every expert in the dataset. Three or four levels are usually enough for business judgment: confident, more likely than not, uncertain, and speculative, each with a stated range.
Collect the label at the time of the decision when you can, and mark it as retrospective when you cannot. An expert who rates their confidence after learning the outcome is not reporting the same thing as one who rated it before. Over time, a dataset that holds both the confidence label and the eventual outcome lets you check calibration: whether the cases marked more likely than not turned out right more often than not.
What evidence would flip the decision, or make another path just as good?
Record the evidence that would have reversed the decision, because it is the most useful thing an expert explanation can teach and the thing most often missing. A reviewer should be able to read the rationale and say which single finding, had it gone the other way, would have produced a different call. If nobody can name one, the decision may rest on a general impression rather than the cited evidence.
Equally valuable is the case where two paths were defensible. Many business decisions have more than one acceptable answer: a claim could be paid in part or denied with an offer of goodwill, a part could be repaired or replaced. When the expert considered both and chose one for a stated reason, that reason is the judgment worth capturing. When a second expert would have chosen the other path for an equally good reason, the dataset should hold both views, labeled as legitimate disagreement, rather than forcing a single gold answer.
Two fields capture this cleanly: a decision-changing evidence field, written as a condition (if the teardown had shown a casting flaw, coverage would apply), and an alternative path field with the reason it was not chosen. These fields make a rationale testable. A model trained on them can be checked against counterfactual versions of the same case.
Illustrative case: a senior service manager explains a warranty denial
This example is illustrative; the company, the case and every count in it are invented. A senior service manager at an equipment dealer denies a warranty claim on a compressor and writes: The unit was not serviced on schedule, the oil analysis shows contamination consistent with neglected filters, and the plan excludes failures caused by skipped maintenance. Denied.
The reviewer splits the note into four claims. Claim 1, no service on schedule, links to the dealer's service history, which shows the last visit 26 months before the failure against a 12-month interval: supported. Claim 2, oil contamination, links to a lab report dated two days after the failure: supported, and dated before the decision. Claim 3, contamination consistent with neglected filters, is a causal inference with no teardown record: labeled judgment, uncertainty more likely than not. Claim 4, the exclusion, links to the plan terms, but the case was opened under a plan version issued after the unit was sold.
The load-bearing test exposes the problem. The denial depends on claim 4, and the reviewer has to retrieve the plan version in force at the sale date. If that older version contains the same exclusion, the rationale passes with claim 3 labeled as judgment. If it does not, the decision rests on the wrong rule, and the case enters the dataset as an example of a rationale that read well and cited the wrong text.
The exception worth recording is a contrary path. A second manager notes that the customer had a documented service appointment cancelled by the dealer, not the owner. That fact does not appear in the original note. The reviewer logs it under disagreement with its record ID, and the rationale is marked partly supported with an open alternative. Both versions are useful data; neither should be silently discarded.
Who is qualified to explain, and what were they rewarded for?
Record the expert's qualification for the specific judgment, not just their title, and record what they were measured on. Courts offer a useful frame for the first part. Federal Rule of Evidence 702, as amended in 2023, allows expert opinion when the proponent shows it is more likely than not that the testimony is based on sufficient facts or data, is the product of reliable principles and methods, and reflects a reliable application of those methods to the facts of the case. A rationale dataset is not a courtroom, but the three tests map neatly onto the checklist: evidence links, a stated method, and a reviewed application to this case.
Incentives belong in the record because they shape explanations. An adjuster measured on claim cost, a support lead measured on handle time, or an underwriter measured on approval rate may write rationales that are accurate and still selective. Capture the role's main performance measure at the time of the decision in a field, without naming individuals, so that a buyer can check whether rationales lean in one direction.
Reviewers need the same scrutiny. A reviewer who reports to the expert, or who wrote the policy being applied, is not independent. Record the reviewer's role and relationship to the decision, and route rationales from senior experts to a reviewer of equal standing or a panel rather than to their own team.
Rationales to hold back, narrow, or send to a second expert
Some rationales cannot be fixed by review and should be held back or narrowed before anything is shared. The decision is usually clear once the checklist has run.
At a glance
Rationales to hold back, narrow, or send to a second expert
- Hold back when the load-bearing claim depends on records that no longer exist, or on a conversation nobody documented.
- Narrow to the supported claims when only part of the rationale holds, and drop the outcome label if it depended on the unsupported part.
- Send to a second qualified expert when the causal inference is the whole decision, or when the expert's confidence label is speculative.
- Send to counsel when the rationale applies a statute, a regulation or contract terms whose meaning is disputed, or when the case was litigated or appealed.
- Exclude when the explanation contains personal details about a customer or employee that the support status does not need, until privacy review has decided how to remove or generalize them.
- Stop the batch when more than a small share of rationales fail the evidence link check, because the problem is in how notes are written, not in individual cases.
Packaging verified rationales from your own records
Verified rationale records are rarer than decision records, and the review work above is what separates the two. A business that already writes explanations into claim notes, referral memos, inspection reports or change tickets holds the raw material; adding claim splits, evidence links and uncertainty labels is the preparation step that turns them into judgment data a buyer can audit.
For businesses that want to license this kind of judgment data, SourceX runs the qualification, rights review, contracting, preparation and delivery work, turning chosen records into a licensed dataset. The company agrees the scope and signs before any records leave its control. Having verified notes is not the same as having the right to license them, so ownership, privacy permission and contract terms are reviewed separately from quality. The first step is a short company assessment, and it needs no data upload.
Frequently asked questions
What is an unsupported claim in an expert rationale?
It is an assertion inside the explanation that no record in the case file supports, such as a cause, a measurement or a rule the expert stated without a document behind it. It may still be correct, but it should be labeled as the expert's judgment with an uncertainty level rather than presented as fact.
Should unsupported rationales be deleted from a dataset?
Not automatically. Remove the unsupported claims that the decision depends on, or exclude the case, but keep labeled judgment claims when the uncertainty is recorded. Cases where a fluent explanation cited the wrong rule are useful as negative examples if they are labeled clearly.
Can a model check expert rationales instead of a person?
A model can help find claims without evidence links or flag dates that fall after the decision, which speeds the review. The judgment about whether a claim is load-bearing, and whether a second path was equally valid, still needs a qualified reviewer who knows the domain.
How many reviewers does each rationale need?
One trained reviewer is enough for routine cases where every load-bearing claim links cleanly to a record. Add a second, independent expert for cases where the causal inference carries the decision, where confidence is low, or where the original expert is senior to the available reviewer.
Does a verified rationale give a company the right to license it?
No. Verification is about quality. Whether the records can be licensed depends on who owns them, what privacy notices and consents cover, and what customer and vendor contracts allow, and counsel reviews those questions on the specific facts.
Sources
- Jacovi and Goldberg separate faithfulness from plausibility in explanations and argue that faithfulness should be judged as a matter of degree. Source
- The ERASER benchmark evaluates rationales by comprehensiveness and sufficiency. Source
- e-SNLI annotators highlighted the words essential to the label and wrote explanations using them, with checks rejecting very short or copied explanations. Source
- The IPCC AR5 uncertainty guidance note defines likely as 66 to 100 percent probability with fuzzy boundaries and prefers stating probability ranges when information allows. Source
- Federal Rule of Evidence 702 requires the proponent to show it is more likely than not that expert testimony rests on sufficient facts or data, reliable methods and their reliable application. Source
- The W3C PROV-O ontology defines derivation as the transformation of one entity into another and attribution of an entity to an agent. Source
See if your company qualifies
A short company assessment. No data uploads are needed.