AI uses for records
Insurance policy exclusion reasoning data: tasks that span documents
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
To build a task whose answer depends on an exclusion in another document, start from a closed file, freeze the declarations, form and endorsements in force on the loss date, and ask a question the obvious document cannot answer. Record gold evidence spans in at least two documents, the reasoning path, the documented decision and a counterexample twin where the exclusion does not apply.
Document stack
Insurance policy exclusion
Key takeaways
- A policy is a set of documents, so the decisive exclusion often sits in an endorsement or a definition rather than the base form.
- Freeze the exact versions in force on the loss date and include superseded documents as distractors.
- Anchor gold answers to the written coverage position that names the provision, not to a claim status field.
- Allow insufficient evidence as an answer when the file is missing an endorsement or the editions do not match.
- Grade the answer and the cited evidence separately, and use counterexample twins to catch lucky guesses.
How do you construct a task whose answer depends on an exclusion in another document?
Start from a closed, resolved file, freeze the policy exactly as it stood on the date of loss, and write a question that cannot be answered from the document it appears to be about. The gold answer must cite at least two locations, such as the coverage grant and an exclusion added by endorsement, and the task must include a near-identical counterexample where the exclusion does not apply.
That is the core of insurance policy exclusion reasoning data. A model that reads only the base form will often give a confident, plausible and wrong answer, because the exclusion, the definition that narrows it, or the endorsement that added it sits somewhere else in the file. The task exists to test whether a system finds that other document and applies it to the facts.
A useful task pack therefore records four things beyond a question and an answer: the exact documents and versions in force, the evidence spans in each, the order in which they must be combined, and the record that shows how the real file was resolved. Each is described below, with an illustrative case and a grading method.
Why the policy you need is a set of documents
A policy is a set of documents read together, so a task built from the base form alone tests the wrong skill. The coverage question is decided by whichever parts were in force on the loss date, and those parts often arrived at different times.
The National Flood Insurance Program's Dwelling Form, published in federal regulations, shows the structure plainly. It defines the policy as the entire written contract and lists its parts: the printed form, the application and Declarations Page, any endorsements issued, and any renewal certificate. It describes the Declarations Page as a computer-generated summary of the application information, showing the term, limits and premium, and says it is part of the policy.
Endorsements are where a lot of exclusion reasoning lives. The NAIC glossary describes an endorsement as an amendment or rider that adjusts coverage and takes precedence over the general contract, and NAIC consumer guidance notes that some endorsements exclude coverage for certain types of claims. A task builder should assume that the decisive text may be in an endorsement issued mid-term, and that the declarations page is the index that says which endorsements belong to which term.
A public example: the flood policy's earth movement exclusion
Public policy forms are a safe place to design and test the method before any private file is used. The NFIP Dwelling Form is federal regulation text, so its wording can be quoted in an example without a confidentiality question.
Its exclusions section says the policy does not insure loss caused directly by earth movement, even if the earth movement is caused by flood, and lists examples including earthquake, landslide, land subsidence, sinkholes and gradual erosion. The same form then carves back losses from mudflow and from land subsidence as a result of erosion that are specifically insured under its definition of flood. The definition of mudflow, in a third place, says that other earth movements such as landslide or slope failure are not mudflows.
So a question such as whether a flood-triggered slope failure that damaged a covered dwelling is covered needs three hops: the exclusion, the carve-back and the definition. A system that stops at the word flood in the coverage grant answers yes; one that stops at the exclusion answers no for the wrong reason when the facts actually describe mudflow. Federal rules also say the standard policy's provisions may not be altered, varied or waived except by the Federal Insurance Administrator's express written consent in an amendatory endorsement, which is another reason to record the endorsement set in force.
What the policy grounded evidence retrieval task pack contains
The pack is one record per task with ten components. Real-file tasks and illustrative tasks use the same structure, with the provenance field saying which is which.
| Component | What it holds | Where it comes from | Check before release |
|---|---|---|---|
| Task ID and version | Stable identifier, pack version and build date | Assigned by the task builder | IDs never reused after a task is retired |
| Starting state | The question as it was actually asked, with the facts known at that time | First notice of loss, adjuster notes or the coverage inquiry | No facts that surfaced only after the decision |
| Document set | Declarations, base form, each endorsement and any prior-term documents, with effective dates | Policy administration system and the claim file | Form and endorsement editions match the declarations |
| Gold evidence spans | Document ID, version, section and the exact span for each piece of evidence | Marked by an insurance operations reviewer | At least two documents; every span quoted exactly |
| Reasoning path | Ordered hops, such as grant, then exclusion, then carve-back, then definition | Written by the reviewer from the spans | Each hop cites a span; no step relies on outside knowledge |
| Answer and disposition | Covered, not covered, partly covered or insufficient evidence, with a short reason | Final coverage position in the file | Matches the denial or payment record |
| Authority boundary | Who could make the decision and where the task stops | Claim authority levels and referral rules | Tasks requiring coverage counsel end at a referral, not a ruling |
| Distractor documents | Superseded endorsements, prior-term forms and look-alike clauses | Same policy history | Each distractor is plausible and wrong for this date |
| Counterexample twin | Same facts with one change that flips the answer | Built from the original task | Flip reason recorded and checked by a second reviewer |
| Outcome link and provenance | What happened after the decision, and whether the task is real-approved or illustrative | Payment, denial letter, appeal or reopening records | Rights and privacy approval recorded for real files |
Is the denial letter or the status field the proof of a coverage decision?
The record that proves the decision is the written coverage position that names the provision relied on, not the claim status field. A status of closed or denied says what happened; it does not say why, or which document decided it.
In property and casualty lines, the NAIC's unfair claims settlement practices model regulation says an insurer should not deny a claim on the grounds of a specific policy provision, condition or exclusion unless the denial refers to it, and that the denial must be given in writing and documented in the claim file. Where a state follows that model, the denial letter is the natural anchor for the gold evidence, because it names the provision the adjuster applied.
The eventual outcome is a separate check. A denial later reversed on appeal or reopened with new evidence is not a clean gold answer for the original question; either the task uses the corrected position with the evidence that supported it, or it is excluded. Keep three fields apart in the pack: the decision as made, the record that documents it, and the final outcome after any appeal, reopening or settlement.
Which version gaps and evidence conflicts change the expected answer?
Expected answers should change whenever the file itself is ambiguous, and insufficient evidence must be an allowed answer. Otherwise the pack teaches a system to guess past gaps that a careful adjuster would flag.
- Endorsement effective after the loss date: the exclusion does not apply to this loss, even though it appears in the current policy. Use this as a counterexample twin.
- Declarations list a form edition that differs from the form attached in the file: mark the task insufficient evidence or retire it until the right edition is found.
- An endorsement named on the declarations is missing from the file: the correct answer is a request for the missing document, not a coverage call.
- Prior-term and current-term forms both present: include the prior-term form as a distractor and record which term the loss falls in.
- Two clauses point different ways, such as an exclusion and a carve-back: the gold reasoning path must show which clause governs and why, citing the governing text.
- Facts in the file conflict, such as a cause of loss recorded differently by two people: the starting state should include both, and the task should test whether the system notices the conflict before deciding.
Illustrative task: a basement inventory loss and a mid-term endorsement
The following task is illustrative and invented for this article; it is not drawn from any real policy or claim. It shows how the pack components fit together around an exclusion added by endorsement.
Starting state: a small distributor reports water damage to inventory stored in a basement after heavy rain. The question as asked by the adjuster is whether the basement inventory loss is covered under the commercial property policy. The document set has five items: the current declarations, the base property form, an endorsement adding an exclusion for business personal property stored below ground level, the prior-term declarations, and the adjuster's inspection notes.
Gold evidence: the base form's coverage grant for business personal property; the endorsement's exclusion text; the current declarations line listing that endorsement and its effective date, which falls 40 days (illustrative) before the loss; and the inspection note placing the stock in the basement. Reasoning path: grant, then endorsement, then effective date, then location of the property. Answer: not covered for the basement stock, with the endorsement as the reason. Authority boundary: the task ends at a recommended position for the adjuster, not a letter to the insured.
Counterexample twin: identical facts, but the loss date falls before the endorsement's effective date. The prior-term declarations, which do not list the endorsement, now govern, and the answer flips to covered subject to the base form's own terms. Distractor: in the original task, the prior-term declarations tempt a system that matches on the policy number without checking the term.
Grading the answer and the evidence path separately
Score whether the answer is right and whether the cited evidence is right as two separate measures, because a correct answer reached from the wrong document is a lucky guess that will fail on the twin.
Multi-hop question answering research established this pattern. HotpotQA, a 2018 dataset of Wikipedia-based question and answer pairs, added sentence-level supporting facts so that systems could be supervised on, and tested for, the evidence behind an answer; its authors note that comparing predicted supporting facts with the ground truth tests explainability. In the released files, each supporting fact is a pair of paragraph title and sentence index.
For policy tasks, extend that pair to document ID, document version, section and span. Then report answer accuracy, evidence recall (did the system cite every required span), evidence precision (did it cite superseded or irrelevant text), and twin consistency (did the answer flip when, and only when, the decisive fact changed). A system that cites the prior-term declarations should lose evidence precision even if its answer happens to match.
What an insurance operations team clears before files leave
Before any real file becomes a task, the operations team needs approval for three separate things: the right to use and license the file contents, the privacy treatment of the people in it, and permission for any policy form text that a third party owns. Clearing one does not clear the others.
At a glance
What an insurance operations team clears before files leave
- Rights: confirm the agency, carrier or administrator agreements let you use claim and policy records this way, and who owns the file.
- Policy form text: standard forms may be owned by a carrier or a forms provider; check whether their wording can be included or must be referenced by form number only.
- People: remove or replace names, addresses, policy and claim numbers, and any health details in injury claims; de-identification reduces risk but does not create a right to license.
- Sensitive files: exclude claims in litigation, fraud referrals and files under regulatory complaint.
- Outcomes: confirm the final disposition is known and stable before the file is used as a gold answer.
Supplying closed claim files for reasoning tasks through SourceX
Insurance operations teams that hold closed files with full policy histories can explore licensing them as source material for this kind of task. No claim file reaches SourceX until the carrier or administrator has agreed the scope and terms in a signed agreement, and the de-identification of policyholder and claimant details is carried out by SourceX through approved processing routes.
Whether a carrier, agency or administrator qualifies is settled by a short company assessment, before any claim file is shared.
Frequently asked questions
What is insurance policy exclusion reasoning data?
It is a set of tasks or records in which the correct coverage answer depends on applying an exclusion, often located in an endorsement or another part of the policy, to the facts of a loss. Each item links the question to the exact evidence spans and the documented decision.
Why not train on claim decisions alone?
A decision field says what happened but not which text decided it. Without the policy version, the endorsement in force and the cited provision, a model can learn to predict denials from surface patterns instead of reading the policy.
How many documents should each task require?
At least two, and usually three or four: a coverage grant, the exclusion, and the definition, carve-back or effective date that decides whether the exclusion applies. Add distractor documents from other terms so retrieval is actually tested.
Can a task use public policy forms instead of real files?
Yes, for designing and testing the method. Public forms such as the federal flood policy can be quoted freely, but the facts and outcomes must then be labelled illustrative, because no real decision stands behind them.
What does a counterexample twin add?
It changes one fact, such as the loss date or where property was stored, so that the exclusion no longer applies. A system that gives the same answer to both tasks is not reading the decisive evidence.
Sources
- The NFIP Dwelling Form defines the policy as the written contract made up of the form, application and Declarations Page, endorsements and renewal certificate, and its earth movement exclusion carves back mudflow as defined. Source
- Federal rules say the Standard Flood Insurance Policy may not be altered, varied or waived except by the Federal Insurance Administrator's express written consent in an amendatory endorsement. Source
- The NAIC glossary defines an endorsement as an amendment or rider that adjusts coverages and takes precedence over the general contract. Source
- NAIC consumer guidance says some endorsements exclude coverage for certain types of claims and become part of the insurance agreement. Source
- The NAIC property and casualty claims settlement model regulation bars denying a claim on a specific provision, condition or exclusion unless the written denial refers to it. Source
- HotpotQA provides sentence-level supporting facts so systems can be supervised on and tested for the evidence behind multi-hop answers. Source
- The HotpotQA release stores each supporting fact as a pair of paragraph title and sentence index. Source
See if your company qualifies
A short company assessment. No data uploads are needed.