Systems and records
Insurance AI evaluation dossier: what a workflow pilot dataset needs
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
An insurance workflow pilot dataset dossier should contain seven linked parts: the task scope and exclusions, the policy version and endorsements in force on the loss date, a rights record for every file, the evidence items with their conflicts and gaps, the authority limits that applied, the final disposition with later events, and an evaluation key that checks the real outcome rather than a closed status.
Evaluation split
Insurance AI evaluation
Key takeaways
- Scope the pilot to one decision with one output, and write down which files are excluded and why.
- Grade against the policy form and endorsements in force on the loss date, not the wording on file today.
- Every file needs its own rights record; holding a file is not the same as being allowed to license it.
- Log evidence conflicts and missing documents, and mark which ones changed the decision.
- Label by the outcome observed over a fixed window, because a closed status can hide a reopening or reversal.
What should an insurance workflow pilot dataset dossier contain?
It should contain everything a reviewer needs to replay one insurance decision and grade a model against it: the scope of the task, the policy wording that governed it, the evidence the handler had, the authority they held, what they decided and what actually happened afterward. If any of those is missing, the pilot measures how well a model guesses a status field, not how well it handles the work.
Most pilot datasets arrive as an export of claim or submission rows plus a folder of attachments. That is enough to train a classifier on past labels, but it is not enough to evaluate an assistant that has to read an endorsement, notice a missing form or stop at an authority limit. The dossier is the layer that joins those pieces and states which ones are real, which are illustrative and who checked them.
This page treats the dossier as a practical packet for an operations leader preparing a pilot with an AI developer or an internal data science team. It covers personal lines and commercial claims and underwriting referrals in general terms; the specific rules of a given state, line or carrier have to come from that carrier's own forms and counsel.
Which single task is the pilot scoped to, and what is deliberately left out?
Pick one decision a handler makes repeatedly and write it as a sentence with a verb and an output: decide whether a reported water loss is covered under the policy form and endorsements in force, and route it to payment, denial review or coverage counsel. A scope that says claims triage, or underwriting support, cannot be graded because nobody can say what a correct answer looks like.
Then list the exclusions out loud. Typical exclusions for a first pilot are litigated files, files with a special investigation referral, bodily injury reserves, and any file where the decision depended on a phone call that left no record. Leaving them out is not a weakness. It keeps the evaluation honest about what the model was asked to do, and it tells the developer which slices they cannot claim performance on.
- Task sentence: one decision, one output, one role that normally makes it.
- Line of business, states and policy form families included.
- Date window of losses or submissions, and why that window.
- Exclusions with a reason for each (litigation, fraud referral, injury, unrecorded verbal decisions).
- Minimum file completeness required for a file to enter the set.
Which policy version and endorsements governed the decision on the loss date?
The governing wording is the policy form plus every endorsement effective on the date of loss, not the form attached to the policy record today. Carriers revise forms and add or remove endorsements at renewal and mid-term, so a file graded against current wording can mark a correct historical decision as wrong.
Record each file's form number and edition date, every endorsement with its own edition and effective date, and the declarations in force on the loss date. Where the claim began as an ACORD loss notice, keep the form and edition: the ACORD forms index lists separate property, automobile, liability and workers compensation loss notices, plus state-specific versions such as ACORD 1 CA, a California property loss notice. Those edition labels are how a reviewer later proves which intake form produced which fields.
| Policy element | What to record | Why it changes the decision |
|---|---|---|
| Base policy form | Form number, edition date, issuing company | Coverage grants and exclusions differ by edition |
| Endorsements | Each endorsement number, edition and effective date | A mid-term endorsement can add or remove the coverage at issue |
| Declarations on loss date | Limits, deductibles, scheduled items, named insureds | Limits and named parties decide whether and how much is payable |
| Cancellation or lapse events | Notice dates and reinstatement dates | Coverage may not have existed on the loss date |
| Intake form | Loss notice form and edition, or portal version | Fields captured at first notice differ by form and channel |
What rights evidence must travel with every file in the dossier?
Every file needs a short rights record that says who holds the records, under what agreement, and whether that agreement allows this use. Holding a claim file in your system, being allowed by privacy law to use it, owning it and having a contractual right to license it are four different questions, and a pilot dossier should answer each one separately.
A third-party administrator or an agency often holds files on behalf of a carrier, and its service agreement may limit use to administering those claims. A carrier may hold policyholder and claimant records that are subject to state insurance privacy rules and its own notices. De-identification reduces privacy risk but does not create a right to license records that a contract reserves to someone else. Counsel reviews the specific agreements, notices and state rules before any file leaves the organization.
- Record holder and capacity (carrier, administrator, agency, vendor).
- Governing agreement and the clause that permits or limits secondary use.
- Privacy notices and state rules counsel identified for the files in scope.
- Third-party material in the file (medical bills, contractor estimates, police reports) and its terms.
- De-identification method applied and who performed it.
- Reviewer, date and decision: include, exclude or include with redaction.
The domain scope, rights and evaluation evidence packet
The packet below is the dossier itself. Each row is one element the pilot needs, what it records, the evidence that backs it and the check a reviewer runs before the file is accepted. A file that fails any row is excluded or held, not patched with an assumption.
At a glance
The domain scope, rights and evaluation evidence packet
Give the packet a version number and a date, and freeze it before the developer sees any model output. If a reviewer later corrects a policy version or an expected answer, issue a new packet version and rescore every run against it, so results from different weeks are never compared across two different keys.
Keep the evidence index at the level of documents and notes, with a hash for each item, rather than copying text into a spreadsheet. The developer can then load the exact files the handler saw, and an auditor can confirm that nothing was added after the decision date. Items created after the decision, such as a later inspection, belong in the outcome record, not in the evidence the model receives.
| Packet element | What it records | Evidence reference | Acceptance check |
|---|---|---|---|
| Task and scope card | Task sentence, lines, states, date window, exclusions | Signed scope memo with version and date | Every file maps to the task sentence |
| Policy version record | Form, edition, endorsements and declarations on the loss date | Policy administration history extract | Loss date falls inside each listed effective period |
| Rights record | Holder, agreement, permitted use, privacy review result | Agreement clause and counsel note | Include decision recorded with reviewer and date |
| Evidence index | Every document and note with source, date and author role | File manifest with hashes | Each decision-relevant item appears once with a date |
| Conflict and gap log | Contradictions between items and documents requested but never received | Diary notes and request letters | Each conflict has a resolution or is marked unresolved |
| Authority record | Handler role, payment or reserve authority, referrals made | Authority matrix in force on the decision date | Decision sits inside the stated authority or shows the referral |
| Decision record | What was decided, when, by which role, citing which provision | Coverage letter or decision note | The cited provision exists in the recorded policy version |
| Outcome record | Payments, reopenings, appeals, subrogation, complaints after the decision | Payment ledger and file history | Observation window has closed for the file |
| Evaluation key | Expected output, acceptable alternatives, fail conditions | Reviewer adjudication sheet | Two reviewers agree, or the disagreement is logged |
| Real or illustrative flag | Whether the case is a licensed real file or a constructed example | Provenance note | Illustrative cases are never scored as real outcomes |
Worked task case (illustrative): a water loss reported after a mid-term endorsement
The case below is illustrative and constructed for this page; it is not a real claim. It shows how the dossier turns one file into a gradable task with a starting state, linked evidence, the applicable rule, the decision and an outcome check, followed by a counterexample that a weak dataset would mislabel.
- Counterexample (illustrative): a similar file closed as denied, then reopened and paid after the insured produced a dated burst-pipe repair record. Its status field says denied. The outcome record says paid on reopening. A dataset that uses the status field as the label teaches the wrong answer; the dossier labels it by the outcome and keeps the reopening as linked evidence.
- Grading note: a model that answers denied, quotes the right endorsement and skips the supervisor step fails, because it acted outside the handler's authority even though its coverage reading was plausible.
| Step | Dossier content | Linked evidence | What the grader checks |
|---|---|---|---|
| Starting state | Homeowners claim opened from a property loss notice; loss date 12 days after a mid-term endorsement | Loss notice form and edition; endorsement effective date | The model identifies the endorsement as in force on the loss date |
| Evidence | Plumber invoice says gradual leak; insured statement says sudden burst | Invoice, recorded statement summary, photos with dates | The model flags the conflict instead of picking one source silently |
| Gap | Requested moisture report never received after two requests | Request letters with dates | The model names the missing document as decision-relevant |
| Rule | Endorsement limits water damage from continuous seepage | Endorsement text, edition and effective date | The cited provision matches the recorded version |
| Authority | Handler may deny only with supervisor review | Authority matrix for the decision date | The model routes to supervisor review, not to a final denial |
| Outcome | Supervisor upheld partial coverage after an inspection; no reopening within the observation window | Payment ledger, inspection note, file history | Expected output is referral with a partial-coverage rationale |
Which evidence conflicts and authority boundaries should change the expected answer?
The ones that changed what the handler was allowed or required to do. A conflict between two dated documents about the cause of loss changes the answer; a typo in a street name usually does not. Mark each conflict in the gap log as decision-changing or not, and let only the decision-changing ones affect the evaluation key.
Authority is the boundary pilots most often miss. Handlers work under payment and reserve limits and referral rules, and a correct coverage reading delivered by someone without authority is still a process failure. Time limits matter as well: the NAIC unfair property and casualty claims model regulation, in its posted 1997 text, sets a 15-day period to acknowledge a claim notice and to reply to pertinent claimant communications, and states adopt their own versions. If the pilot task includes correspondence, the dossier should record the applicable clock for each file rather than assume one.
- Cause-of-loss conflicts between documents from different authors.
- Date conflicts that move the loss inside or outside a coverage period.
- Missing documents that the handler requested and did not receive.
- Payment, reserve or denial authority below what the decision required.
- Regulatory or contractual response clocks that ran during the file.
Which linked record proves the eventual outcome rather than the last status?
The payment ledger and the file history after the decision date, read over a fixed observation window, prove the outcome. A closed status only says the file stopped moving. Reopenings, supplemental payments, appeals, complaints and subrogation recoveries are the events that confirm or overturn the decision, and the dossier should list which of them were checked and for how long.
Choose the observation window per line before building the set, write it on the scope card and exclude files whose window has not closed. A short window flatters the pilot because late reversals never appear; a long one shrinks the set to old files written under older forms. Record the trade-off you chose so the developer can read the results in that light. NIST's AI Risk Management Framework asks that test sets, metrics and tool details be documented and that performance be shown under conditions similar to deployment; the window, the slices and the exclusions are part of that documentation. The datasheets for datasets questions on composition, collection and recommended uses are a useful checklist for the narrative that accompanies the packet.
Where does the dossier meet what insurance regulators ask carriers to document?
It supplies the testing and data evidence a carrier's AI governance program needs. The NAIC membership adopted a model bulletin in December 2023 asking insurers to maintain a written program for the responsible use of AI systems that make or support decisions in regulated insurance practices, covering the system life cycle and systems built by third parties as well as the insurer. Colorado's amended Regulation 10-1-1 sets governance and risk management requirements for insurers' use of external consumer data, algorithms and predictive models, including oversight and monitoring.
A pilot dossier is not a compliance filing, and which rules apply depends on the state, the line and how the system is used. What the dossier gives a carrier's compliance team is a dated record of what was tested, on which records, under which rights and with which exclusions. A vendor that hands over that record shortens the carrier's own review; one that cannot is asked for it later.
Preparing claim and underwriting files for a licensed pilot
If the files come from your own operations and you are thinking about licensing them to an AI developer, the dossier doubles as your preparation checklist. Carriers, administrators and agencies can work with SourceX on that licensing, from rights review and contracting to preparation and delivery; the company approves scope and terms and signs before any file is delivered, and a short assessment with no uploads comes first.
Frequently asked questions
What is an insurance AI evaluation dossier?
It is a packet that turns a set of claim or underwriting files into gradable tasks. It records the task scope, the policy version in force, the rights basis for each file, the evidence with its conflicts and gaps, the handler's authority, the decision and the outcome observed afterward. A reviewer can replay any file and check the model's answer against it.
Can a closed claim status be used as the ground-truth label?
Not on its own. A closed status shows that work stopped, not that the decision held. Check payments, reopenings, appeals and recoveries over a fixed observation window and label by that outcome.
Should the pilot use real files or illustrative cases?
Real licensed files give a true measure of performance, but they need a rights record and privacy review for each file. Illustrative cases help explain the task and test edge cases. Keep them flagged and never score them as real outcomes.
Who should review an insurance pilot dossier?
An insurance operations expert should confirm the task, policy versions, authority rules and expected answers. A privacy and rights reviewer, usually with counsel, should confirm that each file may be used for the pilot and that redaction is adequate.
Does de-identifying claim files make them licensable?
No. De-identification lowers privacy risk, but the right to license depends on who holds the records and what their agreements and notices allow. A third-party administrator, for example, may hold files only to administer them for a carrier.
Sources
- The NAIC announced that its members adopted the Model Bulletin on the Use of Artificial Intelligence Systems by Insurers at the 2023 Fall National Meeting. Source
- The NAIC model bulletin text asks insurers to maintain a written AIS Program for AI systems that make or support decisions in regulated insurance practices, covering the life cycle and systems developed by the insurer or a third party. Source
- Colorado's Division of Insurance adopted amended Regulation 10-1-1 on governance and risk management requirements for insurers' use of external consumer data and information sources, algorithms and predictive models. Source
- The NAIC Unfair Property/Casualty Claims Settlement Practices Model Regulation (1997 text) requires acknowledging a claim notice within 15 days unless payment is made, and replying within 15 days to pertinent claimant communications. Source
- The ACORD forms index lists P&C loss notice forms, including ACORD 1 Property Loss Notice, ACORD 2 Automobile Loss Notice, ACORD 3 Liability Notice of Occurrence/Claim, ACORD 4 Workers Compensation First Report, and state-specific versions such as ACORD 1 CA. Source
- NIST AI 100-1 MEASURE 2.1 asks that test sets, metrics and tool details used in testing be documented, and MEASURE 2.3 asks that performance be demonstrated for conditions similar to the deployment setting. Source
- Datasheets for Datasets proposes that every dataset be accompanied by a datasheet answering questions on motivation, composition, collection process, preprocessing, uses, distribution and maintenance. Source
See if your company qualifies
A short company assessment. No data uploads are needed.