AI uses for records
Insurance coverage uncertainty labels: when the evidence is insufficient
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Build the task from closed files where a decision was actually held. Freeze the policy version and documents that existed at that moment, then label the gold answer as missing item, version unknown, conflicting, outside authority or decidable. Each held label names the single deciding item, who can supply it and any clock running, and every held case is paired with a decidable twin so hesitation is never rewarded by default.
Agent run
Insurance coverage
Key takeaways
- Insufficient evidence is a training target only when the file truly lacked a deciding item at a reconstructed moment, not when nobody had read it.
- Five labels (decidable, missing item, version unknown, conflicting, outside authority) map to five different next actions.
- Every held label must name one deciding item, the party who can supply it and any deadline running on it.
- Pair held cases with decidable twins and score hold precision, hold recall and confident-wrong answers separately.
- A hold is wrong when no plausible content of the missing item could change the outcome.
How do you build an insurance task whose right answer is that the evidence is insufficient?
Start from closed files where a handler or underwriter actually held the decision, freeze the documents that existed at that moment, and make the gold answer a label that says why the question could not yet be decided, what single item would decide it and who can supply it. Pair every held case with a decidable twin from the same line of business so the agent cannot learn that hesitation is always rewarded.
Most insurance training sets are built from outcomes: paid, denied, bound, declined. That teaches a model to always produce a verdict, which is the opposite of what a careful claims or underwriting desk does on a thin file. A file that was decided a week later, after a proof of loss or an endorsement copy arrived, contains a moment when the honest answer was 'not yet', and that moment is the training example.
Reading comprehension research faced the same problem. The SQuAD 2.0 benchmark added more than 50,000 unanswerable questions, written adversarially by crowdworkers to look similar to answerable ones, so that systems had to decide when no answer is supported and abstain. Insurance coverage tasks need the same property: the held cases have to look as decidable as the decided ones on the surface, or the agent learns a shortcut instead of a judgment.
Five labels for the state of a coverage question
A coverage or eligibility question is in one of five states, and each needs a different next move. Collapsing them into one 'uncertain' flag throws away the part an operations team cares about: whether to request a document, reconcile two sources, escalate to someone with authority or simply decide.
The labels below are an original scheme for training and evaluation data, not an industry standard. Rename them to match your own claim or underwriting diary codes if those already carry the same meaning.
| Label | Meaning | Typical trigger in the file | What the label must name |
|---|---|---|---|
| DECIDABLE | The documents in force and the facts on file settle the question | Declarations, forms and endorsements are present and consistent with the loss facts | The provision and evidence spans that decide it |
| MISSING_ITEM | A specific document or fact required by the policy or a rule is absent | No proof of loss, no signed application page, no repair estimate | The missing item, the party who can supply it and any clock running on it |
| VERSION_UNKNOWN | It is unclear which policy edition or endorsement applied on the relevant date | Mid-term change with no effective date on file, renewal issued after the loss | The candidate versions and the date that separates them |
| CONFLICTING | Two sources on file disagree about a fact the decision turns on | Notice of loss and adjuster report give different causes or dates | Both sources, the disputed fact and how the conflict was later settled |
| OUTSIDE_AUTHORITY | The file may be decidable, but not by this role | Amount above the handler's authority, coverage counsel or reservation of rights needed | The authority limit crossed and the role that must decide |
Which policy version, evidence conflict or authority limit changes the answer?
Three things most often turn a decidable-looking question into a held one: the policy version in force on the loss or bind date, a conflict between two documents about the fact that matters, and an authority boundary that means the person or system looking at the file is not the one allowed to decide it.
Policy version is the quiet one. A homeowners or commercial package is a set of documents, and an endorsement issued mid-term can add, remove or narrow coverage from its own effective date. A task that hands the agent today's policy, rather than the declarations and endorsements in force on the date of loss, has the wrong gold answer before anyone labels it. Every case should carry the edition identifiers and effective dates of each form, and the labeler should record VERSION_UNKNOWN whenever those dates cannot be established from the file.
Conflicts deserve their own label because the right action is reconciliation, not a document request. When a first notice of loss says a pipe burst and a field adjuster's note says surface water entered from outside, the coverage outcome may differ completely, and asking the policyholder for another form will not resolve it.
Holding a decision is not a free action either. The NAIC Unfair Claims Settlement Practices Act model lists, among unfair practices, refusing to pay claims without conducting a reasonable investigation and failing to affirm or deny coverage within a reasonable time after the investigation is complete. The companion NAIC model regulation for property and casualty claims expects insurers to affirm or deny liability within a reasonable time. States adopt and vary these rules, so the label guide should record a deadline clock where one applies, and the expected output for a held case is always a next investigative step, never a bare refusal to answer.
The uncertainty and escalation label guide, field by field
This is the asset: the record each held or decided case carries, in the order a reviewer fills it. It works for claims files and for underwriting submissions; fields that only apply to one are marked.
At a glance
The uncertainty and escalation label guide, field by field
- Keep the label rationale as one or two plain sentences that cite the deciding item; it becomes the target text if you train an agent to explain its hold.
- Store the decided twin's case_id on each held case so splits keep the pair together and evaluation can compare them.
| Field | What it holds | Where it usually comes from | Labeling rule |
|---|---|---|---|
| case_id and line | Pseudonymous claim or submission key, line of business, coverage part | Claims or policy administration system | Never the real claim number; keep the mapping inside the company |
| question | The exact coverage or eligibility question posed to the agent | Written by the labeler from the diary entry | One question per case; split compound questions |
| as_of | The timestamp of the decision moment being reconstructed | Diary, note or task timestamps | Only documents received before this time go into the task |
| policy_version | Declarations edition, form numbers and endorsement effective dates in force | Policy documents and endorsement history | If any date is missing, the label cannot be DECIDABLE |
| documents_in_task | List of documents the agent sees, with received dates | Document management index | Remove anything received after as_of |
| label | One of the five states | Two independent labelers, then adjudication | Record both original labels, not just the final one |
| deciding_item | The missing item, the conflicting pair or the authority limit | Labeler, with evidence pointers | Must point to a document, field or rule, not to a feeling |
| supplier_of_item | Policyholder, broker, adjuster, mortgagee, prior carrier, internal role | Party records, authority tables | Name the role, never the person |
| clock | Any policy or regulatory deadline running on the held item | Policy conditions, state rule references | Write 'none found' rather than leaving it blank |
| resolution | What arrived or who decided, the final disposition and its date | Later diary entries and closing status | Lets you check whether the held label was right in hindsight |
Telling a truly thin file from one nobody had read yet
A file is genuinely insufficient only if the deciding item was absent at the decision moment, not merely unread. The single most common labeling error is treating a file as thin because the handler had not opened an attachment that was already in the document store.
Reconstruct the as-of view from document received dates, not from the order of diary notes, and compare it with what the handler asked for next. If the handler requested an item that was already on file, the case is a process miss and belongs in a different dataset; labeling it MISSING_ITEM teaches the agent to ask for documents it already has.
Use two labelers who work from the reconstructed view without seeing the eventual outcome, then adjudicate disagreements with a senior handler. Keep a short disagreement log by label pair; repeated confusion between CONFLICTING and MISSING_ITEM usually means the guide needs a sharper definition of which facts the decision turns on for that line of business.
Finally, check every held case against its resolution. If the item that arrived later did not change anything, the hold may still have been prudent, but it was not about a deciding item, and the label should be reconsidered.
Worked example (illustrative): a flood claim with no proof of loss and a disputed water line
The National Flood Insurance Program's dwelling form is a useful public example because its conditions are published. It requires the policyholder, within 60 days after the loss, to send a proof of loss: a statement of the amount claimed, signed and sworn to by the policyholder, with details such as the date and time of loss and how the loss happened. The form also says an adjuster may help complete it as a courtesy, but the policyholder must still send it within 60 days. FEMA's January 2025 guidance to policyholders repeats the 60-day requirement, so the condition is still current; the policy also lets the insurer accept a signed adjuster's report in place of a proof of loss, which a label guide should record when it happens.
Illustrative case, invented for this guide: a dwelling claim reaches day 40 with photographs, an adjuster's preliminary report and a contractor estimate, but no signed proof of loss. The adjuster's report says water entered at ground level after a nearby creek overflowed; the policyholder's notice says a supply line failed first. Two questions are posed to the agent on day 40.
- The resolution field for this illustration would record when the proof of loss arrived and how the cause conflict was settled, which is what lets you grade the day-40 labels later.
- No figures here describe a real claim; the 60-day period is the published condition of the form, and everything else is illustrative.
| Question at day 40 | Label | Deciding item | Expected agent output |
|---|---|---|---|
| Can the building claim be paid in the claimed amount now? | MISSING_ITEM | Signed, sworn proof of loss; policyholder supplies it; 60-day condition running | Hold payment, request the proof of loss with the remaining days stated, offer the form |
| Is the damage to the finished basement walls a flood loss under this form? | CONFLICTING | Notice of loss versus adjuster report on cause and sequence | Do not decide; ask the adjuster to reconcile cause with dated evidence |
| Is the property address on the declarations the insured location? | DECIDABLE | Declarations page and loss location in the adjuster report agree | Confirm, citing both documents |
| Can the handler agree the contractor estimate as final? | OUTSIDE_AUTHORITY | Estimate amount above the handler's settlement authority in this illustration | Route to the role with authority, with the file summary |
When insufficient is the wrong label
A hold is wrong when the missing or conflicting item could not change the outcome. If a loss is plainly outside the insuring agreement whatever the proof of loss says, the correct label is DECIDABLE with the deciding provision cited, and an agent that requests the form anyway is wasting the policyholder's time.
Apply a materiality test while labeling: for each candidate deciding item, ask whether any plausible content of that item would flip the answer. If no plausible content would, the case is decidable. This is the meaningful exception to the whole method, and it is why every MISSING_ITEM label must name a single item rather than 'more documentation'.
The other failure is over-holding by habit. Some desks escalate every file above a size threshold whether or not anything is uncertain. Those escalations are OUTSIDE_AUTHORITY cases, not evidence gaps, and mixing them with true evidence holds teaches the agent that large amounts are inherently uncertain.
Scoring an agent that is allowed to hold a decision
Score held and decided cases separately, then together, because a model that holds everything looks safe and a model that decides everything looks fast. Selective classification research frames the same trade-off as risk against coverage: Geifman and El-Yaniv describe building a classifier that rejects uncertain inputs so that a desired risk level holds on the cases it does decide.
At a glance
Scoring an agent that is allowed to hold a decision
For insurance tasks, report at least these measures on a held-out set split by line of business and by as_of period.
- Hold precision: of the cases the agent held, the share whose gold label is not DECIDABLE.
- Hold recall: of the gold held cases, the share the agent held.
- Deciding-item match: for correctly held cases, whether the agent named the same missing item, conflicting pair or authority limit as the gold record.
- Confident-wrong rate: decided answers that contradict the gold label, reported apart from ordinary errors because these are the costly ones.
- Coverage at a fixed confident-wrong rate: how many cases the agent decides before its confident-wrong rate exceeds the threshold the business sets.
- Next-step quality: whether the requested item, recipient and stated deadline match the resolution field.
Points where an insurance operations team should stop and bring in specialists
Stop and bring in a specialist when a label turns on interpreting policy language that is itself disputed, when a file is in litigation or appraisal, when a reservation of rights or coverage opinion exists, or when the deadline clock is set by a state rule you have not confirmed. Those cases can stay in the dataset only after coverage counsel or a senior claims professional has written the gold label.
Governance applies to the model side as well. The NAIC model bulletin on insurers' use of AI systems, adopted by NAIC members in December 2023, asks insurers to maintain a written program for AI systems that make or support decisions, covering areas that include claim administration, and it reaches systems acquired from third parties. A team building held-decision data for a carrier should expect its labeling guide, adjudication log and validation results to be reviewed under that kind of program where a state has adopted the bulletin.
Data rights are a separate question from the labeling method. Claim and policy files can belong to a carrier, sit with a third-party administrator or agency under contract, and include personal information about policyholders and claimants. Who may use the files, under which contracts and with what privacy treatment has to be settled before any of them become training or evaluation data, and de-identification alone does not create the right to license them.
Supplying closed files with held decisions through SourceX
Files where a decision was held and later resolved are scarcer than ordinary closed claims, because they carry the request trail and the resolution, and that is what makes them useful for this kind of task. Selected files like these can become licensed datasets for AI developers through SourceX, which runs rights review, contracting, preparation and delivery.
Whether a carrier, agency or administrator qualifies is checked through a brief assessment with no file uploads. No claim file leaves the company before it has approved the scope and terms and signed, and buyers typically look for at least two years of consistent history and tens of thousands of records in one system, though rarer work can qualify with less and each deal is assessed separately.
What makes an evidence request a correct decision rather than a delay?
An evidence request is the right call when a named item could still change the outcome, the policy or a claims rule entitles the insurer to it, and everything foreseeably needed goes out in one written request to the party who holds it. An insurance evidence request dataset therefore needs a second label beside the state labels above: one that grades the request itself as material, grounded, complete and correctly addressed.
Claims rules already treat a request as an administrative act with limits on its content. The California Department of Insurance's 2026 guide to adjusting property claims after a major disaster restates the state's fair claims rules: when an insurer needs more time, its written notice must specify the additional information the adjuster requires, and the insurer may not persist in seeking information that is not reasonably required for resolving a claim dispute. Texas's consumer bill of rights for homeowners and renters, in paraphrase, says the company should ask for any other information it needs when it acknowledges the claim and starts investigating, may ask only for information reasonably needed, and generally may not demand federal income tax returns without a court order unless the claim involves a fire loss, lost profits or a similar exception. Its written decision deadline runs from when it has all the needed information, so a first request that leaves out something knowable on day one costs the claimant time.
The policy supplies the other half of the grounding test, and its wording is often loose. Lemonade's open-source sample renters policy (Policy 2.0 on GitHub, a 2018 template written for a fictional New York renter, not a form on sale) tells the policyholder to "Provide us with the information and records we ask for as soon as possible", gives proof of ownership for high value items as an example, and lets the insurer ask for an examination under oath. The records clause sets no day count and does not limit which records may be requested, so the contract alone cannot separate a reasonable request from an overbroad one. Label each request against the clause and the state rule together, and treat an examination under oath as a decision for counsel, never for an agent.
What can a standard claim record show about a request, and what does it leave out?
A standard claim record shows where the evidence trail ended, not what was asked. Brazil's Open Insurance system, in which consumers of insurance, open pension and capitalization products can allow their information to be shared among companies authorized or accredited by Susep, publishes its API specifications on GitHub. We read the 13 current insurance product specifications at commit c819587 (6 October 2026) and walked every field of every claim object: 11 specifications define one, 9 of those record the date the complete documentation was delivered and list no documentation and incomplete documentation as 2 of 7 reasons for closing without indemnity, and none has a pending-documents status or any field for the request itself.
That is the gap a licensor's export inherits. From status, closing reason and dates you can label how a request trail ended and measure the time from notice of loss to complete documentation. You cannot recover which items were asked for, from whom, in how many rounds, whether any request was unnecessary or whether a status notice went out. Those facts live in diary notes, outbound letters and the document index, which is why the request-trail files described in the supplier section above are the scarce ones. Treat a documentation denial as an outcome those records must explain, never as proof that the request behind it was right.
Method: a claim object is the item schema of each claims response in the specifications' current folder, followed through every nested reference; field names and descriptions were matched without regard to case, descriptions also after removing accents. The count shows what the standard can carry, not what any insurer's data contains. The table groups the 11 specifications that define a claim object by product family.
| Specifications (count) | Date complete documentation arrived | Documentation reasons for closing without indemnity | Pending-documents status | Field for the request itself |
|---|---|---|---|---|
| Housing, patrimonial, auto, rural, transport, responsibility, financial risk, acceptance and branches abroad (8) | Yes: documentationDeliveryDate, filled only if everything was delivered | SEM_DOCUMENTACAO and DOCUMENTACAO_INCOMPLETA, 2 of 7 reasons | None among 6 status values | None |
| Person (1) | Yes, the same field | The same 2 of 7 reasons | None among 9 status values | None |
| Life pension and pension plan (2) | No | No reason field; a refused benefit appears only as a closing status | None among 9 status values | None |
| All 11 specifications | 9 | 9 | 0 | 0 |
How should a casebook grade the next move on a held claim?
Grade the move with four action labels and pair every case with a decidable twin built from the same file. The labels sit on top of the state labels above: DECIDABLE leads to DECIDE_NOW, MISSING_ITEM to REQUEST_NAMED or PAY_PART_AND_REQUEST, and OUTSIDE_AUTHORITY to ROUTE_SPECIALIST, while CONFLICTING and VERSION_UNKNOWN cases keep the reconciliation steps described earlier. PAY_PART_AND_REQUEST applies only where the policy and the state's rules allow paying the undisputed part while the rest waits.
Each case records the jurisdiction, the policy form and edition, and the as-of moment, because the same file can need a different move in another state. The six Texas renters cases below are invented to show the format, with rows 1 and 2 as a twin pair, and they are a design awaiting review: a licensed adjuster and coverage counsel in the case's state should confirm or rewrite every expected label before any case trains or tests an agent.
| Case (illustrative, Texas renters policy) | File at the as-of moment | Expected label | Wrong move it must beat | Record that grades it later |
|---|---|---|---|---|
| 1. Theft claim, decidable control | Police report, receipt for the stolen camera, loss inside the policy period | DECIDE_NOW | Asking for receipts for every item in the home before paying | Payment date, with no supplement or complaint afterward |
| 2. The same theft claim without the camera receipt | Police report only; the records clause names proof of ownership of high value items as an example | REQUEST_NAMED: proof of ownership for the camera, from the policyholder, with any other foreseeable item in the same letter | A second letter weeks later for an item knowable on day one | Date the last requested item arrived, then the decision date |
| 3. Pipe leak with documented furniture damage and an unsupported laptop value | Photos and an adjuster estimate for the furniture; no purchase record for the laptop | PAY_PART_AND_REQUEST, where the policy and state rules allow it | Holding the whole claim until the laptop record arrives | Partial payment, then a supplemental payment or a closing note |
| 4. Theft claim with no fire loss and no income claimed | The adjuster wants federal income tax returns to test the policyholder's finances | REQUEST_NAMED for what is reasonably needed, such as proof of ownership | Demanding tax returns, which the Texas guidance limits to narrow cases or a court order | Text of the request letter and any complaint that followed |
| 5. Contents claim with receipts dated after the loss | Purchase records that postdate the loss; misrepresentation suspected | ROUTE_SPECIALIST to the special investigations unit or counsel, who decide on any examination under oath | The agent scheduling an examination under oath itself | Referral record and counsel's decision, kept under restricted access |
| 6. Liability claim for a guest's injury in the rented unit | Guest's medical bills claimed; no signed authorization on file | ROUTE_SPECIALIST for a privacy-reviewed authorization covering only the injury | Requesting the guest's full medical history | Authorization on file, scope of records received, settlement record |
Sources for this section
- The California Department of Insurance's 2026 adjusting guide says that when an insurer needs more time, its written notice must specify the additional information the adjuster requires, and that the insurer may not persist in seeking information not reasonably required for resolving a claim dispute. Source
- Texas's consumer bill of rights for homeowners and renters (paraphrase) says the company asks for any other information it needs when it acknowledges and starts investigating a claim, may ask only for information reasonably needed, generally may not demand federal income tax returns without a court order outside fire-loss or lost-profit claims, and decides in writing once it has all needed information. Source
- Lemonade's open-source Policy 2.0 sample renters policy (2018, written for a fictional New York renter) asks the policyholder to provide the information and records the insurer asks for as soon as possible, gives proof of ownership for high value items as an example, and reserves an examination under oath, without a day count. Source
- Open Insurance Brasil's published specifications at commit c819587 (6 October 2026): of 13 insurance product data-sharing specifications, 11 define a claim object, 9 record the date complete documentation was delivered and list no documentation and incomplete documentation as 2 of 7 reasons for closing without indemnity, and none has a pending-documents status or a field for the request itself. Source
- The Open Insurance housing claim object describes documentationDeliveryDate as the delivery date of the complete documentation, if all of it was delivered, and applies its denial reasons when a claim is closed without indemnity. Source
Frequently asked questions
What is an insufficient evidence label in insurance AI training data?
It is a gold answer that says a coverage or eligibility question could not be decided at a given moment, plus the specific item that would decide it. A useful label also names who can supply the item and any deadline running. A bare 'uncertain' flag without those parts teaches the agent to hesitate without telling anyone what to do next.
Should an AI agent ever refuse to answer a coverage question?
It should hold the decision, not refuse the task. The expected output for a held case is a next step: request a named document, reconcile two sources or route to the role with authority. Claims rules in many states expect a reasonable investigation and a timely decision, so holding without a next step is not an acceptable outcome.
How many held cases should a training set include?
There is no fixed ratio; match the share of held decisions to what the business actually sees in each line, and keep a decidable twin for every held case. Report results separately for held and decided cases so a model that holds everything cannot look good on average.
Can underwriting submissions use the same label guide as claims?
Yes. The five states apply to eligibility and appetite questions as well: a missing loss run is MISSING_ITEM, an application that contradicts an inspection is CONFLICTING and a risk outside a binding authority is OUTSIDE_AUTHORITY. The as_of field becomes the moment of the referral or quote decision.
Why record both labelers' original labels instead of only the adjudicated one?
Disagreement patterns show where the guide is ambiguous for a line of business, and they give you a natural estimate of how hard each case is. Keeping only the final label hides that signal and makes it impossible to audit how gold answers were reached.
Sources
- The SQuAD 2.0 site says the dataset combines SQuAD 1.1 questions with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones, so systems must abstain when no answer is supported. Source
- The NAIC Unfair Claims Settlement Practices Act model lists refusing to pay claims without a reasonable investigation and failing to affirm or deny coverage within a reasonable time after investigation among unfair claims practices. Source
- The NAIC property and casualty unfair claims settlement model regulation says insurers shall affirm or deny liability on claims within a reasonable time. Source
- FEMA's NFIP Dwelling Form requires a signed and sworn proof of loss within 60 days after the loss and says adjuster help with the form is a courtesy only. It also says the insurer may, at its option, accept a signed adjuster's report of the loss instead of the proof of loss. Source
- Geifman and El-Yaniv's selective classification paper builds a classifier from a trained network that rejects uncertain inputs so a user-specified risk level is met. Source
- NAIC members adopted the Model Bulletin on the Use of Artificial Intelligence Systems by Insurers in December 2023; it asks for a written AIS program covering decisions including claim administration and third-party AI systems. Source
- FEMA's January 2025 release tells NFIP policyholders to submit the proof of loss form within 60 days of the loss, even if they disagree with the adjuster's estimate. Source
See if your company qualifies
A short company assessment. No data uploads are needed.