Underwriting AI evaluation: submissions with bound, declined and referred decisions

What one evaluation record should contain

The evaluation unit is the submission, not the policy, because the copilot's job starts at intake and the decision is made per submission. A usable record bundles the inputs an underwriter actually saw, the rules they applied and the outcome they chose. For commercial lines that typically means the ACORD 125 and line-specific supplements, the broker email, loss runs, statement of values or schedule of vehicles, and any inspection or third-party data pulled at the time.

The outcome side matters as much as the inputs. Capture the decision code, the decline or referral reason, the authority level that approved it, and quoted versus bound premium, limits, deductibles and key endorsements. Free-text rationale from the underwriter's notes field is the hardest part to get, because it usually sits in workbench notes, emails and rules-engine logs rather than in structured fields [3].

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "submission_id": "SUB-000412",
  "received_at": "2025-03-18",
  "line_of_business": "commercial_auto",
  "naics": "484121",
  "state": "TX",
  "documents": ["acord_125.pdf", "acord_127.pdf", "loss_runs_5yr.pdf", "vehicle_schedule.xlsx", "broker_email.eml"],
  "appetite_guide_version": "CA-2025-01",
  "decision": "referred",
  "referral_reason": "radius over 500 miles; 2 at-fault losses in 36 months",
  "authority_level": "senior_underwriter",
  "final_outcome": "quoted_not_bound",
  "quoted_terms": {"liability_limit": 1000000, "deductible": 2500, "premium_band": "B"},
  "rationale_text": "Long-haul exposure outside standard appetite; acceptable with driver MVR review and higher deductible.",
  "pii_treatment": "insured name, FEIN, driver names and VINs replaced with tokens",
  "split": "test"
}

Use a written specification to fix these fields, slices and acceptance thresholds before any data arrives; the evaluation dataset specification guide covers the template.

How to score an underwriting copilot against real decisions

Score decisions and rationale separately, because a copilot can match the outcome for the wrong reason. Decision accuracy is a confusion matrix over bound, quoted, declined and referred, reported per line of business and per appetite tier. Referral recall deserves its own metric: a copilot that auto-clears a risk an underwriter would have escalated is a governance failure, not a minor error.

Rationale quality is graded against the underwriter's own notes. Expert-written rationale can serve as rubric input: each note is decomposed into the factors cited (class code, loss frequency, TIV concentration, CAT zone, prior carrier cancellation) and the model is scored on whether it surfaces the same factors and no invented ones [1]. The guide to designing rubrics with domain experts shows how to turn notes into gradable criteria.

Extraction accuracy is a third layer that is easy to measure. Check field-level recall on the ACORD and loss-run data the copilot pulls, since a wrong TIV or missed loss propagates into every downstream judgment.

Selection bias: declined risks have no loss outcomes

Underwriter decisions, not loss outcomes, are the ground truth for most underwriting eval sets, because declined and unbound submissions never generate loss experience. A set labeled only with bound policies and their later claims measures loss prediction on accepted risks and says nothing about whether the copilot would have declined the right ones. Treat this as a design constraint, not a data defect.

Three practical consequences follow. First, report agreement with underwriters as the primary metric and do not describe it as accuracy against truth. Second, if you add loss outcomes for bound risks, keep them in a separate slice with enough seasoning (often several policy years for liability lines). Third, record which underwriter and authority level decided each case, because inter-underwriter disagreement sets the ceiling on achievable agreement. The outcome-labeled evaluation data page discusses when business decisions can stand in for ground truth.

Stratifying by line, appetite tier and decision type

A random sample of submissions over-represents easy declines and under-represents the referrals that matter most. Stratify by line of business, state, NAICS segment, appetite tier (in appetite, conditional, out of appetite) and decision type, then oversample referrals, large-TIV accounts and cases where the appetite guide changed during the period. The stratified eval sets guide gives allocation methods.

Version the appetite guide alongside each record. A copilot evaluated against a 2026 guide on 2024 submissions will look wrong whenever the carrier tightened or loosened appetite, which is a common source of false failures.

Illustrative example: invented to show structure; it does not describe an available dataset.

Gold-label quality and contamination controls

Even expert labels carry errors, and small test sets are sensitive to them. Audits of widely used benchmarks estimated an average label error rate of at least 3.3%, enough to reorder model rankings [2]. For underwriting, budget a second-underwriter review of a sample and adjudicate disagreements before treating the set as gold; the gold-label audit guide describes acceptance steps.

Keep the eval set out of any training pipeline, including fine-tuning on the same carrier's files from overlapping periods. Private, held-out sets reduce contamination, but if the same party supplies both training and evaluation data, check annotator overlap and conflicts of interest [7]. Use a time-based split so test submissions postdate anything the model has seen.

Confidentiality, regulation and governance evidence

Submission files are among the most sensitive records an insurer holds: they contain insured financials, FEINs, driver and employee names, and sometimes health or payroll data. Agree on a de-identification method per field before delivery, and verify it on a sample, since free-text broker emails and loss-run narratives leak identifiers that structured redaction misses.

Your evaluation evidence also supports the carrier's own governance. The NAIC model bulletin on insurers' use of AI systems, adopted December 4, 2023, expects insurers to maintain a written AI program, and NAIC tracks which states have adopted it (map status as of May 1, 2026) [4]. Measurement results map to the MEASURE function of NIST AI RMF 1.0 [5]. As of October 2026, Colorado's SB26-189 replaces SB 24-205 and, from January 1, 2027, requires developers of automated decision-making technology that materially influences consequential decisions to give deployers technical documentation [6]; confirm with counsel whether your copilot is in scope.

How to source licensed submission files with decisions

Submission files with decisions come from carriers, MGAs, program administrators and wholesale brokers, and each holds a different slice of the pipeline. Wholesalers see many declines across markets; carriers hold authority logs and appetite guides; MGAs often have the cleanest structured rationale. Ask any supplier for the decision taxonomy, notes coverage rate, appetite-guide history and the share of submissions with complete loss runs before you discuss volume.

SourceX sources operational datasets, including documents and finance and legal workflows, from US companies on request, and every release is approved by the supplying company. It does not hold stock, so a request does not guarantee a match. Each dataset is rights-reviewed for ownership and consents, personal details such as names and account numbers are removed or replaced before delivery with the method recorded, and delivery runs through private, access-controlled workflows only after an executed agreement. You can describe the submission data you need to SourceX, and for training rather than evaluation use, see licensing underwriting files for AI training and the underwriting decision rationale data page.

For the wider map of private eval sets and benchmarks, start at the evaluation datasets hub.

Request underwriting evaluation data

Describe the lines, decision types, rationale fields and period you need, not the companies you want them from. SourceX looks for US businesses that hold that data, assesses data and licensing permissions, and agrees pricing and allowed uses in a license; nothing is contracted until a supplier agrees. Start an underwriting eval data request.