Deal economics
Outcome labeled dataset value: what 9,116 unlinked claims do to models
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Outcome labels add value to business records only when each label is linked to a verified event, and the gain shows up as task results, not an asserted premium. In public French motor insurance data, 9,116 of 34,060 policies with counted claims had no claim record. A model trained on those counts forecast 37.5% more claims than the records support and ranked risk worse than one trained on linked outcomes.
Grader rubric
Outcome labeled dataset
Key takeaways
- An outcome field is an assertion; a label worth paying for is one linked to a second record that confirms the event.
- In the public freMTPL2 tables first released in 2017, 9,116 of 34,060 policies with counted claims had no claim record; a 2022 reconciliation set 9,094 of those counts to zero.
- Trained on the unreconciled counts, a claim-frequency model over-forecast verified claims by 34.2% to 37.5% across five folds; trained on linked claims, it stayed within about 2%.
- Graded against the unverified labels, the worse model wins in every fold, so acceptance tests must read verified outcomes.
- Price outcome labels on measured lift over the best no-outcome baseline and on calibration, after rights, scope, preparation and buyer purpose are normalized.
Do outcome labels make business records worth more?
They can, but only labels linked to a verified event, and only by an amount measured on the buyer's own task. An outcome field that says a claim happened, a ticket was resolved or a loan was repaid is an assertion; it becomes a label worth paying for when a second record confirms it, such as the claim payment, the absence of a reopen or the final repayment.
So the investor question is narrower than whether outcomes add value. It is how much better a buyer's model performs with verified outcome linkage than without it, on the same records, and whether that difference survives grading against labels the buyer trusts. That is a matched ablation. This page runs one on public insurance data and then turns it into a valuation worksheet for outcome labeled dataset value.
The distinction matters for pricing because outcome fields are cheap to assert and expensive to verify. A seller can point to a status column on every row. Whether that column was ever reconciled with the system that records the event is a separate question, and the public data below shows how wide the gap between the two can be.
Two tables, two answers: claims counted on policies and the records behind them
In the French motor third-party liability data published in the CASdatasets package, the policy table first released in 2017 counted 36,102 claims on 34,060 of its 678,013 policies, while the separate claims table held 26,639 claim records. For 9,116 policies, 26.8% of those with a counted claim, the claims table had no record at all.
The two tables are meant to agree. The package documentation says the frequency table holds each policy's risk features and claim number, and the severity table holds claim amounts with the policy ID used to link them. Linkage failed in the other direction too: 195 claim records sat on 6 policy IDs that do not appear in the policy table.
The maintainer settled it in June 2022 with a commit titled 'update to remove inconsistencies'. The current release has 677,991 policies and 26,444 claims, and every policy's claim count now equals its linked claim records. Matching the releases policy by policy, 9,094 counts were set to zero and one fell from two to one, while 22 policies were dropped and none added. The reconciled release treats the claims table as the record of what happened.
The unlinked counts were not scattered at random, which is what makes them dangerous as labels. Among policies observed for less than a quarter of a year, 52.1% of counted claims had no claim record; among policies observed for three quarters of a year or more, 19.4% did. By area code the share ran from 21.9% to 32.9%. The releases compared:
| Release (commit, date) | Policies | Claims counted on policies | Claim records | Counted claims with no record |
|---|---|---|---|---|
| c49cbbb, 2017-01-19 (first committed) | 678,013 | 36,102 on 34,060 policies | 26,639 | 9,116 policies |
| 4511e99, 2020-11-16 (new data files) | 678,013 | 36,102 on 34,060 policies | 26,639 | 9,116 policies |
| 531aa23, 2022-06-23 (remove inconsistencies) | 677,991 | 26,444 on 24,944 policies | 26,444 | None |
| 227fb56, current head read 2026-10-28 | 677,991 | 26,444 on 24,944 policies | 26,444 | None |
The matched ablation: one model, three kinds of label
On the same 677,991 policies, a claim-frequency model trained on the policy table's counts forecast 37.5% more claims in held-out data than the linked claim records support, while the same model trained on linked claims came within 1%. Ranking moved as well: the linked-outcome model placed 27.3% of verified claims in its riskiest tenth of policies, against 24.3% for the count-trained model.
The design changes nothing but the label. It uses policies present in both the 2020 and the current release, with features from the current release: vehicle power, vehicle age and driver age in bands, the bonus-malus coefficient and population density on a log scale, brand, fuel, area and region, with exposure as an offset. Policies whose ID ends in 0 or 5 form the test set of 135,601; the other 542,390 train one Poisson regression on the recorded counts and an identical one on the linked claims.
A third arm uses no outcome at all. Without linked outcomes, the natural signal left to a buyer is the insurer's own experience rating, the bonus-malus coefficient, so this arm ranks policies by exposure times that coefficient. It can say which policies to look at first but cannot forecast how many claims to expect.
Every arm is graded against the linked claim records, and the pattern held in all five possible test folds: the count-trained model over-forecast by 34.2% to 37.5%, the linked-outcome model by between minus 2.0% and plus 0.9%. Fold 0 in detail:
| Arm (test fold 0, 135,601 policies) | Forecast against 5,234 linked claims | AUC, any linked claim | Linked claims in riskiest 10% |
|---|---|---|---|
| No outcome: exposure times bonus-malus | No forecast possible | 0.664 | 27.0% |
| Trained on policy-table counts | 7,196 (+37.5%) | 0.668 | 24.3% |
| Trained on linked claim records | 5,282 (+0.9%) | 0.689 | 27.3% |
| Reference: exposure only | No forecast possible | 0.630 | 15.2% |
Why grading against the same unverified labels hides the problem
A model trained on unverified counts looks best when it is graded on unverified counts. Scored against the policy table, the count-trained model beat the linked-outcome model in every fold, with an AUC of 0.651 to 0.657 against 0.626 to 0.631. Scored against the linked claim records, the order reversed: 0.668 to 0.673 against 0.689 to 0.698.
That is the acceptance-test trap. If a buyer's acceptance test reads its labels from the same unreconciled column the seller trained on, the test rewards the wrong model and reports a quality the buyer will not see in use. Verified linkage has to reach the test set, not only the training set.
Widely copied tutorials show how easily the definitions split. In scikit-learn's current examples, the Poisson regression example loads the policy table from OpenML and models its claim counts as recorded, while the Tweedie example joins the claims table and sets the count to zero wherever the joined claim amount is zero, which includes every policy without a claim record, explaining that this keeps frequency and severity consistent. That example gives the full dataset as 678,013 samples, the unreconciled release. One library teaches two outcome definitions for the same records.
The no-outcome arm carries a second lesson. Ranking by the insurer's bonus-malus coefficient put 26.6% to 27.4% of linked claims in the riskiest tenth across folds, more than the count-trained model's 24.3% to 25.2% and a little less than the linked-outcome model's 27.3% to 28.9%. At picking the riskiest policies, unverified outcome labels did worse than using no outcomes at all.
How many records carry a final, verified outcome at all?
Fewer than the outcome columns suggest, so count verified outcomes rather than populated fields. In openintro's widely used sample of 10,000 Lending Club loans issued from January to March 2018, 9,375 were still current at the snapshot; only 454, the 447 fully paid and 7 charged off, had a final outcome.
Benchmark documentation sometimes writes the provisional state into the label itself. The modeldata package's Lending Club subset, drawn from the first quarter of 2016, defines a good loan as one fully paid back or currently on-time. A loan that is on time today is not a verified good outcome; it is an outcome nobody knows yet.
Selective observation narrows the count further. Kleinberg, Lakkaraju and coauthors, studying judges' bail decisions, point out that outcome labels can be missing in a nonrandom way because the decisions under study generate the data that exists (paraphrase). A rejected applicant never produces a repayment record. Report outcome coverage by decision branch and by maturity, with the denominators beside each share.
The rights-normalized valuation worksheet for outcome linkage
The worksheet converts a claim that records have outcomes into evidence a buyer can compare across offers. Each row isolates one condition an outcome premium depends on, and the right-hand column fills it from the public motor insurance test so the shape of a completed row is visible.
Rights come before utility. Outcome tables such as claims, payments, complaints and repayment histories are often more sensitive than the decision records they confirm, and a label that cannot be licensed for the buyer's purpose adds nothing however predictive it is. The rows:
| Condition | What to record | Evidence that settles it | freMTPL2 reading |
|---|---|---|---|
| Linkage key | The key joining each outcome to its record, with match rates both ways | Join counts in both directions, orphans listed | IDpol; 9,116 counted claims without a record and 195 records without a policy in 2017 |
| Verification event | Which second record confirms each outcome | A field-level definition signed off by the data owner | A claim record carrying an amount |
| Maturity | Share of records whose outcome is final at the snapshot | Status counts by cohort and length of observation | Unlinked share highest for policies observed under a quarter year |
| Label used for acceptance | Whether the buyer's test reads verified or recorded outcomes | The acceptance test specification | Graded on recorded counts, the wrong model wins |
| Measured task lift | Gain over the best no-outcome baseline, with its range | Matched ablation across folds | Riskiest-tenth capture 27.3% to 28.9% against 26.6% to 27.4% for bonus-malus ranking |
| Calibration | Forecast level against verified outcomes | Predicted over observed totals for each fold | Count-trained +34.2% to +37.5%; linked between -2.0% and +0.9% |
| Rights to outcome tables | Whether outcome records may be licensed for the task | Counsel's review of contracts, notices and sector rules | Not assessed: public research data |
| Reconciliation cost | Work to repair or drop unlinked outcomes | Preparation estimate and the rule applied | Maintainer set 9,094 counts to zero and dropped 22 policies |
Can a strong baseline or an asking price distort what outcome labels add?
Yes, in opposite directions: a strong no-outcome baseline shrinks the measured lift, while asking prices inflate the apparent premium. In the motor data the insurer's bonus-malus coefficient already ranked risk almost as well as the linked-outcome model at the top decile. The labels' clearest contribution was a forecast at the right level, which matters a great deal for pricing risk and much less for picking cases to review.
The value of outcome linkage therefore depends on the buyer's purpose. A buyer that needs a calibrated rate, a reward signal or a verified evaluation set gets most of it; a buyer that only ranks cases may get little beyond what the supplier's own decisions already encode. Demand for checkable outcomes is visible in model training: Ai2's Tülu 3 report introduced reinforcement learning with verifiable rewards, a training stage that rewards a model only when its answer is verified correct.
Asking prices and public payout ranges cannot settle any of this. They are offers and conditional ceilings, not realized prices for a defined bundle of records, rights and labels, and they rarely say whether outcomes were verified. Compare transactions only after rights, usable scope, preparation and buyer purpose are normalized, and attribute to outcome labels only the lift a matched ablation measured.
What could make an outcome-label premium wrong
Six developments could shrink or remove a premium priced on outcome labels. Each is specific enough to check in diligence:
- Verification gets redefined. The motor data's maintainer chose the claims table as the truth; a buyer who trusted the policy-table counts instead would label 9,094 policies differently.
- Labels mature after the sale. Outcomes still open at delivery can flip, so a premium priced on provisional statuses can disappear as they resolve.
- Selective labels hide the hard cases. Outcomes exist only on the branches a past decision allowed, so a model can look accurate where outcomes were observed and fail where none were recorded.
- Rights to outcome tables are narrower than rights to decisions. Claims, complaints and repayment histories often carry personal data and sector rules, and a buyer may receive the decisions without their confirmation.
- One public dataset is a demonstration, not a market. The ablation covers one insurer's records, one task and one model family; the method transfers, the 37.5% figure does not.
- Cheaper verification substitutes. Where answers can be checked by a program, as with verifiable rewards in Tülu 3, buyers may need fewer business-record outcomes for those tasks.
Questions for a company selling outcome-linked records, SourceX included
Ask to see linkage rates, maturity and an ablation on a task you name, graded on verified outcomes. A seller that reports only the share of rows with an outcome field has measured coverage, not value.
SourceX acquires licenses to selected business records from the companies that hold them and supplies the resulting datasets to AI developers, handling rights review, contracting, preparation, delivery and payment itself. Its deal figures stay private, and company and commercial metrics go to investors on request. Only accredited investors can take part in its Rule 506(c) raise; materials can be requested at sourcex.si/investors. Questions to put to it or any peer:
- For each outcome you sell, which second record verifies it, and what is the match rate in both directions?
- What share of delivered records had a final outcome at the snapshot, by cohort and length of observation?
- On which buyer task did you measure lift over the best no-outcome baseline, and what was the range across folds?
- Is the buyer's acceptance test graded on verified outcomes or on the recorded status?
- Which outcome tables were excluded for rights reasons, and how did that change the measured lift?
Methods note: the freMTPL2 linkage count and ablation
Data: freMTPL2freq and freMTPL2sev from the CASdatasets repository on GitHub, read at commits c49cbbb (2017-01-19), 9aa07b9 (2017-01-21), 4511e99 (2020-11-16), 531aa23 (2022-06-23) and the current head 227fb56, with a reader written for R data files; no package code was run. The three releases before 2022 give identical counts. Region values differ for every policy between the January 2017 and 2020 files and for 12,990 matched policies between 2020 and 2022; no column changed after 2022, and the ablation's features come from the head release.
Linkage rules: each policy's claim count is compared with the number of claim records carrying the same IDpol, with IDs compared as integer strings. A counted claim with no record means a count above zero and no matching record. The ablation uses the 677,991 policies present in both 4511e99 and the head release; the recorded label is the 4511e99 count and the linked label the number of head claim records, which the script checks equals the head count for every policy. Exposure is capped at one year and enters as a log offset.
Model and metrics: Poisson regression with a log link and an L2 penalty of 1 times the sum of squared weights, intercept unpenalized, fitted by Newton's method for up to 30 iterations or until the largest step falls below 1e-9. Vehicle power is capped at 9 and treated as categories; vehicle age bands are 0, 1-2, 3-5, 6-10, 11-15 and 16+; driver age bands are 18-20, 21-25, 26-30, 31-40, 41-50, 51-60, 61-70 and 71+; log bonus-malus and log density are standardized on the training fold; brand, fuel, area and region are categories, with the first level in sorted order as the reference. Test fold k holds policies whose integer IDpol modulo 5 equals k, headline figures use k = 0, and ranges cover k = 0 to 4. AUC counts ties as half; the riskiest 10% is the top tenth of test policies by expected claims, rounded, with ties broken by lower IDpol. Nothing is random.
The unlinked shares by exposure band and by area are computed on the 677,991 matched policies, 34,038 of which carry a counted claim, with capped exposure bands of under 0.25, 0.25 to under 0.5, 0.5 to under 0.75, and 0.75 years or more. The Lending Club counts read csv/openintro/loans_full_schema.csv in Rdatasets at commit 1dcc2bf, where a final outcome is the exact status Fully Paid or Charged Off. Not measured: any price, why the 9,116 counts lack records, whether the dropped claims occurred, other tasks or model families. Run on 2026-10-28; no SourceX data is used.
Frequently asked questions
Do outcome labels increase the value of business records?
They can, when each outcome is linked to a verified event and the buyer's task uses it. In public motor insurance data, a model trained on linked claim records forecast within 1% of the verified total and ranked risk best, while one trained on unverified counts over-forecast claims by 37.5%. The value is a measured task gain, not a fixed premium.
What is verified outcome linkage?
It is a join from each record to a second record that confirms what happened, such as a claim payment, a closed repayment or a reopen check, made on a stable key with match rates reported in both directions. An outcome column that was never reconciled with that second record is a recorded status, not a verified label.
How can a buyer test outcome labels before paying for them?
Run a matched ablation on its own task: the same records, features and model, trained once on recorded outcomes, once on verified outcomes and once with the best no-outcome baseline. Grade every arm on verified outcomes over several folds, and report calibration as well as ranking.
Why can an acceptance test approve a model trained on bad labels?
Because a model trained on unverified labels scores best against those same labels. In the motor data, the count-trained model won when graded on the policy table and lost when graded on linked claim records, in every fold. The acceptance test has to read verified outcomes.
Does SourceX price outcome-labeled records higher?
SourceX does not publish its commercial terms or how it values a dataset. Its how-it-works page says pricing stays indicative until a Dataset Deal Confirmation fixes the exact dataset, payout and rights. Investors can request company and commercial metrics through sourcex.si/investors.
Sources
- The current CASdatasets release of freMTPL2freq and freMTPL2sev: 677,991 policies and 26,444 claim records, with every policy's claim count equal to its linked claim records. Source
- The 2020 CASdatasets files, identical in counts to the 2017 release: 678,013 policies whose counts sum to 36,102 claims on 34,060 policies, against 26,639 claim records; 9,116 counted policies without a record and 195 records on 6 unlisted policy IDs. Source
- The CASdatasets commit of 2022-06-23 titled 'update to remove inconsistencies' replaced the freMTPL2 frequency and severity files. Source
- CASdatasets documentation: freMTPL2freq holds risk features and the claim number per policy, freMTPL2sev holds claim amounts with the policy ID used to link the two. Source
- scikit-learn's Tweedie regression example loads freMTPL2 from OpenML (full dataset 678,013 samples), joins the claims table and sets ClaimNb to zero where the linked claim amount is zero, to keep frequency and severity consistent. Source
- scikit-learn's Poisson regression example loads freMTPL2freq from OpenML and models ClaimNb as recorded. Source
- The openintro loans_full_schema sample (Rdatasets copy): 10,000 Lending Club loans issued January to March 2018; 9,375 current, 447 fully paid and 7 charged off at the snapshot. Source
- The modeldata lending_club documentation (Rdatasets copy): loans from the first quarter of 2016, with the outcome good meaning fully paid back or currently on-time. Source
- The Tülu 3 report introduced reinforcement learning with verifiable rewards (RLVR), which rewards completions only when they are verified correct. Source
- Kleinberg, Lakkaraju and coauthors note that outcome labels can be missing in a nonrandom way because the decisions studied generate the available data. Source
- SourceX's how-it-works page says pricing stays indicative until a Dataset Deal Confirmation fixes the exact dataset, payout and rights, and that SourceX does not publish its commercial terms. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.