Systems and records
Overturned outcome dataset labels: which verdict should a model learn?
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Treat an overturned outcome as two labeled decisions, not one corrected field. Keep each verdict with its date, authority and evidence, classify why it was reversed (an error on the same facts, new evidence, a discretionary concession or a procedural ruling), and mark whether the latest verdict is still provisional. Medicare Advantage insurers overturned 75% of appealed denials in 2014-16, yet only 1% of denials were appealed.
Grader rubric
Overturned outcome dataset
Key takeaways
- An overturned outcome holds two decisions; overwriting the first with the second erases what a first-line model needs to learn.
- The reversal cause sets the label: a same-evidence error makes the first verdict wrong, while new evidence leaves it right for what was known.
- Appeal overturn rates describe a self-selected few; in Medicare Advantage only 1% of denials were appealed in 2014-16.
- A freeze date and per-type review windows decide which outcomes count as final, and the rest are labeled contested or provisional.
- Join only what the reversal cause requires; more records add noise, linkage errors and privacy exposure faster than they add clarity.
What should a dataset do when an official outcome is later overturned?
Keep both outcomes, record the revision between them, and choose the training label by task. An overturned outcome is a decision of record, such as a claim denial, a fraud flag, a credit decline or an audit finding, that a later authority reversed. Overwriting the first verdict with the second destroys exactly the information a model needs to learn either decision well.
Which verdict counts as correct depends on what the model is for. An assistant for first-line reviewers should learn what the final authority would have decided, so a reversed verdict becomes an error example. A model that forecasts workload or appeal risk needs the first verdict as it stood, plus the chance it gets reversed. An evaluation of policy compliance needs to know whether the reversal had anything to do with the policy. One record can carry three different labels.
This guide sets out a decision process an AI team can reproduce for overturned outcome dataset labels: a status vocabulary for outcomes, a four-cause classification of reversals with a label treatment for each, a selection check for outcomes nobody contested, an annotated packet for one reversed case and a context completeness map that limits what gets joined.
Provisional, final, amended: an outcome has a lifecycle
Every official outcome has a status, and the status decides whether it can serve as a label yet. Some standards already encode this. The observation status code system in HL7 FHIR release 4 separates preliminary results, which may be incomplete or unverified, from final ones. It then distinguishes amended results, modified after being final through updates or new information, from corrected results, modified to fix an error, and from entered-in-error records withdrawn after release.
Financial regulation writes the provisional state into the clock. Under Regulation E, a financial institution must normally decide within 10 business days whether a reported electronic transfer error occurred, but it may take up to 45 days if it provisionally credits the consumer's account within those 10 business days. Until the investigation closes, that credit is an outcome awaiting its final determination.
Appeal ladders add authority to time. Original Medicare has five levels of appeal: a redetermination by the claims contractor, reviewed by staff not involved in the initial decision, then a reconsideration, a hearing, a review by the Medicare Appeals Council and finally judicial review. Each level can issue a new official outcome, so a label is only as final as the highest level the case could still reach. The table maps these statuses to label decisions:
| Outcome status | What it means | Public example | Use as a label |
|---|---|---|---|
| Provisional | Issued but awaiting a determination or review | Regulation E provisional credit; FHIR preliminary | Hold out until the determination arrives, or label provisional |
| Final, uncontested | Issued and not challenged within the window | A denial that was never appealed | Usable, but flag as unreviewed and run the selection check |
| Contested | Challenged and under review at a higher level | A claim awaiting reconsideration or a hearing | Censor; count it as neither outcome yet |
| Revised | Replaced by a later decision | FHIR amended or corrected; an appeal reversal | Keep both verdicts and choose by reversal cause |
| Withdrawn | Removed as recorded in error | FHIR entered-in-error | Exclude; a data error is not a reversal |
Why was it overturned? Four causes, four label treatments
The cause of a reversal decides what each verdict can teach, so classify it before choosing a label. The HHS Office of Inspector General drew the central distinction in its 2018 review of Medicare Advantage appeals: an insurer may overturn a denial because the original decision was incorrect, or because the provider or beneficiary added new information on appeal after a denial that was correct on what was known at the time. Those are different training signals, and two more causes sit beside them.
Record the cause as a field backed by evidence, not as an annotator's guess. A cause that cannot be established from the appeal file or the decision letter is coded unknown, and unknown-cause reversals stay out of every label that depends on the cause. The figure traces one illustrative case from denial on day 3 to reversal on day 41 and shows where each cause sends the labels.
| Reversal cause | What changed between verdicts | Label for the first decision | Evidence to keep |
|---|---|---|---|
| Error on the same evidence | Only the judgment | Wrong; the final verdict is the correct label | Both decision records and the unchanged evidence set |
| New evidence on review | The information available | Right for what was known; the final verdict labels the fuller record | The evidence present at each decision, with dates |
| Discretionary concession | A goodwill, settlement or commercial choice | Policy-correct; do not relabel it as an error | Who granted the concession and the stated reason |
| Procedural ruling | A deadline, jurisdiction or form defect | No merit label from either verdict | The procedural ground and the rule cited |
| Unknown | Not established from the file | Exclude from cause-dependent labels | What was searched for and not found |
Store revisions in an append-only outcome ledger
Store each verdict as its own row and never update a row in place; the current outcome is derived from the ledger, not kept in a field someone edits. An append-only ledger lets one dataset serve all three uses described above, because each use reads different rows.
Derive labels from the ledger with written rules. A first-line imitation label reads the earliest verdict by the lowest authority; a correctness label reads the latest final verdict, filtered by reversal cause; a process label reads the whole sequence. Because rows never change, a buyer can trace any label back to the verdict and evidence it came from, and a later refresh adds rows instead of rewriting history.
Each row carries these fields:
- Case identifier and line identifier, so a reversal attaches to the item it actually changed.
- Verdict sequence number and the identifier of the verdict it supersedes, if any.
- Verdict value, decision date and the date the affected party was told.
- Status at entry: provisional, final, contested, revised or withdrawn.
- Decider role and authority level, read from the delegation rules in force that day.
- A pointer to the evidence snapshot the decider had, rather than a copy of the evidence.
- Reversal cause with the document it was coded from, or unknown.
- The earliest date the verdict could become final, and the freeze date applied.
The selection check: most official outcomes were never contested
An overturn rate among appealed cases is not an error rate among all cases, because only a small, self-selected share of outcomes ever gets a second look. The same OIG review found that Medicare Advantage organizations overturned 75% of their own preauthorization and payment denials that were appealed in 2014 through 2016, about 216,000 denials a year, while beneficiaries and providers appealed only 1% of denials.
Read as a dataset problem, that leaves 99% of denials standing as official outcomes without any review. Some would have survived an appeal and some would not, and the data cannot say which. A model trained on them as ground truth learns the first-line decision, errors included. A 2026 OIG report on prior authorization for skilled nursing facility admission found that insurers overturned nearly all appealed denials, so the pattern is not confined to one period or one type of care.
Three habits keep these labels honest. Mark never-contested outcomes final but unreviewed rather than verified. Where the budget allows, have a blind reviewer re-decide a random sample of them to estimate how often they would flip. And publish the appeal rate next to any overturn rate, so a buyer can see how much of the dataset ever faced a second decision.
How long until an outcome is final enough to label?
Long enough for the last reversal that could plausibly arrive, set per outcome type from its appeal or dispute rules, and no longer. Every extra week of waiting delays the dataset; every week cut short turns cases that would have been reversed into confident wrong labels. Ktena and colleagues met the same tension with clicks that arrive after an ad impression is logged, noting that it is unclear what the ideal window length would be, given the tradeoff between delay in model training and the fake negative rate.
For overturns, the remedy is a cut-off rule written before labeling. Choose a freeze date, treat a case as final only if every review it could still reach had expired by then, and label the rest contested or provisional so they stay out of success and error rates. Store the freeze date with the dataset, so whoever refreshes it later can relabel cases whose windows have since closed. Outcomes that are late rather than reversed raise a related censoring problem, covered in the agent trajectory outcome delay guide linked below.
One reversed warranty denial, packed for training (illustrative)
The packet below holds one illustrative case: a manufacturer's warranty claim denied for missing proof of purchase date, then approved after the customer escalated with a dated receipt. It describes no real company or customer, and its dates are counted in days from the claim. Each element names the record that supports it, so a reviewer can check the packet against the source system.
Two choices define it. The first decision is kept with exactly the evidence the agent had on day 3, which lets the case teach a first-line model that the denial was reasonable on that record. And the reversal cause, new evidence, is coded from the manager's note rather than inferred, which tells a buyer not to relabel the first verdict as an error.
Left out on purpose: the customer's identity and contact history, other claims on the account and the product's sales record. None of them changes the cause classification, and each adds linkage and privacy risk. The packet, element by element:
| Packet element | Illustrative value | Supported by | Why it is in the packet |
|---|---|---|---|
| Original decision | Claim denied on day 3: purchase date not shown | Claims system decision record and denial message | The first verdict, kept as issued |
| Evidence at first decision | Claim form, serial number and a photo of the defect | Attachments dated on or before day 3 | Fixes what the first reviewer could see |
| Decider and authority | Tier 1 claims agent, allowed to deny for missing proof | Delegation table version in force on day 3 | Shows the denial was within authority |
| Challenge | Customer escalation on day 30 with a dated receipt | Escalation ticket and its new attachment | Marks the new evidence and when it arrived |
| Revised decision | Approved on day 41 by the warranty manager | Decision record with the reviewer's role | The final verdict and the higher authority |
| Reversal cause | New evidence on review | Manager's note citing the receipt | Sets the label treatment for both verdicts |
| Finality | Final on day 41; no further internal review exists | Warranty policy version in force on day 41 | Lets the case enter training as final |
Context completeness map: what each reversal cause requires
Build the map once per outcome type and apply it to every case, so completeness is judged by one rule across the dataset and a missing element becomes an explicit gap instead of a silent null.
Not every reversal needs the same context, and the map keeps each packet as small as its cause allows. Required means the cause cannot be established, or the label trusted, without the element; not needed means joining it adds risk without changing the label. The map by element and cause:
| Context element | Error on same evidence | New evidence | Concession | Procedural |
|---|---|---|---|---|
| Dated evidence set at each decision | Required | Required | Helpful | Not needed |
| Authority of each decider | Required | Helpful | Required | Helpful |
| Policy version at each decision | Required | Required | Required | Helpful |
| Stated reason for the reversal | Required | Required | Required | Required |
| Rule or deadline cited | Not needed | Not needed | Helpful | Required |
| Account history beyond the case | Not needed | Not needed | Helpful | Not needed |
Would joining more records make the right verdict clearer?
Rarely, and this is the alternative worth testing before any large join. The intuition is that a fuller picture, every related ticket, payment and message, will reveal which verdict was right. Past the elements in the map, extra records mostly add three costs: noise a model can latch onto, linkage errors that attach a challenge to the wrong original decision, and more personal data to protect.
Linkage is the quiet failure. Many outcomes are revised at line level, one item on a claim or one transaction in a dispute, while the challenge is filed against the whole case. A join on case number alone can pair a reversal with the wrong line and turn a correct denial into an apparent error. FHIR's separate entered-in-error status points to a second trap: a record withdrawn as a data mistake is not an overturned decision, and counting it as one inflates the reversal rate.
Test the alternative directly. Build labels twice, once from the minimal packet and once with the extra joins, and have reviewers classify reversal causes blind on a sample from each version. If the extra records change few classifications but raise linkage disputes, the minimal packet wins.
Writing overturn rules into a dataset request
Put the label policy in the request itself, because a supplier's default export usually holds only the latest status. Name the outcome types, ask for every verdict with its date and decider, ask for the reversal cause where the records hold one, state the freeze date and the review windows you will apply, and request the appeal or dispute rate for the period so the selection check can be run.
Add an acceptance test a reviewer can run on a sample: for example, a stated minimum share of reversed cases carry a documented cause, no case marked final had an open review window at the freeze date, and every withdrawn record is excluded. The thresholds are the buyer's to set; what matters is that they are written down before preparation starts.
SourceX sources each dataset from U.S. businesses after a buyer explains what it needs, rather than selling from stock, so overturn rules like these can shape what gets extracted in the first place. The request form at sourcex.si/request-data asks what the buyer needs and on what licensing terms; the revision ledger, freeze date and acceptance test belong in that description. Whether a particular company's records keep reversal causes is established during review, case by case.
Frequently asked questions
Should a model learn the original decision or the overturned one?
It depends on the task and on why the decision was reversed. If the first verdict was wrong on the same evidence, train correctness models on the final verdict and keep the first as an error example. If new evidence arrived on appeal, the first verdict was right for what was known and should not be relabeled as a mistake.
What is a provisional outcome in a dataset?
A provisional outcome has been issued but still awaits a determination or review, such as a provisional credit during a financial institution's error investigation or a preliminary lab result. Label it provisional or hold it out until the final determination, because counting it as final teaches a model outcomes that may still change.
Why is an appeal overturn rate misleading as an error rate?
Only contested outcomes are reviewed, and the people who contest are not a random sample. In Medicare Advantage, insurers overturned 75% of appealed denials in 2014-16, but only 1% of denials were appealed, so the overturn rate says little about the 99% never reviewed.
How long should a dataset wait before treating outcomes as final?
Until every review the case could still reach has expired, based on the appeal or dispute rules for that outcome type. Set a freeze date before labeling, mark cases with open windows contested or provisional, and store the freeze date so the dataset can be relabeled later.
Can SourceX supply datasets that keep outcome revision histories?
SourceX sources datasets on request from U.S. businesses, so it depends on whether a specific company's records keep each verdict, its date, its decider and the reason for reversal. A buyer can describe the label policy and acceptance test in a request at sourcex.si/request-data; nothing is held in stock.
Sources
- HHS OIG (September 2018) found Medicare Advantage organizations overturned 75% of their own appealed preauthorization and payment denials in 2014-16, about 216,000 a year, while only 1% of denials were appealed, and noted overturns can reflect an incorrect original decision or new information added on appeal. Source
- A 2026 HHS OIG report found Medicare Advantage organizations overturned nearly all appealed prior authorization denials for skilled nursing facility admission. Source
- Regulation E section 1005.11 requires a determination within 10 business days, or up to 45 days if the institution provisionally credits the account within 10 business days. Source
- FHIR R4's observation status code system defines preliminary, final, amended, corrected and entered-in-error statuses. Source
- Original Medicare has five appeal levels, from redetermination by staff uninvolved in the initial determination through reconsideration, hearing, Appeals Council review and judicial review. Source
- Ktena and colleagues describe the tradeoff in choosing a waiting window between training delay and the fake negative rate when labels arrive late. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.