AI uses for records
Enterprise AI error severity index: disclosed weights and a rank-flip test
By SourceX Editorial · Updated
10 sections
Short answer
An enterprise AI error severity index weights each model error by its business consequence, using tiers and weights fixed by domain reviewers before any results, and reports three numbers apart: raw error counts, the oversampling of rare cases and the severity weights. Its key output is the flip ratio, the weight at which two systems swap rank. SourceX has not run this study; this page publishes the design.
Retrieval
Enterprise AI error severity
Key takeaways
- Average accuracy counts a wrong payment and a clumsy summary the same; a severity index weights each error by what it breaks.
- Severity weights are value judgments, so domain reviewers must set them before anyone sees model results, and the weights must be published.
- Rare high-consequence cases have to be oversampled to be measurable and then weighted back to their real frequency; reporting both stops a test set from exaggerating or hiding the tail.
- The flip ratio states the exact weight at which two systems swap rank, turning a weighting debate into one number reviewers can check.
- In the illustrative case, the model with higher accuracy at the real case mix is the riskier choice whenever one critical error counts for more than 10 recoverable ones.
What is an enterprise AI error severity index?
It is a score that weights each model error by its business consequence instead of counting all errors equally, built so that every weight, sampling choice and count stays visible. It answers a narrower and more useful question than accuracy: given how often each kind of failure happens in real work, and how much each one costs, which system leaves the lighter risk profile?
The idea comes from risk management. NIST's AI Risk Management Framework (AI RMF 1.0, January 2023) defines risk as a composite measure of an event's probability of occurring and the magnitude or degree of its consequences. Most model evaluations report only the probability half. The index adds the magnitude half for each error and keeps the two halves apart, so neither can be quietly adjusted after results are in.
It does that with three numbers per error class that are never merged before they are reported: how many errors were observed in the test set, how heavily rare cases were oversampled to observe them, and what severity weight reviewers assigned in advance. Its headline output is not one blended score but a flip ratio and a sensitivity range, which show how far any ranking depends on the weights.
Why can a higher average score hide a worse operational risk?
Because an average gives a rare catastrophic failure the same weight as a routine slip, and rare failures are where business damage concentrates. Two systems with similar accuracy can differ sharply in how often they produce the error that reaches a customer's bank account, a regulator or a safety record.
Public evaluations show how wide the range of a single failure is. In GDPval, OpenAI's evaluation of AI deliverables against work by industry professionals, other expert graders re-rated the GPT-5 deliverables that had lost to the human expert. The most common rating was acceptable but subpar; about 29 percent were rated bad or catastrophic, and roughly 3 percent catastrophic, meaning harmful or dangerously wrong if used in practice. A loss in a win rate can mean anything from a stylistic miss to a dangerous answer.
Medical imaging research gave the problem a name. Oakden-Rayner and colleagues described hidden stratification: models with high overall performance can consistently miss rare but important subsets, with relative performance differences of over 20 percent on clinically important subsets in their analysis. Reliability adds a further layer. The tau-bench paper (2024) found that gpt-4o with function calling succeeded on fewer than half of its tasks and that pass^8, the chance of succeeding on all eight trials of a task, fell below 25 percent in the retail domain; Anthropic's agent-evaluation guide defines pass^k the same way, as the probability that all k trials succeed. A single average hides how often, how consistently and how badly a system fails.
How should domain reviewers fix severity before seeing model results?
By writing the severity categories and eliciting the weights first, then freezing them, so that nobody can tune weights to favor a preferred system. Reviewers who have seen a leaderboard cannot unsee it, so elicitation happens before any model output is graded and the frozen sheet is preregistered.
Severity attaches to the error and its consequence, never to the task type or to how rare the case is. A routine invoice can produce a critical error, such as paying the wrong vendor, and a rare case can produce a cosmetic one. Each reviewer answers the elicitation questions independently, and the spread of their answers is kept and reported rather than averaged away.
The question format matters. Asking how many recoverable errors equal one critical error gives a ratio one reviewer can defend and another can challenge, which a bare score from 1 to 5 does not. The elicitation sheet:
| Tier | Consequence test | Question each reviewer answers alone | What to record |
|---|---|---|---|
| Cosmetic | No decision, money, record or person is affected | How many cosmetic errors equal one recoverable error? | Each reviewer's answer and an example |
| Recoverable | Caught and corrected inside the normal workflow before anyone outside is affected | None: this is the reference unit, weight 1 | Examples that anchor the unit for every reviewer |
| Material | Reaches a customer, a ledger or a decision and needs rework, a refund or an apology | How many recoverable errors equal one material error? | Each reviewer's ratio, the range and the median |
| Critical | Money moves wrongly, a legal or safety duty is breached, or harm cannot be reversed | How many recoverable errors equal one critical error? | Each reviewer's ratio, the range and the median |
Which rare cases belong in the task set, and how far should they be oversampled?
Include a rare case only when there is a written reason to expect a high-consequence failure, oversample rare cases enough to observe failures, then weight them back to their real frequency. Rarity earns a case a place in the sampling frame; it never earns it a severity weight.
Start from the source system's own counts. If a support or finance queue shows that rare case types, such as disputed refunds, duplicate payments or exceptions needing manager approval, are a small share of real volume, a random test set of a few hundred tasks may contain none of them, and their error rate cannot be estimated. Stratify instead: a routine stratum drawn at its natural rate and a rare stratum drawn at a much higher rate, each with its real share recorded from the source system with a date.
Every rare case needs a justification note, such as a regulatory duty, a money movement or an irreversible action, written before grading. Cases added because they are merely unusual, or because a model is already known to fail them, stay out of the index or are reported as a separate stress set. The oversampling factor, the stratum's sampling rate relative to the routine stratum, is reported for every stratum.
The Long-Tail Error Severity Index: what the report must disclose
The report keeps every ingredient visible, so a reader can recompute the result or substitute their own weights. The table is the disclosure standard; a report missing any row should not be read as a severity index.
| Component | Definition | Denominator | Reported with |
|---|---|---|---|
| Raw error counts | Errors observed per stratum and severity tier in the test set | Graded tasks in that stratum | Counts as well as rates, and the rubric version |
| Population weights | Real tasks each test task stands for, scaled to 1,000 real tasks | Real task volume from the source system, dated | The oversampling factor per stratum |
| Deployment-weighted error rates | Raw rates re-weighted to the real case mix | 1,000 tasks at the real case mix | Plain accuracy on the same basis |
| Severity weights | Frozen ratio per tier from the elicitation sheet | One recoverable error as the unit | Each reviewer's ratio, range and median |
| Flip ratio | The critical weight at which two systems swap rank | Each pair of systems | Whether the elicited range lies above, below or across it |
| Sensitivity dashboard | The ranking under every prespecified weight set | All weight sets fixed before results | Share of reviewer weight sets under which each ranking holds |
Worked example: the higher-accuracy model with the worse risk profile (illustrative)
In this invented case, the model with higher accuracy at the real case mix carries the worse risk profile once critical errors are weighted. Every number is illustrative, chosen to show the arithmetic, not measured.
A support operation's real volume is 99 percent routine cases and 1 percent rare cases. The test set holds 990 routine tasks and 250 rare tasks, so rare cases are sampled at 25 times the routine rate: per 1,000 real tasks, each routine task stands for one real task and each rare task for 0.04. Model A makes 61 routine errors, 2 of them critical, and 75 rare errors, 50 of them critical. Model B makes 90 routine errors, 1 of them critical, and 25 rare errors, none critical.
Three honest readings give three different answers. On the raw test set, B looks better, with 90.7 percent accuracy against A's 89.0 percent, because the oversampled rare stratum punishes A. At the real case mix, A looks better: 64 errors per 1,000 tasks against B's 91, or 93.6 percent accuracy against 90.9 percent. Weighted by severity, A's loss per 1,000 tasks is 60 recoverable errors plus 4 times the critical weight, and B's is 90 plus 1 times the critical weight, so the two tie when one critical error counts for 10 recoverable ones. Above that flip ratio, B is the safer system.
If the reviewers' frozen ratios for a critical error ran from 15 to 25, every one lies above the flip ratio of 10, and the verdict that B carries the lighter risk profile holds under every reviewer's weights. Had the range run from 5 to 20, the honest report would say the ranking depends on whose weights are used and show both answers. The figure plots both loss lines against the critical weight.
| Measure (illustrative) | Model A | Model B | How it is computed |
|---|---|---|---|
| Raw test-set accuracy | 89.0% (136 errors in 1,240) | 90.7% (115 errors in 1,240) | Errors over all graded tasks, oversampled mix |
| Accuracy at the real case mix | 93.6% (64 errors per 1,000) | 90.9% (91 errors per 1,000) | Routine errors plus 0.04 times rare errors |
| Critical errors per 1,000 real tasks | 4 | 1 | 2 plus 0.04 times 50, against 1 plus 0 |
| Recoverable errors per 1,000 real tasks | 60 | 90 | Remaining errors after critical ones |
| Severity-weighted loss, critical weight w | 60 plus 4w | 90 plus 1w | Equal at w = 10, the flip ratio |
How sensitive is the ranking to the severity weights?
Report the flip ratio for every pair of systems and the share of reviewer weight sets under which each ranking holds. With two tiers the flip ratio has a closed form: the difference in recoverable errors divided by the difference in critical errors, both per 1,000 real tasks, which in the example is 30 divided by 3.
With four tiers there is no single flip point, so the dashboard rescores the ranking under every prespecified weight set: equal weights, each reviewer's frozen ratios, the median ratios, and the lowest and highest ratio per tier. A ranking that holds under all of them is robust to the weighting debate. One that holds under only some is a value judgment the buyer has to own, and the report should say whose weights produce which answer.
Critical errors are rare by construction, so their counts carry wide sampling uncertainty. Report exact binomial intervals for critical error rates in each stratum and resample task families for the index itself. A flip ratio computed from two or three critical errors is a direction, not a finding.
What a severity index cannot settle
It cannot turn a value judgment into a fact. The weights express what one set of reviewers believes a critical error costs relative to a recoverable one; another business with different exposure may reasonably choose other weights, which is why the index publishes the weights and the flip ratio rather than a single blended score. Five limits travel with every result:
- No sector-wide ranking: results hold for the task families, case mix and systems tested, not for an industry or for models in general.
- Severity grading is itself uncertain: when GDPval's graders re-rated GPT-5 deliverables that had lost to the human expert, they judged about 23 percent of them better than the human work, so every material and critical error needs an independent second grade.
- Population shares drift: a rare stratum's real share comes from one source system at one date and must be refreshed when the business changes.
- Rarity is not harm: a case type earns oversampling through a written consequence rationale, never through rarity alone.
- Tiers should map to the deployer's own risk register: NIST's Generative AI Profile, announced in July 2024, names 12 risks, including confabulated or hallucinated output, and a business can align its critical tier with the ones that apply to it.
Methods note: the proposed study and its limits
This is a research commission, not a finished index. SourceX has not run the study, holds no severity results and will publish results, including negative ones and any case where weighting changes nothing, only after the study is actually conducted with rights and disclosure approvals in place. The design:
At a glance
Methods note: the proposed study and its limits
- Question: how often does a severity-weighted ranking of enterprise AI systems differ from an accuracy ranking, and at what weights?
- Population: pre-registered task families from licensed operational records, with routine and rare strata and real shares taken from the source systems.
- Severity: four tiers fixed, and weights elicited independently from at least five domain reviewers, before any model output is graded.
- Variables: for each error, the stratum, tier, first and second grader, and adjudication notes.
- Metrics: raw counts, population weights, deployment-weighted error rates, flip ratios and the sensitivity dashboard, each with its denominator.
- Uncertainty: exact intervals for critical error rates and task-family resampling for index values.
- Not measured: money losses, downstream correction effort or any SourceX data; the worked example is illustrative.
Sourcing rare, consequential cases for a severity index
An index like this needs records that show not just the case but what an error would have touched: the request, the decision, the downstream record it changed and any reversal, refund, dispute or audit that followed. Those links let reviewers tie a severity tier to evidence rather than imagination.
SourceX looks for datasets among U.S. businesses when an AI buyer's request comes in, rather than selling from stock, and before any records are delivered, the company holding them approves the scope and terms and signs. A buyer planning a severity index can submit a requirement with the intended use and an acceptance test, for example a minimum count of rare-stratum cases with linked outcomes in each task family.
Frequently asked questions
How is an AI error severity index different from an error rate?
An error rate counts every mistake equally. A severity index weights each mistake by its business consequence, using weights fixed before results, and reports the counts and weights separately. Two systems with the same error rate can have very different severity profiles.
Who should set the severity weights?
Domain reviewers who own the consequences, such as finance, support or compliance leads, each answering the elicitation questions independently before seeing any model output. Their individual ratios, range and median are published. People who built a system should not set the weights used to judge it.
Why oversample rare cases and then weight them back down?
Oversampling makes rare failures observable; without it, a test set of a few hundred tasks may contain no rare cases at all. Weighting back to real frequency stops the oversampled stratum from dominating the overall error rate. Reporting both keeps the test set from hiding or exaggerating the tail.
What is a flip ratio in a severity-weighted evaluation?
It is the severity weight at which two systems swap rank. With two tiers it equals the difference in recoverable errors divided by the difference in critical errors, both at the real case mix. If every reviewer's weight lies on one side of it, the ranking is robust to the weighting debate.
Has SourceX published Long-Tail Error Severity Index results?
No. This page publishes the study design and an illustrative worked example only. SourceX has not run the study and has no severity results or system rankings to report.
Sources
- NIST's AI RMF 1.0 defines risk as a composite measure of an event's probability of occurring and the magnitude or degree of the consequences of that event. Source
- NIST announced its Generative AI Profile in July 2024, built around a list of 12 risks, including generative AI systems confabulating or hallucinating output. Source
- GDPval has graders from the same occupations rank human and AI deliverables, labeling each AI deliverable better than, as good as or worse than the human one. Source
- In GDPval's failure analysis, graders re-rated GPT-5 deliverables that lost to human experts: about 29 percent bad or catastrophic, roughly 3 percent catastrophic, about 23 percent judged better than the human work. Source
- Hidden stratification: models with high overall performance can miss rare but important subsets, with relative performance differences of over 20 percent on clinically important subsets. Source
- The tau-bench paper (2024) reports that gpt-4o with function calling succeeds on under 50 percent of tasks and that pass^8 falls below 25 percent in retail. Source
- Anthropic's agent-evaluation guide defines pass^k as the probability that all k trials succeed, and pass@k as the likelihood of at least one correct solution in k attempts. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.