Skip to content

AI uses for records

Enterprise AI error severity index: disclosed weights and a rank-flip test

By SourceX Editorial · Updated

10 sections

Short answer

An enterprise AI error severity index weights each model error by its business consequence, using tiers and weights fixed by domain reviewers before any results, and reports three numbers apart: raw error counts, the oversampling of rare cases and the severity weights. Its key output is the flip ratio, the weight at which two systems swap rank. SourceX has not run this study; this page publishes the design.

Retrieval

Enterprise AI error severity

This page covers enterprise AI error severity index: disclosed weights and a rank-flip test.Illustrative

Key takeaways

  • Average accuracy counts a wrong payment and a clumsy summary the same; a severity index weights each error by what it breaks.
  • Severity weights are value judgments, so domain reviewers must set them before anyone sees model results, and the weights must be published.
  • Rare high-consequence cases have to be oversampled to be measurable and then weighted back to their real frequency; reporting both stops a test set from exaggerating or hiding the tail.
  • The flip ratio states the exact weight at which two systems swap rank, turning a weighting debate into one number reviewers can check.
  • In the illustrative case, the model with higher accuracy at the real case mix is the riskier choice whenever one critical error counts for more than 10 recoverable ones.

What is an enterprise AI error severity index?

It is a score that weights each model error by its business consequence instead of counting all errors equally, built so that every weight, sampling choice and count stays visible. It answers a narrower and more useful question than accuracy: given how often each kind of failure happens in real work, and how much each one costs, which system leaves the lighter risk profile?

The idea comes from risk management. NIST's AI Risk Management Framework (AI RMF 1.0, January 2023) defines risk as a composite measure of an event's probability of occurring and the magnitude or degree of its consequences. Most model evaluations report only the probability half. The index adds the magnitude half for each error and keeps the two halves apart, so neither can be quietly adjusted after results are in.

It does that with three numbers per error class that are never merged before they are reported: how many errors were observed in the test set, how heavily rare cases were oversampled to observe them, and what severity weight reviewers assigned in advance. Its headline output is not one blended score but a flip ratio and a sensitivity range, which show how far any ranking depends on the weights.

Why can a higher average score hide a worse operational risk?

Because an average gives a rare catastrophic failure the same weight as a routine slip, and rare failures are where business damage concentrates. Two systems with similar accuracy can differ sharply in how often they produce the error that reaches a customer's bank account, a regulator or a safety record.

Public evaluations show how wide the range of a single failure is. In GDPval, OpenAI's evaluation of AI deliverables against work by industry professionals, other expert graders re-rated the GPT-5 deliverables that had lost to the human expert. The most common rating was acceptable but subpar; about 29 percent were rated bad or catastrophic, and roughly 3 percent catastrophic, meaning harmful or dangerously wrong if used in practice. A loss in a win rate can mean anything from a stylistic miss to a dangerous answer.

Medical imaging research gave the problem a name. Oakden-Rayner and colleagues described hidden stratification: models with high overall performance can consistently miss rare but important subsets, with relative performance differences of over 20 percent on clinically important subsets in their analysis. Reliability adds a further layer. The tau-bench paper (2024) found that gpt-4o with function calling succeeded on fewer than half of its tasks and that pass^8, the chance of succeeding on all eight trials of a task, fell below 25 percent in the retail domain; Anthropic's agent-evaluation guide defines pass^k the same way, as the probability that all k trials succeed. A single average hides how often, how consistently and how badly a system fails.

How should domain reviewers fix severity before seeing model results?

By writing the severity categories and eliciting the weights first, then freezing them, so that nobody can tune weights to favor a preferred system. Reviewers who have seen a leaderboard cannot unsee it, so elicitation happens before any model output is graded and the frozen sheet is preregistered.

Severity attaches to the error and its consequence, never to the task type or to how rare the case is. A routine invoice can produce a critical error, such as paying the wrong vendor, and a rare case can produce a cosmetic one. Each reviewer answers the elicitation questions independently, and the spread of their answers is kept and reported rather than averaged away.

The question format matters. Asking how many recoverable errors equal one critical error gives a ratio one reviewer can defend and another can challenge, which a bare score from 1 to 5 does not. The elicitation sheet:

How should domain reviewers fix severity before seeing model results?
TierConsequence testQuestion each reviewer answers aloneWhat to record
CosmeticNo decision, money, record or person is affectedHow many cosmetic errors equal one recoverable error?Each reviewer's answer and an example
RecoverableCaught and corrected inside the normal workflow before anyone outside is affectedNone: this is the reference unit, weight 1Examples that anchor the unit for every reviewer
MaterialReaches a customer, a ledger or a decision and needs rework, a refund or an apologyHow many recoverable errors equal one material error?Each reviewer's ratio, the range and the median
CriticalMoney moves wrongly, a legal or safety duty is breached, or harm cannot be reversedHow many recoverable errors equal one critical error?Each reviewer's ratio, the range and the median

Which rare cases belong in the task set, and how far should they be oversampled?

Include a rare case only when there is a written reason to expect a high-consequence failure, oversample rare cases enough to observe failures, then weight them back to their real frequency. Rarity earns a case a place in the sampling frame; it never earns it a severity weight.

Start from the source system's own counts. If a support or finance queue shows that rare case types, such as disputed refunds, duplicate payments or exceptions needing manager approval, are a small share of real volume, a random test set of a few hundred tasks may contain none of them, and their error rate cannot be estimated. Stratify instead: a routine stratum drawn at its natural rate and a rare stratum drawn at a much higher rate, each with its real share recorded from the source system with a date.

Every rare case needs a justification note, such as a regulatory duty, a money movement or an irreversible action, written before grading. Cases added because they are merely unusual, or because a model is already known to fail them, stay out of the index or are reported as a separate stress set. The oversampling factor, the stratum's sampling rate relative to the routine stratum, is reported for every stratum.

The Long-Tail Error Severity Index: what the report must disclose

The report keeps every ingredient visible, so a reader can recompute the result or substitute their own weights. The table is the disclosure standard; a report missing any row should not be read as a severity index.

The Long-Tail Error Severity Index: what the report must disclose
ComponentDefinitionDenominatorReported with
Raw error countsErrors observed per stratum and severity tier in the test setGraded tasks in that stratumCounts as well as rates, and the rubric version
Population weightsReal tasks each test task stands for, scaled to 1,000 real tasksReal task volume from the source system, datedThe oversampling factor per stratum
Deployment-weighted error ratesRaw rates re-weighted to the real case mix1,000 tasks at the real case mixPlain accuracy on the same basis
Severity weightsFrozen ratio per tier from the elicitation sheetOne recoverable error as the unitEach reviewer's ratio, range and median
Flip ratioThe critical weight at which two systems swap rankEach pair of systemsWhether the elicited range lies above, below or across it
Sensitivity dashboardThe ranking under every prespecified weight setAll weight sets fixed before resultsShare of reviewer weight sets under which each ranking holds

Worked example: the higher-accuracy model with the worse risk profile (illustrative)

In this invented case, the model with higher accuracy at the real case mix carries the worse risk profile once critical errors are weighted. Every number is illustrative, chosen to show the arithmetic, not measured.

A support operation's real volume is 99 percent routine cases and 1 percent rare cases. The test set holds 990 routine tasks and 250 rare tasks, so rare cases are sampled at 25 times the routine rate: per 1,000 real tasks, each routine task stands for one real task and each rare task for 0.04. Model A makes 61 routine errors, 2 of them critical, and 75 rare errors, 50 of them critical. Model B makes 90 routine errors, 1 of them critical, and 25 rare errors, none critical.

Three honest readings give three different answers. On the raw test set, B looks better, with 90.7 percent accuracy against A's 89.0 percent, because the oversampled rare stratum punishes A. At the real case mix, A looks better: 64 errors per 1,000 tasks against B's 91, or 93.6 percent accuracy against 90.9 percent. Weighted by severity, A's loss per 1,000 tasks is 60 recoverable errors plus 4 times the critical weight, and B's is 90 plus 1 times the critical weight, so the two tie when one critical error counts for 10 recoverable ones. Above that flip ratio, B is the safer system.

If the reviewers' frozen ratios for a critical error ran from 15 to 25, every one lies above the flip ratio of 10, and the verdict that B carries the lighter risk profile holds under every reviewer's weights. Had the range run from 5 to 20, the honest report would say the ranking depends on whose weights are used and show both answers. The figure plots both loss lines against the critical weight.

Worked example: the higher-accuracy model with the worse risk profile (illustrative)
Measure (illustrative)Model AModel BHow it is computed
Raw test-set accuracy89.0% (136 errors in 1,240)90.7% (115 errors in 1,240)Errors over all graded tasks, oversampled mix
Accuracy at the real case mix93.6% (64 errors per 1,000)90.9% (91 errors per 1,000)Routine errors plus 0.04 times rare errors
Critical errors per 1,000 real tasks412 plus 0.04 times 50, against 1 plus 0
Recoverable errors per 1,000 real tasks6090Remaining errors after critical ones
Severity-weighted loss, critical weight w60 plus 4w90 plus 1wEqual at w = 10, the flip ratio
Line chart of severity-weighted loss per 1,000 real tasks against the weight of one critical error: model A's line starts lower and rises faster, crossing model B's at a weight of 10, left of a shaded reviewer range from 15 to 25.
Illustrative: the two models swap rank at a critical weight of 10, and every reviewer weight in the example lies above it.

How sensitive is the ranking to the severity weights?

Report the flip ratio for every pair of systems and the share of reviewer weight sets under which each ranking holds. With two tiers the flip ratio has a closed form: the difference in recoverable errors divided by the difference in critical errors, both per 1,000 real tasks, which in the example is 30 divided by 3.

With four tiers there is no single flip point, so the dashboard rescores the ranking under every prespecified weight set: equal weights, each reviewer's frozen ratios, the median ratios, and the lowest and highest ratio per tier. A ranking that holds under all of them is robust to the weighting debate. One that holds under only some is a value judgment the buyer has to own, and the report should say whose weights produce which answer.

Critical errors are rare by construction, so their counts carry wide sampling uncertainty. Report exact binomial intervals for critical error rates in each stratum and resample task families for the index itself. A flip ratio computed from two or three critical errors is a direction, not a finding.

What a severity index cannot settle

It cannot turn a value judgment into a fact. The weights express what one set of reviewers believes a critical error costs relative to a recoverable one; another business with different exposure may reasonably choose other weights, which is why the index publishes the weights and the flip ratio rather than a single blended score. Five limits travel with every result:

  • No sector-wide ranking: results hold for the task families, case mix and systems tested, not for an industry or for models in general.
  • Severity grading is itself uncertain: when GDPval's graders re-rated GPT-5 deliverables that had lost to the human expert, they judged about 23 percent of them better than the human work, so every material and critical error needs an independent second grade.
  • Population shares drift: a rare stratum's real share comes from one source system at one date and must be refreshed when the business changes.
  • Rarity is not harm: a case type earns oversampling through a written consequence rationale, never through rarity alone.
  • Tiers should map to the deployer's own risk register: NIST's Generative AI Profile, announced in July 2024, names 12 risks, including confabulated or hallucinated output, and a business can align its critical tier with the ones that apply to it.

Methods note: the proposed study and its limits

This is a research commission, not a finished index. SourceX has not run the study, holds no severity results and will publish results, including negative ones and any case where weighting changes nothing, only after the study is actually conducted with rights and disclosure approvals in place. The design:

At a glance

Methods note: the proposed study and its limits

  • Question: how often does a severity-weighted ranking of enterprise AI systems differ from an accuracy ranking, and at what weights?
  • Population: pre-registered task families from licensed operational records, with routine and rare strata and real shares taken from the source systems.
  • Severity: four tiers fixed, and weights elicited independently from at least five domain reviewers, before any model output is graded.
  • Variables: for each error, the stratum, tier, first and second grader, and adjudication notes.
  • Metrics: raw counts, population weights, deployment-weighted error rates, flip ratios and the sensitivity dashboard, each with its denominator.
  • Uncertainty: exact intervals for critical error rates and task-family resampling for index values.
  • Not measured: money losses, downstream correction effort or any SourceX data; the worked example is illustrative.

Sourcing rare, consequential cases for a severity index

An index like this needs records that show not just the case but what an error would have touched: the request, the decision, the downstream record it changed and any reversal, refund, dispute or audit that followed. Those links let reviewers tie a severity tier to evidence rather than imagination.

SourceX looks for datasets among U.S. businesses when an AI buyer's request comes in, rather than selling from stock, and before any records are delivered, the company holding them approves the scope and terms and signs. A buyer planning a severity index can submit a requirement with the intended use and an acceptance test, for example a minimum count of rare-stratum cases with linked outcomes in each task family.

Frequently asked questions

How is an AI error severity index different from an error rate?

An error rate counts every mistake equally. A severity index weights each mistake by its business consequence, using weights fixed before results, and reports the counts and weights separately. Two systems with the same error rate can have very different severity profiles.

Who should set the severity weights?

Domain reviewers who own the consequences, such as finance, support or compliance leads, each answering the elicitation questions independently before seeing any model output. Their individual ratios, range and median are published. People who built a system should not set the weights used to judge it.

Why oversample rare cases and then weight them back down?

Oversampling makes rare failures observable; without it, a test set of a few hundred tasks may contain no rare cases at all. Weighting back to real frequency stops the oversampled stratum from dominating the overall error rate. Reporting both keeps the test set from hiding or exaggerating the tail.

What is a flip ratio in a severity-weighted evaluation?

It is the severity weight at which two systems swap rank. With two tiers it equals the difference in recoverable errors divided by the difference in critical errors, both at the real case mix. If every reviewer's weight lies on one side of it, the ranking is robust to the weighting debate.

Has SourceX published Long-Tail Error Severity Index results?

No. This page publishes the study design and an illustrative worked example only. SourceX has not run the study and has no severity results or system rankings to report.

Sources

  • NIST's AI RMF 1.0 defines risk as a composite measure of an event's probability of occurring and the magnitude or degree of the consequences of that event. Source
  • NIST announced its Generative AI Profile in July 2024, built around a list of 12 risks, including generative AI systems confabulating or hallucinating output. Source
  • GDPval has graders from the same occupations rank human and AI deliverables, labeling each AI deliverable better than, as good as or worse than the human one. Source
  • In GDPval's failure analysis, graders re-rated GPT-5 deliverables that lost to human experts: about 29 percent bad or catastrophic, roughly 3 percent catastrophic, about 23 percent judged better than the human work. Source
  • Hidden stratification: models with high overall performance can miss rare but important subsets, with relative performance differences of over 20 percent on clinically important subsets. Source
  • The tau-bench paper (2024) reports that gpt-4o with function calling succeeds on under 50 percent of tasks and that pass^8 falls below 25 percent in retail. Source
  • Anthropic's agent-evaluation guide defines pass^k as the probability that all k trials succeed, and pass@k as the likelihood of at least one correct solution in k attempts. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies