Skip to content

AI uses for records

AI evaluation severity weights: scoring a wrong payment above a bad summary

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Weight each error by its business consequence, not by the task it came from. A wrong payment usually lands in a top tier because money reaches the wrong party, reversal is slow and it can trigger formal error-resolution duties; a bad summary usually lands lower unless someone acts on it. Report the plain completion rate and a severity-weighted loss side by side, and fail any run with a critical error.

Evaluation split

AI evaluation business

This page covers AI evaluation severity weights: scoring a wrong payment above a bad summary.Illustrative

Key takeaways

  • Attach severity to the consequence of each error, judged by the decision the output feeds, not to the task label.
  • Use a small tier rubric with written consequence anchors and weights the task owner signs, scoring the worst error per task.
  • Report the plain completion rate and the severity-weighted loss side by side; they can rank the same agents in opposite order.
  • Add a hard gate on critical errors such as wrong-payee or duplicate payments that no average can override.
  • Rerun the score under alternative weights and state whether the ranking holds, with intervals per task family.

How should an AI evaluation weight a wrong payment against a bad summary?

Score the consequence of each error on a written severity scale, then multiply. A payment sent to the wrong vendor, for the wrong amount or twice is usually a critical error: money has moved, a person has to chase it back, and the error may set off obligations the business cannot skip. A summary that drops a minor detail is usually a minor error that a reader fixes before acting.

The point of business severity weights is that one average hides the difference. An agent that is slightly better on summaries but occasionally misroutes a payment can post a higher plain pass rate than an agent that never touches money wrongly. A severity-weighted score, reported next to the plain rate and backed by a hard gate on critical errors, makes that trade visible to the person who signs off on deployment.

This page builds that scoring scheme end to end: a four-tier rubric with consequence anchors, a worked example on illustrative counts in which the two numbers rank the agents in opposite order, the evidence each scored case must carry, and how to set and stress-test the weights.

Why severity follows the decision an output feeds, not the task label

Assign severity to the error's downstream effect, because the same task type can produce harmless and costly errors. Labeling every payment task critical and every summary task minor is the most common shortcut, and it breaks on the first counterexample.

Take a call summary that a collections or billing team uses to decide whether to keep charging a customer. If the call included the customer withdrawing authorization for a recurring debit and the summary leaves that out, the next debit is an unauthorized transfer. The summary task produced a top-tier error. Conversely, a payment-preparation task where the agent picks the wrong remittance note format but the right payee, amount and date has produced a cosmetic error.

Risk frameworks already separate these two dimensions. The NIST AI Risk Management Framework treats risk as a composite of the probability of an event and the magnitude of its consequences. An evaluation measures probability by counting errors over a representative set of tasks; the severity weight carries the magnitude. Keep them apart in your scoring sheet so a reviewer can see which one moved.

The severity rubric: four tiers with consequence anchors

Use four tiers, anchor each one to observable consequences, and give each a starting weight that the business owner then adjusts. The anchors below borrow the structure of FIPS 199, which describes potential impact as a limited, serious, or severe or catastrophic adverse effect; the business wording and the weights are this page's editorial starting point, not a standard.

  • Score the worst error in each task, not the sum of every error, so long tasks are not penalized for length.
  • Write the anchor before you see any agent output, and keep the rubric version on every scored case.
  • Treat S3 as a gate as well as a weight: one confirmed S3 error fails the run for that task family, whatever the average.
The severity rubric: four tiers with consequence anchors
TierConsequence anchorTypical examplesStarting weight
S0 cosmeticNo change to any record, decision or customer communication; style or format onlyWrong date format in a summary; extra blank line in a remittance note0 (logged, not scored)
S1 minorA reviewer catches and fixes it in normal review before anyone relies on itSummary omits a non-decision detail; ticket tagged with a near-miss category1
S2 seriousReaches a customer, ledger or downstream decision but is reversible within the normal workflowWrong due date on an invoice reminder; refund routed to the right customer but wrong order line5
S3 criticalHard to reverse, moves money or data to the wrong party, or triggers a formal obligationPayment to the wrong payee, wrong amount or duplicate; debit after authorization was withdrawn25, plus a release gate
UnratedSeverity cannot be judged from the linked evidenceOutcome record missing; rule that applies is unclearExcluded from the score and listed for the task owner

Worked example: two agents, one task set, opposite rankings

In this illustrative case, two agents run the same 200 accounts-payable support tasks: 120 vendor-email summaries and 80 payment-preparation tasks, where the agent proposes payee, amount and date for an approved invoice. Every count below is illustrative and chosen to show the arithmetic, not taken from a real deployment.

Agent A makes 20 errors and Agent B makes 30, so the plain error rates are 10% and 15% and Agent A looks better. Agent A's errors include 2 payments proposed to the wrong payee. Agent B makes more summary mistakes but none on payments.

  • Read the result as: Agent A is better at summaries and unsafe on payments; Agent B is noisier but safe to pilot on payments with summary review in place.
  • The ranking flips because 2 critical errors outweigh 13 extra minor ones at these weights. If the business owner says one wrong payment is worth only 5 minor errors, rerun the sheet and report that the ranking holds or flips under that view.
Worked example: two agents, one task set, opposite rankings
MeasureAgent A (illustrative)Agent B (illustrative)How it is computed
S1 errors1225Count of tasks whose worst error is S1
S2 errors65Count of tasks whose worst error is S2
S3 errors2 (both payments)0Count of tasks whose worst error is S3
Plain error rate10% (20 of 200)15% (30 of 200)Tasks with any scored error divided by tasks
Weighted loss points925012x1 + 6x5 + 2x25 for A; 25x1 + 5x5 for B
Weighted loss per 100 tasks4625Loss points divided by tasks, times 100
S3 gateFailsPassesAny confirmed S3 error fails the payment family

What linked evidence each scored case must carry

Every scored case needs enough linked evidence that a second reviewer reaches the same severity without asking the first. Without it, severity becomes an opinion and the weights amplify the disagreement.

At a glance

What linked evidence each scored case must carry

Judge the outcome from the system of record, not from the agent's last message. The tau-bench benchmark makes the same choice for customer-service agents: it compares the database state at the end of a conversation with an annotated goal state, so a confident closing message does not count as success. For a payment task, the outcome check is the payment record or approval queue, not a status field the agent set or a sentence saying the payment is scheduled.

  • Starting state: the invoice, purchase order and vendor master record as they stood when the task began, with record IDs.
  • Evidence references: the documents the agent was allowed to read, such as the receiving record or the vendor's bank-change request, by ID.
  • Applicable rule: the policy that decides the correct action, for example payment terms, approval limits or a vendor-verification step, quoted with its version.
  • Decision: what the agent proposed or did, captured from tool calls or the record diff.
  • Outcome check: the record that shows what actually happened, such as the payment batch, a reversal, a dispute or a corrected invoice.
  • Counterexample: one near-identical case where the right action differs, so the grader can confirm the agent applied the rule rather than a pattern.
  • Label: real case approved for use, or illustrative case written for the evaluation, never mixed in one score.

Why payment errors usually sit in the top tier

Payment errors sit high because of what happens after them, and you should write those consequences into the anchor so the weight can be defended. Three features usually apply: the money has left, recovery depends on the counterparty, and the error may start a clock.

For consumer accounts in the United States, Regulation E lists an incorrect electronic fund transfer to or from the account as an error. Section 1005.11 then requires the financial institution to investigate and determine whether an error occurred within 10 business days of receiving the consumer's notice, with longer periods only under conditions such as provisional credit. Business-to-business payments follow different rules and contracts, so the obligation varies by payment type, but the pattern is the same: a wrong payment creates work and exposure that a weak summary does not.

Machine-learning practice has long priced errors this way. A scikit-learn example on cost-sensitive learning uses the German credit dataset, where misclassifying a bad credit as good is treated as five times more costly than the reverse, and then builds a fraud metric whose cost depends on each transaction's amount. An evaluation of business agents can borrow both ideas: fixed tier weights for most tasks, and an amount-aware weight where the business already measures exposure by value.

Setting weights the business owner will sign

Elicit weights as trade-offs from the person who owns the task, write down the answer, and test whether it changes the decision. Engineers guessing weights is how evaluations end up optimizing the wrong thing.

Ask paired questions: how many S1 errors would you accept to avoid one S2, and how many S2 to avoid one S3? Use the median answer of two or three owners, record the dissent, and keep the ratio rather than an absolute number. Most scoring libraries support this directly; the scikit-learn metrics guide notes that most implementations let each sample contribute a weighted amount to the overall score through a sample_weight parameter.

  • If the ranking only holds under one weight set, say so in the report rather than picking the set that tells the preferred story.
  • The S3 gate does not depend on the weights, which is why it should exist alongside them.
  • All figures in this table are illustrative and reuse the worked example's counts.
Setting weights the business owner will sign
Weight setS1 / S2 / S3Agent A loss per 100 tasksAgent B loss per 100 tasksRanking
Owner's stated view1 / 5 / 254625B ahead
Low critical weight1 / 5 / 52625Near tie
High critical weight1 / 5 / 507125B ahead
Flat (no severity)1 / 1 / 11015A ahead

Which business task or failure type is missing from your evaluation?

Build a coverage matrix of task families against severity tiers and look for empty cells before trusting any weighted score. A weighted score can only penalize failures the task set gives the agent a chance to make.

In the worked example, a payment family with no vendor bank-detail changes cannot reveal whether the agent pays a changed account before verification, which is one of the costliest failure paths in accounts payable. Add tasks that open each high-severity path, such as duplicate invoices with different numbering, partial receipts, credit notes and changed remittance details, and mark any cell you cannot fill as a known gap in the report.

Draw the routine tasks from the real workload so the plain rate means something, then raise high-severity families to a minimum count so their error rate has a usable interval. Report the floors and the weights that undo them, so the overall figures still describe the real mix of work.

Reporting the result: two numbers, a gate and an interval

Report the plain completion rate, the severity-weighted loss, per-tier counts with their task families, and the gate result, each with uncertainty. A single blended score invites people to trade a payment error for a tidier summary without noticing.

Evan Miller's paper on adding error bars to evaluations treats evaluation questions as draws from a larger population and gives formulas and recommendations for confidence intervals and for comparing two models on the same questions. Apply that to each family separately: when Agent B makes no S3 errors in 80 payment tasks, that is evidence of a low rate, not proof of zero, and a common approximation puts the 95% upper bound near 3 divided by the number of tasks, here about 3.75%.

Agents also vary between runs. The tau-bench authors introduced pass^k, the chance an agent succeeds on all k independent trials of a task, for exactly this reason. For payment families, run each task several times and count a task as passed only if every run passes, because a payment that is right four times out of five is still wrong in production one time in five.

Sourcing back-office tasks with recorded outcomes

Severity weights are only as good as the cases behind them, and the hardest cases to write by hand are the ones with real outcome records: the reversal, the dispute, the corrected invoice. When internal history runs short, outcome-linked back-office cases can be commissioned: SourceX locates them at U.S. businesses per request, with nothing kept on hand, after a buyer lists scope, volume, language, format, timing and licensing terms on its request form, and each dataset still clears rights and privacy review before delivery.

Frequently asked questions

What are business severity weights in an AI evaluation?

They are multipliers that make an error count in proportion to its business consequence, such as a wrong payment counting many times more than a minor summary slip. Each weight attaches to a written tier with consequence anchors, and the task owner signs off on the ratios between tiers.

Should severity be assigned per task type or per error?

Per error, judged by the decision the output feeds. A summary that drops a withdrawn payment authorization can be critical, while a payment task with a formatting slip can be cosmetic. Task type is a useful default for sampling, not for scoring.

Why not fold severity into one overall score?

A single blended number lets frequent small wins hide rare critical failures. Report the plain completion rate, the weighted loss and the per-tier counts separately, and use a hard gate on critical errors that no average can override.

How many payment tasks are enough to trust a zero critical-error result?

It depends on the rate you need to rule out. With zero failures in n tasks, a common approximation puts the 95% upper bound near 3 divided by n, so 80 tasks only rules out rates above roughly 4%. Size the family with an evaluation statistician against the risk the business will accept.

Can the weights use transaction amounts instead of fixed tiers?

Yes, where the business already measures exposure by value, an amount-aware weight is reasonable and mirrors cost-sensitive learning practice. Keep the fixed-tier gate anyway, because some critical errors, such as paying an unverified bank account, matter regardless of amount.

Sources

  • The NIST AI Risk Management Framework (AI RMF 1.0) describes risk as a composite measure of an event's probability of occurring and the magnitude or degree of its consequences. Source
  • FIPS 199 defines low, moderate and high potential impact as a limited, serious, or severe or catastrophic adverse effect on operations, assets or individuals, and a system's category follows its highest rated objective. Source
  • Regulation E section 1005.11 lists an incorrect electronic fund transfer as an error and requires the institution to investigate and determine whether an error occurred within 10 business days of the consumer's notice, with longer periods only under set conditions. Source
  • A scikit-learn example on cost-sensitive learning treats misclassifying a bad credit as good as five times more costly than the reverse, and builds a fraud metric whose cost depends on each transaction's amount. Source
  • The scikit-learn metrics guide notes that most implementations let each sample contribute a weighted amount to the overall score through the sample_weight parameter. Source
  • The tau-bench paper evaluates agents by comparing the database state at the end of a conversation with an annotated goal state and introduces pass^k to measure reliability across repeated trials. Source
  • Adding Error Bars to Evals treats evaluation questions as draws from an unseen super-population and gives formulas and recommendations for confidence intervals and for comparing two models. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies