Skip to content

AI uses for records

Agent evaluation task coverage matrix: what 1,000 tasks still miss

By SourceX Editorial · Updated

9 sections

Short answer

To find coverage gaps in a thousand-task AI evaluation, tag every task by the workflow decision it exercises and the exception it contains, then cross-tabulate. Compare each cell with live workload share and business severity. Empty cells, cells built from one template and high-severity cells too small to bound their failure rate are the gaps; report them per cell with uncertainty instead of one pass rate.

Evaluation split

Agent evaluation task

This page covers agent evaluation task coverage matrix: what 1,000 tasks still miss.Illustrative

Key takeaways

  • Task count says how much was tested; a decision-by-exception matrix says what was tested.
  • Derive rows from the state changes the agent may make and columns from incident and escalation history, not from the suite itself.
  • Judge high-severity cells by the upper bound of their failure rate and count independent task families, not template variants.
  • Run several trials in tier 1 cells and report pass^k beside single-trial success.
  • List empty cells and their compensating controls in the report rather than hiding them in an average.

How do you find coverage gaps in a 1,000-task AI evaluation?

Stop counting tasks and start counting cells. Tag every task in the suite with the business decision it exercises and the exception it contains, cross-tabulate the tags into a matrix, and compare each cell with how often that situation occurs in the live workflow and how much damage a wrong action there would do. Empty cells, thin cells and cells dominated by easy variants are your coverage gaps.

A task count answers how much was tested. A coverage matrix answers what was tested, which is the question a buyer, a risk owner or a deployment lead actually has to sign off on. One thousand tasks drawn from the same happy path can tell you less than two hundred tasks spread deliberately across the decisions and failure modes that matter.

The method has four steps: define the target workload, tag the suite, compare the tags with the workload and with a severity rubric, then report each cell with its own score and uncertainty. The rest of this page gives the tags, the matrix, the sample size rule and a worked audit you can copy.

Why a big task count can still leave whole decisions untested

Large suites tend to inherit the shape of whatever source was easiest to harvest. If tasks were generated from a sample of closed tickets, the common resolutions dominate; if they were written by engineers, the exceptions engineers find interesting dominate. Neither shape matches the work an agent will meet in production.

Benchmark designers have said this openly. The authors of the HELM benchmark describe a two-level design: a broad taxonomy of scenarios and metrics, and the smaller subset they actually implemented, and they argue that writing the taxonomy down first makes what the subset leaves out explicit. The same discipline applies to a private agent suite: list what should be covered before you trust what is covered.

Software testing offers the second lesson. The CheckList method for language models organizes tests as a matrix of capabilities by test type, and its authors report that held-out accuracy often overstates how well models perform. A grid makes an empty square visible in a way a long list of passing tests never does.

Finally, agent behavior is not stable across attempts. The tau-bench paper defines pass^k as the chance that an agent succeeds on all k trials of the same task and reports that agents which look acceptable on one trial become much less consistent over several. A cell covered by one task run once is barely covered at all.

Choosing the two axes: workflow decisions down, exceptions across

Put workflow decisions on the rows and exception types on the columns, because those are the two things a deployment owner reasons about. A decision is a point where the agent commits to an action that changes state in a system of record: approve, hold, route, reject, refund, escalate, close. An exception is a condition that makes the normal rule for that decision unsafe or wrong.

Derive the decision rows from the workflow, not from the suite. Walk the process with the person who owns it and list every state change the agent is allowed to make. In a ticketing system such as Zendesk or Jira that is usually the set of status transitions and field updates the agent can perform; in an ERP such as NetSuite it is the set of approval, hold and posting actions. If the suite has no task for a state change the agent is permitted to make, that row is a gap before you look at any column.

Derive the exception columns from incident history, policy documents and escalation queues. Most operational workflows share a small family of exception types, listed below; rename them to match your process, but keep the list short enough that every cell can be reviewed by a person.

Keep a third tag, the task family, for reporting and uncertainty. Tasks built from the same template, the same customer account or the same source document are correlated, and they should be treated as one cluster when you compute error bars.

  • Missing input: a required field, attachment or approval is absent.
  • Conflicting records: two systems disagree, such as an invoice and a purchase order with different quantities.
  • Policy override: a rule normally applies but a documented exception, limit or authority changes it.
  • Stale or out-of-order state: the record changed after the request was made, or events arrived in the wrong sequence.
  • Ambiguous or shifting request: the user is unclear, changes the goal midway or asks for two incompatible things.
  • Tool or system failure: an API errors, times out or returns partial data.
  • Adversarial or suspicious input: an instruction embedded in content, a changed bank detail, an impersonation attempt.
  • No exception: the clean path, kept as a column so its share of the suite is visible.

The coverage matrix by workflow decision and exception

Use one row per decision-by-exception cell and keep every column filled; a blank is itself a finding. The table below is the template, populated with illustrative rows from an accounts payable agent so the columns are concrete. Counts and shares are illustrative, not observed.

Read the status column as a decision rule. A cell is Covered when it has enough independent task families and trials for its severity tier, Thin when it has tasks but too few to bound the error rate, Skewed when its tasks are all easy variants of the situation, and Missing when it has none. Severity tiers come from the business owner: tier 1 means an irreversible or regulated action, tier 2 a costly but reversible error, tier 3 a nuisance.

Add the input dictionary beside the matrix so the next auditor can reproduce it: task id, decision tag, exception tag, task family, source of the task (production record, edited record, synthetic), grader type (end-state check, rubric, judge model), trials run, and the date the source records were captured. Without the source and date fields you cannot tell whether a cell is covered by realistic cases or by a template someone wrote in an afternoon.

The coverage matrix by workflow decision and exception
Decision x exception cellTasks / families in suiteShare of live workloadSeverity tierStatus and next action
Approve and pay x no exception540 tasks / 12 familiesAbout 70 percentTier 3Covered; sample down, do not add more
Approve and pay x conflicting records (PO quantity mismatch)85 tasks / 6 familiesAbout 9 percentTier 2Covered; check grader reads the posted record
Place on hold x missing input (no receipt)60 tasks / 2 familiesAbout 6 percentTier 2Thin; two templates only, add families from other vendors
Approve and pay x adversarial input (changed bank details)0 tasksUnder 1 percentTier 1Missing; source real or carefully edited cases before any sign-off
Route for approval x policy override (amount above limit)12 tasks / 1 familyAbout 3 percentTier 1Thin and skewed; one template, all just over the limit
Reject x stale state (invoice already paid)4 tasks / 1 familyAbout 2 percentTier 1Thin; duplicate payment risk, raise to the tier 1 floor
Escalate x tool failure (ERP timeout)0 tasksUnknownTier 2Missing; also measure the live rate before weighting
Any decision x ambiguous request38 tasks / 3 familiesAbout 4 percentTier 3Covered for tier 3; review whether clarifying questions are graded

How many tasks does one cell need before its score means anything?

Set a minimum per cell from its severity tier, and judge a cell by the upper bound of its failure rate rather than its point estimate. A cell where 20 of 20 tasks pass has not shown that the agent never fails there; it has shown that the failure rate is probably below a bound that depends on the count.

At a glance

How many tasks does one cell need before its score means anything

A useful rule of thumb is the rule of three: if n independent tasks all pass, an approximate 95 percent upper bound on the true failure rate is 3 divided by n. Twenty clean passes bound the failure rate near 15 percent; one hundred bound it near 3 percent. For a tier 1 cell, decide in advance what upper bound the business can accept, and size the cell to reach it. Count independent task families, not near-duplicate tasks, because ten variants of one template do not behave like ten independent cases.

Report each cell with an interval that respects that structure. The Inspect evaluation framework, for example, can compute a metric separately for each group defined in sample metadata, offers a Wilson score interval for binary scores, and supports clustered standard errors through a cluster parameter; tag your tasks with decision, exception and family fields and you get per-cell scores and family-clustered uncertainty from the same run.

Run more than one trial per task in high-severity cells. A single pass hides inconsistency, and the pass^k view from tau-bench is the right one for any decision where one bad attempt in production is unacceptable.

  • Tier 1 cells: a floor of independent families set by the acceptable upper bound, at least several trials per task, and per-cell reporting only, never folded into an average.
  • Tier 2 cells: a smaller floor, reported per cell and in a weighted total.
  • Tier 3 cells: proportional to workload share; aggregate reporting is acceptable.
  • Any cell below its floor is reported as Thin with its interval, never as a pass.

Which business task or failure type is absent from the evaluation?

The gaps that matter most are usually the ones nobody tagged, so look outside the suite. Compare the matrix with four records of what actually happens: incident and postmortem reports, the human escalation queue, policy and limit changes from the last year, and a fresh random sample of live cases tagged by the process owner. Any decision or exception that appears in those records but not in the matrix is a missing row or column, not just a missing cell.

Write the gaps down even when you cannot fill them yet. The NIST AI Risk Management Framework playbook asks organizations to document risks identified during mapping that will not be measured, with a justification for not measuring them. An evaluation report that lists its own empty cells, with the reason and the compensating control, is far more useful to a buyer than one that reports a single high score.

Watch for three quiet failure types that suites routinely miss: actions that are correct but taken on stale data, refusals or escalations that should have been actions, and multi-step sequences where each step passes but the end state is wrong. Each needs its own column or its own grader check, because a final-message grader will not see them.

Illustrative audit: a 1,000-task accounts payable agent suite

Here is how the audit plays out on a hypothetical suite; every number in this section is illustrative. A team has 1,000 tasks for an agent that processes supplier invoices: matching, holding, routing for approval, rejecting and paying. The headline pass rate is high, and the deployment lead is asked to approve a pilot.

Tagging takes two people about two working days. The result: 540 tasks sit in the clean approve-and-pay cell, built from 12 templates. Only 16 tasks touch any tier 1 cell, and all of them come from just two templates. There are zero tasks with a changed bank detail, zero with an ERP timeout, and four with an invoice that was already paid.

Comparing with a sample of 300 recent live invoices tagged by the AP supervisor shows the clean path is about 70 percent of real work, not the 54 percent of the suite it occupies, so the suite is not even skewed toward easy cases overall. The problem is concentration: almost all exception tasks reuse a handful of templates, and the riskiest decisions have no independent evidence.

The decision changes from approve to approve with conditions. The pilot can run on clean invoices with human review of every hold, rejection and approval routing; the suite gets new families for the three tier 1 cells before the agent may act alone there; and the evaluation report lists the two Missing cells with the compensating control (human approval) until they are filled.

Which assumption flips the verdict, and how to show it

Three assumptions most often change the verdict: the workload weights, the severity tiers and the validity of the grader. Test each one explicitly instead of reporting a single number.

Workload weights decide what a weighted total means. If they come from last quarter and a new supplier or a seasonal peak changes the mix, the total moves even when the agent does not. Show the total under at least two plausible mixes.

Severity tiers decide which cells need floors. Ask the business owner, not the evaluation team, to assign them, and record who signed. A cell moved from tier 2 to tier 1 can turn a pass into a hold.

Grader validity decides whether a pass means the business outcome happened. A grader that reads the agent's final message rather than the record in the system of record will pass an agent that says it placed a hold but did not. Spot-check a sample of passes in every tier 1 cell against the actual end state before trusting the cell.

Which assumption flips the verdict, and how to show it
AssumptionHow to test itWhat to show in the report
Workload weightsRecompute the weighted total under the current mix and a stressed mixBoth totals side by side with the dates of each mix
Severity tier per cellHave the process owner review and sign the tier listTier list with the signer and any disputed cells
Grader reads the true end stateManually audit a sample of passes per tier 1 cellAgreement rate between grader and audit, per cell
Task families are independentCheck source records and templates for shared accounts or documentsFamily counts and clustered intervals, not raw task counts
Trials capture inconsistencyRun several trials in tier 1 and tier 2 cellspass^k next to single-trial success for those cells

Filling empty cells with real operational cases

The fastest way to fill a Missing cell is with real historical cases where the exception actually occurred, edited only enough to remove personal data. Synthetic cases help with volume, but they tend to reproduce the author's idea of the exception rather than the messy record trail a real one leaves.

If your own operations do not produce enough of a rare exception, the cases have to come from businesses whose records do. SourceX looks for such cases at U.S. businesses only once a buyer asks, so the request starts by describing the cells to fill: the scope, volume, language, format, timing and licensing terms needed. The business that holds those cases approves what is in scope, and on what terms, and signs before any case is handed over.

Frequently asked questions

What should a coverage matrix by workflow decision and exception include?

Rows for every state-changing decision the agent may take, columns for each exception type plus the clean path, and in each cell the task and family counts, the live workload share, a severity tier and a status. Keep an input dictionary with task source, grader type, trials and capture date so the matrix can be reproduced.

Is 1,000 tasks enough for an AI agent evaluation?

It depends on how the tasks are spread, not on the total. A thousand tasks concentrated in one clean path can leave the riskiest decisions with no evidence, while a few hundred tasks spread across decisions and exceptions can support a real sign-off. Check each high-severity cell against its own floor.

How many tasks does a rare but critical cell need?

Work back from the failure rate you can accept. If n independent cases all pass, an approximate 95 percent upper bound on the failure rate is 3 divided by n, so bounding it near 3 percent takes about 100 independent cases. Count task families, not template variants.

Should coverage cells be weighted by live traffic?

Use traffic weights for the aggregate score and severity floors for individual cells. Weighting alone lets a rare tier 1 cell vanish into the average, so always report high-severity cells on their own with intervals.

How do I find failure types that are missing from the suite?

Compare the matrix with incident reports, the human escalation queue, recent policy and limit changes, and a fresh sample of live cases tagged by the process owner. Anything in those records that has no row or column in the matrix is a structural gap.

Sources

  • The HELM authors pair a broad taxonomy of scenarios and metrics with the subset they implemented, and argue that a stated taxonomy makes what an evaluation leaves out explicit. Source
  • The CheckList paper structures behavioral tests as a matrix of capabilities by test types and notes that held-out accuracy often overestimates model performance. Source
  • The CheckList library defines minimum functionality, invariance and directional expectation tests, each tagged with a capability. Source
  • The tau-bench paper defines pass^k as success on all k trials of a task and reports that agents are inconsistent across repeated trials. Source
  • Inspect can compute a metric per metadata group, offers a Wilson score interval for binary scores and supports clustered standard errors. Source
  • The NIST AI RMF playbook asks organizations to document identified risks that will not be measured, with a justification for not measuring them. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies