AI uses for records
Agent evaluation and workflow transitions: test coverage between states
By SourceX Editorial · Updated
10 sections
Short answer
Test transition coverage by building a state table from the system of record's status history, listing each real transition with its frequency and the cost of getting it wrong, and counting the evaluation tasks that exercise each one. Report covered, thin and missing transitions, weighted by frequency and severity, because a dataset can fill every field and still test only the path from open to closed.
Evaluation split
Agent evaluation workflow
Key takeaways
- Field coverage and transition coverage fail independently; a dataset can fill every field and test only the happy path.
- Build the state table from workflow configuration and status-change history, and have the workflow owner review it.
- Audit one row per transition with frequency, severity, task count and a verdict of covered, thin or missing.
- Grade end state and, where the path matters, the sequence of state changes; no-change tasks can be passed by doing nothing.
- Report small transition cells as counts with wide intervals, not percentages with normal-approximation error bars.
How do you test whether a dataset covers transitions between business states?
Test transition coverage by building a state table from the system of record's status history, listing every transition real work takes (from state, to state, how often, how costly when wrong), and then counting how many evaluation tasks exercise each one. A transition with no tasks is a blind spot, a transition with two tasks is a guess, and a dataset can populate every field while covering only the happy path from open to closed.
The audit is short to run once you have two inputs: an event history of status changes from production, and the list of states each evaluation task starts in, passes through and must end in. The output is a table that shows, transition by transition, where the evaluation can and cannot tell you how an agent behaves.
The examples on this page, including every count and share, are illustrative. They show the method, not measurements of any real workflow or agent.
Why field coverage is not transition coverage
Field coverage asks whether each column is populated and varied; transition coverage asks whether the evaluation includes cases where the work moves from each state to each valid next state. They fail independently. An evaluation set of support tickets can have every priority, channel, product and tag, and still contain only tickets that went from new to solved in one reply.
Agents fail at transitions because transitions are where rules apply. Moving a ticket to pending implies a question was asked of the customer. Reopening a solved ticket implies the earlier resolution failed. Moving an invoice from on-hold to approved implies someone with authority cleared the hold. If none of those moves appear in the evaluation, a high score says nothing about the decisions that carry the most risk.
Software testing has a name for this. The ISTQB glossary defines state transition testing as a black-box technique in which test conditions are state transitions or sequences of them. Evaluating a business agent is the same problem with messier states, and the same coverage ideas apply.
Build the state table from the system of record
Take the states and allowed transitions from the system's own workflow configuration and the observed transitions from its change history, then reconcile the two. Configuration tells you what is allowed; history tells you what happens and how often.
Most business systems expose both. Atlassian describes a Jira workflow as a set of statuses and transitions that a work item moves through, configured per project. Zendesk's Ticket Audits API returns a read-only history of all updates to a ticket, and its audit events reference describes change events that carry the previous and new value of each ticket property, so status changes can be extracted with both ends of the move. Salesforce field history, ERP document status logs and process mining event logs serve the same purpose for other workflows.
Keep states at the level where business rules differ. If pending-customer and pending-vendor follow different rules, they are two states even if the system stores one status with a reason field. If two statuses behave identically, merge them. The state table should be reviewed by the person who owns the workflow, because they know which distinctions matter.
| State (illustrative support workflow) | Meaning | Allowed next states | Rule that governs leaving it |
|---|---|---|---|
| New | Ticket created, not yet triaged | Open, Solved | Triage assigns owner and category |
| Open | Being worked by an agent | Pending, On-hold, Solved | Agent must act or ask a question |
| Pending | Waiting on the customer | Open, Solved | Customer reply or follow-up policy |
| On-hold | Waiting on a third party or internal team | Open | Dependency resolved or escalation owner acts |
| Solved | Resolution sent | Open (reopen), Closed | Customer reply reopens; otherwise closes after policy period |
| Closed | Locked for edits | None in this workflow | Follow-ups create a new ticket |
The transition coverage audit: one row per transition
The audit table is the asset: one row per transition, with how often it happens in production, how costly an error on it is, how many evaluation tasks exercise it, and a verdict. Sort it by severity first and frequency second, because a rare transition with a high error cost can matter more than a common one.
At a glance
The transition coverage audit: one row per transition
Fill production share from the change history over a recent stable period. Fill severity with the workflow owner, using a short ordinal scale tied to business consequences such as a wrong refund, a missed contractual deadline or a lost customer reply. Count evaluation tasks by replaying each task's expected path, not by reading task titles.
Add a row for every transition the history shows but the configuration does not allow, and for every allowed transition the history never shows. The first kind usually means an administrator moved records by hand or a migration rewrote statuses, which the evaluation should neither reward nor imitate. The second kind may be dead configuration, or a rare but legitimate move that deserves a deliberately constructed task.
| Transition (from to to) | Production share (illustrative) | Severity if wrong | Eval tasks exercising it | Verdict |
|---|---|---|---|---|
| New to Open | High | Low | 96 | Covered |
| Open to Solved | High | Medium | 88 | Covered |
| Open to Pending | Medium | Medium | 14 | Covered |
| Pending to Open | Medium | Medium | 6 | Thin |
| Open to On-hold | Low | High | 2 | Thin, high risk |
| On-hold to Open | Low | High | 0 | Missing, high risk |
| Solved to Open (reopen) | Low | High | 0 | Missing, high risk |
| Pending to Solved (no reply policy) | Low | Medium | 1 | Thin |
| New to Solved (duplicate or spam) | Low | Low | 3 | Thin, acceptable |
Which coverage numbers to report
Report four numbers and the table, not one score. A single coverage percentage hides the distribution that matters.
Each number answers a different question, and a set can look strong on one while failing another. Weighted coverage is usually high because common transitions are easy to include; severity-weighted coverage is where gaps show.
- Transition coverage: share of valid transitions in the state table with at least a minimum number of tasks, for example ten. In the illustrative audit, 3 of 9 transitions meet that bar.
- Frequency-weighted coverage: share of production transitions, weighted by how often each occurs, that fall on covered transitions. This is what a reader usually assumes coverage means.
- Severity-weighted coverage: the same calculation weighted by severity instead of frequency, which exposes rare, costly moves.
- Sequence coverage: share of observed two-step sequences, such as Open to Pending to Open, that appear in at least one task. Many failures live in loops rather than single moves.
- Invalid-transition tests: number of tasks in which the correct behavior is to refuse or not make a move the rules forbid, such as editing a closed ticket.
Grade the transition, not only the end state
End-state grading is necessary but does not prove the right transition happened. Compare the final state with the expected state, and for transitions where the path matters, also check the sequence of state changes the agent produced.
Public agent benchmarks show both the strength and the limit. The tau-bench paper evaluates agents by comparing the database state at the end of a conversation with the annotated goal state, and its released code computes reward by hashing the environment's data after the episode and comparing it with the hash produced by replaying the ground-truth actions. A later analysis, Establishing Best Practices for Building Rigorous Agentic Benchmarks, reports that on tau-bench's airline tasks a trivial agent that returns empty responses passed 38 percent of tasks, because tasks that require no database change and no specific response text can be passed by doing nothing.
The lesson for transition coverage is direct. Tasks whose expected end state equals their start state, such as correctly leaving a ticket on hold, need an explicit check that no forbidden move happened and that the required action, such as a note to the customer, was taken. Tasks that pass through an approval state need a path check that the approval state was entered, not only that the final state is right.
Worked example: auditing an illustrative support-agent evaluation set
Suppose a buyer receives an illustrative evaluation set of 120 support tasks for a ticket-handling agent. The set's documentation shows good field coverage: every product line, channel and priority appears, and free-text quality is high. The transition audit tells a different story.
Replaying expected paths shows that 96 tasks start at New and most end at Solved after a single Open step. Only 2 tasks touch On-hold, none return from On-hold, and none involve a reopen. The workflow owner rates both of those moves as high severity: a ticket left on hold after the dependency resolves breaches response commitments, and a mishandled reopen tells the customer their problem is fixed when it is not.
The audit produces three actions. Add tasks that start mid-workflow, from a real ticket already in On-hold or Solved, so the agent faces the transition directly instead of hoping a conversation wanders there. Add sequence tasks for the Open to Pending to Open loop. Add two invalid-transition tasks where the right behavior is to create a follow-up ticket instead of editing a closed one. Then rerun the audit and report the before and after tables side by side.
One exception is worth recording rather than fixing. The audit shows only 3 tasks for New to Solved, which covers duplicates and spam. The workflow owner rates that move low severity and the agent will not be deployed on spam handling, so the buyer marks the row thin but acceptable and states the exclusion in the evaluation's scope. An audit that forces every row to the same count wastes tasks on moves nobody will rely on.
How to report uncertainty when cells are small
Report per-transition results as counts with intervals, and do not rely on normal-approximation error bars when a cell has a handful of tasks. Most transition cells in a real audit are small, which is the point of the audit.
A position paper by Bowyer, Aitchison and Ivanova argues that uncertainty methods based on the central limit theorem are appropriate for benchmarks with thousands of examples but usually underestimate uncertainty on smaller, specialized ones, and recommends alternative frequentist or Bayesian methods. A transition with six tasks and five passes should be reported as five of six with a wide interval, not as 83 percent.
Run each task several times when the agent is stochastic. The tau-bench paper introduces pass^k, the probability that all k trials of a task succeed, as a reliability measure distinct from single-run success. For high-severity transitions, a buyer usually cares more about pass^k than about average success. Record the number of trials per task next to every reported cell, so readers can tell a single lucky run from a repeated, stable result.
Which missing transitions should shrink the evaluation's claim?
Stop and narrow the claim when a high-severity transition has no tasks and you cannot obtain real cases for it soon. State the evaluation's scope as the transitions it covers and exclude the rest from any claim about the agent.
Ask for specialist review when the state table itself is in doubt. If the workflow owner and the system configuration disagree about which moves are allowed, or if a transition's severity depends on contract terms or regulation, the people who own those rules should set the table before anyone counts tasks.
Do not fill a missing transition with synthetic conversations alone if the transition depends on real-world consequences, such as a vendor responding late or a customer reopening after a partial fix. Synthetic tasks can test the mechanics; real records show the conditions under which the move actually happens.
Where evaluation records with real transitions come from
Transition coverage depends on records that keep history: status change logs, ticket audits, approval trails and event logs, linked to the outcome of each case. Current-state exports cannot supply it, because they show where each case ended and erase how it got there.
When you specify an evaluation dataset, include the state table and the transition audit in the request, so the supplier knows which moves must be represented and at what minimum counts. Ask for mid-workflow starting states and for cases that loop or reopen, and require the history fields needed to replay each path.
SourceX can source records with intact status history from U.S. businesses when a buyer asks; it does not keep datasets on the shelf. Its request form covers scope, volume, language, format, timing and licensing needs, and attaching a transition coverage table states scope more precisely than any field list.
Frequently asked questions
What should the transition coverage audit and state table include?
The state table lists each business state, its meaning, the allowed next states and the rule for leaving it. The audit adds one row per transition with production frequency, severity if handled wrongly, the number of evaluation tasks that exercise it, and a verdict of covered, thin or missing. Both should be reviewed by the workflow owner.
Which transitions do evaluation sets most often leave out?
Look at the audit rows marked missing or thin with high severity. Reopens, returns from hold, approval loops and refusals of forbidden moves are the usual gaps, because evaluation sets built from finished cases tend to start at creation and end at resolution. Each gap names the kind of task to add.
When should a buyer narrow the evaluation's claim instead of adding tasks?
Narrow the claim when a high-severity transition has no tasks and no real cases are available soon. Request specialist review when the allowed transitions or their severity depend on contracts, regulation or disputed workflow rules. In both cases report coverage limits alongside the scores.
Is transition coverage the same as process mining?
No, but they share inputs. Process mining discovers how work flows from event logs; transition coverage uses the same kind of log to check whether an evaluation set represents those flows. A process mining tool can produce the observed transitions and frequencies the audit needs.
How many tasks does each transition need?
There is no universal number. Set a minimum per transition based on the decision the evaluation supports, report every cell as passes out of tasks with an interval, and expect small cells to give wide intervals. High-severity transitions deserve more tasks and repeated trials.
Can end-state grading alone measure transitions?
Only partly. Comparing the final state with the expected state confirms where the case ended, but tasks that should leave the state unchanged can be passed by doing nothing, and tasks that must pass through an approval state need a path check. Use end-state and path checks together where the path matters.
Sources
- The ISTQB glossary defines state transition testing as a black-box test technique in which test conditions are state transitions or sequences of them. Source
- Atlassian describes a Jira workflow as a set of statuses and transitions that a work item moves through. Source
- Zendesk's Ticket Audits API returns a read-only history of all updates to a ticket, with each audit holding a list of events. Source
- Zendesk's ticket audit events reference describes change events that record the previous and newly updated value of each ticket property. Source
- The tau-bench paper evaluates agents by comparing the database state at the end of a conversation with the annotated goal state and introduces pass^k as a reliability metric over k trials. Source
- tau-bench's released environment code computes reward by comparing a hash of the data after the episode with the hash produced by replaying the ground-truth actions, and checks required outputs where a task lists them. Source
- Establishing Best Practices for Building Rigorous Agentic Benchmarks reports that a trivial agent passed 38 percent of tau-bench airline tasks because tasks without database changes or required text can be passed by doing nothing. Source
- Bowyer, Aitchison and Ivanova argue that CLT-based uncertainty methods suit benchmarks with thousands of examples but usually underestimate uncertainty on small, specialized benchmarks. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.