AI uses for records
Agent evaluation latency tradeoffs: how to measure downstream rework
By SourceX Editorial · Updated
10 sections
Short answer
Measure downstream rework by timing each task from request to verified completion, not to the first answer. Fix per task family which events count as rework (reopens, repeat contacts, corrections, reversals) and how long to watch for them, then run fast and slow configurations on the same matched tasks, repeat trials, and report median and tail completion time and severity-weighted rework with intervals.
Evaluation split
Agent evaluation latency
Key takeaways
- Response latency only means something when it is split by whether the answer was right.
- The number a business pays for is time to verified completion, which includes reopens, corrections and reversals.
- Define rework events and a watch window per task family before collecting any data.
- Compare configurations on matched tasks with repeated trials, and report the 90th or 95th percentile as well as the median.
- Expect a faster setting to win in some task families and lose in others; a blended average hides both.
How do you measure downstream rework caused by faster AI answers?
Measure time to verified completion for each task, not response time. Start the clock when the request arrives, stop it only when an outcome record shows the work was accepted and stayed accepted, and add every minute of correction, reopening and handoff that followed inside a rework window you fix in advance. Then compare a faster configuration with a slower one on the same matched tasks.
That definition moves the measurement out of the model trace and into the business record. A help desk ticket that an agent closes in eight seconds and a customer reopens two days later did not take eight seconds. It took eight seconds, plus the reopen, plus the second technician, plus whatever the wrong first answer broke on the way.
The rest of this guide turns that definition into a plan an evaluation lead and a business task owner can run together: which clocks to record, what counts as rework, how long to wait, how to compare configurations fairly, and which tasks the plan will miss unless you look for them.
Why a fast response time says little about a wrong answer
A wrong answer can be the fastest answer in the log, so response latency only means something once it is split by whether the answer was right. Google's Site Reliability Engineering book makes the same point for web services: it tells operators to separate the latency of successful requests from the latency of failed ones, because a failed request can come back almost instantly and flatter the average.
The same chapter names a failure mode that maps directly onto AI answers. An implicit failure is a response that returns success but carries the wrong content, and the book notes that only end-to-end checks catch it. A model that confidently answers a billing question with last year's policy is an implicit failure: the trace says success, the customer says otherwise.
Averages hide a second problem. The SRE book's chapter on service level objectives recommends percentiles over means because a healthy-looking average can conceal a long tail of very slow requests. Rework behaves like that tail. Most tasks finish cleanly, and a minority return days later with most of the cost.
Perceived speed is also a poor guide. In METR's randomized trial with experienced open-source developers, allowing early-2025 AI tools increased task completion time by 19 percent, while the developers themselves estimated afterward that the tools had made them about 20 percent faster. A team that asks users whether the faster agent saved time is measuring the wrong thing.
Which clocks should the plan record, and where does each one live?
Record four clocks per task: model time, agent time, human handling time and business completion time. The first two come from telemetry, the last two from the systems where work is tracked, and the gap between them is where rework hides.
OpenTelemetry's generative AI semantic conventions already name several of the telemetry clocks, including time to first token on the server, the end-to-end duration of one agent invocation, and the number of tool and inference calls an agent makes during that invocation. The conventions are marked as in development, so pin the version you instrument against.
| Clock | Starts | Stops | Typical record | What it cannot see |
|---|---|---|---|---|
| Model latency | Request reaches the model | First or last output token | gen_ai.server.time_to_first_token and request duration histograms | Whether the content was correct |
| Agent invocation time | Agent run begins | Final response or error | gen_ai.invoke_agent.duration, plus tool and inference call counts | Anything a person does after the agent stops |
| Human handling time | A person opens the task | That person hands off or closes it | Ticket worklogs, CRM activity time, case audit trail | Waiting time between handlers |
| Time to first close | Request created | First resolved or closed status | Created and resolved timestamps in Zendesk, Jira Service Management, ServiceNow or Salesforce Service Cloud | Reopens, reversals and follow-up contacts |
| Time to verified completion | Request created | Outcome confirmed and unchanged through the rework window | Status history joined to the outcome evidence named in the plan | Rework that surfaces after the window closes |
What counts as rework, and how long should you wait before calling a task done?
Rework is any unplanned effort spent correcting, repeating or undoing work the task was supposed to have finished, and the waiting period should match how long errors in that task family usually take to surface. Write both down per task family before any data is collected, or the definitions will drift toward whichever configuration someone prefers.
At a glance
What counts as rework, and how long should you wait before calling a task done
Software delivery offers a tested analogy. DORA added deployment rework rate to its delivery metrics: the share of deployments that are unplanned and happen because of an incident in production. The idea transfers directly. A rework event is unplanned, it is caused by an earlier output, and it is counted against that output rather than as new work.
- Reopen: the same ticket, case or order returns to an open status after a resolved or closed status.
- Repeat contact: the same requester raises the same issue through a new ticket, chat or call inside the window.
- Correction: a person edits a field, amount, code or document the agent produced, with the edit visible in the audit trail.
- Reversal: a refund, credit memo, cancellation, rollback or voided transaction tied to the original task.
- Escalation after close: the task is routed to a higher tier or a specialist after the first close.
- Silent defect: an error found later by audit, reconciliation or a downstream team, linked back to the task by an identifier.
The end to end latency and rework measurement plan
Use the plan below as a working document. Each row is a decision the evaluation lead and the business task owner sign off together, and each needs evidence a reviewer can inspect later. Version it and date it, because changing a rework window halfway through a comparison invalidates the comparison.
| Plan element | Decision to record | Evidence a reviewer checks | Common failure |
|---|---|---|---|
| Target population | Which requests, channels, customer segments and date range the result applies to | Query or filter that defines the frame, with counts per segment | Testing on a convenient queue and reporting it as the whole workload |
| Task families | The grouping of tasks that share inputs, tools and acceptance rules | A family list with a definition and an example for each | Mixing password resets with outage triage in one average |
| Coverage matrix | Planned task counts per family and severity level, with minimum floors for rare, costly families | Matrix of planned versus achieved counts | Rare families absent, so their rework never appears |
| Completion evidence | The record that proves a task was finished correctly in each family | Named field, status or document per family | Treating a closed status as proof |
| Rework events and window | Which events count as rework and how many days to watch for each family | Event definitions and window lengths, fixed before data collection | Windows shortened after early results look bad |
| Severity rubric | How the business weighs a rework event, from cosmetic to customer harm or financial loss | Rubric with anchored examples, approved by the task owner | Counting a typo fix and a wrong refund as one event each |
| Comparison design | Which configurations are compared and how tasks are matched or randomized between them | Assignment log and matching rules | Faster setting tried only on easy days or easy queues |
| Metrics | Median and 90th or 95th percentile time to verified completion, rework rate, severity-weighted rework, human minutes per task | Metric definitions with formulas and units | Reporting mean response time alone |
| Uncertainty | Interval method, repeated trials per task, and the smallest difference that would change the decision | Intervals per family, not only overall | Declaring a winner on a difference smaller than run-to-run noise |
| Stop and review rules | Conditions that pause the comparison, such as a high-severity reversal | Named reviewer and threshold, written down | Harm discovered in the final report instead of during the run |
How do you compare a faster configuration with a slower one fairly?
Run both configurations on the same matched tasks, repeat each task more than once, and report differences per task family with intervals. Speed changes often come from a smaller model, fewer tool calls, a shorter context or skipped verification steps, and each of those can shift which tasks fail, not only how many.
Matching matters because workload changes by hour, weekday and season. A paired design, where each historical task is replayed against both configurations in a sandbox, removes most of that variation. If live traffic is used instead, assign tasks at random within each family and record the assignment so a reviewer can confirm nothing steered the easy work to one side.
Repetition matters because agents are not deterministic. The tau-bench authors introduced pass^k, the chance that all k independent attempts at a task succeed, and showed that agent reliability falls sharply as k rises even when single-attempt success looks reasonable. A configuration that is faster but less consistent will show its rework in repeated trials before it shows it in production.
Report the tail, not just the middle. Following the SRE practice of setting targets at high percentiles, give the median and the 90th or 95th percentile of time to verified completion for each configuration. A faster setting that improves the median while stretching the 95th percentile has moved cost from the many to the few, and the task owner should see that trade directly.
Know where speed is worth buying at all. Nielsen Norman Group's response-time limits put roughly 1 second as the point where a user's flow of thought stays unbroken and about 10 seconds as the limit for holding attention. Shaving an answer from 6 seconds to 3 may matter for a live chat; it rarely matters for a task whose verified completion is measured in days.
Worked example (illustrative): an IT service desk agent at two speed settings
The figures here are invented to show the arithmetic; they describe no real deployment. An internal IT team compares two settings of the same service desk agent on 400 historical tickets replayed in a sandbox, each run three times. Setting A runs a verification step against the configuration database before answering; setting B skips it.
Setting B answers faster: median agent invocation time falls from 14 seconds to 5. Time to first close falls too. Then the rework window of seven calendar days, chosen because most reopened access tickets in this illustrative team surface within a week, changes the picture.
Under setting B, 9 percent of tickets in the access-request family are reopened or corrected, against 3 percent under setting A, and each reopen adds a median of 35 minutes of technician time. Across that family, setting B's 95th percentile time to verified completion is longer than setting A's, even though its median is shorter. In the password-reset family the rework rates are indistinguishable across the three trials, so setting B is the better choice there.
The exception is the decision the plan exists to surface. The team ships setting B for password resets and keeps setting A for access requests. A single blended number would have hidden both findings.
Which tasks and failure types will this plan miss unless you look for them?
Any task whose errors surface after the rework window, cross a system boundary or never trigger a ticket will look clean. List those gaps in the plan so the result is read with them in mind, and narrow the claim when a gap cannot be closed.
At a glance
Which tasks and failure types will this plan miss unless you look for them
- Slow-burn errors: a wrong tax code or contract term found at quarter-end reconciliation, long after any practical window.
- Cross-team rework: a wrong answer from the support agent creates work in billing or engineering, logged in a different system with no link back.
- Abandoned requests: the customer gives up and never reopens, which reads as success unless churn or repeat-contact data is joined.
- Workarounds: staff quietly fix agent output before sending it, so no correction appears in the audit trail; sample worklogs or interview handlers.
- Rare high-severity families: too few cases to estimate a rate, so report each case individually and route them to the task owner.
- Policy changes during the run: a new refund rule changes what correct means; split the analysis at the change date.
When does the result need a statistician or the task owner rather than a dashboard?
Bring in an evaluation statistician when intervals for the families you care about overlap, when fewer than a few dozen tasks sit in a severity cell, or when tasks were not randomized. Bring in the business task owner whenever a high-severity rework event appears, whenever completion evidence is ambiguous, and before any family's result is generalized beyond the population in the plan.
Stop the comparison outright if a faster setting produces a reversal that reached a customer's money, access or data. The point of the plan is to find those cases in a sandbox or a controlled pilot, not to accumulate enough of them for a clean chart.
Getting resolved-work histories that show rework, not just first answers
This plan depends on records most evaluation sets lack: full status histories, reopen and reversal events, worklogs and outcome evidence linked to each task. Public benchmarks rarely carry them, and synthetic tasks cannot show how real errors return.
SourceX looks for histories with reopen and reversal events intact only when a buyer asks, drawing on U.S. businesses rather than a stock catalog, and the request covers scope, volume, language, format, timing and licensing needs. A finished plan with its task families, rework events and windows is already most of that specification. Most buyers want a few years of steady history, commonly two or more, with tens of thousands of records in a single system; unusual work can qualify with much less.
Frequently asked questions
What is the difference between latency and time to verified completion?
Latency is how long the system takes to respond. Time to verified completion runs from the request to an outcome that was confirmed and stayed unchanged through a fixed rework window. The second includes human handling, reopens and corrections, so it is the number a business actually pays for.
How long should the rework window be?
Long enough to catch most errors in that task family and short enough to report before the decision is due. Base it on how long reopens and corrections have historically taken to appear in your own ticket or order data, set it per family, and fix it before collecting results.
Can I measure rework without production traffic?
Partly. Replaying historical tasks in a sandbox shows correctness and consistency across repeated trials, which predicts rework. Reopens, repeat contacts and reversals only appear with real users, so a controlled pilot with a holdout group is the usual second stage.
Which metrics should go in the final report?
Per task family: median and 90th or 95th percentile time to verified completion, rework rate, severity-weighted rework, and human minutes per task, each with an interval. Add model and agent latency for context, never as the headline.
Why not just ask users whether the faster agent saves time?
Self-reports can point the wrong way. In METR's randomized trial, developers believed AI tools had sped them up by about 20 percent while measured completion time rose by 19 percent. Use timestamps and outcome records, and treat surveys as a secondary signal.
Sources
- Google's SRE book says to distinguish the latency of successful and failed requests, and defines implicit failures as success responses with wrong content that only end-to-end checks catch. Source
- Google's SRE book recommends percentile latency indicators because averages can hide a long tail of slow requests. Source
- OpenTelemetry's generative AI metrics conventions, in development status, define server time to first token, agent invocation duration, and per-invocation tool call and inference call counts. Source
- The tau-bench paper introduces the pass^k metric for agent reliability across repeated trials and reports that reliability falls as k increases. Source
- METR's randomized trial found early-2025 AI tools increased experienced developers' completion time by 19 percent, while developers estimated a 20 percent speedup. Source
- DORA describes deployment rework rate as the ratio of deployments that are unplanned and happen as a result of an incident in production. Source
- Nielsen Norman Group gives about 1 second as the limit for an uninterrupted flow of thought and about 10 seconds as the limit for keeping attention. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.