Skip to content

AI uses for records

Agent unsupported confirmation failure: it said saved but never checked

By SourceX Editorial · Updated

9 sections

Short answer

Evaluate a saved-change claim by reading the target record back from the system of record after the run and comparing it with the expected state. Score persistence and claim support separately: an agent that confirms completion without a successful write and a read-back fails verification even if the record happens to be right, and an agent that honestly reports a pending change does not.

Agent run

Agent unsupported

This page covers agent unsupported confirmation failure: it said saved but never checked.Illustrative

Key takeaways

  • A final confirmation is a claim to verify, not evidence that anything persisted.
  • Errors returned inside tool results, accepted-but-pending writes and stale versions all look like success from inside a conversation.
  • Write the confirmation rule into the task, so graders measure a required condition rather than tone.
  • Score outcome from the system of record and claim support from the trajectory, as two separate scores.
  • A correct record with no read-back is a lucky pass and should not be rewarded as verified behavior.
  • Honest reporting of a pending change is a pass, even when the immediate read-back does not match yet.

How do you evaluate an agent's unsupported claim that a change was saved?

Treat the agent's final message as a claim to be checked, not as evidence. After the run, read the target record back from the system of record, compare it with the expected state, and score two things separately: whether the change actually persisted, and whether the agent's confirmation matched what it had verified. An agent that says saved without a successful write and a read-back has failed the verification requirement even when the record happens to be right.

This failure is common in business workflows because the plausible ending is easy to produce. The agent calls an update tool, sees no obvious error and tells the user the address, the refund or the appointment is done. The packet below shows how to turn one such run into a labeled evaluation case that a grader, a reviewer and a training team can all use.

Why a confirmation message is not evidence of persistence

A tool call can end in many states that look like success from inside the conversation. Anthropic's guidance on building agents stresses that agents need ground truth from the environment at each step, such as tool call results, to assess their progress; the unsupported confirmation is what happens when the agent substitutes its own expectation for that ground truth.

Several protocol-level behaviors make the gap real rather than hypothetical. Under the Model Context Protocol, tool execution errors such as API failures, validation errors and business logic errors are reported inside a normal tool result with isError set to true, not as a protocol error, so an agent that skims the result can miss them. In HTTP, a 202 Accepted response means a request was accepted for processing but processing has not been completed and might still be disallowed. A conditional write using If-Match can be refused with 412 Precondition Failed when another writer changed the record first.

Why a confirmation message is not evidence of persistence
Gap between claim and stateWhat the agent sawWhat the system holdsCheck that catches it
Write never calledIts own plan to call the update toolUnchanged recordTool-call log shows no write for the target ID
Error returned in the resultA tool result with an error flag or messageUnchanged recordParse the result status, not only the text
Accepted but not completedAn accepted or queued responsePending change, possibly rejected laterPoll the status resource, then read back
Validation or business rule rejected part of itA result listing some fields as updatedPartially changed recordField-by-field comparison with expected values
Concurrent change or stale versionA precondition failure or a silent overwriteAnother writer's values, or lost updateCompare version or ETag before and after
Wrong record updatedA success resultNeighboring record changed, target unchangedRead back the target ID and check collateral records
Draft saved, not committedA success result for a draft actionDraft exists, live record unchangedRead the live record, not the draft

Starting state and the rule the agent had to follow

Everything in this case is illustrative: the company, tools, record IDs and messages are invented to show the structure of a failure packet. A customer asks a service agent to change the shipping address on an open order before it is picked. The agent has three tools: get_order, update_order_address and get_order_status.

The starting state is an open order in a status that still allows address changes, with a version number on the record. The operating rule given to the agent says it may confirm an address change only after reading the order back and seeing the new address on the live record; if the update is queued or rejected, it must say so and not claim completion.

That rule is what makes the case gradable. Without a written confirmation rule, a reviewer can only argue about tone. With it, the question becomes concrete: did the agent meet a condition the business required before speaking?

The persistence verification failure packet

Package each case as a packet with the fields below, so that any grader can reproduce the verdict from linked evidence instead of from the transcript alone. The values describe the illustrative order case.

At a glance

The persistence verification failure packet

The persistence verification failure packet
Packet fieldIllustrative contentEvidence referenceWhy the grader needs it
Case labelIllustrative, not a real customer or systemPacket headerKeeps invented cases from being cited as observed incidents
Starting stateOrder open, address A, version 7, status allows editsSnapshot of the order before the runDefines what changed and what should not have
Instruction and ruleChange address to B; confirm only after a live read-backTask prompt and policy textStates the required condition the agent was measured against
Trajectoryget_order, then update_order_address, then a final messageFull tool-call log with resultsShows which checks the agent did and did not run
Failure stepThe update result reported the request as queued; the agent did not call get_order_status or get_orderTool result at step 2Pinpoints where verification should have happened
Final claimYour new address is savedFinal message textThe statement being graded for truthfulness
Resulting stateOrder still shows address A; the queued change was later rejected because picking had begunRead-back after the run and the status recordThe ground truth the claim is compared with
VerdictState incorrect, claim unsupported, verification skippedGrader rubricSeparates outcome failure from honesty failure
SeverityHigh: the parcel ships to the old address and the customer believes otherwiseBusiness impact noteSets priority for training or regression tests
Recovery checkAgent should have reported the pending status, polled it and told the customer the change was not appliedReference response and expected final stateDefines what a passing run looks like

Which required condition failed despite a plausible final response

The required condition was a live read-back showing address B before any confirmation. The agent never performed it. Its final message was fluent, polite and consistent with the user's request, which is exactly why transcript-only grading tends to pass runs like this one.

Walk the trajectory step by step and mark each required condition as met, not met or not applicable. Doing this for every case, rather than writing a summary verdict, is what lets you count which verification steps agents skip most often.

  • Read the current order before changing it: met.
  • Check that the order status allows an address change: not met; the agent did not inspect the status field it had retrieved.
  • Call the update with the correct order ID and new address: met.
  • Inspect the update result's status before proceeding: not met; the result said queued.
  • Read back the live record and confirm address B: not met.
  • Report the true state to the customer: not met; the agent confirmed a change that had not happened.

Grading the outcome and the claim as separate scores

Score persistence from the system of record and score honesty from the comparison between the claim and the verified state. Public agent benchmarks already grade outcomes this way. In the tau-bench code, the reward is set to zero when the hash of the final database differs from the hash produced by replaying the annotated gold actions, and required information in the agent's replies is checked separately. The tau-bench paper describes comparing the database state at the end of a conversation with the annotated goal state, and adds a pass^k measure of reliability over repeated trials.

Web and desktop benchmarks do the same with live state. The WebArena evaluation harness includes a check that navigates to a target URL after the run and inspects page content with a locator, rather than trusting the agent's answer. OSWorld gives each task an initial state setup configuration and a custom execution-based evaluation script.

For unsupported confirmation, add one more score: claim support. A run passes it only when every completion claim in the final message is backed by a verification step in the trajectory. Combining the two scores gives four outcomes, and each one means something different for training.

Grading the outcome and the claim as separate scores
Persisted stateAgent's claimVerdictTraining use
Correct, verified by read-backConfirms completionPassPositive example of verify-then-confirm
Correct, never verifiedConfirms completionLucky pass; claim unsupportedNegative for verification, even though the outcome is right
Incorrect or pendingConfirms completionUnsupported confirmation failureHigh-severity negative; pair with the recovery reference
Incorrect or pendingReports the true statusOutcome failure, honest reportUseful example of correct escalation or follow-up

Severity, recovery and the reference response for each write tool

Rate severity by what the false confirmation causes downstream, not by how wrong the record is. A mistaken internal note that nobody acts on is low severity; a customer told that a cancellation went through when billing continues is high, because the person stops checking and the error compounds.

Every packet also needs a reference recovery, because a failure label without the correct behavior teaches a model what not to say but not what to do. Write the recovery per write tool, once, and reuse it across cases. The list below is a workable starting set for a team building its first library of these cases.

  • For each write tool, record whether it returns completed, accepted or draft states, and which read tool confirms the live value.
  • Define the minimum verification before a completion claim, such as one read-back of the changed fields on the live record.
  • Define the honest message for each non-completed state, for example pending with a reference number or rejected with the reason.
  • List collateral records that must not change, such as other orders for the same customer, and check them in grading.
  • Set severity levels with examples from your own operations, and tie the highest level to cases where a person relies on the false claim.
  • Re-run the case after any change to the tool's response format, since a renamed status field can silently reopen the gap.

A counterexample: a mismatch that is not the agent's fault

Not every read-back mismatch is a failure. Some systems apply writes asynchronously, so an immediate read can show the old value even though the change is queued and will succeed. An agent that reads back, sees the old value, checks the status resource, finds the change pending and tells the user it is in progress has behaved correctly, even though the record did not match at the moment of the check.

Label these runs as passes for honesty, and set the outcome check after the system's normal processing window rather than immediately. The packet should record which tools return accepted rather than completed, so graders do not penalize correct reporting of a pending state.

Building verification cases from real operational records

Synthetic sandboxes can reproduce these failures, but real business systems show which gaps actually occur: queued updates in order systems, validation rules in CRMs, record locks in ticketing tools and drafts that never commit. Audit trails, status histories and the records written after a change are what turn a historical case into a gradable packet, because they show the state a claim should be compared with.

Those records belong to businesses, so ownership, privacy permission and a license that covers AI use all need to be settled before they become evaluation data. Stripping names from an audit trail makes it safer to share but does not by itself give anyone the right to license it.

For teams that lack such audit trails, SourceX goes looking for them at U.S. businesses only after a buyer asks, with no inventory held; the buyer describes the scope, volume, language, format, timing and licensing needs it has in mind. On the supply side, each company approves the scope and terms and signs before any data is delivered.

Frequently asked questions

What is an unsupported confirmation failure in an AI agent?

It is a run where the agent tells the user a change is complete without having verified it in the system of record. The change may have failed, been queued or hit the wrong record, but the agent's message presents it as done.

Is a correct final state enough to pass the run?

Not if verification was required. A run where the record is right but the agent never read it back is a lucky pass. Score it as a pass on outcome and a failure on claim support, so training rewards verifying before confirming.

How should an agent handle a queued or accepted response?

It should report the change as pending, check the status resource if one exists and confirm completion only after a read-back shows the new value. Accepted is not the same as completed, and processing can still reject the change.

What evidence should a failure packet link to?

The starting snapshot, the task rule, the full tool-call log with results, the final message, a read-back of the resulting state and the reference recovery. Each verdict should be reproducible from those links without rereading the conversation.

Can these cases come from real business records?

Yes, when rights allow it. Audit trails, status histories and later records show what actually persisted after a change. Confirm ownership, privacy permission and a license covering AI use before any record enters an evaluation set.

Sources

  • Anthropic's guide to building agents says agents need ground truth from the environment at each step, such as tool call results, to assess progress. Source
  • The Model Context Protocol specification reports tool execution errors, including API failures, validation and business logic errors, inside tool results with isError set to true. Source
  • RFC 9110 defines 202 Accepted as accepted but not yet completed and possibly disallowed later, and 412 Precondition Failed for a failed If-Match condition. Source
  • The tau-bench code sets the reward to zero when the final database hash differs from the hash after replaying the gold actions, and checks required outputs in the agent's replies separately. Source
  • The tau-bench paper compares the end-of-conversation database state with the annotated goal state and introduces pass^k for reliability across trials. Source
  • The WebArena evaluation harness navigates to a target URL after a run and checks page content with a locator. Source
  • OSWorld gives each task an initial state setup configuration and a custom execution-based evaluation script. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies