AI uses for records
Agent partial refund failure: the refund was right, the order was wrong
By SourceX Editorial · Updated
9 sections
Short answer
Evaluate it by comparing the end state of every system the task touches with a written goal state, not by the refund response or the agent's reply. A partial refund that succeeds in the payment processor while the order record still shows fully paid, with no refund line or restock decision, is a failed task. Grade each condition separately and keep a counterexample where processor-only refunds are valid.
Agent run
Agent partial refund failure
Key takeaways
- A partial refund splits across the processor, the order record, inventory and the helpdesk, and each can be right while another is wrong.
- Status-based graders pass this failure because every tool call returns success.
- Stripe's refunded flag stays false on a partial refund, so grade amount_refunded and the refund object instead.
- The goal state names the order's financial status, the refund line, the restock type and an unchanged inventory count.
- A webhook-sync counterexample stops the grader from failing a valid processor-only path.
- Repair is compensation without a second money movement, scored separately from the original run.
How do you evaluate a correct refund that leaves the order in the wrong state?
Grade the end state of every system of record the task touches against a written goal state, and treat the refund as one condition among several. If the payment processor shows the right partial refund but the order management system still shows the order as fully paid, with no refund line and no restock decision, the task failed, however accurate the agent's closing message was.
Most agent evaluations check one of two things: whether the tool call succeeded, or whether the final reply sounds right. A partial refund is a good stress test because it splits across systems. Money moves in the processor, the order record carries the financial status and line-level refund detail, inventory decides whether a unit comes back to stock, and the helpdesk holds the promise made to the customer. Each can be right while another is wrong.
The case below is constructed and labeled illustrative. It uses Stripe and Shopify field names because their public documentation defines partial refund behavior precisely, but nothing here depends on those vendors, and the same packet works for Adyen with Salesforce Order Management, or Braintree with NetSuite.
Starting state: one damaged unit on a three-unit line (illustrative)
Write the starting state of each system before the run, because the goal state is defined as a change from it. In this case a customer ordered three units of one item and one unit of another, all delivered. One unit arrived damaged; the customer sends a photo and asks for a refund for that unit. Store policy, version 4 in the illustration, says a damaged unit under a set value threshold is refunded without a return, and the unit is written off, not restocked.
Snapshot each system as data, not as a description: export the processor charge, the order with its lines and refunds, the inventory record and the ticket into fixtures the harness can load. A prose description of the starting state invites a grader to infer things the agent could not see, and it makes the case impossible to rerun after the tools change.
| System | Record | Starting state | Relevant policy rule |
|---|---|---|---|
| Payment processor | One charge for the whole order | Captured; nothing refunded; refunded flag false | Refunds go to the original payment method |
| Order management system | Order with two lines (3 units and 1 unit) | Financial status paid; fulfillment status fulfilled | Every refund is recorded on the order with line and quantity |
| Inventory or warehouse system | On-hand count for the damaged item | Unchanged since shipment | Damaged units kept by the customer are not restocked |
| Helpdesk | Ticket with photo attachment | Open, assigned to the agent | Reply states what was refunded, to which method, and that no return is needed |
| Agent tool set | Processor refund tool, order lookup, order refund tool, ticket reply tool | All available and authorized | Policy version 4 provided in the system prompt |
The run: every tool call returned success
Every call in the illustrative trace returned success, which is exactly why a status-based grader passes it. The failure is an omission and a choice of path, not an error message.
Why would an agent skip step 4? In the illustration the processor refund tool is described as issuing refunds, and the order refund tool as issuing refunds and updating the order, so the agent reads them as alternatives and picks the one that moves money. That is a tool design problem as much as an agent problem, and the case should record the tool descriptions exactly as the agent saw them, because changing a description is often the cheapest fix.
| Step | Agent action | System response | What the agent concluded |
|---|---|---|---|
| 1 | Looked up the order and the charge | Order paid and fulfilled; one charge found | Eligible for a damaged-item refund |
| 2 | Checked the photo and policy version 4 | Damage visible; unit under the threshold | Refund one unit, no return |
| 3 | Called the processor refund tool for one unit's price on the order's charge | Refund created with status succeeded | Refund complete |
| 4 | Skipped the order refund tool | No call made | Not needed, money already returned |
| 5 | Replied on the ticket and solved it | Ticket solved | Customer told one unit was refunded to the original card, no return needed |
Which condition did the order record fail while the reply looked right?
The failed condition is that the order record must reflect the refund at line level with the correct restock decision. The money movement was right; the order system never learned about it, so the order still reads as fully paid.
The processor's own fields show why a processor-only check is not enough. In Stripe's API, a refund has a status that can be pending, requires_action, succeeded, failed or canceled, and the charge's refunded flag stays false when a charge is only partially refunded, while amount_refunded records the partial amount. A grader that checks refunded equals true would fail this correct partial refund; a grader that checks the refund status alone passes the whole run.
On the order side, Shopify's Admin API, as one example, has a display financial status of PARTIALLY_REFUNDED for an order where the refunded amount is less than the amount paid, and each refund line item carries a restock type: CANCEL for unfulfilled items, RETURN for fulfilled items that come back, and NO_RESTOCK when inventory is untouched. The goal state here is a partially refunded order with one refund line of quantity one and NO_RESTOCK.
The consequences are concrete. Finance reconciliation shows a processor refund with no matching order refund. A later return request on the same order sees three refundable units instead of two. Reporting on refund rates by product misses the damage entirely. None of that shows in the transcript.
The cross-system consistency failure packet
The packet is one record per case, written so a second evaluator can rebuild the environment, rerun the agent and reach the same verdict without asking the author. Every row below is required; a case missing the counterexample or the recovery check is a draft, not a test.
At a glance
The cross-system consistency failure packet
Generate the goal state rather than typing it. Load the starting fixtures, apply the reference actions a careful human operator would take under the policy, and export the result; that export is the goal state. Typed goal states drift from what the systems actually do, for example forgetting that recording a refund on the order also changes its financial status.
| Field | Content for this case | Evidence it points to | Grader use |
|---|---|---|---|
| Case label | Illustrative, constructed; no real customer or merchant | Author note and creation date | Kept out of any claim about real failure rates |
| Task and policy | Refund one damaged unit without return; policy version 4 text attached | Policy document with version and effective date | Defines which outcome is correct |
| Starting state | Per-system snapshot from the starting-state table | Seed fixtures or database export | Baseline for the diff |
| Goal state | Processor: one succeeded partial refund on the charge. Order: partially refunded, one refund line, quantity one, no restock. Inventory: unchanged. Ticket: solved with an accurate reply | Expected fixture after replaying the reference actions | Compared field by field with the observed state |
| Trace | Five steps with tool names, arguments and responses | Run log | Locates the failure step |
| Failure step and rule | Step 4: order refund not recorded; rule that every refund is recorded on the order | Policy section and trace line | Assigns the failure to a condition, not to the reply |
| Observed end state | Processor correct; order still paid with no refund line; inventory unchanged; ticket solved | Post-run export from each system | Produces the pass or fail per condition |
| Severity | High: financial records disagree and a second refund becomes possible | Severity scale used by the evaluation team | Weights the failure in aggregate scores |
| Recovery check | Order shows one refund line with no new money movement; no duplicate refund on the charge | Post-repair export | Separates a clean repair from a second error |
| Counterexample link | Webhook-sync variant where the processor-only path is correct | Sibling case ID | Prevents the grader from punishing a valid path |
Which linked records prove the decision and the eventual outcome?
The proof is a chain of identifiers that crosses all four systems, not any single record. If one link is missing, the grader cannot tell a correct partial refund on the right order from a correct-looking refund on the wrong charge.
Build the chain into the case. The processor refund should reference the charge or payment intent it refunds, and carry the order ID in a metadata field; Stripe's refund request accepts a set of key-value pairs as metadata for exactly this kind of cross-reference. The order record's refund should carry the processor's refund ID as its transaction reference. The ticket should reference the order. With those three links, the grader can join the systems mechanically.
- Processor: the refund object exists, its status is succeeded, it references the order's charge, and its amount equals one unit's price under the policy's rounding and tax rules.
- Processor: the charge's amount_refunded equals the sum of refunds on it, and refunded stays false because the refund is partial.
- Order: financial status is partially refunded and exactly one refund record exists, with one line, quantity one, and a restock type meaning no restock.
- Order: the refund record's transaction reference matches the processor refund ID.
- Inventory: on-hand count for the item is unchanged from the starting state.
- Helpdesk: the reply states one unit, the original payment method and no return needed, and the ticket references the order.
- Across systems: no second refund exists on the charge, and nothing else in the starting snapshot changed.
Variants and a counterexample that keep the grader honest
Write sibling cases that change one thing at a time, so you know which condition the grader is actually testing. The counterexample is the most important one: in some shops the processor is the system of record for refunds and the order system updates itself from processor events, so a processor-only refund is the correct path and the grader must wait for the sync before judging.
Benchmarks built on goal states already work this way. In tau-bench, the reward compares a hash of the final database with the hash obtained by replaying the ground-truth actions on a fresh copy, and sets the reward to zero if they differ, with a separate check that required outputs appear. Its retail policy also requires the agent to list the action details and get explicit user confirmation before consequential database updates such as returns.
| Variant | What changes | Correct verdict | What it proves about the grader |
|---|---|---|---|
| Base case | Processor refund only; order not updated | Fail on order state | Grader checks the order, not only the processor |
| Wrong restock type | Refund recorded on the order with RETURN for the kept unit | Fail on inventory | Grader checks restock, not only the refund line |
| Cancel on a fulfilled line | Refund recorded with CANCEL on a delivered unit | Fail on line and inventory state | Grader distinguishes fulfilled from unfulfilled handling |
| Wrong message only | All systems correct; reply says full refund | Fail on the customer promise | Grader reads the reply, not only state |
| Duplicate refund | Agent refunds in the processor, then again through the order system | Fail on processor total | Grader sums refunds per charge |
| Counterexample: webhook sync | Processor-only refund; order updates from processor events within the sync window | Pass, judged after the window | Grader waits for eventual consistency before failing |
Repairing the case without refunding twice
A correct repair records the existing refund on the order without moving money again. The tempting fix, issuing the refund through the order system, can create a second refund on the same charge, which turns a bookkeeping failure into a financial one.
This is a compensation problem, not a rollback. Microsoft's Saga pattern guidance describes splitting a business transaction into local transactions per service, with compensating transactions that undo completed steps when a later step fails; its Compensating Transaction pattern adds that simply restoring the original state can overwrite concurrent changes, so compensation is usually application specific. In the case, that means the repair is to add the missing order record, linked to the existing processor refund, by whatever mechanism the order platform provides for recording an external refund. Check the platform's documentation for that mechanism before writing the expected repair into the case.
Score the repair on its own short checklist: the order now carries exactly one refund record for the unit; that record references the existing processor refund; the processor shows no new refund on the charge; inventory is still unchanged; and the customer received no second message that contradicts the first. A repair that passes four of five is still a failed repair if the missing one is the duplicate check.
Score the repair separately from the original run. An agent that notices its own omission and records the refund without double refunding should get credit for recovery, but the original run still failed: the customer-facing state was wrong until the repair, and in production nobody may have noticed.
Real refund and order-state histories for evaluation sets
Constructed cases like this one teach the shape of the failure; real histories teach how often each variant happens and what the messy versions look like. The records that matter are processor refund events joined to order refund records, inventory adjustments and the support conversation, with policy versions attached, from a business that has actually reconciled them.
Joined refund and order-state histories can be requested from SourceX, which starts searching U.S. businesses for them only when a request specifies the payment, order and helpdesk systems, plus volume, language, format, timing and license terms. Refund records carry customer and payment details, so it matters that SourceX de-identifies them through approved processing routes, and that the supplying merchant signs off on scope and terms before any of them ship.
Keep constructed and real cases in separate splits and report them separately. A grader tuned on clean constructed packets can look strong and then miss the partial-sync, manual-override and late-chargeback patterns that only appear in real operating records.
Frequently asked questions
Why do agent evaluations miss a partial refund that leaves the order wrong?
Because they grade the tool response or the final message. The processor returns success and the reply is accurate, so only a check of the order record, inventory and refund totals across systems catches the omission.
Is refunding only in the payment processor always a failure?
No. Where the order system updates itself from processor events, the processor path can be correct. The grader should check the order state after the documented sync window instead of immediately, which is why the webhook variant is kept as a counterexample.
Which Stripe fields show a partial refund?
The refund object has a status such as succeeded, and the charge's amount_refunded records the refunded amount. The charge's refunded flag stays false until the charge is fully refunded, so it should not be used to confirm a partial refund.
What restock type fits a damaged unit the customer keeps?
In Shopify's terms, NO_RESTOCK, because inventory is not affected. RETURN applies to fulfilled items coming back and CANCEL to unfulfilled items, and either one would overstate stock for a unit that never returns.
Can this packet be built from real support and order records?
Yes, if the records join across systems by order, charge and refund IDs and the business has the rights to license them. Label real cases as approved and constructed ones as illustrative, and keep them in separate splits.
Sources
- Stripe's OpenAPI specification defines refund status values (pending, requires_action, succeeded, failed, canceled), the charge's refunded flag that stays false on partial refunds, amount_refunded, and metadata on refund creation. Source
- Shopify's Admin API order display financial status includes PARTIALLY_REFUNDED, for orders where the amount refunded is less than the amount paid. Source
- Shopify's refund line item restock types are CANCEL for unfulfilled items, RETURN for fulfilled items that are returned, and NO_RESTOCK when the item is not restocked. Source
- tau-bench computes reward by comparing a hash of the final database with the hash obtained by replaying the ground-truth actions, and also checks required outputs. Source
- tau-bench's retail agent policy requires listing action details and obtaining explicit user confirmation before consequential database updates such as returns. Source
- Microsoft's Saga pattern breaks a transaction into local transactions per service and uses compensating transactions to undo completed steps when a step fails. Source
- Microsoft's Compensating Transaction pattern notes that restoring the original state can overwrite concurrent changes, so compensation is usually application specific. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.