Skip to content

AI uses for records

Agent partial refund failure: the refund was right, the order was wrong

By SourceX Editorial · Updated

9 sections

Short answer

Evaluate it by comparing the end state of every system the task touches with a written goal state, not by the refund response or the agent's reply. A partial refund that succeeds in the payment processor while the order record still shows fully paid, with no refund line or restock decision, is a failed task. Grade each condition separately and keep a counterexample where processor-only refunds are valid.

Agent run

Agent partial refund failure

This page covers agent partial refund failure: the refund was right, the order was wrong.Illustrative

Key takeaways

  • A partial refund splits across the processor, the order record, inventory and the helpdesk, and each can be right while another is wrong.
  • Status-based graders pass this failure because every tool call returns success.
  • Stripe's refunded flag stays false on a partial refund, so grade amount_refunded and the refund object instead.
  • The goal state names the order's financial status, the refund line, the restock type and an unchanged inventory count.
  • A webhook-sync counterexample stops the grader from failing a valid processor-only path.
  • Repair is compensation without a second money movement, scored separately from the original run.

How do you evaluate a correct refund that leaves the order in the wrong state?

Grade the end state of every system of record the task touches against a written goal state, and treat the refund as one condition among several. If the payment processor shows the right partial refund but the order management system still shows the order as fully paid, with no refund line and no restock decision, the task failed, however accurate the agent's closing message was.

Most agent evaluations check one of two things: whether the tool call succeeded, or whether the final reply sounds right. A partial refund is a good stress test because it splits across systems. Money moves in the processor, the order record carries the financial status and line-level refund detail, inventory decides whether a unit comes back to stock, and the helpdesk holds the promise made to the customer. Each can be right while another is wrong.

The case below is constructed and labeled illustrative. It uses Stripe and Shopify field names because their public documentation defines partial refund behavior precisely, but nothing here depends on those vendors, and the same packet works for Adyen with Salesforce Order Management, or Braintree with NetSuite.

Starting state: one damaged unit on a three-unit line (illustrative)

Write the starting state of each system before the run, because the goal state is defined as a change from it. In this case a customer ordered three units of one item and one unit of another, all delivered. One unit arrived damaged; the customer sends a photo and asks for a refund for that unit. Store policy, version 4 in the illustration, says a damaged unit under a set value threshold is refunded without a return, and the unit is written off, not restocked.

Snapshot each system as data, not as a description: export the processor charge, the order with its lines and refunds, the inventory record and the ticket into fixtures the harness can load. A prose description of the starting state invites a grader to infer things the agent could not see, and it makes the case impossible to rerun after the tools change.

Starting state: one damaged unit on a three-unit line (illustrative)
SystemRecordStarting stateRelevant policy rule
Payment processorOne charge for the whole orderCaptured; nothing refunded; refunded flag falseRefunds go to the original payment method
Order management systemOrder with two lines (3 units and 1 unit)Financial status paid; fulfillment status fulfilledEvery refund is recorded on the order with line and quantity
Inventory or warehouse systemOn-hand count for the damaged itemUnchanged since shipmentDamaged units kept by the customer are not restocked
HelpdeskTicket with photo attachmentOpen, assigned to the agentReply states what was refunded, to which method, and that no return is needed
Agent tool setProcessor refund tool, order lookup, order refund tool, ticket reply toolAll available and authorizedPolicy version 4 provided in the system prompt

The run: every tool call returned success

Every call in the illustrative trace returned success, which is exactly why a status-based grader passes it. The failure is an omission and a choice of path, not an error message.

Why would an agent skip step 4? In the illustration the processor refund tool is described as issuing refunds, and the order refund tool as issuing refunds and updating the order, so the agent reads them as alternatives and picks the one that moves money. That is a tool design problem as much as an agent problem, and the case should record the tool descriptions exactly as the agent saw them, because changing a description is often the cheapest fix.

The run: every tool call returned success
StepAgent actionSystem responseWhat the agent concluded
1Looked up the order and the chargeOrder paid and fulfilled; one charge foundEligible for a damaged-item refund
2Checked the photo and policy version 4Damage visible; unit under the thresholdRefund one unit, no return
3Called the processor refund tool for one unit's price on the order's chargeRefund created with status succeededRefund complete
4Skipped the order refund toolNo call madeNot needed, money already returned
5Replied on the ticket and solved itTicket solvedCustomer told one unit was refunded to the original card, no return needed

Which condition did the order record fail while the reply looked right?

The failed condition is that the order record must reflect the refund at line level with the correct restock decision. The money movement was right; the order system never learned about it, so the order still reads as fully paid.

The processor's own fields show why a processor-only check is not enough. In Stripe's API, a refund has a status that can be pending, requires_action, succeeded, failed or canceled, and the charge's refunded flag stays false when a charge is only partially refunded, while amount_refunded records the partial amount. A grader that checks refunded equals true would fail this correct partial refund; a grader that checks the refund status alone passes the whole run.

On the order side, Shopify's Admin API, as one example, has a display financial status of PARTIALLY_REFUNDED for an order where the refunded amount is less than the amount paid, and each refund line item carries a restock type: CANCEL for unfulfilled items, RETURN for fulfilled items that come back, and NO_RESTOCK when inventory is untouched. The goal state here is a partially refunded order with one refund line of quantity one and NO_RESTOCK.

The consequences are concrete. Finance reconciliation shows a processor refund with no matching order refund. A later return request on the same order sees three refundable units instead of two. Reporting on refund rates by product misses the damage entirely. None of that shows in the transcript.

The cross-system consistency failure packet

The packet is one record per case, written so a second evaluator can rebuild the environment, rerun the agent and reach the same verdict without asking the author. Every row below is required; a case missing the counterexample or the recovery check is a draft, not a test.

At a glance

The cross-system consistency failure packet

Generate the goal state rather than typing it. Load the starting fixtures, apply the reference actions a careful human operator would take under the policy, and export the result; that export is the goal state. Typed goal states drift from what the systems actually do, for example forgetting that recording a refund on the order also changes its financial status.

The cross-system consistency failure packet
FieldContent for this caseEvidence it points toGrader use
Case labelIllustrative, constructed; no real customer or merchantAuthor note and creation dateKept out of any claim about real failure rates
Task and policyRefund one damaged unit without return; policy version 4 text attachedPolicy document with version and effective dateDefines which outcome is correct
Starting statePer-system snapshot from the starting-state tableSeed fixtures or database exportBaseline for the diff
Goal stateProcessor: one succeeded partial refund on the charge. Order: partially refunded, one refund line, quantity one, no restock. Inventory: unchanged. Ticket: solved with an accurate replyExpected fixture after replaying the reference actionsCompared field by field with the observed state
TraceFive steps with tool names, arguments and responsesRun logLocates the failure step
Failure step and ruleStep 4: order refund not recorded; rule that every refund is recorded on the orderPolicy section and trace lineAssigns the failure to a condition, not to the reply
Observed end stateProcessor correct; order still paid with no refund line; inventory unchanged; ticket solvedPost-run export from each systemProduces the pass or fail per condition
SeverityHigh: financial records disagree and a second refund becomes possibleSeverity scale used by the evaluation teamWeights the failure in aggregate scores
Recovery checkOrder shows one refund line with no new money movement; no duplicate refund on the chargePost-repair exportSeparates a clean repair from a second error
Counterexample linkWebhook-sync variant where the processor-only path is correctSibling case IDPrevents the grader from punishing a valid path

Which linked records prove the decision and the eventual outcome?

The proof is a chain of identifiers that crosses all four systems, not any single record. If one link is missing, the grader cannot tell a correct partial refund on the right order from a correct-looking refund on the wrong charge.

Build the chain into the case. The processor refund should reference the charge or payment intent it refunds, and carry the order ID in a metadata field; Stripe's refund request accepts a set of key-value pairs as metadata for exactly this kind of cross-reference. The order record's refund should carry the processor's refund ID as its transaction reference. The ticket should reference the order. With those three links, the grader can join the systems mechanically.

  • Processor: the refund object exists, its status is succeeded, it references the order's charge, and its amount equals one unit's price under the policy's rounding and tax rules.
  • Processor: the charge's amount_refunded equals the sum of refunds on it, and refunded stays false because the refund is partial.
  • Order: financial status is partially refunded and exactly one refund record exists, with one line, quantity one, and a restock type meaning no restock.
  • Order: the refund record's transaction reference matches the processor refund ID.
  • Inventory: on-hand count for the item is unchanged from the starting state.
  • Helpdesk: the reply states one unit, the original payment method and no return needed, and the ticket references the order.
  • Across systems: no second refund exists on the charge, and nothing else in the starting snapshot changed.

Variants and a counterexample that keep the grader honest

Write sibling cases that change one thing at a time, so you know which condition the grader is actually testing. The counterexample is the most important one: in some shops the processor is the system of record for refunds and the order system updates itself from processor events, so a processor-only refund is the correct path and the grader must wait for the sync before judging.

Benchmarks built on goal states already work this way. In tau-bench, the reward compares a hash of the final database with the hash obtained by replaying the ground-truth actions on a fresh copy, and sets the reward to zero if they differ, with a separate check that required outputs appear. Its retail policy also requires the agent to list the action details and get explicit user confirmation before consequential database updates such as returns.

Variants and a counterexample that keep the grader honest
VariantWhat changesCorrect verdictWhat it proves about the grader
Base caseProcessor refund only; order not updatedFail on order stateGrader checks the order, not only the processor
Wrong restock typeRefund recorded on the order with RETURN for the kept unitFail on inventoryGrader checks restock, not only the refund line
Cancel on a fulfilled lineRefund recorded with CANCEL on a delivered unitFail on line and inventory stateGrader distinguishes fulfilled from unfulfilled handling
Wrong message onlyAll systems correct; reply says full refundFail on the customer promiseGrader reads the reply, not only state
Duplicate refundAgent refunds in the processor, then again through the order systemFail on processor totalGrader sums refunds per charge
Counterexample: webhook syncProcessor-only refund; order updates from processor events within the sync windowPass, judged after the windowGrader waits for eventual consistency before failing

Repairing the case without refunding twice

A correct repair records the existing refund on the order without moving money again. The tempting fix, issuing the refund through the order system, can create a second refund on the same charge, which turns a bookkeeping failure into a financial one.

This is a compensation problem, not a rollback. Microsoft's Saga pattern guidance describes splitting a business transaction into local transactions per service, with compensating transactions that undo completed steps when a later step fails; its Compensating Transaction pattern adds that simply restoring the original state can overwrite concurrent changes, so compensation is usually application specific. In the case, that means the repair is to add the missing order record, linked to the existing processor refund, by whatever mechanism the order platform provides for recording an external refund. Check the platform's documentation for that mechanism before writing the expected repair into the case.

Score the repair on its own short checklist: the order now carries exactly one refund record for the unit; that record references the existing processor refund; the processor shows no new refund on the charge; inventory is still unchanged; and the customer received no second message that contradicts the first. A repair that passes four of five is still a failed repair if the missing one is the duplicate check.

Score the repair separately from the original run. An agent that notices its own omission and records the refund without double refunding should get credit for recovery, but the original run still failed: the customer-facing state was wrong until the repair, and in production nobody may have noticed.

Real refund and order-state histories for evaluation sets

Constructed cases like this one teach the shape of the failure; real histories teach how often each variant happens and what the messy versions look like. The records that matter are processor refund events joined to order refund records, inventory adjustments and the support conversation, with policy versions attached, from a business that has actually reconciled them.

Joined refund and order-state histories can be requested from SourceX, which starts searching U.S. businesses for them only when a request specifies the payment, order and helpdesk systems, plus volume, language, format, timing and license terms. Refund records carry customer and payment details, so it matters that SourceX de-identifies them through approved processing routes, and that the supplying merchant signs off on scope and terms before any of them ship.

Keep constructed and real cases in separate splits and report them separately. A grader tuned on clean constructed packets can look strong and then miss the partial-sync, manual-override and late-chargeback patterns that only appear in real operating records.

Frequently asked questions

Why do agent evaluations miss a partial refund that leaves the order wrong?

Because they grade the tool response or the final message. The processor returns success and the reply is accurate, so only a check of the order record, inventory and refund totals across systems catches the omission.

Is refunding only in the payment processor always a failure?

No. Where the order system updates itself from processor events, the processor path can be correct. The grader should check the order state after the documented sync window instead of immediately, which is why the webhook variant is kept as a counterexample.

Which Stripe fields show a partial refund?

The refund object has a status such as succeeded, and the charge's amount_refunded records the refunded amount. The charge's refunded flag stays false until the charge is fully refunded, so it should not be used to confirm a partial refund.

What restock type fits a damaged unit the customer keeps?

In Shopify's terms, NO_RESTOCK, because inventory is not affected. RETURN applies to fulfilled items coming back and CANCEL to unfulfilled items, and either one would overstate stock for a unit that never returns.

Can this packet be built from real support and order records?

Yes, if the records join across systems by order, charge and refund IDs and the business has the rights to license them. Label real cases as approved and constructed ones as illustrative, and keep them in separate splits.

Sources

  • Stripe's OpenAPI specification defines refund status values (pending, requires_action, succeeded, failed, canceled), the charge's refunded flag that stays false on partial refunds, amount_refunded, and metadata on refund creation. Source
  • Shopify's Admin API order display financial status includes PARTIALLY_REFUNDED, for orders where the amount refunded is less than the amount paid. Source
  • Shopify's refund line item restock types are CANCEL for unfulfilled items, RETURN for fulfilled items that are returned, and NO_RESTOCK when the item is not restocked. Source
  • tau-bench computes reward by comparing a hash of the final database with the hash obtained by replaying the ground-truth actions, and also checks required outputs. Source
  • tau-bench's retail agent policy requires listing action details and obtaining explicit user confirmation before consequential database updates such as returns. Source
  • Microsoft's Saga pattern breaks a transaction into local transactions per service and uses compensating transactions to undo completed steps when a step fails. Source
  • Microsoft's Compensating Transaction pattern notes that restoring the original state can overwrite concurrent changes, so compensation is usually application specific. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies