Skip to content

AI uses for records

Ecommerce split shipment dataset: one order, three shipments, two problems

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Build a split-shipment agent task around shipments, not the order. Freeze each line, its shipment, carrier events, inventory, promotion and return rules, and payment state at the moment of contact; state the valid action for each problem and for the order as a whole, including how one problem changes another; then prove the outcome with end-state records, not the agent's final message.

Evaluation split

Ecommerce split shipment

This page covers ecommerce split shipment dataset: one order, three shipments, two problems.Illustrative

Key takeaways

  • Order-level statuses such as partially fulfilled hide which shipment had which problem; build tasks at the shipment and line level.
  • Freeze the starting state at contact time and keep later carrier scans, restocks and refunds in the outcome block.
  • List the dependencies, such as a bundle discount or a shared payment, that make one problem change the valid action for another.
  • Prove outcomes with end-state assertions on fulfillment, refunds and notices, checked after a maturity window.
  • Include a counterexample, such as canceling the whole order, that fails at least one assertion.

How do you build an agent task from a split shipment with more than one problem?

Build it around the shipments, not the order. Freeze the order as it stood when the customer made contact: each line, which shipment it belongs to, each shipment's own status and tracking, the inventory and promised dates behind any unshipped line, the promotion and return rules in force, and the payment state. Then state the valid action for each problem separately and check the outcome in the systems of record afterward.

An order with three shipments and two problems is a useful agent test precisely because the problems interact. A delayed line can break a bundle discount on a delivered one; a missing item in one parcel changes what a cancellation of another should refund. An agent that resolves each issue in isolation, or reads one order-level status, will often pick an action that is valid for one shipment and wrong for the order.

This page shows how to turn real order, fulfillment and support records into that kind of task, with a packet format, an illustrative case and the checks that prove the outcome.

Why does an order-level status hide what happened to each shipment?

Commerce platforms track fulfillment below the order, and the task has to as well. Shopify's Admin API models each group of items shipping from one location as a fulfillment order, and an order can split into several of them when products ship from different locations or shipping profiles. When a merchant requests fulfillment for only some items, Shopify splits the fulfillment order into one with the submitted items and one with the rest.

Each fulfillment order has its own status, including open, scheduled, on hold, in progress, closed, cancelled and incomplete. The order itself carries a display status such as partially fulfilled, which the documentation describes as the status shown to merchants. Reading only that order-level field throws away the information a split-shipment task depends on.

Agent benchmarks show the same compression. The retail domain in the open tau2-bench project gives an order a single status of pending, processed, delivered or cancelled, with fulfillment details recorded as tracking ids and item ids. Schema.org's vocabulary separates them more finely: an order has an orderStatus, each order item has its own orderItemStatus, and a ParcelDelivery links back to the order through partOfOrder. Use that separation as the shape of your packet.

What must the starting state freeze before the agent acts?

Freeze everything that was true and knowable at the moment the customer made contact, and nothing that became known later. A carrier scan that arrives two days after the contact, a restock that happened the next week or the refund an agent eventually issued are outcomes, and putting them in the starting state lets a model read the answer.

At a glance

What must the starting state freeze before the agent acts

The frozen state should include the following.

  • Order header: placement time, channel, shipping address and any address change with its timestamp.
  • Lines: product, variant, quantity, price basis, promotion links and the shipment each line is assigned to.
  • Shipments: location, status, carrier, tracking events up to the contact time and the promised delivery window shown to the customer.
  • Inventory for unshipped lines: on-hand and expected-receipt dates as the system showed them at contact time.
  • Rules in force: promotion terms, return window, refund method rules and the delay notice policy, each with a version or effective date.
  • Payment state: authorizations, captures, partial refunds and gift card balances already applied.
  • Contact history: the customer's messages and any earlier agent replies, in order.

Which dependent item, rule or inventory change alters the valid action?

The valid action changes whenever one shipment's problem touches another shipment's lines through a shared rule, a shared payment or a customer promise. List those dependencies explicitly in the task so a reviewer can see why the expected action is right.

In the United States, delays on unshipped lines also bring a legal rule into the task. The FTC's Mail, Internet, or Telephone Order Merchandise Rule requires a seller that cannot ship within the stated time, or within 30 days if no time was stated, to offer the buyer the option to consent to the delay or cancel for a prompt refund. The FTC's business guide adds that when some but not all items ship, the seller refunds the difference between what the buyer paid and what the shipped items would have cost.

Which dependent item, rule or inventory change alters the valid action?
ChangeIllustrative triggerValid action shifts toEvidence the task must carry
Backorder past the promised dateThird shipment's item now expected after the stated ship dateDelay notice with a revised date and a cancel-for-refund option for that linePromised date, revised inventory date, notice record
Bundle or threshold promotionA discount applied only because two linked items were ordered togetherRecompute the discount for the lines actually kept before refundingPromotion terms version and line links
Missing item in a delivered parcelCarrier shows delivered; one of two items absentReship or refund per policy, plus a carrier or warehouse investigationPacking record, scan weight or photo if kept, tracking events
Wrong variant deliveredSize or color differs from the lineExchange or return under the return policy, not a cancellationLine variant, received item report
Address change after a shipment leftCustomer updates address mid-orderApplies only to shipments not yet handed to the carrierAddress change timestamp against each shipment's handoff
Item restocked during the conversationInventory arrives after a delay notice was sentOffer to ship if the customer has not canceledInventory event time and notice response
Refund method constraintPart paid by gift card, part by cardSplit the refund by the documented method rulePayment records and refund method policy

What goes into the split fulfillment task and outcome packet?

The packet has three parts: the frozen starting state, the decision the agent must make with the rules that govern it, and the outcome checks run against the systems of record. Each part names the source system so a reviewer can trace every field.

Use the structure below as a template. Field names are generic; map them to the platform, warehouse system and help desk the records came from.

What goes into the split fulfillment task and outcome packet?
Packet partFieldsTypical sourceCheck before release
Order headerOrder key, placement time, channel, address historyCommerce platformAddress changes carry timestamps
Lines and linksLine key, variant, quantity, promotion link, assigned shipmentCommerce platformEvery line maps to exactly one shipment or to unshipped
ShipmentsShipment key, location, status, carrier events to contact time, promised windowPlatform fulfillment records and carrier feedNo carrier event later than contact time
Inventory snapshotOn-hand and expected dates for unshipped lines at contact timeInventory or warehouse systemSnapshot time precedes contact
RulesPromotion, return, refund method and delay notice policy with versionsPolicy documents and platform settingsVersion in force on the order date
PaymentsAuthorizations, captures, prior refunds, tender splitPayment processor and platformTotals reconcile to the order
ContactCustomer messages and earlier replies in orderHelp deskPersonal details replaced with keys
DecisionExpected action per problem and for the order as a whole, with rule referencesReviewer, from the record and policyTwo reviewers agree or a third decides
Outcome checksEnd-state assertions: shipments, refunds, notices, later contactsAll of the above, after a maturity windowAssertions use records, not the agent's final message
CounterexampleOne plausible wrong action and why it failsReviewerFails at least one outcome check

How should an illustrative order with three shipments and two problems be resolved?

This case is invented to show the format. A customer orders four items: a jacket and matching gloves sold as a bundle with a bundle discount, a backpack, and a water bottle. The platform splits the order into three fulfillment orders: the jacket from one warehouse, the backpack from a second, and the gloves and bottle from a third.

Steps

How should an illustrative order with three shipments and two problems be resolved

Shipments 1 and 2 are delivered. Shipment 3 shows delivered, but the customer writes that the bottle arrived and the gloves did not. A day later the gloves are also listed as out of stock at the third warehouse, with an expected receipt about three weeks out, past the promised delivery window. The customer asks to keep everything else and get the gloves as soon as possible.

  • Problem one, the missing gloves in a delivered parcel: open a warehouse and carrier investigation and, since the item cannot reship from stock, treat the gloves as an unshipped line.
  • Problem two, the stock delay: send a delay notice with the revised date and the option to cancel the gloves line for a refund, as the order rule requires when the promised time cannot be met.
  • Dependency: if the customer cancels the gloves, the bundle discount on the jacket no longer applies under the illustrative promotion terms, so the refund for the gloves line is reduced by the discount the jacket kept. The agent must explain this before the customer chooses.
  • Expected end state: shipments 1 and 2 untouched; a new fulfillment record for the gloves if the customer accepts the delay, or a line-level cancellation and a refund computed under the promotion terms if not; an investigation record for the missing parcel contents.
  • Counterexample: canceling the whole order because one shipment has a problem. It fails because the jacket, backpack and bottle were delivered and kept, and it would refund items the customer still has.

What record shows the order really ended up right?

Prove the outcome with records written by the systems that act on it, not with what the agent said. A polite final message that promises a refund is not evidence that a correct refund was issued.

Benchmarks built for agents take the same view. The tau2-bench evaluation documentation describes replaying a reference set of actions on a fresh environment to get a target database end state and comparing the agent's end state to it, so any sequence of actions that reaches an equivalent state passes. For real records, the equivalent is a set of end-state assertions checked after a maturity window.

  • Fulfillment: the unaffected shipments are unchanged, and the affected line has exactly one new fulfillment or one cancellation.
  • Money: the refund records match the line, promotion and tender rules, with no refund for kept items.
  • Notices: the delay notice exists with a revised date and a cancel option, sent before the original promised time passed.
  • Customer: no repeat contact about the same problem within the maturity window, or, if there is one, it is linked to the case.
  • Investigation: the missing-item record is opened and closed with a reason code.

How should agents be scored on multi-shipment cases?

Score each problem and the order as a whole, and require both. An agent that fixes the delay correctly but refunds the bundle wrongly has failed the order, even if most assertions pass.

Keep three scores apart: end-state correctness from the assertions above; rule use, meaning the agent cited and applied the promotion, return and delay rules in force; and communication, meaning the customer was told the consequence of each choice before choosing. The tau2-bench documentation multiplies a database score by a communication score for the same reason: a correct state reached by a misleading conversation is not a success.

When building test splits, keep all contacts about one order in the same split and group by promotion type and warehouse, so a model cannot learn one merchant's patterns and pass on recognition.

Where do split-shipment packets go wrong?

The most common faults are leaked outcomes, labels taken from status fields and invented rules. A carrier scan time later than the contact, or a refund already visible in the payment block, gives the answer away. A label copied from the order's final status records what the system ended at, which may include errors that a person later corrected.

Promotion and return terms must be the real versions in force on the order date. Where you cannot recover the exact terms, mark the case as illustrative or drop it rather than writing plausible rules.

Order and shipment records also identify people. Names, addresses, phone numbers, tracking numbers and free-text messages can all point to a household, so replace them with stable keys before any packet leaves the company, and keep the timestamps that define the sequence.

Can order and fulfillment histories be licensed for AI?

Retailers and fulfillment providers with years of linked order, shipment and support records hold the raw material for these packets. When such histories are licensed through SourceX, its share of the work covers qualification, rights review, the contract, preparation, delivery and payment on the merchant's behalf.

Nothing ships until the merchant has approved the scope and terms and signed, and order archives too large to move easily can remain in the merchant's own storage or go out on encrypted drives instead of being hosted by SourceX. The first step is a short company assessment with no order data uploaded.

Frequently asked questions

What is a split shipment in ecommerce?

A split shipment is one order delivered in more than one package, usually because items ship from different locations, have different shipping profiles or are not all in stock. Platforms such as Shopify track each part as its own fulfillment order with its own status.

Why not label split-shipment tasks with the order status?

Because an order-level status compresses several shipments into one value, such as partially fulfilled or delivered. It cannot show which item had which problem, and it often reflects later corrections. Label each problem's correct action and check the end state in the records.

Does the FTC rule apply when only part of an order is delayed?

The FTC's Mail, Internet, or Telephone Order Merchandise Rule covers shipping delays, and its business guide addresses partial shipments, including refunding the difference for items not shipped. Merchants should confirm with counsel how it applies to their order flows.

How many problems should one agent test contain?

Two linked problems is often enough to test whether an agent reasons about dependencies, such as a delay that affects a bundle discount. More problems make the case harder to review; add them only if each changes the valid action.

Can real order histories be used to build agent tests?

Yes, if the company has the rights to license them and personal details are removed first. Names, addresses, tracking numbers and message text identify households, so they should be replaced with keys while timestamps and links are kept.

Sources

  • Shopify models items shipping from one location as a fulfillment order; an order can split into several when items ship from different locations or shipping profiles, and a partial fulfillment request splits a fulfillment order in two. Source
  • Shopify's FulfillmentOrderStatus values are open, scheduled, on hold, in progress, closed, cancelled and incomplete. Source
  • Shopify's OrderDisplayFulfillmentStatus includes partially fulfilled, shown when some of an order's items have been fulfilled, and reflects the status displayed in the admin. Source
  • 16 CFR 435.2 requires a seller that cannot ship within the stated time, or 30 days if none is stated, to offer the buyer the option to consent to a delay or cancel for a prompt refund. Source
  • The FTC business guide says that when some but not all ordered items ship, the seller must refund the difference between the amount paid and the amount for the shipped items. Source
  • Schema.org defines orderStatus for an Order, orderItemStatus for an OrderItem, and partOfOrder linking a ParcelDelivery to its order. Source
  • The tau2-bench retail policy gives an order one status (pending, processed, delivered or cancelled) and records fulfillments as tracking ids and item ids. Source
  • The tau2-bench evaluation documentation replays reference actions on a fresh environment to derive a target database end state, passes any trajectory that reaches an equivalent state, and multiplies DB and communication rewards. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies