AI uses for records
Agent stale inventory failure: testing a stock promise made from an old snapshot
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
To test an agent that promises stock from a stale inventory snapshot, let orders consume stock after the snapshot is taken, then ask the agent for a commitment. Grade the system of record, not the reply: pass only if the agent read a quantity fresh enough for the promise, used available rather than on-hand stock, reserved or qualified what it promised, and told the customer what was actually true.
Agent run
Agent stale inventory
Key takeaways
- A stale read is invisible in the transcript; it shows up only when you compare the agent's promise with the live inventory record at that moment.
- Available, on hand and committed are different quantities, and an agent that reads the wrong one fails even with fresh data.
- A stock answer becomes a commitment when it states a ship date, takes an order or reserves units.
- Every failure case needs passing siblings, so graders reward rechecking rather than refusing to answer.
- Run each case several times and grade by end state, because freshness failures are often intermittent.
How do you test an agent that promises stock from a stale inventory snapshot?
Create a gap between the snapshot the agent can read and the live record, then ask for something the gap matters to. In practice, load a snapshot, commit enough orders afterwards that the live available quantity falls below what the customer asks for, and have a simulated customer request a firm quantity and ship date.
Then grade what happened in the systems, not what the agent said. A reply such as twelve units are in stock and will ship today reads well and may even be accurate about the snapshot. The case fails if, at the moment of the promise, the live available quantity could not cover the request and the agent neither rechecked nor qualified its answer.
The rest of this page walks one illustrative case end to end, names the conditions that failed, and gives a packet format that lets a team reproduce, grade and vary the case.
Teams often skip this case because the agent did nothing obviously wrong. It called a tool, read a number and repeated it. That is exactly why it belongs in an evaluation set: production failures of this kind surface days later as complaints and delay notices, long after the conversation that caused them has been closed as resolved.
When does a stock answer become a commitment?
A stock answer becomes a commitment when the customer can reasonably act on it: it states or implies a ship date, it accepts an order, or it reserves units. Telling a customer that an item is usually stocked is information; telling them it ships today is a promise the business has to keep or correct.
The quantity the agent reads matters as much as its age. Shopify's inventory model, for example, separates on_hand from available, committed, reserved and incoming quantities, and counts unfulfilled orders and draft order reservations as committed. An agent that reads on_hand will over-promise even from a current record. Public product markup has its own vocabulary: schema.org's ItemAvailability values include InStock, OutOfStock, BackOrder, PreOrder and LimitedAvailability, which a feed may refresh on a different schedule from the order system.
Shipping promises also carry obligations in some markets. In the United States, the FTC's business guide to the Mail, Internet, or Telephone Order Merchandise Rule says a seller needs a reasonable basis for any shipping time it states, or for shipping within 30 days if it states none, and must tell buyers about delays and offer them the option to cancel. This is general information, not legal advice; the point for evaluation is that a promised ship date is a business event with consequences, so a test should treat it as one.
For an evaluation, write the commitment boundary into the case itself. List which phrases count as a promise, such as a stated ship date or a confirmed quantity, and which count as information, such as usually in stock. Graders then apply the same line every time instead of judging tone.
The starting state of the illustrative case
The case is invented and every value is illustrative. A distributor's ordering agent answers business customers by chat. It can call a product lookup tool that reads a nightly inventory snapshot, a live inventory tool that reads the order management system, and an order tool that creates sales orders. The snapshot was taken at 02:00; the conversation happens at 15:40.
The gap is deliberately large so the failure is unambiguous: twelve units appeared available in the snapshot, and by mid-afternoon open orders had committed every unit on hand. Smaller gaps, where the snapshot overstates availability by one or two units, belong in the fault grid later on, because they test whether the agent rechecks on every commitment or only when the numbers look suspicious.
| Record | At snapshot (02:00) | At promise (15:40) | Source |
|---|---|---|---|
| SKU on hand | 40 units | 40 units | Warehouse system |
| SKU committed to open orders | 28 units | 40 units | Order management system |
| SKU available to promise | 12 units | 0 units | Order management system |
| Incoming purchase order | 24 units, due in 9 days | 24 units, due in 9 days | Purchasing system |
| Snapshot age shown to agent | Not exposed | 13 hours 40 minutes | Product lookup tool metadata |
Tracing the run from snapshot read to customer promise
The run is short, which is why it is easy to miss. Every tool call succeeded, and the transcript contains no error.
Two design choices made the failure possible. The lookup tool returned a quantity without its age, so the agent had no signal that the number was thirteen hours old. And the order tool reported success for an order the system had accepted only as a backorder, so the agent had no signal that its promise was already broken. A test that varies only the model will miss both; the packet below records tool behavior as part of the starting state for that reason.
- The customer asks whether 10 units can ship today for an order they want to place now.
- The agent calls the product lookup tool, which returns available 12 from the 02:00 snapshot. The response carries no timestamp the agent reads.
- The agent does not call the live inventory tool, although the request includes a quantity and a same-day ship date.
- The agent replies that 10 units are in stock and will ship today, and calls the order tool to create the sales order.
- The order management system accepts the order and marks it backordered, because available is 0. The order tool returns success with the order number.
- The agent confirms the order to the customer with the order number and repeats the same-day ship date.
- The next morning the warehouse cannot pick the order; the customer receives a delay notice several hours after the promised ship time.
Which required conditions failed behind a correct-sounding reply?
Four conditions failed, and none of them shows in the agent's final message. That is the core lesson of the case: the reply was fluent, specific and consistent with what the agent had read.
At a glance
Which required conditions failed behind a correct-sounding reply
Freshness needs a definition the grader can apply. HTTP caching offers a useful model: RFC 9111 treats a stored response as fresh while its age is below its freshness lifetime and stale after that. Give each inventory read an age and a freshness bound for the decision it supports. A bound of hours may be fine for answering whether an item is carried; a same-day commitment needs a live read. Where tools are exposed through the Model Context Protocol, the schema's optional lastModified annotation can carry a resource's timestamp, but it is optional, so the test should check whether the agent is shown and uses an age at all.
- Freshness: the quantity was 13 hours 40 minutes old, against a bound of minutes for a same-day promise.
- Quantity semantics: the agent treated the snapshot's available figure as current available to promise; it was neither current nor reserved.
- Commitment check: the agent stated a ship date without reserving units or confirming that the order would allocate.
- Outcome reporting: the order tool's success hid a backorder status, and the agent reported a same-day shipment that the system had already declined.
The freshness and commitment failure packet
This is the asset: a twelve-field packet, version 1.0 dated 2026-10-18, that lets another team reproduce and grade the case. Store it with the seed data and the expected end state, not only the transcript.
- Linked evidence: snapshot record with its timestamp, live inventory reading at promise time, order record with status history, customer-facing messages.
- Grading rule: pass only if the end state and the agent's claims agree and every condition holds.
| Field | What it holds | In this case (illustrative) |
|---|---|---|
| Case label | Illustrative or approved real case, with approver | Illustrative |
| Starting state | Seeded inventory, orders and tool behavior | Snapshot available 12; live available 0 |
| Customer request | Quantity, date and whether an order is placed | 10 units, ship today, order now |
| Freshness bound | Maximum read age for this commitment type | 5 minutes for a same-day promise |
| Failure step | First step where a required condition failed | Answered from snapshot without live read |
| Applicable rule | Business policy or obligation the promise touches | Promise only allocated stock; state real ship dates |
| Agent claim | What the agent told the customer | In stock, ships today |
| Resulting state | System of record after the run | One backordered order; zero units allocated |
| Severity | Effect on customer and business | High: false ship date, delay notice needed |
| Recovery check | What a correct agent does after detecting the gap | Corrects the date, offers backorder or cancellation |
Which linked records prove the decision and the outcome?
Four records, each with a timestamp, are enough to grade the case without reading the transcript at all. The snapshot record shows what the agent could have seen and how old it was. The live inventory reading, captured by the harness at the moment of the promise, shows what was true. The order record and its status history show what the system did with the order. The outbound messages show what the customer was told and when.
Join them by order number and time. The case fails when the claim in the messages contradicts the order status or the live reading at the same moment. Keeping these records with the case also makes it reusable as training data: a model can learn from the contrast between what the snapshot said and what the order system did.
Capture the live reading inside the harness rather than reconstructing it afterwards. Inventory changes constantly, and a reading taken an hour after the run can make a failed promise look reasonable or a correct one look wrong. Record the reading, its timestamp and the query that produced it, so a reviewer can see exactly what the grader compared against.
Counterexamples that keep the grader honest
A grader that only sees failures learns to reward hedging. Pair the case with siblings that should pass, and one that looks like a failure but is not.
tau-bench, a benchmark of tool-using agents that includes a retail domain, grades runs by comparing the final database state with an annotated goal state and measures reliability with pass^k, the chance that all of k trials succeed. Use the same approach here: run each sibling several times and require consistent behavior, because freshness failures often depend on timing and appear in only some runs.
The last sibling matters most for fairness. Stock that moves after a correct, allocated promise is the business working normally, and an agent should not be penalized for it. Grade each promise against the live reading taken at the moment it was made, not against the state at the end of the run.
- Pass: the agent calls the live tool, finds 0 available, and offers the 24 incoming units with the purchase order's due date stated as an estimate.
- Pass: the snapshot is minutes old and within the bound; the agent promises and the order allocates.
- Pass: the agent answers a general availability question from the snapshot, states its age, and makes no ship-date promise.
- Looks like a failure, but passes: the agent promises from a live read and allocates, then another order is cancelled and stock moves; the promise was correct when made.
Varying the case: a fault-injection grid
One case is a demonstration; a family of cases is a test. Vary one factor at a time and keep the expected end state for each.
Keep the grid small at first. Six variables with three values each already produce hundreds of combinations, most of which add little. Start with snapshot age crossed with age exposure, because those two decide whether an agent can notice staleness at all, then add the order tool responses, which decide whether it can notice a broken promise afterwards. Keep each combination's expected end state alongside the seed data so a new model version is graded against the same answers.
| Variable | Values to try | What it tests |
|---|---|---|
| Snapshot age | Minutes, hours, more than a day | Whether the agent uses age at all |
| Age exposure | Timestamp present, absent, misleading | Whether the agent reads metadata or assumes freshness |
| Quantity field | Available, on hand only, both | Whether the agent knows which quantity to promise |
| Live tool behavior | Fast, slow, timing out | Whether the agent falls back to the snapshot silently |
| Order tool response | Allocated, backordered, partial | Whether the agent reports the real status |
| Customer pressure | Neutral, urgent, insisting | Whether urgency makes the agent skip the live read |
Sourcing real stock-promise incidents for an evaluation set
Illustrative cases show the shape of a failure; real incidents show its frequency and variety. Businesses with order management and customer service histories hold the records that matter here: availability answers, the inventory state at the time, orders that went to backorder and the delay notices that followed.
SourceX, which ironically holds no inventory of its own, finds such records at U.S. businesses only when a buyer asks. The request form takes the scope, volume, language, format, timing and licensing a team needs, and it is the place to specify linked inventory, order and message records with timestamps.
Frequently asked questions
What is a stale inventory failure in an AI agent?
It is a case where an agent makes a promise based on an inventory value that was true when it was read but is no longer true when the promise is made. The tool call succeeds and the reply looks correct. The failure only appears when the promise is compared with the live record at that moment.
Why can't the transcript alone show the failure?
The transcript shows what the agent read and said, and both can be internally consistent. It does not show the live available quantity at the time, or the status the order system gave the order. Grading needs those system records alongside the conversation.
Should agents always read live inventory?
Not for every answer. A snapshot is fine for general questions if the agent states its age and makes no promise. Commitments with a quantity and date should use a live read, or a reservation, within a freshness bound set for that kind of promise.
What is the difference between on hand and available inventory?
On hand counts units physically at a location. Available subtracts units already committed to orders or set aside, so it is the quantity that can still be promised. Systems name these states differently, so the evaluation should check which field each tool returns.
How many runs does a freshness case need?
Enough to see whether the agent behaves consistently, because timing changes from run to run. Reporting the share of cases where every one of several trials succeeds, as pass^k does, is stricter than counting a single success. Choose k before you run the cases.
Sources
- Shopify's inventory model separates quantity states including on_hand, available, committed, reserved and incoming; committed covers unfulfilled orders and draft order reservations. Source
- Schema.org's ItemAvailability enumeration lists product availability values including InStock, OutOfStock, BackOrder, PreOrder and LimitedAvailability. Source
- The FTC's business guide says sellers need a reasonable basis for stated shipping times, or for shipping within 30 days if none is stated, and must notify buyers of delays with an option to cancel. Source
- RFC 9111 defines a response as fresh while its age is below its freshness lifetime and stale afterwards. Source
- The Model Context Protocol schema defines optional annotations including lastModified, an ISO 8601 timestamp of when a resource was last modified. Source
- tau-bench, which includes a retail domain, grades agent runs by comparing final database state with an annotated goal state and defines pass^k as the chance all k trials succeed. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.