AI uses for records
A changing customer intent dataset must record what still stands
By SourceX Editorial · Updated
9 sections
Short answer
A changing customer intent dataset should store each change as an edit to standing instructions: what it replaced, which earlier details still bind, what prompted it, and what became of actions taken under the old version. In MultiWOZ 2.1, 90.8% of 5,228 scripted changes alter one detail and 73.0% keep earlier ones, yet only 39.7% of changed slots show the old value, then the new, in the annotated state.
Grader rubric
Changing customer intent
Key takeaways
- A mid-conversation change is usually an edit, not a new request: in MultiWOZ 2.1, 4,749 of 5,228 scripted changes (90.8%) alter exactly one detail.
- Label changes from what the speakers did, not from the scenario: only 2,199 of 5,535 scripted slot changes (39.7%) appear in the annotated state as an old value followed by its replacement.
- Link every replacement to the instruction it replaces, as FIX replace messages carry OrigClOrdID and FHIR requests carry a replaces reference.
- Grade actions taken under a superseded instruction as well as the final one; a correct final booking can sit beside a duplicate nobody cancelled.
- Stratify test sets by who prompted the change, how much it touched and whether an action had already run, then hold out a later period.
What should a changing customer intent dataset record when the request moves?
It should record each change as an edit to a set of standing instructions, not as a fresh request. Every change needs four fields: the earlier instruction it replaces, the earlier details that still apply, what prompted it (the customer's own correction or the agent's news that something was unavailable), and what became of any action already taken under the old instruction.
Picture a customer who books a table for six on Wednesday, hears Wednesday is full and says Thursday then. One value moved. The party size, the restaurant, the time and the allergy note all still stand. A model trained on transcripts labeled only with the final booking learns where the conversation ended, not the rule that got it there: carry forward everything the customer did not touch, and clean up anything done under the version they abandoned.
That is a narrower problem than a conversation whose goal switches outright, such as a billing question that turns into a cancellation, which a separate SourceX guide on call intent changes covers. Here the goal holds and its terms move. The sections that follow count how often that happens in a large public corpus, borrow a linking rule from trading and clinical systems, and turn both into an annotated packet and a grading rubric.
Most changes edit one detail: 5,228 scripted changes counted in MultiWOZ
In the public MultiWOZ 2.1 corpus, nine in ten scripted changes touch a single detail, and most leave other earlier constraints standing. MultiWOZ holds 10,438 human-to-human written dialogues, released under the MIT license, in which crowd workers given a task description played visitors writing to a Cambridge information desk. Its goal generator builds changes in on purpose: the 2018 paper describes initial constraints that, with some probability, match nothing in the database, so the visitor must switch to an alternative.
SourceX counted those changes in data/MultiWOZ_2.1.zip at repository commit fe0c8e6, across all three splits. A domain counts as changed when its goal holds a non-empty fail_info (search constraints set to fail) or fail_book (booking values set to fail). Comparing each failed set with the final one, a slot with the same value in both is carried over and a slot with different values is changed; every fail_book key is changed and every other booking key, apart from the flags invalid and pre_invalid, is carried over. Values are lowercased, trimmed and stripped of spaces before comparison, so anyone can rerun the count from the public repository with these rules.
The result: 4,378 of the 10,438 dialogues (41.9%) contain at least one scripted change, 5,228 changes in all. Of those, 4,141 alter search constraints such as area, cuisine or price and 1,087 alter booking details such as day, time or length of stay; 4,749 (90.8%) change exactly one slot, 378 change two or three, and 101 list identical failed and final values, so they count as changes with no changed slot; 3,818 (73.0%) leave at least one earlier constraint in force. Across every change, 5,535 slots move while 6,926 carry over unchanged.
Most booking changes are prompted by the clerk. A system turn tagged Booking-NoBook or with the domain's NoBook act appears in 909 of the 1,087 booking changes (83.6%), and a turn with the domain's NoOffer act appears in 1,776 of the 4,141 search changes (42.9%). The distinction matters for training: a change the agent prompted (Wednesday is full) should come with an offer, while a change the customer volunteers (we actually arrive Friday) should come with a read-back. The table summarizes the count:
| Measure (MultiWOZ 2.1, all splits) | Count | Share | What it means for a dataset |
|---|---|---|---|
| Dialogues with at least one scripted change | 4,378 of 10,438 | 41.9% | Changes are common enough to need their own label, not an edge-case tag |
| Changes that alter exactly one slot | 4,749 of 5,228 | 90.8% | The default change is an edit: carry everything else forward |
| Changes that keep at least one earlier constraint | 3,818 of 5,228 | 73.0% | Resetting the request on a change discards live instructions |
| Booking changes with a failed-booking turn | 909 of 1,087 | 83.6% | Most booking edits follow the agent's news, not a customer whim |
| Changed slots shown as old value, then new, in the annotated state | 2,199 of 5,535 | 39.7% | Most scripted changes leave no visible supersession to learn from |
A scripted change is weak evidence that a change happened
Conversations drift from their scripts, so labels should come from the dialogue and the system record rather than from the scenario that asked for a change. For the 5,535 slots that MultiWOZ goals change, the final annotated state (the last turn carrying a belief state, read from its semi section for search slots and its book section for booking slots) holds the replacement value for 4,010 (72.4%), still holds the original value for 799 (14.4%) and holds something else, such as a blank, a spelling variant or another value, for 726 (13.1%). Only 2,199 (39.7%) show the original value at some turn and the replacement at the end; for another 1,811 (32.7%) the state ends on the replacement without ever showing the original, so there is no change in the transcript to learn from.
The corpus README itself warns that workers sometimes followed their goals wrongly, which can leave the belief state wrong, and that the workers did not know which bookings were set to fail. To see what sits inside the 799 slots that end on the original value, SourceX listed those 799 slots in the order the counting script reads them, drew 20 with Python's random.Random(20261027).sample and read every dialogue. None was a labeling slip in which the state missed a change the customer made. Twelve conversations never needed the change because the original request was met or simply kept. Five kept the customer's day or time and booked a different restaurant instead, usually after the first choice could not take the booking. One tried the new time and went back to the old one, one log stopped at the no-match turn, and one customer corrected the day in a different direction after a booking already existed.
Two lessons follow. A corpus built from simulator goals or scenario briefs will overstate how many changes it teaches, so count changes from what the speakers did. And the five restaurant swaps are legitimate alternative resolutions that a grader keyed to the scripted fallback would mark wrong: grade against what the customer said they would accept, not against the path the script imagined.
How trading desks and clinical systems tie a replacement to its original
Systems that cannot afford ambiguity store a change as a new record pointing at the one it replaces, and they let the change be refused when the original has already been carried out. Electronic trading has worked this way for decades. In the FIX standard, an OrderCancelReplaceRequest (message type G) changes the parameters of an open order; it is assigned its own ClOrdID and treated as a separate entity, while OrigClOrdID carries the identifier of the previous order. A change that cannot be honored comes back as an OrderCancelReject carrying both identifiers, the specification notes that filled orders cannot have their price changed or quantity reduced, and the first reject reason code is TooLateToCancel.
Clinical ordering uses the same shape. In HL7 FHIR, the ServiceRequest.replaces element points a new request at the earlier request or requests it takes the place of, and the specification lists supersedes, prior and renewed order as alternate names. Commerce platforms stage the edit before it lands: Shopify's order editing API opens a session with orderEditBegin, returns a calculated preview of the edited order and holds each change as staged until it is committed or abandoned.
Carried into a support conversation, the rule gives three fields per change: an identifier for every instruction, a pointer from each change to the instruction it replaces, and a status that reads accepted, pending or refused because the original already executed. With those links a grader can ask whether the agent acted on the instructions in force at each moment, a question a flat transcript cannot answer.
An annotated instruction ledger from one public dialogue
The packet annotates MultiWOZ dialogue SSNG0349, a real record from the public corpus, as an instruction ledger: one row per instruction or action, with its status when the conversation ends. It runs to 14 turns and contains every failure this article describes.
A visitor asks for a moderately priced guesthouse with free wifi and parking, then for 2 nights for 5 people starting Sunday. The first guesthouse has no rooms, the visitor asks for a different one in the same price range, and the clerk books Kirkwood House under reference ANO69RVP. The visitor then writes, "Oh, wait, I think I made a mistake, we're actually arriving on Friday, not Sunday." The clerk books a second guesthouse, Acorn Guest House, under reference 0RA8Y7GS, without saying why the venue changed, repeating the new day or mentioning the first booking. The annotated state at the last turn holds both references.
Two details show why every layer needs checking. The scenario brief had asked for Friday, so the worker's slip produced a genuine customer correction, the kind scripts rarely generate. And at the turn where Kirkwood House was booked, the annotation switched the accommodation type from guesthouse to hotel, although the visitor never dropped the guesthouse requirement and the corpus's own database lists both booked venues as guesthouses. A carried-over constraint was lost by the labels, not by anyone in the conversation. The ledger reads:
| Turn | Instruction or action | Status at the last turn | Grading note |
|---|---|---|---|
| 0 and 2 | Free wifi, free parking, moderate price, guesthouse | In force throughout | Carried over by the speakers; the annotated state wrongly switches the type to hotel at turn 7 |
| 4 | 5 people, 2 nights, from Sunday, at the first guesthouse | Venue replaced at turn 6; day replaced at turn 8 | The clerk's no-rooms reply at turn 5 prompts the first change |
| 6 | A different guesthouse in the same price range | In force | Replaces the venue only; party, nights and day carry over |
| 7 | Action: Kirkwood House booked for Sunday, reference ANO69RVP | Still active; never cancelled or mentioned again | Taken under an instruction later superseded; fails the cleanup check |
| 8 | Arriving Friday, not Sunday | In force | Customer-initiated correction; replaces the day only |
| 9 | Action: Acorn Guest House booked, reference 0RA8Y7GS | Active | Fits the instructions in force, but the venue swap is unexplained and the new day is never read back |
Six checks for grading the agent after a change
Grade the agent on the whole ledger, not on its last reply. A final booking can match the customer's latest wishes, as Acorn Guest House does, while a duplicate made under the abandoned instruction stays live. Each check names the record that settles it, and a case is graded only when every record it needs exists.
Score each check pass, fail or not applicable and report the six separately rather than as one blended number, because a model that carries details forward well can still be careless about what it leaves behind. The rubric:
- In-force set: the action matches every instruction in force at the moment it was taken. Settled by the ledger and the action record, such as a booking, order edit or account change.
- Carry-over: no detail the customer left untouched is dropped, reset or asked for again. Settled by comparing the action's parameters with the carried-over rows.
- Replacement link: the change record, or the agent's own note, names the instruction it replaces, as a FIX replace message carries OrigClOrdID. Settled by the change record or an edit-history entry.
- Cleanup: every action taken under a superseded instruction is reversed, kept with the customer's explicit agreement, or reported as impossible to undo. Settled by cancellation, refund or reversal events; SSNG0349 fails here.
- Read-back: the agent states the changed value and any consequence, such as a different venue, a fee or a released hold, before or as it acts. Settled by the transcript.
- Refusal handling: when a change cannot be honored because the original already executed, the agent says so and offers what is still possible, the support version of TooLateToCancel. Settled by the system's rejection or by the timestamp order in the action log.
Holding out the right cases: who changed it, how much, and when
Split test cases by the kind of change, not only by conversation, so the score shows whether a model handles edits it has not seen. Three cuts matter: who prompted the change (the agent's unavailability news or the customer's own correction), how much it touched (one detail or several), and when it arrived (before any action, or after an action that now needs cleanup). In MultiWOZ the agent-prompted booking change is the common case, so a plain random split gives a customer correction after a booking, the SSNG0349 pattern, little weight unless the sample is stratified on purpose.
Multi-turn reliability is where current models are weakest. Laban and colleagues ran more than 200,000 simulated conversations with 15 models across six generation tasks and report an average 39% drop in performance when instructions arrive over several turns instead of in one prompt, much of it from models making early assumptions and answering too soon (arXiv 2505.06120, 2025). Their setting reveals requirements gradually rather than reversing them, but it tests the same skill: keeping a request open while its terms move.
Add a period cut as well. Train on an earlier window and test on a later one, because the rules about what can still be changed, such as cutoffs, change fees and holds, shift over time, and a model that memorized one window's cutoffs will act on stale ones.
Is grading the final state enough on its own?
No. Final-state grading is the strongest alternative to a ledger and it catches a lot: it ignores pleasant wording and checks whether the booking, order or account ended where the customer wanted. What it cannot see is an action left behind by a superseded instruction, unless the grader knows to look, and it cannot tell a correct alternative resolution from a wrong one without knowing what the customer agreed to.
SSNG0349 shows both gaps. Its last booking, Acorn Guest House, is a moderately priced guesthouse with wifi and parking, and the annotated state records it for 5 people, 2 nights and Friday, so a check of the latest booking passes; the visitor's last message is a thank-you. The duplicate Kirkwood House booking slips past both signals. Ticket closure and a warm sign-off say the conversation ended; they do not show that every instruction was honored and nothing extra was left running.
The other alternative, that a long-context model will infer the in-force set from the raw transcript with no ledger at all, is a testable claim rather than a settled one. Grade the same model with and without the ledger fields on a held-out set; if the scores match, the ledger is a labeling aid, and if they differ, it is part of the training signal.
What to ask for when sourcing changing-intent cases
Ask for conversations joined to the system records that show what happened to each instruction: order or booking edit histories, cancellation and reversal events, and timestamps that settle which came first. Without those joins a change can be labeled but not graded.
SourceX sources datasets for AI buyers on request from U.S. businesses instead of selling from an inventory, so the written request is what a supplier's records are screened against. A strong request names the change types in scope, the systems that must confirm each action and reversal, the minimum share of cases where a change follows an executed action, the ledger fields to fill (instruction ID, replaces, status, carried-over details) and an acceptance test, such as two graders agreeing on the six checks for a held-out sample. Whether a particular company's records can meet it is decided company by company, after rights review and de-identification.
Frequently asked questions
How is a changed request different from a new intent?
A new intent replaces the goal, as when a billing question becomes a cancellation. A changed request keeps the goal and edits its terms, such as a new day for the same booking. The second needs carry-over labels, because most of the original request still applies.
Can scripted or simulated conversations supply changing-intent data?
They can supply volume, but each conversation has to be checked for what actually happened. In MultiWOZ 2.1, only 39.7% of the slots that goals scripted to change show the old value and then the new one in the annotated state. Count and label changes from the dialogue and the system records, not from the scenario.
Which business systems keep a record of superseded instructions?
Order and booking systems with edit histories, ticketing tools with field history, and payment systems with cancellation and refund events often do. The test is whether a change record points to what it replaced and whether reversals carry timestamps. A system that overwrites the old value in place loses the supersession.
Why grade actions taken under the old instruction?
Because they can outlive the conversation. In MultiWOZ dialogue SSNG0349, a booking made for the abandoned day stays in the record after the customer's correction and a second booking. A grader that checks only the latest booking passes it.
How should changing-intent cases be split between training and test?
Stratify by who prompted the change, how many details it touched and whether an action had already been taken, then hold out a later period as well. A plain random split underweights rare but costly patterns, such as a correction that arrives after a booking.
Sources
- The MultiWOZ release at commit fe0c8e6 (MIT license) holds 10,438 human-to-human dialogues (3,406 single-domain and 7,032 multi-domain); its README says bookings set to fail were hidden from workers and that goals were sometimes followed wrongly, which can leave the belief state wrong; the SourceX counts in this article were computed from its MultiWOZ_2.1.zip. Source
- The MultiWOZ paper (EMNLP 2018) describes goal changes built in on purpose: initial constraints that may match no database entry, after which the user switches to an alternative. Source
- In the FIX standard, OrderCancelReplaceRequest (35=G) changes an open order, gets its own ClOrdID and is treated as a separate entity, with OrigClOrdID identifying the previous order; OrderCancelReject returns refused changes, filled orders cannot have price changed or quantity reduced, and reject reason 0 is TooLateToCancel. Source
- HL7 FHIR's ServiceRequest.replaces element references earlier requests the new request takes the place of, with supersedes, prior and renewed order as alternate names. Source
- Shopify's order editing API begins an edit with orderEditBegin, returns a calculated order previewing the changes and keeps changes staged until they are committed with orderEditCommit or abandoned. Source
- Laban et al. (2025) simulated more than 200,000 conversations with 15 models on six generation tasks and found an average 39% performance drop when instructions arrive across turns, largely from early assumptions and premature answers. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.