Skip to content

AI uses for records

Business conversation repair dataset: labeling misunderstandings and recoveries

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Represent each repair as an episode, not a single turn: the trouble source, the turn where someone signals the misunderstanding, the repair itself and what happened next, each with speaker, timestamps and a link to the audio or chat log. Keep these conversational labels separate from the verified business outcome, such as the correct order shipping, so a model learns to recover rather than merely to apologize.

Call to transcript

Business conversation repair

This page covers business conversation repair dataset: labeling misunderstandings and recoveries.Illustrative

Key takeaways

  • Label each repair as an episode with a trouble source, repair initiation, repair and resolution, not as a single corrected turn.
  • Record who noticed the problem and who fixed it, because each repair type calls for different agent behavior.
  • Take the outcome from an order, ticket or billing system, and mark it provisional until it has had time to settle.
  • Train on recovery turns and mask the agent's misreading so the model is not rewarded for the error.
  • Hold calls whose recording basis cannot be shown, and treat speaker labels as per-file until mapped to roles.

How can conversation repair examples be represented in AI training data?

Store each misunderstanding and its recovery as a bounded episode inside the full conversation, with four marked parts: the trouble source, the repair initiation, the repair and the resolution. Every part points to exact turns, speakers and times, and the episode carries a separate outcome record from a business system that shows whether the recovery actually worked.

That split matters because the conversation alone can look like a success when it was not. An agent can say sorry, rephrase and get a thank you, while the order system shows the wrong item shipped anyway. A business conversation repair dataset is valuable to the degree that it can tell those cases apart, and it can only do that if the outcome comes from a system of record rather than from the tone of the last turn.

The same structure works for chat, email threads and phone calls. Calls add audio, speaker separation and timing, which carry signals text loses, such as a pause before a correction or two people talking over each other.

Which kinds of repair should a business dataset tell apart?

At minimum, label who noticed the problem and who fixed it, and where the fix happened relative to the trouble. Conversation analysts have used that framing since Schegloff, Jefferson and Sacks's 1977 paper on the organization of repair, which describes a strong skew toward self-repair over repair by the other speaker. For a support or sales dataset, the practical value is that each type calls for a different agent behavior.

The table below adapts the framing to business conversations. Example turns are illustrative.

Two of these types deserve extra attention when sampling. Customer-initiated repair of the agent's response is the classic actually, that's not what I meant moment, and it is the hardest for agents because the misreading has already shaped the agent's plan. Repair after action is rarer but the most costly, because something in a business system has to be undone, and the dataset should record whether it was. A pack that over-samples easy self-repairs will make a model look better at recovery than it is.

Which kinds of repair should a business dataset tell apart?
Repair typeWho noticesIllustrative turnWhat the label should record
Customer self-repairThe customer, mid-turnI need the 12-inch, sorry, the 14-inch bladeCorrected value and whether the agent used the corrected one
Agent-initiated clarificationThe agent, before actingDo you mean the account ending 41 or the new one?Ambiguity type and whether the question was needed
Customer-initiated repair of agent's responseThe customer, after the agent acts on a misreadingActually, that's not what I meant, I wanted to pause itTrouble source turn, misreading and recovery turn
Agent self-repairThe agent, after its own errorCorrection, that delivery date was for the other orderError type and whether the customer had already relied on it
Repair after actionEither side, after a system change was madeI see the cancellation went through, I only asked to pauseSystem action to undo and whether it was undone

Why public dialogue data says so little about recovering from a misunderstanding

Because most task-oriented dialogue datasets were written or simulated to complete tasks cleanly, and misunderstandings are either rare or reduced to a yes or no. Repairs that unfold over several turns, or that surface after the agent has acted, are what real operations produce and what public data mostly lacks.

The Schema-Guided Dialogue dataset is a good illustration of what is covered. Its annotation includes a system CONFIRM act for checking a slot value before a transactional service call and user acts such as AFFIRM and NEGATE, plus a per-turn dialogue state with slot values. That captures a confirmation step, but a NEGATE carries no slot or value, so the dataset records that the user said no without recording what they meant instead or how the misunderstanding arose.

Research datasets aimed at repair are starting to fill the gap, though not with business conversations. BlockWorld-Repairs, presented at EMNLP 2024, collects third position repairs, where a person corrects an agent after seeing its response, in a block-manipulation instruction task. A study of Google Assistant and Siri in English and Spanish found ten assistant repair strategies but no reproduction of the human huh? style of repair, and a gap between the strategies users preferred and those the assistants used most. Both point to the same need: labeled recoveries from real, multi-turn work.

What should the misunderstanding and recovery conversation pack include?

It should include four layers: the source conversation with alignment, the repair episode labels, the outcome record and the rights record. The field list below is the pack specification; it is written so a data team can use it as is and so a buyer can check a sample against it.

At a glance

What should the misunderstanding and recovery conversation pack include

Keep the episode as pointers into the full conversation instead of copying turns into a separate file. A conversation can contain several episodes, sometimes nested, and pointers let a training pipeline cut whatever window it needs while the episode labels stay consistent. Store negative cases too: conversations where an ambiguity existed but nobody needed to repair it, so a model can learn when not to ask a clarifying question.

What should the misunderstanding and recovery conversation pack include?
LayerFieldRuleIllustrative value
Sourceconversation_id, channel, languageOne ID per call or thread across all filescall-5521, phone, en
Sourceturns with speaker role and start and end timeRoles from a fixed list; times in seconds from call startagent, 63.2 to 66.9
Sourceaudio_ref and transcript_versionLink each turn to its audio span and transcript revisionaudio segment 14, transcript v2
Episodetrouble_source_turn and misreadingPoint to the turn that caused the problem and state the misreadingturn 9, part number read as 2240
Episoderepair_initiation_turn, repair_typeUse the repair type vocabularyturn 15, customer repair of agent response
Episoderepair_turns and resolution_turnAll turns that fix the problem, ending at the confirmed corrected valueturns 16 to 19
Outcomeoutcome_source, outcome_label, observed_atFrom an order, ticket or billing system, never from the transcriptorder system, correct part shipped
Outcomeoutcome_maturityFinal, provisional or unknown, with the wait period appliedfinal after return window closed
Rightsrecording_basis, permitted_use, redactionsRecorded from the business's own records; reviewed by counselnotice played, training use reviewed

An annotated episode: the wrong replacement part

A customer calls a parts distributor to reorder a filter cartridge. Every detail in this episode is illustrative. The customer reads a part number from an old box; the agent mishears one digit, repeats the description of a different cartridge, and the customer agrees because the description sounds close.

A few turns later the agent reads back a delivery date and the product name. The customer says that is not the one in the box, it should fit the larger housing. That turn is the repair initiation. The agent asks the customer to read the number again digit by digit, confirms it back, cancels the first line and adds the correct one. The episode ends when the customer confirms the corrected order.

The weak acceptance at turn 11 is the instructive detail. A transcript-only label might mark turn 11 as confirmation and the episode as a smooth order. The pack marks it as acceptance under a misreading, which gives an evaluator a second test point: a model that reads back the part number at turn 10, instead of a description, would have prevented the repair entirely. The exception to keep in mind is that not every late correction is a repair of a misunderstanding; if the customer simply changes their mind about the part, the episode is a change of request and belongs in a different label.

An annotated episode: the wrong replacement part
TurnSpeaker and timeWhat is said (paraphrased)Episode label
9Customer, 41.0 to 47.5 secondsReads the part number from the old boxTrouble source
10Agent, 47.9 to 53.2 secondsRepeats a description of a different cartridgeMisreading
11Customer, 53.6 to 54.4 secondsSays yes, that sounds rightWeak acceptance, not a repair
15Customer, 71.3 to 76.0 secondsSays it is not the one in the box; it must fit the larger housingRepair initiation
16 to 18Agent and customer, 76.4 to 101.8 secondsDigit-by-digit read and read-back; line cancelled and replacedRepair
19Customer, 102.1 to 104.0 secondsConfirms the corrected orderResolution
OutcomeOrder system, 9 days laterCorrect cartridge shipped; no return openedVerified recovery, final

Does the recording support the transcript and the claimed outcome?

Check it before labeling, not after. A repair label is only as good as the turn boundaries and speaker labels under it, and an outcome label is only as good as the link from the call to the right order or ticket.

Transcription errors are a particular trap for repair data, because the trouble source is often a misheard word. If the transcript records what the agent thought it heard rather than what the customer said, the episode loses its cause. Annotators should listen to the trouble source turn in every episode, write what was actually said in a separate field, and keep the machine transcript as is, since the gap between the two is itself useful training signal for speech models.

Timing and speaker conventions already exist and are worth reusing. WebVTT, the W3C text track format, gives every cue a start and end timestamp and lets a voice span name the speaker. Diarization services return speaker segments with start and end times; Amazon Transcribe, for example, labels speakers spk_0, spk_1 and so on, and its documentation notes that these labels are assigned per file and are not consistent across separate recordings. Map them to roles such as agent and customer per call, and never assume spk_0 means the same person in two files.

  • Spot-listen a sample of repair episodes against the transcript and fix any turn whose text or speaker is wrong before labeling the rest.
  • Confirm that each episode's outcome comes from a business system linked by an order, ticket or account ID, not inferred from the closing words of the call.
  • Set an outcome maturity rule per outcome type, such as waiting until a return window or reopen window has closed, and mark younger episodes provisional.
  • Record the transcript version used for labels, since a new transcription pass can move turn boundaries and break episode pointers.

Turning repair episodes into training targets and test items

Use the episode labels to decide which turns become targets, which become context and which become evaluation items. The most common mistake is training a model on the agent's misreading as if it were a good response, simply because it is an agent turn.

For supervised fine-tuning, the recovery turns are the targets and the trouble source, misreading and repair initiation are context, so the model learns what to do once a customer signals a problem. Mask the misreading itself from the training loss. For preference data, pair the observed recovery with a weaker alternative, such as an apology without a read-back, and rank them using the verified outcome rather than a rater's sense of tone.

For evaluation, cut the conversation just after the repair initiation and ask the model for the next agent turn, then score whether it identifies the trouble source and confirms the corrected value. A second test cuts the conversation after the weak acceptance at turn 11 and checks whether the model notices the risk before the customer does. Both tests need the outcome record to separate a recovery that worked from one that only sounded right.

Report results by repair type rather than as one recovery score. A model can be good at answering explicit corrections and poor at noticing weak acceptance, and the two call for different fixes: more recovery targets in the first case, and more read-back and confirmation behavior in the second. Keep a held-out slice of episodes from businesses or time periods not used in training so the test measures recovery on unfamiliar conversations, not memory of familiar ones.

Episodes to drop or hold before they reach a training set

Drop or hold an episode when the recording basis is unclear, when the outcome cannot be linked, or when the misunderstanding involves sensitive details. A small, clean set of verified recoveries is more useful than a large set of guesses.

Treat these as gates applied before annotation effort is spent, and record the reason for each exclusion so the final pack can report how many episodes were removed and why.

  • Hold any call whose recording notice or consent basis cannot be shown from the business's records, and let counsel decide whether training use is allowed in the relevant jurisdictions.
  • Drop episodes whose outcome system link is missing or ambiguous, rather than labeling the outcome from the conversation.
  • Hold episodes where the misunderstanding concerns health, financial hardship, identity checks or payment details for privacy review, since redaction can remove the very turn that carries the repair.
  • Separate bot and macro turns from human agent turns, and do not mix automated-agent repairs with human repairs in one label without marking which is which.
  • Exclude unresolved episodes from targets but keep them as evaluation items, labeled unresolved, because they show where recovery fails.

Finding real repair episodes in support and sales records

The best sources are operations where conversations and outcomes are both kept: contact centers that retain recordings and transcripts alongside order or ticket systems, field service dispatch desks that log changes after calls, and sales desks whose order amendments are tied to call records. Repairs are common in these settings, and the system logs supply the verified outcome.

A team can describe the episodes it wants, such as phone and chat conversations with linked order outcomes, to SourceX, which fills each request from U.S. businesses rather than from stock, guided by the scope, volume, language, format, timing and licensing needs stated. No recording leaves a business until it has approved the scope and signed.

Frequently asked questions

What is conversation repair?

It is how speakers deal with problems in hearing, speaking or understanding during a conversation, from a quick self-correction to a later turn that says the other person misunderstood. In business conversations it covers clarifying questions, corrections and the recovery after an agent acts on a misreading.

Can synthetic conversations replace real repair episodes?

They can cover rare repair types and help test formats, but they tend to be tidier than real calls and lack verified outcomes. Use them to fill gaps and compare model behavior on synthetic and real slices before relying on either.

Should the agent's mistake be removed from the training data?

Keep it as context but exclude it from the training targets. The model needs to see the misreading to learn how to recover from it, but it should not be rewarded for producing it.

Why link conversations to outcomes from business systems?

Because a polite closing does not prove the problem was fixed. An order, ticket or billing record shows whether the corrected request was actually carried out, which is the label that separates real recovery from a conversation that only sounded resolved.

Do voice recordings add anything beyond the transcript for repair data?

Often, yes. Pauses, overlapping speech and emphasis can mark the moment a customer realizes something is wrong, and audio lets you check that the transcript and speaker labels are right. Audio also raises extra consent and voice privacy questions that counsel should review.

Sources

  • Schegloff, Jefferson and Sacks's 1977 paper in Language describes the organization of repair in conversation and a strong skew toward self-repair over other-repair. Source
  • BlockWorld-Repairs, presented at EMNLP 2024, is a released dataset of multi-modal third position repair sequences in a block manipulation instruction task. Source
  • A study of Google Assistant and Siri in English and Spanish found ten assistant repair strategies, an inability to replicate human-like huh? repair, and a gap between user-preferred and most-used strategies. Source
  • The Schema-Guided Dialogue dataset defines a system CONFIRM act before transactional service calls, user AFFIRM and NEGATE acts with empty slot and values, and a per-turn dialogue state with slot values. Source
  • WebVTT cue timings carry start and end timestamps, and a cue voice span annotates the name of the voice speaking. Source
  • Amazon Transcribe speaker partitioning labels speakers spk_0 onward with start and end times, and the labels are not consistent across separate audio files. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies