Skip to content

AI uses for records

Business call interruption dataset: labeling when callers cut in

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Interruption handling in a business voice agent is learned from overlap events, not from transcripts. Each event needs separate speaker channels, millisecond onsets, what the agent was doing when the caller cut in, why the caller cut in, how the agent responded and whether the call recovered. A business call interruption dataset labels those events with a fixed guide and keeps them apart from later, verified call outcomes.

Call to transcript

Business call interruption

This page covers business call interruption dataset: labeling when callers cut in.Illustrative

Key takeaways

  • Interruption handling is learned from labeled overlap events, which transcripts usually erase.
  • Separate backchannels, corrections, objections, escalations and side talk, because each needs a different agent response.
  • Keep caller and agent on separate channels and keep the agent's playback log, or yield time cannot be measured.
  • Label response quality per overlap type and keep later business outcomes in a linked field, never in the label.
  • Recording consent and licensing rights are separate checks that come before any audio is shared.

What training evidence supports interruption handling in business voice agents?

The evidence is a set of labeled overlap events cut from real business calls: moments when the caller started speaking while the agent, human or automated, was still talking. For each event a voice team needs the audio on separate channels, the exact onset and length of the overlap, the agent's state at that instant, the caller's reason for cutting in, the agent's response and what the next few turns show. A transcript alone cannot carry this, because most transcripts flatten overlapping speech into tidy alternating lines.

Voice frameworks make the need concrete. LiveKit Agents exposes interruption settings including a minimum speech duration before speech counts as an interruption (0.5 seconds by default), a timeout after which an interruption with no follow-up speech is classed as false (2.0 seconds by default), and an option to resume the agent's speech after a false interruption. Pipecat ships a turn start strategy that requires a minimum number of words while the bot is speaking but only one word when it is not. Every one of those thresholds is a guess unless a team can test it against labeled business calls.

This guide sets out the overlap types to distinguish, what the recording must preserve, a field-level turn interruption annotation guide, response-quality labels, an illustrative worked event and the calls that should not be labeled at all.

Not every overlap is an interruption

The first labeling decision is whether the caller meant to take the turn. Treating every overlap as a barge-in trains an agent that stops talking whenever a caller says uh-huh, which callers experience as nervous and slow. Treating none as a barge-in trains the agent that talks over the customer. Research benchmarks draw the same line: Full-Duplex-Bench evaluates pause handling, backchanneling, smooth turn-taking and user interruption as separate behaviors, and treats an utterance as a backchannel when it lasts under one second and contains fewer than two words.

Use a closed list of overlap types and require one per event. The duration and word thresholds in the benchmark are a useful starting heuristic for pre-labeling, but the human label should rest on what the caller did next, since a short no can be a real objection.

Not every overlap is an interruption
Overlap typeWhat the caller is doingDesired agent behaviorTypical cue
BackchannelSignaling attention without taking the turnKeep talkingShort token such as okay or right, no follow-up content
CorrectionFixing something the agent just said or assumedStop, acknowledge, take the correctionRepeated or contradicting value, such as a different date
Answer aheadAnswering a question before the agent finishes asking itStop and accept the answerCaller supplies the requested field mid-prompt
Objection or refusalRejecting the offer, policy or next stepStop and address the objectionNo, wait, that is not what I asked
Urgency or escalationNeeds a person or has a time-critical problemStop and route as policy requiresRequest for a human, safety words, raised voice
Side talk or noiseSpeaking to someone else, background audioKeep talking or pause brieflyOff-axis voice, television, a second speaker
Agent over callerThe agent started while the caller still held the turnYield and let the caller finishAgent onset inside an unfinished caller phrase

What the recording must preserve before labeling starts

Keep caller and agent on separate channels and keep the original timing. Mixed mono audio makes it hard to tell who started first and how long both voices overlapped, and any diarization step adds its own errors. The pyannote.metrics documentation notes that diarization error rate accounts for overlapping speech, which can raise missed detection when a system has no overlap detection module. In other words, the very regions an interruption dataset cares about are where automatic speaker separation is weakest.

At a glance

What the recording must preserve before labeling starts

Do not apply a forgiveness collar at overlap boundaries. Speaker diarization research commonly removes a 500 millisecond collar around each turn boundary from scoring, because manual boundaries are imprecise, and pyannote.metrics supports a collar parameter that defaults to zero. For interruption work the boundary is the signal, so annotate onsets on the waveform and score without a collar, or report both.

For automated agents, keep the agent's playback log: when each synthesized utterance started and stopped playing on the line, and whether it was cut. Without it, the dataset cannot show how long the agent kept talking after the caller started, which is the measurement most teams want.

  • Two channels, caller and agent, at the original sample rate.
  • Word-level transcript timestamps aligned to each channel.
  • Agent playback start, stop and truncation times for automated calls.
  • Agent state events: speaking, listening, waiting on a tool or lookup, on hold, transferring.
  • Transfer and hold markers, IVR prompts and queue messages tagged as non-agent audio.
  • Recording disclosure and consent evidence per call.

The turn interruption annotation guide

Label one record per overlap event using the fields below. The guide separates what happened acoustically from what the caller intended and from how the agent responded, so a voice team can tune detection thresholds, intent handling and recovery behavior independently. Two annotators label each event, and an adjudicator settles disagreements on type and intent.

The turn interruption annotation guide
FieldDefinitionAllowed values or formatValidation check
event_id and call_idStable IDs for the overlap event and its callOpaque IDs, no customer identifiersUnique; call_id resolves to one recording
onset_ms and overlap_msCaller speech onset relative to call start, and length of simultaneous speechIntegers in millisecondsOnset falls inside an agent speaking interval
agent_stateWhat the agent was doing at onsetSpeaking, waiting on tool, reading back, hold message, closingOne value; matches the state log
overlap_typeThe type from the overlap tableClosed list of seven typesOne value; backchannel cannot carry a turn intent
caller_intentWhy the caller took the turnCorrect, answer, object, escalate, clarify, noneRequired unless type is backchannel or side talk
agent_responseWhat the agent did nextYielded, continued, paused then resumed, restarted utteranceMatches playback log for automated calls
yield_msTime from caller onset to agent silenceInteger in milliseconds, or null if it did not yieldNull only when response is continued
repair_turnsTurns spent recovering the lost or wrong informationInteger count with turn IDsTurn IDs exist in the transcript
response_qualityWhether the response fit the overlap typeAppropriate, premature yield, failed to yield, lost contentRequired; rationale note for anything but appropriate
outcome_linkPointer to the later business outcome record, if anyRecord ID, outcome date and maturity statusOutcome dated after the call; never edited into the event label

Which calls to sample so rare interruptions show up

Sample by overlap type and call context, not at random. In most business calls backchannels vastly outnumber escalations and objections, so a random sample produces a dataset that teaches an agent to ignore overlaps. Pre-label overlaps automatically from the two channels, then draw events so that each type in the taxonomy has enough examples to measure, and record the sampling weights so results can be reweighted to real call traffic.

Stratify by the agent state at onset as well. Corrections during read-backs, objections during disclosures and escalations during hold messages are the moments that break voice agents, and they occur in specific parts of a call. Include calls handled by human agents alongside automated ones: human agents show what good yielding looks like, while automated calls show the failure modes a team is trying to fix.

Measure annotator agreement on overlap_type and caller_intent before scaling the labeling, and publish the agreement figure with the dataset. Where agreement is low for a type, tighten the definition or merge it with its neighbor; a category annotators cannot apply consistently is not one a model can learn.

Was the agent's response right for that overlap?

Response quality is where the training value sits, so define it per overlap type rather than as a general good or bad. A premature yield is stopping for a backchannel or for side talk. A failed yield is talking through a correction, an objection or an escalation. Lost content is yielding correctly but then dropping what the agent was saying, so the caller never hears the confirmation number or the cancellation terms.

Pay special attention to the moments when the agent is busy. Pipecat includes a strategy that mutes user input while a function call is executing, so a caller's correction during a lookup can be discarded by design. LiveKit can likewise drop buffered caller audio while the agent cannot be interrupted, and by default suppresses speech its adaptive detector classes as a backchannel near the start and end of each agent turn. Label agent_state precisely at those moments, because they are the events a team needs to decide whether those settings fit its calls.

End-of-turn errors belong in the same dataset. The Smart Turn project notes that most voice agents base turn detection on voice activity detection, which cannot use linguistic or acoustic content, and offers an audio model that uses prosody instead. When the agent starts speaking while the caller is mid-sentence, label it agent over caller; those events are the training evidence for better endpointing.

Keeping call signals apart from verified outcomes

Attach business outcomes through a link, not by rewriting the event label. An interruption that the agent handled well can still precede a canceled order, and a clumsy exchange can end in a completed booking. If the outcome is written into response_quality, a model learns that whatever preceded a sale was good handling.

Record each outcome with its own date and a maturity status: pending while it could still change, settled once the order shipped, the appointment happened or the refund window closed. Count repair turns and call length as conversational signals, and treat orders, cancellations, complaints and repeat contacts as outcomes from systems of record such as the order management or ticketing system. The outcome-alignment method for whole calls is covered in more depth in a separate guide; here the point is only that the interruption label stays fixed once adjudicated.

Choose the outcome window before linking anything. A repeat call about the same issue within a short, declared period is a reasonable sign that the information lost in an interruption mattered; a cancellation months later usually has other causes. Write the window, the systems queried and the matching key into the dataset card, and mark events whose outcome is still pending rather than dropping them, so the outcome field never quietly selects for calls that ended well.

Worked example (illustrative): a correction during a read-back

This event is invented for illustration. An automated agent for a field service company is reading back a booking: the address, then the appointment window. Two seconds into the window, the caller says no, the afternoon, I said the afternoon. The overlap lasts just under a second before the agent stops.

Labels: overlap_type correction, caller_intent correct, agent_state reading back, agent_response yielded, yield_ms recorded from the playback log. The agent then restarts the read-back from the beginning, including the address the caller had already confirmed, which costs two repair turns. response_quality is appropriate for the yield and the annotator adds a note that the restart repeated confirmed content. The outcome_link points to the job record, settled once the visit took place in the afternoon window.

The exception: in a second illustrative call the caller says okay, okay twice during the same read-back and the agent stops both times. Those are backchannels, so response_quality is premature yield, and that pair of events is the evidence for a higher word or duration threshold during read-backs.

Calls to leave out or route to a specialist

Leave out calls where the overlap cannot be attributed: mixed mono recordings with no reliable separation, speakerphone calls with several people on the caller side, and calls with heavy hold music or IVR audio on the agent channel. Interpreted calls need a separate guide, because the interpreter's overlaps follow different norms.

Keep synthetic overlaps out of the same split as real ones. Mixing two clean recordings so that one voice lands on top of another produces an overlap with the right acoustics but no real intent behind it: nobody was correcting anything. Such clips can help train a detector, but labeling them with caller_intent would invent a reason the caller never had, so tag them as synthetic and evaluate only on real calls.

Route to a voice evaluation lead any event type that annotators split on after adjudication, and route to an operations reviewer any event where the right agent behavior depends on company policy, such as whether an agent may stop a required disclosure when the caller objects. Calls involving distress, safety, medical or financial hardship topics need a policy decision before they enter a training set at all.

State the limits of the dataset with it: the overlap norms of one language, region and call type do not transfer automatically, and a guide tuned on booking calls says little about collections or technical support.

Recording rights before interruption audio is licensed

Interruption data is raw call audio, so consent and licensing rights come first. Federal law permits a private party to intercept a call it is part of, or one where a party has given prior consent, unless the purpose is criminal or tortious, but some states set a stricter rule; California Penal Code section 632, for example, prohibits recording a confidential communication without the consent of all parties, and recording consent obtained for quality purposes is not automatically permission to license audio for AI training. Access to recordings, privacy permission, ownership and contractual licensing rights are separate questions, and de-identifying a transcript does nothing for the voices in the audio. Counsel reviews the specific call flows and notices.

AI teams that need business call audio can describe the call types, volume, language, format, timing and licensing terms through the SourceX request form; datasets are sourced from U.S. businesses for each request rather than held in stock. The business that owns the recordings approves the scope and signs before any audio is delivered.

Frequently asked questions

What should a turn interruption annotation guide include?

It should define a closed list of overlap types, the caller's intent, the agent's state at the moment of overlap, the agent's response, the time it took to yield, the repair turns that followed and a judgment of whether the response fit the overlap. A link to the later business outcome belongs in a separate field so it never rewrites the label.

Does the recording support the transcript for interruption labels?

Only if caller and agent are on separate channels with original timing, and for automated agents only if the playback log shows when synthesized speech started and stopped. A mixed recording or a transcript that alternates speakers cleanly cannot show who spoke first or for how long both spoke.

How is a backchannel different from an interruption?

A backchannel signals attention without taking the turn, such as a short okay, and the agent should keep talking. An interruption takes the turn to correct, object, answer ahead or escalate. Short duration and few words are a useful first filter, but the label should rest on what the caller did next.

Can contact center recordings be used for an interruption dataset?

They can be the best source, because they hold real corrections and objections, but recording consent, notices, contracts and the company's right to license all need review first. Consent given for quality monitoring is not automatically permission to license audio for AI training.

Why score yield time in milliseconds?

Because the difference between an agent that stops promptly and one that keeps talking over the caller is often less than a second, and frameworks set thresholds at that scale. Measuring yield time from the playback log lets a team compare threshold settings against what callers actually did.

Sources

  • LiveKit Agents interruption options include a 0.5 second minimum speech duration, a 2.0 second false-interruption timeout, resuming speech after a false interruption, and backchannel suppression near the start and end of agent turns. Source
  • Pipecat's minimum-words turn start strategy requires the configured number of words while the bot is speaking and a single word when it is not. Source
  • Pipecat's function call mute strategy mutes all user input while a function call is executing. Source
  • Smart Turn is an open audio turn detection model; most voice agents use voice activity detection for turn detection, which cannot account for linguistic or acoustic content. Source
  • Diarization error rate accounts for overlapping speech; a 500 millisecond collar is common in diarization research and pyannote.metrics defaults the collar to zero. Source
  • Full-Duplex-Bench evaluates pause handling, backchanneling, smooth turn-taking and user interruption, and defines a backchannel as under one second with fewer than two words. Source
  • Federal law does not prohibit a private party from intercepting a communication it is party to, or where a party has given prior consent, unless the purpose is criminal or tortious. Source
  • California Penal Code section 632 prohibits intentionally recording or eavesdropping on a confidential communication without the consent of all parties. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies