AI uses for records
Call silence and hold time labels: what pauses teach a voice agent
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
A business voice dataset should keep every non-speech span as a timed event, not a single silence total: its start and end, the channel, a cause class (such as hold, mute, system lookup, customer task, hesitation, transfer, redaction mute or technical dead air), the evidence that proves the cause, the turns on either side, and who was waiting. Outcome labels stay separate and are added only once the result is verified.
Call to transcript
Call silence hold time
Key takeaways
- Label each pause as an event with a cause, not as a share of call time.
- Every cause label needs evidence: a hold signal, a platform event, desktop activity or the words around the gap.
- Short gaps between turns are normal conversation, so set thresholds deliberately and record them.
- Redaction mutes and technical dead air look like silence but must never be labeled as hesitation.
- Reconcile labeled holds against the platform's own hold count and duration before release.
- Keep conversational signals apart from downstream outcomes, which mature later.
What should a business voice dataset preserve about silence, holds and hesitation?
Preserve each pause as its own record with exact timing, the channel it occurred on, why it happened and how you know. The cause is the valuable part: a voice agent learns different lessons from a customer thinking before answering, an agent searching a billing system and a deliberate hold.
Most call archives keep none of this. They hold a recording, a transcript with word timings if you are lucky, and contact-level totals such as hold time. The transcript drops silences entirely, and the totals blend every kind of quiet into one number. Rebuilding the events later is possible only if the platform events and channel-separated audio were kept.
The labels serve concrete agent behaviors: when to wait rather than reprompt, when to announce that it is checking something, how long a hold can last before checking back, and how to tell a finished turn from a pause for thought. Each depends on knowing the cause of the silence, not just its length.
Why is one silence number not enough?
An aggregate silence figure mixes causes that call for opposite agent behavior, so it cannot be used as a training label. It also depends on definitions that differ by platform.
Amazon Connect's Contact Lens, for example, defines non-talk time as hold time plus any silence longer than 3 seconds where neither participant is talking, and its documentation says that duration cannot be customized. Amazon Transcribe Call Analytics reports non-talk time alongside talk time, interruptions, loudness and talk speed as call characteristics. Both are useful operating metrics. Neither tells you whether a 12-second gap was a customer reading a card number aloud slowly, an agent waiting on a slow screen or a muted line.
Short gaps need a deliberate threshold too. In a study of 10 languages, Stivers and colleagues found that speakers generally avoid overlap and keep silence between turns to a minimum, with each language's average gap falling within 250 milliseconds of the cross-language mean. Ordinary turn transitions are therefore brief, and labeling every sub-second gap as an event would bury the pauses that matter. Record the threshold you chose and keep the raw timings so others can choose differently.
Where does each pause come from, and what proves its cause?
Assign a cause only when you have evidence for it, and record which evidence you used. Audio alone can show that nobody spoke; it rarely shows why.
The strongest evidence comes from systems rather than listening. In SIP calls, RFC 3264 describes putting a media stream on hold by changing its direction attribute, to sendonly for a stream that was sendrecv or to inactive for one that was recvonly, so hold shows up in signaling. Contact center platforms also log hold at contact level: Amazon Connect contact records include CustomerHoldDuration, NumberOfHolds and LongestHoldDuration, which give you totals to reconcile against.
| Cause class | What it is | Evidence that proves it | Common confusion |
|---|---|---|---|
| hold | Agent placed the customer on hold | Hold signal or platform hold event, often hold audio on the customer leg | Long lookup without a hold |
| mute | One party muted while still connected | Platform mute event; speech absent on one channel only | Hold |
| system_lookup | Agent silent while working in a tool | Desktop or CRM activity in the gap; phrases like 'one moment' | Agent hesitation |
| customer_task | Customer finding a document or card | Request immediately before, information supplied after | Customer hesitation |
| hesitation | Pause before a reply, often with filled pauses | Question before, delayed or hedged answer after, no system activity | Customer task |
| transfer_or_queue | Call moving between agents or queues | Transfer event, new agent leg, queue audio | Hold |
| ivr_or_prompt | Silence inside automated menus | IVR flow logs and prompt timestamps | Customer hesitation |
| redaction_mute | Audio removed during preparation | Redaction manifest span | Any natural silence |
| technical_dead_air | Packet loss, one-way audio or dropped leg | Network or media quality logs; both channels silent unexpectedly | Hold or mute |
The silence and hold event labeling schema
Each event record links to a call and its turns, carries its own timing and cause, and states how it was detected. The ten fields below are required; the list after the table adds optional fields that make the set more useful for evaluation.
At a glance
The silence and hold event labeling schema
Times are in milliseconds from the start of the recording, measured on the channel-separated audio where available. Store the detection settings with the record, because a different voice activity detector or threshold produces different boundaries.
- Optional: filled_pause_tokens, the hesitation markers in the next turn, such as 'um' or a restarted sentence.
- Optional: check_back_ms, time from hold start to the first check-back during the hold.
- Optional: redaction_overlap_ms, any overlap with a redaction span, which must be zero for natural cause classes.
- Optional: outcome_ref and outcome_matured_at, a pointer to a verified downstream outcome and the date it became final.
- Optional: labeler_id and agreement, who labeled the cause and whether a second labeler agreed.
| Field | Type | Definition | Rule |
|---|---|---|---|
| event_id | string | Unique id of the pause event | Unique within the dataset |
| call_id | string | Pseudonymous id of the call | Matches a call in the manifest |
| start_ms and end_ms | integer | Event boundaries from recording start | end_ms greater than start_ms; within call length |
| channel | enum | agent, customer or both | both only when every channel is silent |
| cause_class | enum | One of the nine cause classes | unknown allowed, never guessed |
| cause_evidence | list | Evidence types used: signaling, platform_event, desktop_activity, lexical, manifest, listening | At least one entry unless cause_class is unknown |
| prev_turn_id and next_turn_id | string | Transcript turns either side of the gap | Both present unless the event starts or ends the call |
| waiting_party | enum | Who was waiting: customer, agent, both or none | Required for hold, system_lookup and customer_task |
| announced | boolean | Whether the pause was announced before it began | Points to the announcing turn when true |
| detection | object | Detector name, version and thresholds | Identical across a release unless versioned |
A valid record and a failing record
These records are illustrative and do not come from any real call or customer. Both describe a pause in the same billing call, after the customer asks why a late fee appeared.
- Valid: event_id E-0412; call_id C-88; start_ms 241200; end_ms 287900; channel both; cause_class hold; cause_evidence signaling and platform_event; prev_turn_id T-31, where the agent says she will check the account and place the customer on a brief hold; next_turn_id T-32, where she thanks the customer for waiting; waiting_party customer; announced true; detection Silero VAD with the release's documented thresholds.
- Why it passes: timing is inside the call, the cause has two independent kinds of evidence, the announcing turn is linked, and the hold reconciles with the contact record's single hold.
- Failing: event_id E-0413; call_id C-88; start_ms 289100; end_ms 296400; channel customer; cause_class hesitation; cause_evidence listening; prev_turn_id T-32; next_turn_id T-33; waiting_party none; announced false.
- Why it fails: the redaction manifest shows a redaction span covering most of the gap, because the customer was reading a card number. The silence is a redaction_mute, so the hesitation label would teach a model that customers pause before giving payment details. The validator rejects it because redaction_overlap_ms is not zero for a natural cause class.
Which rules should a validator enforce?
Run automatic checks on every release so labels stay consistent with the audio, the transcript and the platform's own records. Each rule below either rejects a record or flags it for a second labeler.
At a glance
Which rules should a validator enforce
Boundaries depend on the detector. Silero VAD's speech-timestamp function, for instance, defaults to a speech probability threshold of 0.5, a minimum silence of 100 milliseconds before splitting, a minimum speech chunk of 250 milliseconds and 30 milliseconds of padding on each side of speech, and that padding shortens every measured silence. Diarization tools such as pyannote.audio return speaker turns with start and end times, which give the turn ids an event must sit between. Record the tool versions and settings with the release.
- Timing: start_ms is before end_ms, both lie within the call, and events on the same channel do not overlap.
- Turn alignment: the gap between prev_turn_id's end and next_turn_id's start contains the event, within the stated padding.
- Hold reconciliation: per call, labeled hold events match the platform's hold count, and their total matches its hold duration within a stated tolerance.
- Channel logic: an agent-channel speech segment inside a hold event is flagged, since the agent should not be heard by the held customer.
- Redaction: natural cause classes have zero overlap with redaction spans; redaction_mute events match a manifest span.
- Evidence: any cause other than unknown lists at least one evidence type, and hesitation never relies on listening alone when system activity exists in the gap.
- Thresholds: no event is shorter than the release's minimum event length, and the detector settings match the release record.
- Outcomes: outcome_ref is set only when outcome_matured_at is present and earlier than the release date.
Do the audio, the transcript and the later outcome agree about each pause?
Check that the audio supports each labeled turn and gap before trusting any downstream label. Transcripts compress or omit restarts, filled pauses and quiet speech, and a pause that looks long in the transcript may contain speech the transcriber dropped.
Keep conversational signals and business outcomes on separate tracks. A customer's long hesitation before accepting a payment arrangement is a conversational fact recorded at the moment of the call. Whether they kept the arrangement is an outcome that only exists weeks later in a billing or collections system. Link the two through outcome_ref and record when the outcome matured, so a model is never trained on an outcome that was still open when the data was cut.
Where an outcome cannot be verified from a system of record, leave it blank. A silence label is still useful without it; a guessed outcome makes the whole record misleading.
What these labels can teach a voice agent, and where they mislead
Cause-labeled pauses support specific behaviors: waiting through a customer's search for a card instead of reprompting, announcing system lookups, checking back during long holds, and not treating a hesitation as the end of a turn. They also support evaluation, since you can replay real gaps and score whether an agent interrupts, fills or waits appropriately.
They mislead when stretched. Pause norms vary between languages, speakers and settings, so a threshold tuned on one population will misfire on another. A correlation between long hesitation and a poor outcome is not a cause, and an agent trained to push through hesitation can pressure customers. Treat the labels as evidence about conversation, not as rules to optimize.
Audio also carries obligations that transcripts may not. Recording consent, notice and voice-biometric rules vary by jurisdiction, so counsel should review whether the recordings can be used for training before any audio is shared.
Sourcing calls that carry hold events and verified outcomes
The labels in this schema depend on data most archives do not export by default: channel-separated audio, platform hold and transfer events, desktop activity, redaction manifests and outcomes from a system of record. When you ask a supplier for call data, name each of those explicitly and ask how they are joined.
Being allowed to read call recordings is not the same as having consent to use them, owning them as business records or holding a contractual right to license them for training, and each needs its own answer.
Call datasets with hold events are something SourceX sources from U.S. businesses on request rather than keeping in inventory; the buyer's request states the scope, volume, language, format, timing and licensing needs. The business holding the calls approves the scope and terms and signs the agreement before any recordings are delivered.
Frequently asked questions
What is the difference between hold time and silence on a call?
Hold time is a deliberate state, usually visible in call signaling or the contact center platform's records. Silence is any span without speech and can come from holds, mutes, lookups, hesitation, redaction or technical problems. Label them separately because they call for different agent behavior.
Can you detect holds from the audio alone?
Sometimes, from hold music or one-way audio, but not reliably. Use signaling or platform hold events where they exist and reconcile labeled holds against the platform's hold count and duration.
How long should a pause be before it is labeled?
There is no universal threshold. Ordinary gaps between turns are short, so choose a minimum event length that suits your use, record it with the release, and keep raw timings so others can re-threshold.
Why does redacted audio matter for silence labels?
Muted redaction spans look like natural silence. If they are labeled as hesitation, a model can learn that customers pause before sharing payment or personal details, which is an artifact of preparation, not behavior.
Should silence labels include whether the call succeeded?
Keep outcomes in a separate field linked to a verified system-of-record result and the date it became final. Conversational signals are fixed at call time; outcomes mature later and can change.
Sources
- Amazon Connect contact records include CustomerHoldDuration, NumberOfHolds and LongestHoldDuration for holds while the customer is connected to an agent. Source
- Contact Lens defines non-talk time as hold time plus silences longer than 3 seconds where neither participant talks, and the duration cannot be customized. Source
- Amazon Transcribe Call Analytics reports call characteristics including talk time, non-talk time, interruptions, loudness and talk speed. Source
- RFC 3264 section 8.4 puts a unicast media stream on hold by marking it sendonly if it was sendrecv, or inactive if it was recvonly. Source
- Silero VAD's speech timestamp function defaults to a 0.5 threshold, 250 ms minimum speech, 100 ms minimum silence and 30 ms speech padding. Source
- pyannote.audio is an open-source speaker diarization toolkit whose pipeline outputs speaker turns with start and end times. Source
- Across 10 languages, speakers avoid overlap and minimize silence between turns, with average gaps within 250 ms of the cross-language mean. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.