AI uses for records
Business call emotional context labels that code behavior, not feelings
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Label what the recording shows, not what the caller feels. Code observable behaviors per speaker turn, such as repeated requests, corrections, interruptions, explicit complaints, supervisor requests and volume rises against the speaker's own baseline, each with a timestamp and a quote or measured value. Keep any perceived-emotion rating in a separate optional field, and attach outcomes like refunds or cancellations only after business records confirm them.
Call to transcript
Business call emotional
Key takeaways
- Behavior labels can be replayed and checked; labels for how someone felt cannot.
- Each code needs a time span plus a verbatim quote or a measured value, and a written not-this boundary.
- Measure volume against the same speaker's earlier baseline, never against a fixed level across callers.
- Outcomes come from CRM, billing or subscription records and attach only after their window closes.
- Record redaction spans so silence created by redaction is not counted as hold time.
- Labeling agents' emotions raises EU AI Act questions that behavior codes avoid; counsel should review any such use.
Why should frustration labels describe behavior rather than feelings?
Because behavior can be checked against the recording and feelings cannot. An annotator who writes frustrated is guessing at an inner state; an annotator who writes caller repeated the account question three times and asked for a supervisor at 06:12 is describing evidence any reviewer can replay.
The research on inferring emotion from outward signals supports that caution. A 2019 review in Psychological Science in the Public Interest by Barrett and colleagues found that people do sometimes scowl when angry or smile when happy more often than chance, but that how emotions are expressed varies substantially across cultures, situations and individuals, and that similar facial configurations can express more than one emotion category. Voice cues raise the same problem: a loud caller may be angry, on a speakerphone, or simply loud.
There is also a regulatory reason in some markets. The EU AI Act defines an emotion recognition system as an AI system for identifying or inferring emotions or intentions of natural persons on the basis of their biometric data, and its recital 18 says the mere detection of readily apparent expressions, gestures or movements, including characteristics of a voice such as a raised voice, is not emotion recognition unless used to identify or infer emotions. Article 5(1)(f) prohibits AI systems that infer the emotions of a natural person in the workplace, with medical and safety exceptions. Call center agents are at work, so a dataset that labels agents' emotions invites a question that a behavior label avoids.
What counts as an observable frustration signal on a business call?
An observable signal is something a second reviewer can find at a timestamp in the audio or the transcript without knowing what the speaker felt. Most fall into five groups: repetition and repair, overlap and interruption, explicit statements, escalation moves, and measurable acoustic or timing changes.
Commercial call analytics already measures several of these, which is useful but not the same as a label. Amazon Transcribe Call Analytics, for example, reports interruptions, loudness on a normalized 0 to 100 scale, non-talk time, talk speed in words per minute and talk time per participant. Its own documentation frames them as associations: frequent interruptions may be associated with rudeness or anger, and loud speech is often indicative of being upset. Its sentiment output, a score from 5 to -5 plus a per-turn positive, neutral, mixed or negative value, works out of the box without customization. Treat these as model features to check against, not as ground truth for your labels.
The signals that matter most for business outcomes are often not acoustic at all. A caller who has to restate the order number after a transfer, or who says I already did that twice, is showing friction in the process. Those moments are what an agent model needs to learn to prevent.
The behavior grounded annotation rubric
Code each speaker turn with zero or more behavior codes from a fixed list, each tied to a time span and a verbatim quote or a measured value. The rubric below is a starting set of ten codes; it labels what happened, never why the person did it.
Every code has a not-this column because most disagreement comes from near misses. Calibrate annotators on those boundaries before they label volume.
| Code | Observable definition | Evidence required | Not this |
|---|---|---|---|
| REPEAT_REQUEST | Speaker asks for the same thing again after it was already asked or answered | Both turns with timestamps | A first request rephrased in the same turn |
| REPAIR_CORRECTION | Speaker corrects what the other party said, heard or entered | Quote of the correction and the turn it corrects | Self-correction of a slip mid-sentence |
| INTERRUPTION | Speaker starts talking while the other is mid-sentence and the other stops | Overlap span from both channels or diarized audio | Short backchannels such as okay or right |
| EXPLICIT_DISSATISFACTION | Speaker states dissatisfaction in words | Verbatim quote | Annotator's impression of tone without words |
| ESCALATION_REQUEST | Speaker asks for a supervisor, a manager, a complaint process or a callback from someone else | Verbatim quote | Agent offering a transfer unprompted |
| EXIT_STATEMENT | Speaker says they will cancel, switch provider, dispute a charge or stop paying | Verbatim quote | A question about how cancellation works |
| RAISED_VOLUME | Speaker's loudness rises well above their own earlier baseline on the same call | Measured values for the span and the baseline | Comparison with a different speaker or a different call |
| LONG_HOLD_OR_SILENCE | Non-talk span longer than the threshold your guide sets | Start and end times, plus hold markers if the system logs them | Silence created by redaction |
| TRANSFER_REPEAT | Caller is transferred again after an earlier transfer on the same contact | Transfer events from the phone system or both handoff turns | A single warm transfer |
| RESOLUTION_ACK | Speaker states the issue is resolved or thanks the agent for a fix | Verbatim quote | Polite closing that does not mention the issue |
Turn record: what each business call emotional context label must point to
Each label should point to a call, a speaker role, a turn and a time span, with the evidence stored beside it. If a label cannot be replayed from those pointers, it cannot be audited, corrected or trusted by the team training on it.
At a glance
Turn record: what each business call emotional context label must point to
Speaker attribution comes first, because every code depends on who said what. Diarization output assigns generic labels; Amazon Transcribe, for example, labels up to 30 speakers as spk_0 through spk_29 with start and end times per segment, and someone still has to map those labels to caller and agent. Diarization also makes errors. The pyannote project's own benchmark table, last updated September 2025, reports diarization error rates on the CALLHOME telephone set between 16.6 and 28.5 percent depending on the pipeline. Dual-channel recordings, with caller and agent on separate channels, remove most of that ambiguity, so ask for them where they exist.
The rubric fits established dialogue annotation practice. ISO 24617-2, the international standard for dialogue act annotation, lets one functional segment carry several communicative functions across dimensions such as turn management and own or partner communication management, which is where repairs and interruptions live. The Switchboard SWBD-DAMSL coders manual is an older example of a shallow discourse tag set applied to telephone conversations with a written manual for coders. Both show the value of a written definition per tag, which is what the not-this column supplies.
- call_id, channel or diarized speaker label, and the mapped role (caller, agent, third party)
- turn_index, start_time and end_time in seconds from call start
- transcript_span for the turn, with the transcript engine and version
- code from the rubric, plus evidence_quote or measured_value and baseline_value
- annotator_id, annotation date and guide version
- perceived_affect as a separate optional field, empty by default, with the rater pool recorded if used
- redaction_spans that overlap the turn, so silence from redaction is not read as hold time
Keeping what was said apart from what happened afterward
Store downstream outcomes as separate labels that come from business records, and attach them only after the observation window has closed. A call that sounds tense may end in a renewal; a calm call may end in a cancellation the next day. Mixing the two teaches a model that tone predicts outcome when the dataset never tested it.
Outcome maturity matters as much as the outcome. If the window for a repeat contact is seven days, a call recorded yesterday has no outcome yet, and labeling it no repeat contact is wrong rather than empty. The table shows the pattern; the windows are illustrative and should be set per business.
| Outcome label | Source record | Illustrative maturity window | Link rule |
|---|---|---|---|
| Repeat contact on the same issue | CRM or ticketing case history | 7 days after the call | Same customer and case or issue category |
| Case reopened | Ticket status history | 14 days after closure | Reopen event on the case linked to the call |
| Credit, refund or adjustment issued | Billing or ERP adjustment record | 30 days after the call | Adjustment references the account and the issue |
| Cancellation or downgrade | Subscription or account record | 30 days after the call | Effective date after the call date |
| Formal complaint filed | Complaint log | 30 days after the call | Complaint cites the contact or the issue |
| Resolved on first contact | Case history plus no repeat contact | After the repeat-contact window closes | Only when no linked contact or reopen occurred |
Three checks before a labeled call enters the set: rights, alignment and outcome trace
Check three things before a labeled call enters the dataset: that the audio may be used, that the transcript and speaker labels match the audio, and that each outcome traces to a record. A label set built on a misaligned transcript or a guessed outcome is worse than a smaller clean one.
At a glance
Three checks before a labeled call enters the set: rights, alignment and outcome trace
Redaction deserves its own check. Call analytics redaction can replace spoken personal information with silence in the audio file; Amazon Transcribe's post-call redaction does exactly that, and replaces the text with a PII tag in the transcript. Unless redaction spans are recorded, those silences look like holds and inflate LONG_HOLD_OR_SILENCE. Keep the spans and exclude them from timing codes.
- Rights: the recording's notice and consent basis, and whether it covers use for AI training, are confirmed for the call's jurisdiction
- Alignment: a sample of turns is replayed to confirm the transcript text and timestamps
- Attribution: speaker labels are mapped to roles and spot-checked at every overlap
- Redaction: redaction spans are listed and excluded from timing and volume measures
- Outcome trace: every outcome label carries the source record identifier and the window used
- Maturity: calls whose windows have not closed are marked pending, not negative
Worked example: a billing dispute scored with the rubric
Illustrative example: invented to show the method; the call, times and counts are not real. A customer calls about a duplicate charge. The call lasts about nine minutes on dual-channel audio, with one transfer.
At 01:40 the caller gives the invoice number. After a transfer at 03:05, the second agent asks for it again, and the caller says I gave that to the last person: REPEAT_REQUEST on the agent turn and EXPLICIT_DISSATISFACTION on the caller turn, both with quotes. At 04:20 the caller corrects the amount the agent read back: REPAIR_CORRECTION. At 05:10 the caller says if this is not fixed I will dispute it with my bank: EXIT_STATEMENT. Measured loudness for the caller over 05:00 to 05:30 is well above the caller's first-minute baseline: RAISED_VOLUME with both values stored. A 70-second silence at 06:00 overlaps a logged hold event: LONG_HOLD_OR_SILENCE. At 08:30 the caller says okay, that sorts it, thanks: RESOLUTION_ACK.
The outcome labels arrive later. Billing shows a reversal two days after the call, and the case history shows no repeat contact within seven days, so the call gets credit issued and resolved on first contact once the window closes. No label says the caller was angry.
The exception: on another call, a caller on a speakerphone in a warehouse is loud from the first second. Their loudness never rises above their own baseline, so RAISED_VOLUME does not apply, even though a listener might call them shouting. A rule based on a fixed loudness level across callers would have labeled that call as heated from start to finish.
Measuring agreement and deciding when the rubric is not enough
Double-code a sample of calls and report agreement per code, with the disagreements replayed and resolved against the recording. Because each behavior code points to evidence, a disagreement can usually be settled by listening; a disagreement about how someone felt cannot, which is why perceived-affect ratings, if you keep them at all, should stay in their own field with every rater's raw answer.
Some situations need a specialist rather than a better rubric. Bring in counsel and a voice evaluation lead before labeling agent emotions for any EU deployment, scoring individual agents from inferred emotional state, labeling calls that reveal health conditions, financial hardship or a caller who may be a minor, or using recordings whose consent basis is unclear. Narrowing the scope, for example to caller-side behavior codes on dual-channel recordings with verified outcomes, often gives a cleaner dataset than broadening it.
Asking for call data with behavior codes and verified outcomes
When you request call data for this kind of rubric, ask for dual-channel audio where it exists, transcripts with timestamps and engine versions, redaction spans, transfer and hold events from the phone system, and the case, billing or subscription records needed to verify outcomes, with the windows stated.
With SourceX, the request is where the rubric's needs go, beside scope, volume, language, format, timing and licensing terms, because recordings are found at U.S. businesses to match each request rather than kept in a library. The company that owns the calls signs off on the scope before any audio moves, which is the moment to settle whether outcome records and phone-system events travel with it.
Frequently asked questions
Is a call sentiment score the same as a frustration label?
No. A sentiment score is a model's estimate, often per turn or per call segment, and it can be a useful feature. A behavior label records a specific event, such as a repeated request or an escalation request, with a timestamp and evidence a reviewer can replay.
Should a dataset label agents' emotions as well as callers'?
Behavior codes can apply to both speakers, but inferred emotions are different. In the EU, the AI Act prohibits AI systems that infer the emotions of a natural person in the workplace, with medical and safety exceptions, so labeling agent emotions for an EU use needs counsel's review first. This is general information, not legal advice.
Can behavior codes be labeled from transcripts alone?
Some can. Repeats, corrections, explicit statements, escalation requests and exit statements are visible in a good transcript. Interruptions, volume changes and silences need the audio and accurate timestamps, ideally on separate channels.
When should a downstream outcome be attached to a call?
Only after the observation window for that outcome has closed and the outcome has been found in a business record. Until then, mark the outcome as pending; recording no outcome before the window ends creates false negatives.
Does labeling a raised voice count as emotion recognition?
Under the EU AI Act's recital 18, the mere detection of readily apparent expressions such as a raised voice is not emotion recognition unless it is used to identify or infer emotions. Keeping the label as a measured behavior, not an inferred feeling, follows that line, but counsel should confirm the use case.
Sources
- Amazon Transcribe Call Analytics reports interruptions, loudness on a normalized 0 to 100 scale, non-talk time, talk speed and talk time; it says frequent interruptions may be associated with rudeness or anger; sentiment is scored from 5 to -5 and per turn without customization; post-call redaction replaces spoken PII with silence in audio. Source
- Amazon Transcribe speaker diarization distinguishes up to 30 speakers, labeled spk_0 through spk_29, with start and end times for each segment. Source
- The EU AI Act defines an emotion recognition system (Article 3(39)), prohibits AI systems that infer emotions of a natural person in the workplace and education institutions except for medical or safety reasons (Article 5(1)(f)), and in recital 18 excludes mere detection of readily apparent expressions, including a raised voice, unless used to infer emotions. Source
- Barrett and colleagues (2019) found that emotional expression varies substantially across cultures, situations and individuals and that similar facial configurations can express more than one emotion category. Source
- ISO 24617-2:2020 specifies multidimensional dialogue act annotation in which functional segments can carry several communicative functions across dimensions including turn management and own and partner communication management. Source
- The Switchboard SWBD-DAMSL coders manual defines a shallow discourse-function tag set for annotating telephone conversations. Source
- The pyannote.audio benchmark table (last updated September 2025) reports diarization error rates on CALLHOME of 28.5, 26.7 and 16.6 percent for its legacy, community-1 and precision-2 pipelines. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.