Skip to content

AI uses for records

Voice agent regional accent robustness, measured without stereotyping

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Assess accent robustness by letting callers describe their own accent through opt-in consent, never inferring it from voice, name or location. Report how many consenting callers and calls each self-described group covers, measure errors that matter to the call, from misheard words to repair turns and wrong outcomes verified in business records, and publish gaps only with minimum group sizes, intervals and paired comparisons.

Agent run

Voice agent regional accent

This page covers voice agent regional accent robustness, measured without stereotyping.Illustrative

Key takeaways

  • Accent labels should come from the caller, on an opt-in basis, with prefer-not-to-say as a valid answer; inferring accent from audio or names is the stereotyping you are trying to avoid.
  • Coverage comes first: report how many consenting speakers, calls and minutes each self-described group has before reporting any error rate.
  • Word error rate is only the first error measure; repair turns, unnecessary transfers and outcomes verified in the booking or ticket system show the cost to the caller.
  • Compare each group with the overall result and with itself across model versions, using intervals and paired tests, rather than ranking groups against each other.
  • Accent labels can reveal ethnic origin and voice recordings can be biometric data, so consent, purpose and retention need counsel review.

How do you assess voice agent robustness across accents without stereotyping?

Measure it on consenting callers who describe their own accent, compare outcomes rather than impressions, and report only groups large enough to support a conclusion. The method has three rules: the caller names the accent, the error measures follow the call through to a verified business outcome, and the report compares each group with the whole, with uncertainty shown.

Stereotyping enters an evaluation in predictable places: when an annotator guesses a caller's origin from how they sound, when a team groups callers by surname or phone area code, when synthetic voices that imitate an accent stand in for real speakers, and when results are published as a league table of nationalities. Each of those turns a robustness test into a set of assumptions about people. The protocol below avoids all four.

Why accent gaps are worth measuring at all

Speech systems do perform unevenly across speakers, and the gaps have been measured in peer-reviewed work. Koenecke and colleagues, writing in PNAS in 2020, tested five commercial speech recognition systems on interviews and found an average word error rate of 0.35 for Black speakers against 0.19 for white speakers. That study compared racial groups and dialect use rather than accents in general, but it shows that average accuracy can hide large differences between speakers.

NIST's Special Publication 1270 on bias in AI separates systemic, statistical or computational, and human sources of bias and notes that all three can arise without anyone intending them. For a voice agent, a statistical source is a recognizer trained on too little speech like a caller's; a human source is an annotator who rates some speakers as harder to understand; a systemic source is a call flow designed around one way of saying addresses or dates. A useful evaluation is built to tell these apart.

Who gets to name a caller's accent?

The caller does. Ask for a self-description only after consent for the evaluation, make it optional, accept free text, and keep prefer not to say as a first-class answer. An accent label inferred by a person or a model is a guess about someone's background, and it fails exactly where robustness testing matters, on speakers who do not fit the guesser's categories.

Public speech corpora show the pattern. Mozilla's Common Voice records age, gender and accents for a clip only if the speaker opted in to report them, and it generates train, dev and test splits non-deterministically to avoid perpetuating demographic skews from one release to the next. The Edinburgh International Accents of English Corpus collected about 40 hours of conversational English with more than 40 self-reported accents from speakers of 51 first languages, along with background such as the places speakers had lived.

Self-descriptions are messy: two people with similar speech may describe it differently. Keep the caller's own words, then map them to a small reporting taxonomy that a reviewer has checked, record the mapping version, and let reporting groups be coarse enough to reach a minimum size. Never let the mapping invent precision the caller did not give.

What should the consented coverage and error reporting protocol include?

The protocol runs from consent to publication, with a stop condition at each step. Use it as a checklist; a step that cannot meet its stop condition halts the evaluation for that group rather than being quietly skipped.

What should the consented coverage and error reporting protocol include?
StepWhat to doWhat to recordStop condition
1. ConsentAsk callers for consent to use the call for evaluation and, separately, to give an accent descriptionConsent text version, time, scope and withdrawal routeNo separate consent for the accent question
2. Self-descriptionCollect the caller's own words, optional, with prefer not to sayRaw answer, mapping to reporting group, mapping versionLabels assigned by staff, models or metadata
3. Coverage tableCount consenting speakers, calls and minutes per reporting group and per call typeCounts per cell, share declined, share unansweredA reporting group below the minimum size; merge or drop it
4. Reference transcriptsTranscribe a sample with trained transcribers blind to the caller's descriptionTranscriber, guidelines version, second-pass agreementTranscribers see accent labels
5. Turn and repair annotationMark speaker turns, timestamps, repair turns and transfersTurn ids, repair type, who initiated itTurn timing cannot be aligned to audio
6. Outcome linkageJoin each call to the business record it should have produced or changedRecord id, expected value, actual value, outcome maturity dateOutcome not yet final; hold the call
7. AnalysisCompute each metric per group and overall, with intervals by resampling speakersMetric definitions, interval method, pre-registered comparisonsComparisons added after seeing results
8. Publication reviewReview wording, groups and examples before releaseReviewer names, removed items and reasonsAny audio clip or quote that could identify or caricature a caller

Which errors count: from transcript to verified outcome

Count errors at three levels: what the recognizer heard, how the conversation went, and whether the business outcome was right. Word error rate alone misses the cost to the caller; a caller who is understood on the third try has a low error rate on the final turn but a worse experience than one served first time.

At a glance

Which errors count: from transcript to verified outcome

For the first level, NIST's sclite scoring tool aligns reference and hypothesis transcripts and tallies errors for each speaker, which is the unit you need for group reporting. For the other two levels, the call record needs fields that keep conversational signals apart from the outcome checked later in a system such as a scheduling platform, a Zendesk ticket or an order system. A recording backs both its transcript and the outcome claimed for the call only when every field below can be filled from the audio and the business record, never from the agent's own summary of the call.

  • Audio rights: consent scope, recording notice played, retention date.
  • Speaker turns with start and end timestamps aligned to the audio.
  • Reference transcript and the agent's live transcript, aligned turn by turn.
  • Repair turns: where the caller repeated, spelled or corrected something, and which item was being repaired.
  • Slot accuracy for the items that drive the outcome, such as names, dates, account numbers and addresses.
  • Transfer context: whether the call went to a person, and whether the transfer followed a recognition failure.
  • Verified outcome: the business record created or changed, the expected value, and the date the outcome became final.
  • Outcome maturity: whether enough time has passed to know if the outcome held, such as a booked appointment that was kept.

How do you report a gap without ranking groups of people?

Report each group against the overall figure and against its own result on the previous model version, with intervals, and suppress any group below the minimum size. A ranked table of groups invites readers to treat the differences as facts about people rather than facts about the system.

Paired comparisons on the same audio are the most informative view, because they remove differences in who called. NIST's sc_stats tool compares two or more recognition systems run on identical test data using tests such as matched pairs, sign, Wilcoxon and McNemar; the same logic applies when comparing a new agent version with the old one within each group.

How do you report a gap without ranking groups of people?
Report elementRuleWhy
Group sizePublish only groups that meet a pre-set minimum of speakers, not just callsMany calls from a few speakers say little about a group
Primary metricVerified outcome success, then repair turns per call, then word error rateOrders the measures by cost to the caller
ComparisonEach group versus overall, and each group across versions on the same callsAvoids a league table of groups
UncertaintyIntervals from resampling speakers, shown next to every figureCalls from one speaker are not independent
Coverage noteShare of callers who declined or left the question blankShows who the results do not describe
ExamplesError patterns described by item type, such as street names, never by audio clips of identifiable callersPrevents caricature and re-identification

Worked example (illustrative): an appointment-booking agent

Consider an illustrative evaluation of a voice agent that books service appointments. Over a sample period, the team asks consenting callers an optional accent question after the call. All figures here are invented for illustration. Of 2,400 consenting callers, 31% decline the question or leave it blank; the remaining answers map to six reporting groups, two of which fall below the minimum of 60 speakers and are merged into a wider group.

Word error rates differ only modestly between groups, and a team that stopped at the recognizer would call the agent robust. The outcome view tells a different story. In one group, callers needed a repair turn on 22% of calls against 12% overall, almost all on street names and unit numbers, and 6% of that group's bookings were later found at the wrong address in the scheduling system, against 2% overall. The finding is not that this group is hard to understand; it is that the agent confirms addresses by reading them back at speed and accepts a yes without spelling. The fix, spelling confirmation for addresses, is then retested within every group on the same audio.

The exception: one merged group showed a worse outcome rate driven by a single call type, after-hours emergency calls, that the group happened to use more. Splitting by call type removed most of the gap, which is why the coverage table counts calls by type within each group.

When accent labels or voice data should not be collected

Some deployments should not collect accent labels at all, and some should not keep voice data for evaluation. Accent descriptions can reveal ethnic or national origin, and recordings can be used to identify a person by voice.

In the EU, GDPR Article 9 generally prohibits processing personal data revealing racial or ethnic origin unless one of its listed exceptions applies, such as explicit consent. In Illinois, the Biometric Information Privacy Act lists voiceprints among biometric identifiers and requires written notice of the purpose and length of term and a written release before a private entity collects them. Other jurisdictions have their own rules, and employment, children and health calls add more. Counsel should review the specific consent text, purpose and retention before collection starts.

Where labels cannot be collected, measure robustness by condition instead: call type, channel, background noise and the items that drive outcomes, such as names and addresses. Cluster failures by what was misheard rather than by who said it. This is weaker evidence about accent, but it does not create a sensitive label to protect, and it often points to the same fixes: slower read-backs, spelling confirmation for names and addresses, and a clear route to a person after two failed repairs.

Sourcing consented calls with outcomes attached

The difficult input for this protocol is real calls whose recordings, consent records and verified outcomes can be joined. Contact centers hold all three in separate systems, and the join is what makes the evaluation trustworthy.

SourceX gathers call datasets like these on request from U.S. businesses rather than from a catalog, and the buyer specifies call types, languages, volume, format, timing and licensing needs. Recordings and outcome records are released only after the supplying business has approved the scope and terms and signed, and de-identification of audio and transcripts is carried out by SourceX through approved processing routes. Whether self-described accent labels can be included is decided case by case, with the consent each caller actually gave.

Frequently asked questions

Can I infer a caller's accent with a classifier instead of asking?

Not for a fairness evaluation. A classifier's label is a guess about the caller's background, it fails most on the speakers you most need to test, and it creates a sensitive label the caller never agreed to. Use opt-in self-descriptions, or measure by call condition instead.

Is word error rate enough to measure accent robustness?

No. Word error rate shows what the recognizer heard, but not how many repairs the caller needed or whether the booking, ticket or order came out right. Report verified outcome success and repair turns first, with word error rate as a diagnostic.

Can synthetic accented voices replace real callers?

They can help find specific failure points, such as how an agent handles unusual street names. They are a poor stand-in for real speakers, because they reproduce someone's idea of an accent and can harden stereotypes. Use them for debugging, not for the reported results.

How large should each accent group be?

Set a minimum number of speakers per reporting group before collecting data, and merge or withhold groups below it. Count speakers rather than calls, since many calls from a few people make a group look better measured than it is.

Do I need separate consent for the accent question?

Treat it as separate. Consent to record a call for quality or evaluation does not clearly cover a question about the caller's background, and accent descriptions can reveal ethnic origin. Counsel should review the wording for each jurisdiction where callers are located.

Sources

  • Five commercial speech recognition systems averaged a word error rate of 0.35 for Black speakers and 0.19 for white speakers in Koenecke et al. (PNAS, 2020). Source
  • NIST SP 1270 sorts AI bias into systemic, statistical or computational, and human categories and notes they can arise without intent. Source
  • Common Voice reports accents only when a speaker opted in, and generates splits non-deterministically to avoid perpetuating demographic skews. Source
  • The Edinburgh International Accents of English Corpus has about 40 hours of conversation with more than 40 self-reported accents from speakers of 51 first languages. Source
  • NIST's sclite aligns reference and hypothesis transcripts and tallies errors for each speaker. Source
  • NIST's sc_stats compares recognition systems run on identical test data with matched pairs, sign, Wilcoxon and McNemar tests. Source
  • Illinois BIPA lists voiceprints as biometric identifiers and requires written notice of purpose and term and a written release before collection. Source
  • GDPR Article 9 generally prohibits processing personal data revealing racial or ethnic origin unless a listed exception applies. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies