Skip to content

AI uses for records

Voice agent background noise evaluation: beyond the quiet room

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A voice agent background noise evaluation should replay consented recordings captured in each real acoustic condition, use controlled noise mixes only to sweep levels, and score verified task outcomes from the system of record rather than word error rate alone. Compare each condition with a quiet reference of the same scenarios, log measured SNR and ambient level per clip, and report every condition separately.

Agent run

Voice agent background noise

This page covers voice agent background noise evaluation: beyond the quiet room.Illustrative

Key takeaways

  • Build the test as a matrix of real acoustic conditions, each paired with a quiet reference of the same scenarios.
  • Synthetic mixes hold speech constant, but people talk louder and longer in noise; use mixes for sweeps and real clips for the release decision.
  • Score verified task success from the system of record first; word error rate explains failures but is not the outcome.
  • Include noise-only segments and treat any action they trigger as a release blocker.
  • Record the consent basis for every voice on the clip, including bystanders, and have counsel review the specific sites.

How should a voice agent evaluation test realistic shop-floor noise?

Test it the way the floor sounds: replay consented recordings captured in each real acoustic condition, add controlled noise mixes only to sweep levels you could not record, and score whether the task was completed correctly according to the system of record. Compare every condition against a quiet reference built from the same scenarios, and report each condition separately rather than as one blended score.

An agent that passes in a quiet room has passed one cell of a much larger grid. On a production floor the noise is loud, uneven and often speech-like; workers speak differently when it is loud; they may wear hearing protection; and the device may be a headset, a handheld or a wall unit. Each of these changes what the agent hears and what the worker hears back.

The matrix in this guide makes those conditions explicit, ties each cell to a measured noise level and a verified outcome, and records the consent each recording rests on. It is a worksheet for building the test set and a decision tool for reading its results.

Why isn't clean speech plus a noise file the same as a loud floor?

Synthetic mixing is a good instrument for controlled sweeps, but it holds the speech constant while a real floor changes it. Use mixes to find where performance starts to drop, and real recordings to confirm that the drop looks the same in practice.

The standard recipe scales a noise clip against clean speech to hit a target signal-to-noise ratio. Microsoft's Deep Noise Suppression Challenge code does this in its snr_mixer function: it normalizes both signals, computes their RMS levels and scales the noise so the mix lands at the requested SNR. The challenge's synthesizer configuration file sets the SNR range from -5 to 20 dB with 21 levels and can convolve clean speech with real or synthetic room impulse responses to add reverberation.

Room acoustics can also be simulated. Pyroomacoustics implements the image source model and ray tracing to generate room impulse responses for sources and microphones in 2D and 3D rooms, which is useful for reasoning about a large reflective hall before anyone records in it.

What none of this captures is the talker. Research on the Lombard effect describes speech produced in noise as louder, longer and higher in fundamental frequency than the same speech in quiet. A clean read utterance mixed with machine noise at 0 dB is therefore not the utterance a worker produces next to that machine, and turn timing, emphasis and pauses shift along with level.

Which acoustic conditions belong in the consented evaluation matrix?

Each row is an acoustic condition that the deployment will meet; each row says how to capture it for real, which controlled mix stands in for it, the failure most likely to appear and the consent note that applies. Replace the illustrative conditions with the ones measured at your sites.

  • Every real clip needs a paired quiet-reference clip of the same scenario where possible, so the comparison isolates the acoustic condition.
  • Keep real and mixed clips in separate result columns. Pooling them hides exactly the gap the matrix exists to measure.
Which acoustic conditions belong in the consented evaluation matrix?
ConditionReal capture requirementControlled counterpartFailure to watchConsent and capture note
Quiet referenceSame scripts and workers in an office or break roomClean audio, no mixBaseline for every other cellStandard participant consent
Steady machine noiseAt the workstation, normal shift, device in use positionRecorded machine noise mixed at a range of SNRsDigits and part numbers misheardNotice posted for the recording zone
Impulsive impactsNear presses, hammers or pallet dropsImpact clips at their peak levelsWords clipped or dropped mid-utteranceSame as steady noise; log the peak events
Competing speech and PADuring announcements and nearby conversationsBabble or announcement clipsAgent answers a voice that was not the callerBystander speech captured; plan masking or exclusion
Alarms and reversing beepsIn forklift lanes and loading docksTonal alarm clipsFalse barge-in or a turn cut offLog alarm type and timing
Hearing protection wornWorkers wearing their usual earmuffs or plugsNo synthetic equivalentWorker mishears the agent and confirms the wrong itemRecord which protection was worn
Reverberant hallLarge open bays, far from walls and near themSimulated room impulse responsesSmeared consonants and late turn detectionNote microphone distance and position
Device variationHeadset, handheld and fixed unit in the same spotSame mix played through each deviceOne device fails where others passRecord device model and settings

What should each cell score when word error rate is not the outcome?

Score whether the work got done correctly, then use recognition metrics to explain why it did not. A voice agent on a floor exists to complete a lookup, a pick confirmation or a work order update, so the primary metric is the verified outcome.

Two tools help with the recognition layer. JiWER computes word error rate and character error rate from minimum edit distance, and since version 4.0 it accepts empty references, which its maintainers describe as allowing a test of whether models hallucinate on silent audio. Extend that idea to noise-only clips: segments where a press cycles or an alarm sounds and nobody speaks. The Whisper model card notes that its models may output text that was not spoken in the audio, and recommends evaluating in the specific context and domain before deployment; noise-only segments are where you find out.

What should each cell score when word error rate is not the outcome?
MetricDefinitionSource of truthWhy noise moves it
Verified task successThe intended record changed correctly, or the right answer was given and acted onSystem of record entry, such as the WMS or CMMSAny misheard entity can complete the wrong task cleanly
Entity accuracyExact match on part numbers, quantities, bin locations and IDsReference transcript checked against the recordDigits and short codes fail first in noise
Word and character error rateEdit-distance error rates against a reference transcriptHuman reference transcriptExplains recognition failures; not the outcome itself
Noise-only insertionsText or actions produced when nobody addressed the agentNoise-only segments with empty referencesMachines and alarms can trigger invented speech
Turn-taking errorsFalse barge-ins, cut-off turns and missed end of speechTimestamps aligned to audioImpulses and alarms look like speech onsets
Repair turnsTurns spent correcting or repeatingAnnotated dialogA pass after four repeats is not a pass on a floor

How loud was it, really, and how should level and SNR be logged for every clip?

Log two numbers for every clip: the ambient sound level at the work position and the speech-to-noise ratio at the microphone. They answer different questions, and a condition label such as loud is not a measurement.

For the SNR, measure speech level over active speech only. ITU-T Recommendation P.56 standardizes the objective measurement of active speech level so that results from different organizations are comparable, and many later ITU-T recommendations refer to it when they specify test levels. A simple RMS over the whole clip, pauses included, understates speech level and makes the same clip look noisier than it was.

For the ambient level, take an A-weighted reading with a calibrated sound level meter or the site's existing noise survey. The level matters for people as well as microphones: the OSHA occupational noise standard requires a hearing conservation program when exposures reach an 8-hour time-weighted average of 85 decibels, and sets 90 decibels as the permissible exposure limit. Floors in that range are where hearing protection is common, and the matrix has a row for it because protection changes both how workers speak and how well they hear the agent's reply.

Store both values with the clip, along with the device, microphone distance and the condition label. When results are later sliced by condition, the measured levels let you check that the slices really differ.

Does the recording support the transcript and the claimed downstream outcome?

Before a clip enters the matrix, confirm that its audio, transcript, turns and outcome all describe the same event. A transcript that cannot be aligned to the audio, or an outcome that cannot be traced to the call, turns a scored failure into a guess.

At a glance

Does the recording support the transcript and the claimed downstream outcome

  • Audio rights: the consent or notice basis for every voice on the clip, including bystanders, is recorded and covers evaluation use.
  • Speaker turns: each turn is attributed to the worker, the agent or a bystander, with overlaps marked rather than resolved by guesswork.
  • Timestamps: turn start and end times are aligned to the audio, so false barge-ins and cut-offs can be scored.
  • Transcript alignment: the reference transcript was produced from this audio, not from a script the worker was supposed to read.
  • Repair turns: corrections and repetitions are labeled, so a recovered task is scored differently from a clean one.
  • Transfer context: if the call moved to a supervisor or another system, the hand-off and what was passed along are captured.
  • Outcome maturity: the downstream record is final, for example the pick confirmed and the count reconciled, not just a pending entry created during the call.

Which assumption changes the release decision, and how should uncertainty be shown?

The assumption that most often flips the decision is that a controlled mix predicts real performance in the same condition. Test it directly wherever both exist, and let the real cells govern the release decision when the two disagree.

Report every cell with its count and an interval, not as a single pooled rate. A noisy cell with a small number of calls can look fine or alarming by chance, and an aggregate across cells is dominated by whichever condition you recorded most, usually the quiet one.

Write the decision rule before scoring. A workable form is: release to a condition only when the lower bound of verified task success in that condition is within a stated margin of the quiet reference, entity accuracy on the critical fields meets a stated floor, and noise-only insertions that trigger an action are zero in the sample. Conditions that fail stay behind a fallback, such as push-to-talk, a screen confirmation or a scan.

Other assumptions to list explicitly: that the recorded workers represent the people who will use the agent, that the devices match deployment hardware and firmware, and that the sites recorded are representative of the noisiest sites in the rollout.

How would an illustrative parts-lookup agent on a stamping line be judged?

The following scenario is illustrative; the percentages are invented to show how to read the matrix and are not results from any system. A parts-lookup agent on a stamping line answers bin-location questions for part numbers spoken by operators wearing headsets.

  • Decision in this illustrative case: release only for steady-noise stations with screen confirmation of the bin, hold impulsive-noise stations behind push-to-talk, and fix the end-of-speech detector before retesting the noise-only segments.
  • The hearing-protection cell is the exception worth noticing: recognition was fine, and the failure was on the return path, which only real recordings with the worker's reply captured would reveal.
  • The gap between the real and mixed steady-noise rows is why the mix cannot stand in for the floor; it is also the number to track as the mixing recipe is improved.
How would an illustrative parts-lookup agent on a stamping line be judged?
CellClips (illustrative)Verified task success (illustrative)Reading
Quiet reference6095 percentBaseline
Steady press noise, real6088 percentWithin the stated margin; check entity errors
Steady press noise, mixed at matching SNR6093 percentMix overstates real performance
Impulsive impacts, real4072 percentFails the rule; cut-off part numbers dominate
Hearing protection worn, real4080 percentAgent heard correctly; operators misheard replies
Noise-only segments502 triggered actionsFails: any triggered action blocks release

A shop-floor recording captures more than the person talking to the agent. Co-workers, supervisors and announcements end up on the audio, so the consent basis has to cover them or the clip has to exclude or mask them.

Recording and consent rules differ by jurisdiction and by whether the speakers are employees, contractors or visitors, and voice can count as biometric information in some places. Counsel should review the notice, consent and retention terms for the specific sites before collection, and the matrix should record which basis applies to each clip.

Practical controls that keep the evaluation set usable: mark recording zones, use close-talking microphones where the deployment does, record which bystander segments were masked, keep raw audio and masked audio separately with access limited to the people scoring it, and set a deletion date. Never use evaluation audio to build voiceprints unless that use was specifically covered.

Where do recordings from sites that actually sound like this come from?

Real floor recordings with linked outcomes are the scarce ingredient, and they usually sit with the operating companies, not with speech corpus vendors. A request for them should name the acoustic conditions, devices, outcome records and consent basis the matrix needs.

Through SourceX, those conditions go into the request-data form as scope, alongside volume, language, format, timing and licensing needs, and SourceX then searches for matching floor recordings at U.S. businesses rather than offering audio it already holds. Bulky audio never has to be hosted by SourceX, since it can stay in the supplier's own storage or travel on encrypted drives, and the supplier approves the scope and terms and signs before any recording is delivered.

Frequently asked questions

What SNR range should a voice agent noise test cover?

Cover the range measured at the actual work positions, from the quietest station to the loudest, and extend it a few steps beyond for headroom. Public mixing recipes, such as the Deep Noise Suppression Challenge configuration that spans -5 to 20 dB, are a starting point, not a substitute for site measurements.

Can I evaluate a voice agent with clean audio plus added noise only?

You can use added noise for controlled sweeps, but not as the final word. Workers speak louder and longer in noise, may wear hearing protection and use different devices, and none of that is present in a clean recording mixed with a noise file.

How many calls does each acoustic condition need?

Enough that the interval around verified task success is narrow enough to apply your release rule. Start with a few dozen real calls per condition, report the interval, and add calls to the cells where the interval straddles the decision threshold.

How do I test whether a voice agent reacts to noise when nobody is speaking?

Include noise-only segments, such as press cycles and alarms with no one addressing the agent, and score any transcribed text or triggered action as an insertion. JiWER's support for empty references makes this straightforward to compute alongside regular error rates.

Do I need consent from people in the background of a recording?

Usually the consent or notice basis has to cover everyone whose voice is captured, or their speech has to be masked or excluded. The rules vary by jurisdiction and worker status, so have counsel review the specific sites before recording.

Sources

  • The Deep Noise Suppression Challenge synthesizer configuration sets an SNR range of -5 to 20 dB with 21 levels and can add real or synthetic room impulse responses. Source
  • The challenge's snr_mixer function normalizes clean speech and noise, computes RMS levels and scales the noise to reach a target SNR. Source
  • Pyroomacoustics implements the image source model and ray tracing to generate room impulse responses for 2D and 3D rooms. Source
  • The Whisper model card says the models may output text not actually spoken in the audio and recommends robust evaluation in the specific context and domain before deployment. Source
  • JiWER computes WER and CER from minimum edit distance and, since version 4.0, accepts empty references so hallucination on silent audio can be tested. Source
  • OSHA requires a hearing conservation program at an 8-hour time-weighted average of 85 decibels and sets a 90 decibel permissible exposure limit. Source
  • Research describes the Lombard effect as an increase in vocal intensity, duration and fundamental frequency when speaking in noise. Source
  • ITU-T Recommendation P.56 standardizes the objective measurement of active speech level so results from different organizations are comparable. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies