AI uses for records
Voice agent background noise evaluation: beyond the quiet room
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A voice agent background noise evaluation should replay consented recordings captured in each real acoustic condition, use controlled noise mixes only to sweep levels, and score verified task outcomes from the system of record rather than word error rate alone. Compare each condition with a quiet reference of the same scenarios, log measured SNR and ambient level per clip, and report every condition separately.
Agent run
Voice agent background noise
Key takeaways
- Build the test as a matrix of real acoustic conditions, each paired with a quiet reference of the same scenarios.
- Synthetic mixes hold speech constant, but people talk louder and longer in noise; use mixes for sweeps and real clips for the release decision.
- Score verified task success from the system of record first; word error rate explains failures but is not the outcome.
- Include noise-only segments and treat any action they trigger as a release blocker.
- Record the consent basis for every voice on the clip, including bystanders, and have counsel review the specific sites.
How should a voice agent evaluation test realistic shop-floor noise?
Test it the way the floor sounds: replay consented recordings captured in each real acoustic condition, add controlled noise mixes only to sweep levels you could not record, and score whether the task was completed correctly according to the system of record. Compare every condition against a quiet reference built from the same scenarios, and report each condition separately rather than as one blended score.
An agent that passes in a quiet room has passed one cell of a much larger grid. On a production floor the noise is loud, uneven and often speech-like; workers speak differently when it is loud; they may wear hearing protection; and the device may be a headset, a handheld or a wall unit. Each of these changes what the agent hears and what the worker hears back.
The matrix in this guide makes those conditions explicit, ties each cell to a measured noise level and a verified outcome, and records the consent each recording rests on. It is a worksheet for building the test set and a decision tool for reading its results.
Why isn't clean speech plus a noise file the same as a loud floor?
Synthetic mixing is a good instrument for controlled sweeps, but it holds the speech constant while a real floor changes it. Use mixes to find where performance starts to drop, and real recordings to confirm that the drop looks the same in practice.
The standard recipe scales a noise clip against clean speech to hit a target signal-to-noise ratio. Microsoft's Deep Noise Suppression Challenge code does this in its snr_mixer function: it normalizes both signals, computes their RMS levels and scales the noise so the mix lands at the requested SNR. The challenge's synthesizer configuration file sets the SNR range from -5 to 20 dB with 21 levels and can convolve clean speech with real or synthetic room impulse responses to add reverberation.
Room acoustics can also be simulated. Pyroomacoustics implements the image source model and ray tracing to generate room impulse responses for sources and microphones in 2D and 3D rooms, which is useful for reasoning about a large reflective hall before anyone records in it.
What none of this captures is the talker. Research on the Lombard effect describes speech produced in noise as louder, longer and higher in fundamental frequency than the same speech in quiet. A clean read utterance mixed with machine noise at 0 dB is therefore not the utterance a worker produces next to that machine, and turn timing, emphasis and pauses shift along with level.
Which acoustic conditions belong in the consented evaluation matrix?
Each row is an acoustic condition that the deployment will meet; each row says how to capture it for real, which controlled mix stands in for it, the failure most likely to appear and the consent note that applies. Replace the illustrative conditions with the ones measured at your sites.
- Every real clip needs a paired quiet-reference clip of the same scenario where possible, so the comparison isolates the acoustic condition.
- Keep real and mixed clips in separate result columns. Pooling them hides exactly the gap the matrix exists to measure.
| Condition | Real capture requirement | Controlled counterpart | Failure to watch | Consent and capture note |
|---|---|---|---|---|
| Quiet reference | Same scripts and workers in an office or break room | Clean audio, no mix | Baseline for every other cell | Standard participant consent |
| Steady machine noise | At the workstation, normal shift, device in use position | Recorded machine noise mixed at a range of SNRs | Digits and part numbers misheard | Notice posted for the recording zone |
| Impulsive impacts | Near presses, hammers or pallet drops | Impact clips at their peak levels | Words clipped or dropped mid-utterance | Same as steady noise; log the peak events |
| Competing speech and PA | During announcements and nearby conversations | Babble or announcement clips | Agent answers a voice that was not the caller | Bystander speech captured; plan masking or exclusion |
| Alarms and reversing beeps | In forklift lanes and loading docks | Tonal alarm clips | False barge-in or a turn cut off | Log alarm type and timing |
| Hearing protection worn | Workers wearing their usual earmuffs or plugs | No synthetic equivalent | Worker mishears the agent and confirms the wrong item | Record which protection was worn |
| Reverberant hall | Large open bays, far from walls and near them | Simulated room impulse responses | Smeared consonants and late turn detection | Note microphone distance and position |
| Device variation | Headset, handheld and fixed unit in the same spot | Same mix played through each device | One device fails where others pass | Record device model and settings |
What should each cell score when word error rate is not the outcome?
Score whether the work got done correctly, then use recognition metrics to explain why it did not. A voice agent on a floor exists to complete a lookup, a pick confirmation or a work order update, so the primary metric is the verified outcome.
Two tools help with the recognition layer. JiWER computes word error rate and character error rate from minimum edit distance, and since version 4.0 it accepts empty references, which its maintainers describe as allowing a test of whether models hallucinate on silent audio. Extend that idea to noise-only clips: segments where a press cycles or an alarm sounds and nobody speaks. The Whisper model card notes that its models may output text that was not spoken in the audio, and recommends evaluating in the specific context and domain before deployment; noise-only segments are where you find out.
| Metric | Definition | Source of truth | Why noise moves it |
|---|---|---|---|
| Verified task success | The intended record changed correctly, or the right answer was given and acted on | System of record entry, such as the WMS or CMMS | Any misheard entity can complete the wrong task cleanly |
| Entity accuracy | Exact match on part numbers, quantities, bin locations and IDs | Reference transcript checked against the record | Digits and short codes fail first in noise |
| Word and character error rate | Edit-distance error rates against a reference transcript | Human reference transcript | Explains recognition failures; not the outcome itself |
| Noise-only insertions | Text or actions produced when nobody addressed the agent | Noise-only segments with empty references | Machines and alarms can trigger invented speech |
| Turn-taking errors | False barge-ins, cut-off turns and missed end of speech | Timestamps aligned to audio | Impulses and alarms look like speech onsets |
| Repair turns | Turns spent correcting or repeating | Annotated dialog | A pass after four repeats is not a pass on a floor |
How loud was it, really, and how should level and SNR be logged for every clip?
Log two numbers for every clip: the ambient sound level at the work position and the speech-to-noise ratio at the microphone. They answer different questions, and a condition label such as loud is not a measurement.
For the SNR, measure speech level over active speech only. ITU-T Recommendation P.56 standardizes the objective measurement of active speech level so that results from different organizations are comparable, and many later ITU-T recommendations refer to it when they specify test levels. A simple RMS over the whole clip, pauses included, understates speech level and makes the same clip look noisier than it was.
For the ambient level, take an A-weighted reading with a calibrated sound level meter or the site's existing noise survey. The level matters for people as well as microphones: the OSHA occupational noise standard requires a hearing conservation program when exposures reach an 8-hour time-weighted average of 85 decibels, and sets 90 decibels as the permissible exposure limit. Floors in that range are where hearing protection is common, and the matrix has a row for it because protection changes both how workers speak and how well they hear the agent's reply.
Store both values with the clip, along with the device, microphone distance and the condition label. When results are later sliced by condition, the measured levels let you check that the slices really differ.
Does the recording support the transcript and the claimed downstream outcome?
Before a clip enters the matrix, confirm that its audio, transcript, turns and outcome all describe the same event. A transcript that cannot be aligned to the audio, or an outcome that cannot be traced to the call, turns a scored failure into a guess.
At a glance
Does the recording support the transcript and the claimed downstream outcome
- Audio rights: the consent or notice basis for every voice on the clip, including bystanders, is recorded and covers evaluation use.
- Speaker turns: each turn is attributed to the worker, the agent or a bystander, with overlaps marked rather than resolved by guesswork.
- Timestamps: turn start and end times are aligned to the audio, so false barge-ins and cut-offs can be scored.
- Transcript alignment: the reference transcript was produced from this audio, not from a script the worker was supposed to read.
- Repair turns: corrections and repetitions are labeled, so a recovered task is scored differently from a clean one.
- Transfer context: if the call moved to a supervisor or another system, the hand-off and what was passed along are captured.
- Outcome maturity: the downstream record is final, for example the pick confirmed and the count reconciled, not just a pending entry created during the call.
Which assumption changes the release decision, and how should uncertainty be shown?
The assumption that most often flips the decision is that a controlled mix predicts real performance in the same condition. Test it directly wherever both exist, and let the real cells govern the release decision when the two disagree.
Report every cell with its count and an interval, not as a single pooled rate. A noisy cell with a small number of calls can look fine or alarming by chance, and an aggregate across cells is dominated by whichever condition you recorded most, usually the quiet one.
Write the decision rule before scoring. A workable form is: release to a condition only when the lower bound of verified task success in that condition is within a stated margin of the quiet reference, entity accuracy on the critical fields meets a stated floor, and noise-only insertions that trigger an action are zero in the sample. Conditions that fail stay behind a fallback, such as push-to-talk, a screen confirmation or a scan.
Other assumptions to list explicitly: that the recorded workers represent the people who will use the agent, that the devices match deployment hardware and firmware, and that the sites recorded are representative of the noisiest sites in the rollout.
How would an illustrative parts-lookup agent on a stamping line be judged?
The following scenario is illustrative; the percentages are invented to show how to read the matrix and are not results from any system. A parts-lookup agent on a stamping line answers bin-location questions for part numbers spoken by operators wearing headsets.
- Decision in this illustrative case: release only for steady-noise stations with screen confirmation of the bin, hold impulsive-noise stations behind push-to-talk, and fix the end-of-speech detector before retesting the noise-only segments.
- The hearing-protection cell is the exception worth noticing: recognition was fine, and the failure was on the return path, which only real recordings with the worker's reply captured would reveal.
- The gap between the real and mixed steady-noise rows is why the mix cannot stand in for the floor; it is also the number to track as the mixing recipe is improved.
| Cell | Clips (illustrative) | Verified task success (illustrative) | Reading |
|---|---|---|---|
| Quiet reference | 60 | 95 percent | Baseline |
| Steady press noise, real | 60 | 88 percent | Within the stated margin; check entity errors |
| Steady press noise, mixed at matching SNR | 60 | 93 percent | Mix overstates real performance |
| Impulsive impacts, real | 40 | 72 percent | Fails the rule; cut-off part numbers dominate |
| Hearing protection worn, real | 40 | 80 percent | Agent heard correctly; operators misheard replies |
| Noise-only segments | 50 | 2 triggered actions | Fails: any triggered action blocks release |
Whose consent does the matrix depend on, bystanders included?
A shop-floor recording captures more than the person talking to the agent. Co-workers, supervisors and announcements end up on the audio, so the consent basis has to cover them or the clip has to exclude or mask them.
Recording and consent rules differ by jurisdiction and by whether the speakers are employees, contractors or visitors, and voice can count as biometric information in some places. Counsel should review the notice, consent and retention terms for the specific sites before collection, and the matrix should record which basis applies to each clip.
Practical controls that keep the evaluation set usable: mark recording zones, use close-talking microphones where the deployment does, record which bystander segments were masked, keep raw audio and masked audio separately with access limited to the people scoring it, and set a deletion date. Never use evaluation audio to build voiceprints unless that use was specifically covered.
Where do recordings from sites that actually sound like this come from?
Real floor recordings with linked outcomes are the scarce ingredient, and they usually sit with the operating companies, not with speech corpus vendors. A request for them should name the acoustic conditions, devices, outcome records and consent basis the matrix needs.
Through SourceX, those conditions go into the request-data form as scope, alongside volume, language, format, timing and licensing needs, and SourceX then searches for matching floor recordings at U.S. businesses rather than offering audio it already holds. Bulky audio never has to be hosted by SourceX, since it can stay in the supplier's own storage or travel on encrypted drives, and the supplier approves the scope and terms and signs before any recording is delivered.
Frequently asked questions
What SNR range should a voice agent noise test cover?
Cover the range measured at the actual work positions, from the quietest station to the loudest, and extend it a few steps beyond for headroom. Public mixing recipes, such as the Deep Noise Suppression Challenge configuration that spans -5 to 20 dB, are a starting point, not a substitute for site measurements.
Can I evaluate a voice agent with clean audio plus added noise only?
You can use added noise for controlled sweeps, but not as the final word. Workers speak louder and longer in noise, may wear hearing protection and use different devices, and none of that is present in a clean recording mixed with a noise file.
How many calls does each acoustic condition need?
Enough that the interval around verified task success is narrow enough to apply your release rule. Start with a few dozen real calls per condition, report the interval, and add calls to the cells where the interval straddles the decision threshold.
How do I test whether a voice agent reacts to noise when nobody is speaking?
Include noise-only segments, such as press cycles and alarms with no one addressing the agent, and score any transcribed text or triggered action as an insertion. JiWER's support for empty references makes this straightforward to compute alongside regular error rates.
Do I need consent from people in the background of a recording?
Usually the consent or notice basis has to cover everyone whose voice is captured, or their speech has to be masked or excluded. The rules vary by jurisdiction and worker status, so have counsel review the specific sites before recording.
Sources
- The Deep Noise Suppression Challenge synthesizer configuration sets an SNR range of -5 to 20 dB with 21 levels and can add real or synthetic room impulse responses. Source
- The challenge's snr_mixer function normalizes clean speech and noise, computes RMS levels and scales the noise to reach a target SNR. Source
- Pyroomacoustics implements the image source model and ray tracing to generate room impulse responses for 2D and 3D rooms. Source
- The Whisper model card says the models may output text not actually spoken in the audio and recommends robust evaluation in the specific context and domain before deployment. Source
- JiWER computes WER and CER from minimum edit distance and, since version 4.0, accepts empty references so hallucination on silent audio can be tested. Source
- OSHA requires a hearing conservation program at an 8-hour time-weighted average of 85 decibels and sets a 90 decibel permissible exposure limit. Source
- Research describes the Lombard effect as an increase in vocal intensity, duration and fundamental frequency when speaking in noise. Source
- ITU-T Recommendation P.56 standardizes the objective measurement of active speech level so results from different organizations are comparable. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.