AI uses for records
Voice agent numerical accuracy: testing spoken amounts, dates and names
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Measure it per value, not per word. Tag every spoken amount, date, quantity, account number and name in a reference transcript, score whether the agent captured the exact value after normalization, then follow each value to the business record it fed, such as the order, booking or payment entry. Report exact-value accuracy, confirmation behavior and the share of errors that reached a record, separately for each value type.
Agent run
Voice agent numerical
Key takeaways
- Word error rate weights every word equally, so it hides the few values that cause most of the harm.
- Score each spoken value as its own row: spoken form, reference value, captured value and record written.
- Normalize reference and captured values with the same versioned rules, and audit the normalizer.
- Separate first-pass accuracy from final accuracy to see what read-backs and repairs contribute.
- Count business impact from the records the call changed, after outcomes have had time to appear.
- Redact sensitive numbers before audio is shared, and check spoken digits by hand.
How do you measure business impact from voice transcription errors in names, amounts and dates?
Score the values, then trace them. A voice agent's numerical accuracy is the share of spoken values, such as amounts, dates, quantities, reference numbers and names, that end up correct in the system the call was meant to update. That takes three measurements: did the recognizer hear the value, did the conversation confirm or repair it, and did the right value land in the downstream record.
Word error rate cannot answer those questions on its own. The NIST sclite documentation states that its default scoring treats word errors with equal weight, so a dropped filler word and a wrong digit in an account number count the same. A system can improve its word error rate while getting more dates wrong, and a business only feels the second change.
The evaluation pack below works at the level of value mentions. Each spoken value gets a row that links the audio, the reference value, what the agent captured, any read-back or correction, and the record it changed. Business impact is then counted from the records, not inferred from the transcript.
Why word error rate hides the errors that matter most
Most words in a call carry little risk; a few carry most of it. Word error rate averages over all of them, and its result also depends on formatting choices that have nothing to do with whether a value was right.
The Whisper authors note that a transcript humans judge correct can still score a high word error rate because of minor formatting differences, and they built a text normalizer to standardize text before scoring. Their English number normalizer converts spelled-out numbers to digits, keeps suffixes such as ordinals, and reads successive single digits like one oh one as a nominal 101. Normalization is necessary, but it changes what counts as an error, so the evaluation must state which normalizer it used and check it on a sample.
Dates add an ordering problem on top of recognition. The W3C note on SSML say-as values explains that without a stated field order a string like 1/2/2004 could be read as either January 2 or February 1, which is why it defines formats such as mdy, dmy and ymd. A transcript that writes a date as digits has already made that choice, and the evaluation should record it.
| Spoken pattern | Typical error | Downstream consequence | How the pack catches it |
|---|---|---|---|
| Teens and tens: fifteen versus fifty | Wrong order of magnitude in a quantity | Order or booking with the wrong count | Exact-value match on the quantity field |
| Digit strings with oh, double or triple | Missing or extra digit in an ID | Lookup of the wrong account or order | Digit-level comparison after normalization |
| Ordinal dates: the thirteenth versus the thirtieth | Wrong day of month | Appointment or delivery on the wrong day | Canonical date comparison with month resolved |
| Relative dates: next Friday, the end of the month | Resolved against the wrong reference day | Booking a week off target | Store the call date and the resolved date |
| Amounts with a decimal part spoken in words | Cents dropped or merged into the whole number | Payment or adjustment entered wrongly | Compare in minor units as an integer string |
| Spelled names and email addresses | Wrong letter or homophone | Record created under a misspelled name | Character-level match on the spelled form |
What should the spoken number verification evaluation pack include?
One row per value mention, linked to its call, its turn and the record it affected. Keep the verbatim spoken form and the canonical value side by side so a reviewer can see both what was said and what it means.
At a glance
What should the spoken number verification evaluation pack include
Values in the table are illustrative. Store amounts as minor units in an integer string and dates in one canonical year-month-day form, and never mix display formats inside the comparison columns.
| Field | What it holds | Illustrative value | Check before use |
|---|---|---|---|
| Call, turn and speaker | Call ID, turn index, caller or agent | Call 0412, turn 7, caller | Turns match the diarized audio |
| Start and end time | Timestamps of the value in the audio | 62.4 to 63.9 seconds | Audio at those times contains the value |
| Value type | Amount, date, quantity, identifier, name | Quantity | One type per row |
| Spoken form | Verbatim words as said | fifteen cases | Matches a human listen |
| Reference value | Canonical value the caller intended after any repair | 15 | Adjudicated by a second listener |
| Captured value | What the agent extracted or wrote | 50 | Taken from logs, not a re-run |
| Read-back and correction | Whether the agent confirmed and whether the caller corrected | No read-back | Linked to the turn IDs |
| Downstream record and field | System, record ID and field written | Order 77-1182, quantity | Value read from the system of record |
| Outcome status | Whether the record value held up later | Changed after a delivery dispute | Dated, with the record that shows it |
| Audio and rights flags | Noise, overlap, transfer and consent status | Line noise; recorded with notice | Rights cleared for evaluation use |
Normalization rules that keep the comparison honest
Normalize the reference and the captured value with the same rules, keep the verbatim form, and audit the normalizer separately. If the normalizer is wrong, its mistakes show up as model errors.
Inverse text normalization is the step that turns spoken forms into written ones. NVIDIA's open NeMo text processing package includes English grammars for money, dates and telephone numbers; its date tagger, for example, maps january fifth twenty twelve to month, day and year fields, and its telephone tagger maps spoken digit strings to grouped numbers. Using a documented grammar gives you a rule set that can be versioned and tested.
Production recognizers are often biased toward these patterns already. Google Cloud Speech-to-Text documents class tokens such as a digit-sequence class for adaptation, so a deployment may recognize long numbers differently from an unadapted model. Record whether adaptation was on for each call, because an evaluation run without it does not describe the deployed agent.
- Dates: resolve to year-month-day, using the call date for relative expressions, and store the field order the caller used.
- Amounts: convert to minor units as an integer string; keep the currency separately.
- Identifiers: digits and letters only, no spaces or separators; expand double and triple; read oh as zero only in digit context.
- Quantities: integers with the unit in its own field.
- Names: compare the spelled form when the caller spelled it; otherwise mark as name heard, not name verified.
- Normalizer audit: hand-check a sample of normalized values per type before any score is reported.
Seven links from call audio to a changed record, and how to check each
An outcome claim holds only when each link in the chain is checked: the audio exists and may be used, the turn boundaries are right, the transcript aligns to the audio, repairs are captured and the downstream record has matured. A gap in any one of them turns an outcome claim into a guess.
At a glance
Seven links from call audio to a changed record, and how to check each
Work through the checklist per call before rows enter the pack. A call that fails an item can still be used for a narrower test, such as recognition only, if the limitation is written next to it.
Checklist · 7 items
- Audio rights: the recording was made with the notice or consent its jurisdiction requires and may be used for evaluation; counsel confirms this for the call set, not per call.
- Speaker turns: diarization assigns each value to the caller or the agent correctly.
- Timestamps: each value row points to audio that contains it.
- Transcript alignment: the reference transcript is aligned to the audio, not typed from memory or notes.
- Repair turns: corrections, restarts and read-backs are kept as separate turns.
- Transfer context: values collected before a transfer are tracked into the next leg of the call.
- Outcome maturity: the downstream record is old enough that later corrections, returns or disputes would have appeared.
0 of 7 ticked. Tick each item as you go.
Scoring read-backs and repairs, not just recognition
Score the conversation as well as the recognizer. An agent that mishears a date but reads it back and accepts the correction produced a correct record; an agent with better recognition that never confirms can still write more errors.
Report the measures below per value type, because the right policy differs by type. Confirming every quantity on a short call may be worth the extra seconds; confirming every spoken name may not. The SLUE benchmark shows why recognition alone is not enough: in its original baselines, the best pipeline that ran named entity recognition on speech recognizer output reached about 69.6 F1, against 81.4 F1 for the text model on human transcripts.
| Measure | Definition | Why it matters |
|---|---|---|
| First-pass accuracy | Share of values captured correctly before any read-back | Isolates recognition and extraction |
| Final accuracy | Share of values correct at the end of the call | What the business actually receives |
| Confirmation coverage | Share of values of a type that the agent read back | Shows whether the confirmation policy is followed |
| Correction uptake | Share of caller corrections the agent applied | Catches agents that confirm but ignore the answer |
| Record leakage | Share of value errors that reached a system of record | The direct business-impact measure |
| Confirmation time | Seconds added by read-backs per call | The cost side of the confirmation policy |
Worked example: a delivery change with a misheard date and quantity
This example is illustrative; the call, records and counts are invented to show the method. A caller asks a voice agent to move a delivery to the thirteenth and to change the order to fifteen cases. The recognizer outputs thirtieth and fifty.
The agent reads back the date, the caller says no, the thirteenth, and the agent corrects it. It does not read back the quantity. The order record shows the new date correctly and a quantity of 50. Three weeks later the receiving record shows 35 cases refused at the dock.
The pack captures two rows. The date row scores wrong on first pass, right at the end, with a read-back and a correction applied: no record leakage. The quantity row scores wrong on first pass and wrong at the end, with no read-back, and it leaked into the order record with a later outcome record proving the impact. Across a full run, the team sees that quantities leak far more often than dates, and the fix is a confirmation rule for quantities, not a new recognizer.
The exception is a caller who misspeaks and then corrects themselves before the agent acts. The reference value is the intended value after the caller's own repair, and the agent is scored on whether it used that value. Payment amounts follow the same pattern; the pack stores them in minor units, so the method never needs real figures in the report.
Where value-level evaluation reaches its limits
The method needs the downstream record, enough time for outcomes to appear and permission to use the audio. When any of those is missing, narrow the claim to what you can show, such as recognition accuracy on value spans, and say so.
Some values should never be in an evaluation set in clear form. Payment card numbers, government identifiers and other sensitive numbers are redacted or replaced before audio and transcripts are shared, and a privacy reviewer decides what replacement keeps the test meaningful. Spoken digit strings are also where automated redaction tends to miss, which is a reason to check them by hand.
- Stop when the downstream system cannot be joined to the call by a stable ID.
- Narrow to recognition-only scoring when outcome records are too recent to have matured.
- Bring in a voice evaluation lead when one value type drives most record leakage, to separate recognizer, normalizer and dialog causes.
- Bring in an operations reviewer to confirm which record changes were caused by the call and which by later events.
Sourcing calls that can be traced to the records they changed
Value-level evaluation needs call audio, aligned transcripts and the business records those calls updated, joined by IDs that survive export. Most public speech corpora have none of the downstream records, which is why this test usually depends on licensed operational data.
A buyer who needs calls joined to the records they changed can ask SourceX, which sources such sets from U.S. businesses on request, with scope, volume, language, format, timing and licensing needs laid out in the request. SourceX also runs rights review and preparation with the supplying business, so the call-to-record join and the redaction of sensitive numbers can be fixed before any audio is prepared.
Frequently asked questions
What is a good numerical accuracy metric for a voice agent?
Use exact-value accuracy per value type, measured at the end of the call, plus the share of errors that reached a system of record. Report first-pass accuracy alongside it so you can see how much the confirmation policy is doing. A single blended number hides which value types cause harm.
Why not just use word error rate?
Word error rate weights every word equally and depends on formatting conventions, so it moves with filler words and number formatting as much as with real value errors. It also stops at the transcript, while the impact happens in the record the call updated. Use it as a recognizer health metric, not as the impact measure.
Should a voice agent read back every amount and date?
Not necessarily. Measure record leakage and confirmation time per value type, then confirm the types whose errors reach records and matter most. Many teams confirm identifiers, amounts and dates and accept unconfirmed values that are easy to fix later.
How should oh, double and triple be handled in spoken digits?
Expand double and triple into repeated digits and treat oh as zero only inside a digit sequence. Apply the same rule to the reference and the captured value, and keep the verbatim spoken form in its own field. Check the normalizer on a sample, because its errors otherwise count against the model.
Can synthetic speech replace real calls for this test?
Synthetic speech is useful for coverage, for example rendering dates in mdy or dmy order with SSML say-as markup. It does not reproduce real callers' restarts, corrections, accents and line noise, and it has no downstream records. Use it to fill gaps, and keep real calls for the impact measure.
Sources
- The NIST sclite documentation states that its default word error scoring treats all word errors with equal weight. Source
- The Whisper paper notes that transcripts humans judge correct can receive high word error rates from formatting differences, and describes the text normalizer used before scoring. Source
- Whisper's English number normalizer converts spelled-out numbers to digits, keeps suffixes and reads successive single digits as a nominal number. Source
- NeMo text processing provides English inverse text normalization grammars for money, dates and telephone numbers. Source
- The W3C note on SSML say-as values defines date field orders such as mdy, dmy and ymd and explains that a date string without a stated order is ambiguous. Source
- Google Cloud Speech-to-Text documents class tokens for speech adaptation, including a class for digit sequences of any length. Source
- In the SLUE benchmark's original baselines, the best pipeline running named entity recognition on speech recognizer output reached about 69.6 F1, against 81.4 F1 for the text model on gold transcripts. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.