Spanish-English code-switching evaluation sets from US customer interactions

Why monolingual Spanish tests miss code-switching failures

Monolingual Spanish benchmarks miss the failure modes that appear when a customer writes "mi factura dice late fee pero ya hice el payment." Intra-sentential switches break tokenizers, intent classifiers, entity extractors and ASR language-ID gates that assume one language per turn. Typical failures include dropping the English noun phrase that carries the key entity, translating a product name that should stay verbatim, and replying fully in English to a customer who opened in Spanish.

This is why a mixed-language set belongs on its own URL and its own dashboard, distinct from the native-versus-translated questions covered in non-English instruction data. Translated test sets are especially weak here: machine translation tends to normalize a switched utterance into one language, which deletes the phenomenon you are trying to measure. For the speech-training side of the same problem, see code-switched speech data.

What public code-switching corpora can and cannot tell you

Public corpora are good for method development but rarely represent a US contact-center domain. The Bangor Miami corpus, a common reference for Spanish-English code-switching speech, consists of informal conversations recorded in Miami among bilingual speakers [1]. It is valuable for calibrating ASR and language-ID behavior, yet it contains no account lookups, billing disputes, policy constraints or agent actions.

Customer-service benchmarks have the opposite gap. tau-bench tests agents against simulated users, APIs and policy documents [7], and the Action-Based Conversations Dataset requires agents to follow guidelines and take actions [8], but both are English-only. Before you build on any public corpus, read its access and license terms; research-use terms can block commercial evaluation pipelines. The tradeoffs are covered in private evaluation sets versus public benchmarks.

Slicing the set by code-switch density and switch type

Report scores per slice rather than one blended number, because a model can look fine on average while failing every heavily mixed turn. Tag each turn with three things: the matrix (dominant) language, the number of switch points, and the switch type (inter-sentential, intra-sentential, or single-word insertion). From those tags, compute a simple density measure such as switch points per 10 tokens and bucket turns into bands.

Useful slices for support and sales traffic:

• Monolingual control: pure Spanish and pure English turns from the same customers, so you can isolate the cost of mixing.
• Lexical insertion: Spanish matrix with English nouns for products, fees, plan names and app screens ("el app no me deja hacer login").
• Intra-sentential switches: a clause boundary inside one sentence, often where intent and entity end up in different languages.
• Turn-level alternation: the customer switches language between turns, often after an agent switches first.
• Mixed-language entities: addresses, names and account references dictated across both languages, including spelled-out numbers in Spanish.

Allocate examples so that rare, high-impact slices are not starved; the method in stratified evaluation sets for rare and high-risk cases applies directly. Hold out by customer or conversation, not by turn, so the same speaker's phrasing does not leak across splits.

Scoring comprehension and response-language choice separately

Score understanding and language choice as two independent axes, because a correct answer in the wrong language is still a failed interaction. Comprehension items check whether the model extracted the right intent, entities and constraints from the mixed turn. Response-language items check whether the reply matches the expected language policy: mirror the customer, follow the stated preference, or default to the account's preferred language.

Write the language policy down before labeling, since raters otherwise disagree on whether mirroring a Spanglish turn with Spanglish is correct. For support agents, layer this on top of policy-following checks such as those in customer support agent policy evaluation; the language rule is just another policy the agent can violate.

For voice pipelines, aggregate word error rate hides where errors occur. Mixed Error Rate combines word- and character-level error rates across languages but does not isolate switch points [2], and recent work argues for switch-aware scoring that measures errors at and around language boundaries [3]. Score ASR on switch-point words separately and propagate transcripts into the downstream intent eval, as described for voice agent evaluation.

Native bilingual raters and label quality

Use raters who are native or heritage speakers of US Spanish varieties and fluent in English, because judgments about natural switching, register and cultural nuance depend on lived usage. Human-evaluation work on cultural nuance in LLM translation relies on native-speaker judgment precisely because token-level metrics miss pragmatic meaning [4]. Match rater variety to traffic: Mexican, Caribbean and Central American Spanish differ in lexicon, and a rater from one region may mark another region's normal phrasing as an error.

Plan for adjudication. Label errors in widely used test sets average at least 3.3% and are enough to change model rankings [6], and judgments about natural switching and expected response language add their own disagreement. Double-label a calibration subset, track agreement per slice, and send disagreements to a third adjudicator with written guidelines.

Consent and de-identification for recorded bilingual calls

Call audio and transcripts need a documented basis for recording and for reuse in evaluation. California Penal Code section 632 prohibits recording a confidential communication without the consent of all parties [5], so check that the supplier's IVR disclosure covered every party on the call and that the notice played in both languages where the call flow allowed either. Ask whether the original customer terms permit use of interactions for model testing, not only for quality assurance.

De-identification is harder across two languages. Name and address detectors trained on English miss Spanish surnames, compound surnames and Spanish-language street formats, and dictated account numbers may appear as Spanish number words. Ask for the redaction method, a per-language recall check on a sample, and consistent pseudonyms so that entity-tracking items still work. Healthcare traffic adds HIPAA de-identification requirements.

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

Request template for a Spanish-English evaluation set

Describe the data, not a specific company, and specify the slices and labels you need before talking to suppliers. A clear request also makes it easier to check, at sample review, whether a candidate dataset actually contains enough mixed turns.

Illustrative example: invented to show structure; it does not describe an available dataset.

An illustrative record for one labeled turn:

{
  "conversation_id": "conv_8821",
  "turn": 3,
  "speaker": "customer",
  "text": "ya pagué el balance pero el app sigue diciendo past due",
  "matrix_language": "es",
  "switch_points": 2,
  "switch_type": "intra_sentential",
  "intent": "payment_not_reflected",
  "entities": {"status_string": "past due"},
  "expected_response_language": "es",
  "rater_ids": ["r07", "r12"],
  "adjudicated": false
}

Keeping the set private and refreshing it

A bilingual eval set loses value once its items appear in training data, so treat it as a held-out asset with access controls. Keep the gold labels out of shared prompt repositories, rotate a slice each quarter with new interactions, and track score drift on the stable core. The broader controls are in contamination-resistant evaluation design, and the full map of eval data types is on the evaluation datasets hub.

Avoid backfilling sparse slices with generated Spanglish. Synthetic code-switching tends to place switches where a model expects them rather than where people actually switch, a problem examined in synthetic evaluation data limits.

How SourceX sources bilingual customer interactions

SourceX sources operational datasets from US companies, including support and sales histories, and manages the licensing process; see evaluation sets built from real business work and customer support agent data. Data is sourced on request rather than held in stock, so a request for bilingual interactions does not guarantee a match. You describe the data you need, SourceX looks for US businesses that hold it, and every release is approved by the supplying company.

Each dataset goes through rights review of ownership and consents and is delivered under a license that defines records, uses, term and delivery. Names, emails, phones and account numbers are removed or replaced before delivery, the method is recorded and a sample is checked, though no method is perfect. For multilingual work beyond evaluation, see training data for multilingual enterprise models, or describe your bilingual evaluation requirement.

Source a Spanish-English code-switching evaluation set

SourceX serves AI teams wherever they are based and runs each request through Find, Assess, Agree, Transact and Manage; nothing is contracted until a supplier agrees. Delivery runs through private, access-controlled workflows only after an executed agreement and supplier approval. Start a buyer request at sourcex.si/buyers.