Regression test suites for production LLM applications

What a regression suite does that a golden set does not

A golden set measures how good the application is; a regression suite answers a narrower question on every change: did anything that used to work stop working?

One practitioner's rule for evaluation sets is to sample from production, freeze, version like code and never let the set grow organically [1]. A regression suite has to grow, so make growth an explicit release and compare results only within a suite version. The representative set is covered in building a golden evaluation dataset from real business records, and the terms in the golden dataset and eval set glossary entries.

A suite holds three kinds of case: fixtures, one per fixed failure, which must always pass; slice sets, grouped by intent, customer segment or risk and gated on pass rate; and invariants checked on every output, such as valid JSON against the response schema and no forbidden tool calls.

From production failure to permanent test case

The loop that keeps fixed failures fixed is short: capture the failing session, define the correct behavior, add the case, fix the system and confirm the case passes. One engineering guide runs this triage weekly and mines signals well beyond thumbs-down ratings: support tickets, human corrections, escalations, refunds, abandoned workflows, failed JSON output, forbidden tool calls and monitor alerts [2].

1. Capture the full trace: user input, system prompt version, model identifier, decoding settings, retrieved passages, tool results and prior turns. An isolated user message is often not enough [2].
2. Reproduce before fixing. Replay the case several times and confirm it fails; if it fails only sometimes, record the rate for the gate.
3. Write the expected behavior, not only expected text. "Denies the prorated refund and cites the annual-plan clause" survives rewording; a verbatim reference does not.
4. Tag from a controlled list, such as hallucinated policy, wrong tool, ignored passage or format break, so slice pass rates mean something.
5. De-identify without erasing the bug. Keep the raw trace in a restricted store and use consistent surrogates in the suite, because blanket placeholders can remove the cue that caused the failure; see de-identifying evaluation data without breaking the test.
6. Fix, confirm green and release the case as a must-pass fixture.

A case record that can be replayed after the system changes

A regression case must carry enough context to replay offline, its grader definitions, and provenance that explains why it exists and whether you may keep it. The same fields work as one JSON Lines row per case.

Illustrative example: invented to show structure; it does not describe an available dataset.

case_id: reg-support-000731
suite_version: "2026.10.2"
kind: fixture                       # fixture | slice | invariant
status: active                      # active | quarantined | retired
origin:
  type: production_incident         # production_incident | licensed_record | synthetic_variant | red_team
  incident_ref: INC-4182            # pointer to the restricted raw trace
  first_failed_on: {model_id: "provider-model-2026-06-01", prompt_version: "support-sys-v41"}
  fixed_by: "PR 2214"
  license_scope: internal           # evaluation_only for cases from licensed records
  deidentification: consistent_surrogates_v2
slice: refunds/annual_plans
failure_mode: policy_hallucination
severity: high
input:
  messages_ref: inputs/000731.json
  context_snapshot:
    retrieved:
      - {doc_id: "kb-refund-policy", doc_version: "2026-08", sha256: "9f2c..."}
    tool_results_ref: tools/000731.json
expected:
  behavior: "Denies prorated refund; cites annual-plan clause 4.2; offers downgrade"
  policy_version: "refund-policy-2026-08"
  graders:
    - {type: tool_call, name: lookup_subscription, args_match: exact}
    - {type: cites_doc, doc_id: "kb-refund-policy"}
    - {type: llm_judge, rubric: "refund-rubric-v3", judge_model: "pinned", pass_label: correct_denial}
gate:
  trials: 3
  required_passes: 3

• context_snapshot with content hashes lets a prompt change be tested against the same evidence after the knowledge base moves on, and policy_version shows when the expectation, not the system, has gone stale.
• incident_ref keeps personal data out of the suite repository.
• license_scope matters once cases come from another company's records; see evaluation-only data license terms.
• status: quarantined pulls a flaky case out of the gate without deleting it.

Which changes should trigger which tests

Run the suite on any change that can alter output, not only code changes: prompt edits, model swaps, retrieval changes, tool schemas, decoding settings and the documents expected answers depend on. Each change tends to break different things, so map triggers to slices.

If your application calls a model through an alias rather than a pinned version, the model can change with no change in your repository, so log the identifier returned with each response. Test generation against frozen context snapshots, and test retrieval separately against gold document IDs on a versioned index. Ragas's v0.1 documentation, for example, structures a test record as a question, retrieved contexts, the generated answer and a ground_truth reference [3]; see question-answer-citation triples for RAG evaluation.

Cost decides cadence: deterministic fixtures and invariants on every pull request, model-graded slices nightly or before merge, and held-out slices only on release candidates. That tiering is a suggestion; vendor guides from Langfuse and Arize describe the underlying practice of running application versions against stored evaluation datasets before changes ship, as evidence of common practice rather than a standard [4][5].

Matching the grader to the failure

Use the cheapest grader that reliably detects the specific failure, and reserve a calibrated LLM judge for qualities that cannot be checked mechanically. Every model-graded case adds cost and a second model whose own changes can move your scores.

Calibrate a judge before gating on it, measuring agreement with human labels with a chance-corrected statistic such as Cohen's kappa rather than raw agreement [6]. LangChain's guide notes that this agreement depends heavily on how well the evaluator prompt captures what good means for the use case [7]. Pin the judge model and rubric in the suite version, or a judge upgrade will look like an application regression; see LLM-as-a-judge calibration sets.

Setting gates that catch regressions instead of noise

Gate fixtures on every trial, and gate slices on a paired, case-by-case comparison with the previous release, using thresholds written down before the run. An unpaired comparison of two pass rates on a small slice tends to block releases on noise or let real regressions through.

• Small slices detect only large drops. One practitioner analysis, framed at 80% power and 5% significance, notes that halving the effect you want to detect roughly quadruples the cases needed [8]. In its example, detecting 82% versus 85% with a two-proportion z-test takes about 2,400 labeled examples per variant, while a 100-example set detects only differences nearer 10 to 12 points [8].
• Pair the comparison. Documentation for one eval-statistics package notes that when both systems answer the same items, a paired comparison is more sensitive than an independent-samples one [9].
• Distrust normal-approximation intervals on small slices. An ICML 2025 position paper argues that CLT-based intervals tend to be too narrow on evals with fewer than a few hundred items [10].
• Account for sampling. At nonzero temperature, one run per case mixes noise with regression, so run several trials, require every fixture trial to pass, and score slice cases by majority result. See agent evaluation task suites for multi-trial metrics and how many examples an LLM eval set needs for sizing.

Worked example: a model upgrade on a 300-case slice

Illustrative example: invented to show structure; it does not describe an available dataset.

A support assistant's refunds slice has 300 cases. The current model passes 276 (92.0%) and the candidate passes 267 (89.0%). The gate, written in advance, blocks if the drop is at least 2 points and significant one-sided at the 5% level; any failing fixture blocks regardless.

• Unpaired reading. A two-proportion z-test on 92.0% versus 89.0% with 300 cases each gives z ≈ 1.25 and a one-sided p ≈ 0.11, which looks like noise.
• Paired reading. 14 cases flipped from pass to fail, 5 from fail to pass, and 281 did not change. An exact McNemar (sign) test on the 19 discordant cases gives a one-sided p ≈ 0.03, so the gate blocks.
• Diagnosis. The 14 newly failing cases are the real output of the run. If most share a failure mode, such as a changed refusal style on annual plans, fix it or accept it in writing.

Model upgrades: why provider benchmarks cannot replace your suite

A new model version can score higher on public benchmarks and still fail your fixtures, because those benchmarks use other tasks, prompts and formats, and some are contaminated. In a post dated 23 February 2026, OpenAI said it had stopped reporting SWE-bench Verified because score gains increasingly reflected training-time exposure to the benchmark [11]. Your own suite, kept out of training data, is the evidence that bears on your prompts.

• Run the full suite, including slices nobody tuned prompts against, with several trials per case.
• Compare output length and format distributions too; a more verbose model can break parsers or UI limits before any judge notices.
• Keep the judge model fixed, and gate the exact prompt and model combination you will ship, since prompts tuned to the old model may carry workarounds.
• When choosing between providers, see running a model bake-off with your own eval set.

Seeding a suite before launch, when there are no failures yet

Before launch there are no production failures to mine, so the first suite version comes from internal testing, generated cases or real historical records from businesses that do the same work. Real records matter most for high-severity slices, because people doing the job decided their outcomes.

• Dogfooding and internal red-teaming find obvious breakage but reflect your staff's phrasing.
• Generated cases are cheap but can inherit the generator's blind spots; see when synthetic evaluation data misleads.
• Licensed operational records, such as support tickets with the resolution applied, invoices with the GL code posted or contracts with accepted redlines, supply real inputs and recorded decisions as expected behavior.

Licensing for a regression suite raises questions a one-off evaluation does not: does the scope cover repeated automated runs for the life of the product, may cases go to hosted model APIs and third-party judges, what happens to derived fixtures when the license ends, and can later time windows be licensed for refresh?

SourceX sources operational datasets from US companies on request, including support and sales histories, engineering records, documents, and finance and legal workflows; these are kinds of data it sources, not inventory, and a request does not guarantee a match. Buyers describe the data, SourceX looks for US businesses that hold it, and every release is approved by the supplying company. Each dataset goes through rights review and is delivered under a license defining the records included, permitted uses, term and delivery.

Names, emails, phone numbers and account numbers are removed or replaced before delivery, with the method recorded per dataset, so check that it preserves what your cases test. If your pre-launch slices need real cases, describe the workflows and outcomes your suite must cover.

Keeping the suite trustworthy as it grows

A regression suite decays in predictable ways: references are wrong or go stale, flaky cases accumulate, prompts get tuned against it, and cases leak. Each needs a standing control.

• Wrong references. Northcutt et al. estimated an average label error rate of at least 3.3% across the test sets of 10 widely used datasets and showed such errors can change model rankings [12]; references written mid-incident deserve the same suspicion. Re-review a sample of fixtures each quarter, and treat a case every candidate fails as a possible bad reference.
• Flaky cases. Quarantine cases whose results vary on an unchanged system, with an owner and a deadline; a gate that fails at random gets overridden.
• Overfitting. Keep a tuning split for prompt work and a gate split nobody reads while editing prompts. Cases reused as few-shot examples or fine-tuning data stop testing behavior.
• Leakage. Cases sent to hosted APIs and external judges leave your control; see keeping a private eval set private.
• Saturation and drift. Keep old fixtures, since they guard their fix, but refresh capability slices every model passes (eval set refresh cadence) and compare slice mix with live traffic (drift between licensed and production data).

What regression records give risk and compliance reviewers

Stored per-run results from a versioned suite are the kind of measurement evidence AI risk frameworks describe, and in the EU, if the application is a high-risk AI system, the suite's test data can fall under data-governance rules even when no model is trained.

NIST's voluntary AI RMF 1.0 (NIST AI 100-1) organizes its Core around four functions: Govern, Map, Measure and Manage [13]. Its Generative AI Profile, NIST AI 600-1, lists 12 risks unique to or worsened by generative AI, including confabulation, data privacy, and value chain and component integration [14]. Slices for hallucinated policy answers, personal-data exposure and third-party model upgrades map onto those three.

Article 10 of the EU AI Act covers high-risk AI systems. For those not using techniques involving model training, which may describe an application that only prompts a third-party model, Article 10(6), as amended by Regulation (EU) 2026/1744, applies paragraphs 2, 3 and 4 and Article 4a(1) to testing data sets alone, including that they be relevant, sufficiently representative and, to the best extent possible, free of errors and complete [15]. As of October 2026, Regulation (EU) 2026/1744 has reportedly moved the Annex III high-risk date to 2 December 2027, and also amended Article 10 [16]. Classification is a question for counsel; see EU AI Act Article 10 for training, validation and test data.

Pre-release checklist for an LLM regression gate

• Every fixed incident has a fixture that failed before the fix and passes after it.
• Each case stores its context snapshot and the model and prompt versions it first failed on.
• Slice thresholds and the significance rule were set before the run, and comparisons are paired on the same suite version.
• Fixtures run the trials the gate assumes; flaky cases are quarantined with an owner.
• Judge model and rubric are pinned, judge calibration is current, and nobody tunes prompts on the gate split.
• Licensed cases carry a license scope covering repeated CI runs and every endpoint the suite calls.

For public benchmarks, private sets and licensed test data more broadly, see the evaluation and benchmarking datasets hub and evaluation datasets built from real business work.

Need real cases to seed or extend a regression suite?

Describe the workflows, slices and expected outcomes your suite has to cover, and the uses your CI process needs licensed, such as repeated automated runs and calls to hosted model APIs or judge models. SourceX looks for US companies that hold matching records, checks the data and the supplier's licensing permissions, and manages the license and delivery. Describe your evaluation data needs.