Skip to content

Systems and records

Production configuration regression dataset: tests passed, setups differed

By SourceX Editorial · Updated

10 sections

Short answer

A production configuration regression dataset links each failure to the condition its tests never had: a database backend, a setting, a runtime mode or a rollout state. In SQLAlchemy's 2.0 changelog, 26 of 74 regressions named such a condition and 16 fixes added a test pinned to it. Of 170 Wikimedia production configuration reverts in 2024-2025, 48 cited a failure or version mismatch; one led to a test change.

System export

Production configuration

This page covers production configuration regression dataset: tests passed, setups differed.Illustrative

Key takeaways

  • Record the condition, not only the bug: backend and version, setting, runtime mode and rollout state at the moment of failure.
  • In SQLAlchemy's 2.0.1 to 2.0.55 changelog, 26 of 74 regressions (35.1 percent) named an environment condition, 21 of them through a database dialect tag.
  • Of the 25 environment fixes found, 16 (64.0 percent) added a test tied to the backend or feature, against 17.0 percent of the other 47.
  • Of 30 traceable changes that caused a SQLAlchemy regression, 25 had changed tests themselves: the break shipped with passing tests attached.
  • Wikimedia's 48 failure reverts of 2024-2025 came a median 0.6 hours after the change, none of the reverted changes touched tests, and one led to a test change within 30 days.
  • Use pinned records for evaluation and revert-only records for diagnosis training, and ask a supplier for the CI configuration that ran beside each change.

What is a production configuration regression dataset?

It is a set of failure records in which a change passed the checks it was given and then broke under a condition that production had and the test run lacked. Each record names that condition (a database backend or version, a setting, a runtime mode, a rollout state), the change that introduced the failure, the evidence that it passed, the signal that exposed it, the fix, and whatever test came afterwards.

The condition is the point. A regression caused by an untested input teaches a model to write more input cases; one caused by a configuration gap teaches it to ask which environment a change will meet. Records that mix the two unlabeled bury the second lesson inside the first.

Two public histories supply the counts here, SQLAlchemy's 2.0 changelog and fix commits and the revert log of Wikimedia's production MediaWiki configuration. The buyer's question is narrow: does a supplier's engineering history keep enough of the environment to reproduce each failure, or only enough to read about it?

Why does a green test run say little about the setup it never ran?

Because a test suite runs under one configuration and production runs under many. SQLAlchemy's testing guide says a plain pytest run executes the full suite against SQLite, and that running against any other backend requires a database of that vendor at a specific URL; a large subset of tests is marked to run on every available backend, and others run only when a matching backend is present (README.unittests.rst, main at commit 5c2970e, October 27, 2026).

The Twelve-Factor App's dev/prod parity factor names the same gap, with SQLite in development and MySQL in production as its example, and advises against different backing services across environments because small differences let code that worked in development fail in production (12factor.net, checked October 27, 2026; paraphrased from a search excerpt).

Google's report on the June 12, 2025 Google Cloud outage says a new quota policy check in Service Control lacked a feature flag, that its code path was never exercised during the region-by-region rollout because only a specific policy could trigger it, and that a later policy change containing unintended blank fields set it off (status.cloud.google.com; paraphrased from a search excerpt). The configuration that mattered had not arrived yet.

How often did SQLAlchemy's regressions depend on the environment?

About one in three did. SQLAlchemy's 2.0 changelog lists 55 released patch versions, from 2.0.1 on February 1, 2023 to 2.0.55 on October 16, 2026, with 582 change entries; 74 of them describe a regression, each under its own ticket. Twenty-six (35.1 percent) name an environment condition, counted on the main branch at commit 5c2970e on October 27, 2026.

The changelog tags every entry, so the database side needs no guessing: 21 regressions carry a dialect tag, 13 PostgreSQL, 3 MySQL, 3 SQLite and one each for SQL Server and MariaDB. Eight name a driver in the text, 7 of them asyncpg, and four a runtime condition, asyncio three times and the Cython extensions once. One entry shows how specific a production setup can be: a revised connection ping broke asyncpg behind PgBouncer in transaction mode (ticket 10226). SQLite, the suite's default, appears three times too.

The fixes split along the same line. Seventy-two regressions map to a commit whose Fixes: line names the ticket; 64 of those changed files under test/. Among the 25 environment-specific fixes, 16 (64.0 percent) added test lines that tie the test to a backend or feature, such as the class-level only_on or backend markers, a requires decorator or a file under test/dialect/; among the other 47, 8 (17.0 percent) did. None of the four fixes for runtime conditions matched a pin.

Thirty-four entries name the ticket whose change caused the regression. For 30 of those the causing change can be found by its own Fixes: line, and 25 of the 30 (83.3 percent) had changed tests itself: the break shipped with passing tests attached. Nineteen entries name the release that introduced the regression, and the median gap from that release to the fixing release was 48.0 days, 45.0 for the nine environment cases and 98.0 for the ten others.

The table splits the 74 regressions by the condition each entry names, with the fixes whose new test pins it:

How often did SQLAlchemy's regressions depend on the environment?
Condition named in the entryRegressionsExample ticketFixes whose test pins it
Dialect tag only15#12488, lowercase on update in a MySQL server default12 of 14 found
Dialect tag plus a driver6#10226, asyncpg behind PgBouncer in transaction mode3 of 6
Driver only1#9173, insertmanyvalues for positional dialects such as asyncpg1 of 1
Runtime only3#9418, pickled rows between Cython and pure Python0 of 3
Driver plus runtime1#9237, asyncpg connections reset outside the event loop0 of 1
No condition named48#13609, pyformat-like literal text turned into binds8 of 47 found

What did 170 reverts of Wikimedia's production configuration record?

Mostly fast rollbacks that changed nothing but configuration. Between January 1, 2024 and December 31, 2025 the wikimedia/operations-mediawiki-config history holds 3,756 commits and 170 reverts that name the commit they undo; 162 of the reverted changes touched only settings, wiki lists, version pins or static files, and one touched tests (commit 26154d3, October 27, 2026).

Revert reasons are free text, so a keyword rule labels them. Forty-five reasons describe a failure seen after merge (errors, 5xx alerts, type errors, slow queries, broken pages) and three a version or rollout mismatch, such as "group1 is not on 1.46.0-wmf.7 yet"; together that is 48, or 28.2 percent. Another 35 are planned ends such as finished maintenance, 19 are not-ready or process slips, 27 give no reason and 41 fit no rule.

The 48 failure reverts landed a median of 0.6 hours after the change they undid, 44 of them within a day. One reason reads "wikifunctionsclient_usage table has not been deployed on the production DB": the configuration assumed a table production lacked. None of the 48 reverted changes touched tests, 46 carried a task ID on a Bug: line, and in the 30 days after each revert only one of those tasks saw a commit that changed tests. Six changes came back under the same subject within 30 days, one with a test change.

The repository's composer.json defines a test script (typo checks, linting, JSON validation, PHPUnit, phpcs, phan). Git does not record whether it ran on a given change, so the count claims only that what failed in production rarely flowed back into tests.

The configuration regression trace: which fields make a record reproducible?

Eight fields turn a failure story into a record a model can learn from or be graded on. The first four are fixed when the change ships; the last four arrive later and must be stored apart, so a prompt built from the early fields cannot read the answer.

Each row below names the field, what it holds, what the two public histories show about it and what a buyer loses without it:

The configuration regression trace: which fields make a record reproducible?
FieldWhat it holdsPublic evidence of the gapIf it is missing
Environment conditionBackend and version, settings difference, runtime mode, flag and rollout state at failureSQLAlchemy entries name one in 26 of 74 regressions; Wikimedia reasons name a version in 3 of 170The record reads as a code bug and teaches input testing
Introducing changeThe commit or configuration change and its files34 of 74 SQLAlchemy entries name the causing ticket; every Wikimedia revert names its targetNobody can rebuild the state before the failure
Passing evidenceThe CI run, the configuration it used and the tests it ran25 of 30 traceable SQLAlchemy causing changes had changed tests; Wikimedia's git keeps no CI resultNo proof the suite was green, so no proof of a gap
Exposure scopeWhich servers, sites or tenants had the conditionWikimedia reasons cite wiki groups and train versionsImpact cannot be sized
Failure signalError rate, alert or report, with the time and where it was seen45 Wikimedia reasons cite errors, alerts or broken pagesDetection time is unknown
Fix typeCode change, configuration-only change, revert or flag flip162 of 170 reverted Wikimedia changes were configuration onlyRemediation tasks are mislabeled
Test that followedThe test added or changed, and whether it sets the condition16 of 25 SQLAlchemy environment fixes pinned it; 1 of 48 Wikimedia failure reverts led to a test changeThere is no guard to grade a candidate fix against
Elapsed timeExact time from change to detection and to fixMedian 0.6 hours (Wikimedia failure reverts); 48.0 days from breaking to fixing release (SQLAlchemy)Fast rollbacks and latent defects look alike

An outcome rubric for configuration regression records

Grade what happened after each failure on a fixed label set, because the label decides how a record may be used. A record whose fix pins the condition can grade a model's fix; one that was only reverted can teach diagnosis, not grade a repair that never happened. SQLAlchemy's backend markers show the mechanism: a test tied to a backend runs only where that backend is present, which is what a grader for a configuration failure needs.

The two histories fail in opposite places. SQLAlchemy's process turns most regressions into tests, and about two thirds of its environment fixes pin the condition, while Wikimedia's configuration practice turns failures into reversals within the hour and almost never into tests. The rubric applies each history's own counted labels:

An outcome rubric for configuration regression records
Outcome labelDefinition, as countedSQLAlchemy, 25 environment fixesWikimedia, 48 failure revertsBest use
Condition pinnedThe fix adds test lines that tie the test to the backend or feature16Not countedEvaluation: the pinned test grades a candidate fix
Test added, condition not pinnedThe fix changes tests, but no added line ties it to the condition4Not countedTraining, with a check that the test fails in the named setup
Fixed without a test changeThe fix touches no file under test/5Not countedDiagnosis practice; ask the supplier for a reproduction
Reverted, not retriedNo commit with the same subject within 30 days of the revertNot applicable42Diagnosis and rollback-decision training; mark the need unresolved
Retried without a test changeSame subject returns within 30 days, no tests/ changeNot applicable5Pair the failed and retried versions; the difference is the label
Retried with a test changeSame subject returns within 30 days with a tests/ changeNot applicable1The complete record: failure, change and guard
Two side-by-side funnels. SQLAlchemy 2.0: 74 regressions narrow to 26 that name an environment condition and 16 whose fix test pins it. Wikimedia configuration: 170 reverts narrow to 48 failure or version reverts and 1 followed by a test change.
How often a failure tied to the environment came back as a test that sets that environment. Source: SQLAlchemy and Wikimedia git histories, counted October 27, 2026.

The strongest objection: most failures were not about configuration

That is true in both histories, and it limits what these records can claim. Forty-eight of SQLAlchemy's 74 regressions name no environment condition, and some of the 26 that do might fail on an untested input in any setup; a PostgreSQL tag shows where a user found the bug, not that every other backend was safe.

The keyword rules also miss. An entry tagged only orm describes a versioning feature that failed on SQLite and similar databases, and the rule, which reads dialect tags rather than prose, does not count it; several of the 41 unlabeled Wikimedia reasons are plainly failures, such as a query killed for a bad plan. Treat the condition-named shares as lower bounds.

Then the business objection: a passing test, a revert or a closed ticket may not settle the need that prompted the change. Of the 48 Wikimedia failure reverts, 42 had not returned under the same subject 30 days later, and the log cannot say whether the feature shipped another way or lapsed. A record that stops at the revert should be labeled unresolved, with the ticket that states the original need attached.

How would a buyer use configuration regression records?

In three ways. Training for environment-aware debugging needs the condition, the introducing change and the failure signal together; an evaluation set needs fixes whose test fails under the named condition and passes after; a rollback-decision set needs exposure scope and timing.

At a glance

How would a buyer use configuration regression records

Before licensing, run these checks on a sample the buyer draws, not one the supplier picks:

  • Reproduction: rebuild the before and after states of 20 records under the stated configuration; the failure should appear only under the named condition.
  • Condition coverage: report the share of records that name a backend, setting, runtime or rollout state, with the exact rule used to find them.
  • Pin rate: report how many fixes add a test that sets the condition (SQLAlchemy's environment fixes: 16 of 25).
  • Leakage guard: keep the later fields (fix, test, postmortem) out of any prompt built from the decision-time fields.
  • Time split: hold out the latest quarter, because configurations drift.
  • Secrets and topology: confirm hostnames, credentials and internal addresses are removed or replaced before any record leaves the company.

What a company's own history adds, and how to scope a request

Private histories keep what public ones drop: the CI configuration that ran beside each change, deployment manifests, feature-flag states, the incident ticket and the customer report. Joined, they show the suite green under one configuration and the failure under another.

A request should name the conditions of interest (database versions, tenant settings, regional rollouts), the eight trace fields, the outcome labels that matter, the acceptance test and the intended use. SourceX licenses business data directly from the companies that hold it, for AI training and evaluation; datasets are sourced when a buyer asks rather than kept on a shelf, and the supplying company approves the scope and terms and signs before any data moves. No particular company's history is implied to be available.

Methods note: SQLAlchemy and Wikimedia counts, rules and limits

SQLAlchemy: github.com/sqlalchemy/sqlalchemy, main branch, partial clone with file contents fetched on demand, at commit 5c2970e9e5cb (committed October 26, 2026, read October 27); no repository code was run. Entries are the change blocks in doc/build/changelog/changelog_20.rst under versions 2.0.1 to 2.0.55 that carry a released date (55 versions; 2.0.5.post1 counts as 2.0.5; 582 entries). A regression entry contains the whole word regression in any case, in its text or tags (74); entries are keyed by their first ticket, and no key repeated.

An environment condition is any of: a dialect tag (postgresql, mysql, mariadb, sqlite, oracle, mssql); a whole-word, case-insensitive driver name in the text (asyncpg, psycopg, psycopg2, pg8000, pyodbc, pymssql, aioodbc, aiosqlite, pysqlite, pymysql, mysqlclient, aiomysql, asyncmy, mysqlconnector, oracledb, cx_oracle); or a runtime condition (the asyncio tag, the whole words asyncio, greenlet, PyPy, Windows, macOS or Cython, or Python 3 followed by a minor version). The fix is the earliest non-merge commit reachable from main with a Fixes: line containing the ticket number; two tickets had none and one had two, and the script prints them.

A pin is an added line in a test/ file containing any of only_on(, requires., skip_if(, fails_on(, fails_if( or against(, or one of the class attributes only_on, backend and requires written with two underscores on each side, or any changed test file under test/dialect/. A causing ticket is a ticket reference that starts within 60 characters after caused by, due to, introduced or fix for, in the same sentence; a sentence ends at a period, question mark or exclamation mark followed by a space, so the periods in a version number such as 2.0.30 do not end it. Its change is found by its own Fixes: line. An introducing release is the first 2.0.N in the entry text lower than the entry's own version, and the gap is the difference between the two released dates in days. Shares and medians are rounded half up to one decimal, once; medians of an even count average the two middle values.

Wikimedia: github.com/wikimedia/operations-mediawiki-config at commit 26154d3f826f (October 27, 2026), partial clone; the window holds no merge commits. A revert has a UTC committer time from 2024-01-01 to 2025-12-31, a subject starting with Revert and a quotation mark, and a This reverts commit or This partially reverts commit line with a 40-character hash on master; two without that line were dropped and no revert of a revert fell in the window. Configuration-only means every path is under wmf-config/, dblists/, static/, logos/, docroot/, w/, portals/ or private/, or is dblists-index.php, a wikiversions JSON file, a langlist file, debug.json or robots.txt.

Labels take the first match on the Reason for revert text, read from the heading to the first trailer line (such as Bug: or Change-Id:) or the end of the message, blank lines included; words and phrases match whole, in any case: version or rollout mismatch (wmf. with digits, train, group0, group1, group2); failure observed after merge (error, errors, 5xx, 500s, 503s, 404s, exception, fatal, crash, broke, broken, breaks, breakage, fail, fails, failed, failing, latency, slow, logspam, plummeted, misses, caused, causing, causes, not work, did not work, didn't work, does not work, doesn't work, not working, type error); not ready or process slip (19 phrases in the script, among them not ready, too early, backport window and accidentally); planned end (ended, over, done, completed, finished, no longer, not necessary, obsoleted, resolved, re-enable, reenable, fixed); no reason given; other. A test change followed when a commit within 30 days after the revert changed tests/ and shares a Bug: task ID with the revert or the reverted change; a retry is a later commit with the reverted change's exact subject within 30 days.

Not measured: CI results, SQLAlchemy's CI matrix, production traffic, issue threads, and anything inside any company. Saved outputs hold ticket numbers, hashes, times and labels; revert reasons, which can contain usernames, were not saved. The Google and Twelve-Factor statements come from search excerpts on their own sites and are paraphrased.

Frequently asked questions

What makes a regression a production configuration regression?

The change passed the checks it was given and failed only under a condition the test run lacked, such as a database version, a setting, a runtime mode or a rollout state. If the same failure appears in every setup, it is an input or logic bug, and the record should say so.

How common are environment-specific regressions in a mature project?

In SQLAlchemy's 2.0 changelog, 26 of 74 regressions (35.1 percent) named an environment condition, 21 of them through a database dialect tag. That share depends on the rule used to find conditions, so treat it as a lower bound rather than a benchmark.

Why is a configuration-only revert weak evidence on its own?

It shows that something failed and was undone, usually fast, but not why the checks missed it or whether the need was met later. In Wikimedia's configuration log, 42 of 48 failure reverts in 2024-2025 had not returned under the same subject within 30 days.

Can configuration regression records be used to evaluate coding agents?

Yes, when the fix carries a test that sets the condition and fails before the fix. Those pinned records act as graders; records without a pin are better suited to diagnosis training, where the model must name the condition.

What should a supplier remove before sharing these records?

Credentials, internal hostnames and addresses, customer identifiers in settings, and any detail that would let an outsider reach the systems. The condition itself, such as the database version or the setting name, usually stays, because it is the point of the record.

Sources

  • SQLAlchemy's testing guide says a plain pytest run executes the full suite against SQLite, and other backends need a database of that vendor at a specific URL. Source
  • SQLAlchemy's 2.0 changelog, whose tagged entries for 2.0.1 to 2.0.55 supply the 74 regressions and their dialect, driver and runtime conditions. Source
  • SQLAlchemy's commit history at 5c2970e, from which the fix commits, test changes, pins and causing changes were counted. Source
  • Wikimedia's production MediaWiki configuration history at 26154d3, from which the 170 reverts of 2024-2025 and their labels were counted. Source
  • The configuration repository's composer.json defines a test script with typo checks, linting, JSON validation, PHPUnit, phpcs and phan. Source
  • The Twelve-Factor dev/prod parity factor advises against different backing services across environments, with SQLite versus MySQL as its example. Source
  • Google's report on the June 12, 2025 Google Cloud outage describes a Service Control code path that rollout never exercised until a policy change with blank fields triggered it. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies