Production SQL Query Corpora and Stored Procedures for AI Training

Why raw production SQL differs from benchmark text-to-SQL

Raw production SQL teaches what benchmarks rarely show: long, multi-step logic written against very wide, poorly named schemas. Spider 2.0 moved evaluation to 632 enterprise workflow problems on real application databases and found models far weaker there than on earlier academic sets [1]. BIRD made the same point from the data side, arguing that dirty cell values, domain knowledge and query efficiency are what separate real database work from toy schemas [3].

A raw corpus has no natural-language question attached, and that is the point. It supplies the distribution of CTE chains, window functions, MERGE statements, dialect quirks and vendor-specific functions that a completion or generation model must reproduce. If you need gold question-to-SQL pairs instead, see our guide to text-to-SQL training data from real enterprise schemas; this page covers the raw material those pairs are often built from.

What a production SQL corpus should contain

A complete corpus has six layers, and each one loses value if separated from the others. Ask for all of them in the request, even if a supplier can provide only some.

• Query history. Statement text with timestamps, statement type, dialect and engine version. In Snowflake this typically comes from the ACCOUNT_USAGE QUERY_HISTORY view; in BigQuery, from the INFORMATION_SCHEMA.JOBS view. Check each platform's current retention window, because older history may already be gone.
• Stored procedures, functions and views. Full PL/SQL, T-SQL, PL/pgSQL or Snowflake Scripting bodies, with view definitions that show how business logic is layered.
• Schema DDL with comments. CREATE TABLE statements, keys, constraints and column comments, versioned by date. Spider 2.0 shows why: enterprise tasks depend on understanding large schemas and their metadata, not only the query [1].
• Migration scripts. Flyway, Liquibase or Alembic histories that explain how the schema reached its current shape.
• Execution statistics. Runtime, rows produced, bytes scanned and error codes. In BigQuery, fields such as total_bytes_processed and referenced_tables carry this signal.
• Optional sample rows. Small, licensed extracts that expose messy values, under the same de-identification as any other record [3].

Schema documentation that explains metric definitions is a separate asset; see data dictionaries and schema docs as AI grounding data.

Fields to require on every query record

Every query record should carry enough metadata to rebuild its context: the schema version it ran against, how it was produced, and whether it succeeded. Without the schema link, a query referencing a dropped column becomes a silent negative example.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "query_id": "q_000184213",
  "engine": "snowflake",
  "dialect_version": "2026-08",
  "schema_snapshot_id": "ddl_2026-07-30_wh_finance",
  "statement_type": "SELECT",
  "query_text": "WITH open_inv AS (SELECT account_id, SUM(amount_usd) AS bal FROM ar.invoices WHERE status = :STR_1 AND customer_email = :EMAIL_1 GROUP BY 1) SELECT ...",
  "literal_masking": {"method": "typed_placeholder", "masked_count": 2},
  "parameterized_hash": "8c41f0...",
  "template_cluster_size": 1,
  "client_tag": "analyst_adhoc",
  "status": "SUCCESS",
  "error_code": null,
  "runtime_ms": 4120,
  "rows_produced": 318,
  "bytes_scanned": 912384000,
  "referenced_objects": ["ar.invoices", "crm.accounts"],
  "license_use_tags": ["pretraining", "completion", "eval"]
}

Keep failed statements with their error codes rather than dropping them. Pairs of a failed query and its corrected rerun from the same session are some of the most useful repair examples in a log.

Masking literals without breaking the SQL

Literals are where personal data hides in SQL, so mask them by type while keeping each statement parseable. Names, emails, phone numbers and account numbers appear in WHERE clauses, IN lists, INSERT VALUES and hard-coded CASE branches, and research on releasing search-engine query logs showed long ago that free text in logs can expose people unless it is deliberately protected [4].

A sound approach parses each statement with a dialect-aware parser such as sqlglot or the engine's own tooling, replaces string and numeric literals with typed placeholders (:EMAIL_1, :ACCT_1, :DATE_1), and then reparses to confirm the output is still valid. Also check identifiers: tables named after clients, schemas named after people and comments containing tickets or emails. Ask the supplier to record the masking method and a sampled reparse and leak-check rate, and treat any claim of perfect removal with caution.

Credentials are a separate failure mode. Connection strings, CREATE USER statements with passwords and API keys inside external function definitions turn up in procedure bodies; our guide on scanning code datasets for secrets covers verification.

Deduplicating templated and machine-generated queries

BI tools and orchestrators can dominate a query log, so deduplicate by parameterized shape before you measure the corpus. A dashboard refreshing every five minutes can emit thousands of near-identical statements that differ only in date literals, which skews training toward a narrow template.

Group statements by a hash computed after literal normalization (Snowflake exposes a parameterized query hash in its history; Postgres pg_stat_statements normalizes constants in a similar way), keep one or a few representatives per cluster, and record the cluster size as a weight. Tag the client source, such as Looker, Tableau, dbt or an ETL service account, so you can rebalance human-written ad hoc SQL against generated SQL. dbt models and Airflow-driven SQL are better licensed as code with their repositories; see data pipeline code datasets.

Using execution statistics as labels

Execution statistics turn a query log into supervised signal for efficiency, not only correctness. BIRD treats query efficiency as a first-class evaluation target alongside accuracy [3], and production logs carry exactly that signal at scale.

Runtime, bytes scanned and rows produced let you rank functionally similar queries, build rewrite pairs where an analyst replaced a slow query with a faster one, and filter out runaway statements. Normalize by warehouse size or slot reservation, because the same query can run much faster on a larger warehouse. Ask whether statistics come from the engine's system views or from a separate monitoring tool, and whether cache hits are flagged, since cached results report misleading costs.

From query logs to eval sets and synthesis seeds

Query logs make strong seeds for enterprise evaluation and synthetic data, provided each query keeps its link to the schema version it ran against. BenchPress describes using existing SQL logs as the starting point for human-in-the-loop text-to-SQL benchmark curation, with an LLM drafting candidate questions for each query and human annotators reviewing and refining them [2].

For held-out evals, freeze a schema snapshot, sample queries by template cluster rather than by row count, and keep the eval split out of anything used for pre-training. For synthesis, generate questions against real DDL and validate generated SQL by executing it against a sandbox that matches the snapshot. If you need cross-dialect pairs such as Oracle PL/SQL to Snowflake, use SQL dialect migration datasets instead.

Evaluating a sample before you license

Run a structured sample check before committing to a full corpus. The checklist below covers the failure modes that most often make a SQL corpus less useful than its row count suggests.

Illustrative example: invented to show structure; it does not describe an available dataset.

Delivery format matters for scale. Parquet keeps query text and statistics columnar with footer metadata for efficient reads [6], and a Snowflake share gives read-only access without copying data between accounts [5]. For a general method, see evaluating a code dataset sample before you license it.

Rights and licensing points specific to SQL

SQL corpora raise rights questions about the queries, the schemas and any row data, which can have different owners. Analysts' queries and procedures are usually company work product, but some logic may come from vendor packages, consultants or ERP add-ons with their own license terms, and schema names can reveal client relationships.

Confirm that the license names each layer (queries, procedures, DDL, statistics, sample rows), the permitted uses such as pre-training, completion fine-tuning and evaluation, and how derived synthetic data may be used. For the grant itself, see rights for foundation-model pre-training data. Repository-level code with full history is covered in proprietary code datasets, and finance warehouses often sit alongside the models described in spreadsheet and financial model datasets.

How SourceX sources production SQL data

SourceX sources operational datasets, including engineering records, from US companies on request rather than from stock, so a SQL corpus request does not guarantee a match. You describe the data you need, such as engine, dialects, layers and statistics, and SourceX looks for US businesses that hold it; every release is approved by the supplying company. You can start that description on the SourceX buyers page.

Each dataset is rights-reviewed for ownership and consents, personal details such as names, emails and account numbers are removed or replaced before delivery with the method recorded and a sample checked, and delivery runs through private, access-controlled workflows only after an executed agreement. SourceX does not train models and does not source scraped web content. More code data types are mapped in the code and software engineering data hub and the wider AI data guides.

Request a production SQL query corpus

If your team needs real query history, stored procedures and schema DDL for SQL generation, completion or eval, describe the engines, layers and uses you need. SourceX assesses data and licensing permissions with suppliers, and nothing is contracted until a supplier agrees. Tell SourceX what SQL data you need.