Schema Linking and Schema Retrieval Data for Wide Enterprise Warehouses

Why wide warehouses need a separate retrieval stage

Schema retrieval exists because an enterprise schema no longer fits in the prompt. Spider 2.0 reports that its enterprise databases often exceed 1,000 columns [3], and its 632 problems come from real applications rather than toy databases [4]. Once DDL for every table in a Snowflake, BigQuery or Databricks catalog is serialized with types and comments, it can run to hundreds of thousands of tokens, so the pipeline has to select a subset first.

That selection step is a ranking problem with its own failure modes. If the retriever drops fct_orders.net_revenue_usd, the SQL generator cannot recover it, however strong the model. If it returns 200 candidate columns, the generator is back to reading noise. Training and evaluating that retriever needs data labeled for retrieval, not just question-SQL pairs; for the full pairs themselves, see text-to-SQL training data from real enterprise schemas.

What a schema linking label actually contains

A usable label marks, for each question, which schema elements are relevant and at what grain. Snowflake's SQL-Schema-Retrieval dataset reframes Spider and BIRD this way, deriving relevant tables and columns from those referenced in each gold query [1], and the framing has already been mirrored and reused [2]. That gives four label layers you can request or build:

• Table relevance: binary label per (question, table), including tables used only in joins.
• Column relevance: label per (question, column), split by role: projected, filtered, grouped, joined or aggregated.
• Join path: the foreign-key or inferred key chain connecting the relevant tables, which matters when the catalog has no declared constraints.
• Value grounding: the literal that a filter matches, such as status = 'CLOSED_WON', which BIRD treats as a core difficulty of real databases [6].

Gold-SQL parsing has a known blind spot. It labels what one correct query used, not every acceptable alternative, so dim_customer.region may be marked negative when dim_account.region would answer equally well. Ask for multi-reference labels or an "acceptable alternative" flag on at least the evaluation split.

Illustrative label record

The record below shows the minimum structure for one training example at column grain.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "question_id": "q_000418",
  "question": "Net revenue by sales region for closed deals last quarter",
  "dialect": "snowflake",
  "schema_snapshot": "catalog_2026-06-30",
  "gold_sql_hash": "sha256:9c1e...",
  "relevant_tables": ["analytics.fct_opportunity", "analytics.dim_account"],
  "relevant_columns": [
    {"col": "fct_opportunity.net_amount_usd", "role": "aggregate"},
    {"col": "fct_opportunity.stage_name", "role": "filter", "value": "Closed Won"},
    {"col": "fct_opportunity.close_date", "role": "filter"},
    {"col": "dim_account.sales_region", "role": "group_by"},
    {"col": "fct_opportunity.account_id", "role": "join"}
  ],
  "hard_negatives": [
    {"col": "stg_sfdc__opportunity.amount", "reason": "staging_copy"},
    {"col": "fct_opportunity.amount_legacy", "reason": "deprecated"},
    {"col": "dim_account_v2.region", "reason": "near_duplicate_table"}
  ],
  "acceptable_alternatives": ["dim_account.territory_region"],
  "label_source": "query_log_plus_analyst_review"
}

The reason field on negatives is what turns a list of wrong answers into a diagnostic. It lets you report recall separately for "beat the staging copy" and "beat the deprecated column" instead of one blended number.

Hard negatives: real warehouse clutter versus synthetic distractors

The negatives that teach a schema retriever the most are the ones real warehouses accumulate on their own. A typical dbt-managed warehouse holds stg_, int_ and fct_ versions of the same entity, columns renamed after a migration but never dropped, _bak and _v2 tables, and vendor-replicated schemas from Fivetran or Airbyte with source-system names. These look lexically and semantically close to the right answer, which is why embedding retrievers confuse them.

Synthetic pipelines approximate this by injecting distractor tables and columns. NVIDIA's NeMo Data Designer text-to-SQL pipeline does so explicitly, so models learn to ignore irrelevant schema elements, and validates output per dialect [5]. That is useful for volume, but injected distractors follow the generator's naming habits; they rarely reproduce a ten-year-old column called rev2 that finance still trusts, or a view that silently filters out test accounts.

Mined negatives carry a second risk: some of them are actually correct. Dense-retrieval research found that many top-ranked "negatives" were unlabeled positives and used a cross-encoder to filter them before training [7]. The schema equivalent is a near-duplicate table that genuinely answers the question. Budget a review pass, or the acceptable-alternatives field above, before you treat mined columns as negatives. The retrieval cluster covers the document version of this problem in superseded versions, drafts and near-duplicates for retrieval testing.

Metrics that predict end-to-end text-to-SQL accuracy

Measure schema retrieval at two levels: recall of the gold elements, then execution accuracy when the generator only sees what was retrieved. Retrieval metrics alone can mislead, because a retriever with high table recall can still drop the one join key that makes the query run.

Report the gap between execution accuracy with the oracle schema and with the retrieved schema. That gap is the retriever's cost to the pipeline, and it is what justifies paying for better labels.

Description-aware retrievers and grounding data

Column names alone are a weak signal in wide warehouses, so the strongest retrievers index names together with descriptions, metric definitions and sample values. A column called amt_3 is unretrievable by meaning until something says it is net revenue after refunds in USD. Pair relevance labels with data dictionaries, schema docs and metric definitions when you train, and record which descriptions were available at labeling time so you can test the retriever without them.

This also shapes how you build the index. A bi-encoder over embeddings of column descriptions handles first-stage recall; a cross-encoder or LLM reranker over the top candidates handles near-duplicates. Each stage needs labels at its own grain: table-level for the first pass, column-level with role tags for the reranker.

Specification checklist for buying schema linking data

Specify schema shape, label grain and negatives before discussing volume; a large set of questions over a 40-table schema will not train a retriever for a 4,000-table catalog.

• Schema scale: table and column counts per database, and whether multiple schemas or databases are in scope per question.
• Schema snapshot: full DDL with types, comments and declared keys, versioned to the date labels were made.
• Label source: gold SQL from analysts, parsed query logs, or both; state how log queries were filtered for correctness.
• Label grain: table, column, role tag, join path and value grounding, as listed above.
• Negatives: mined real clutter with reason codes, plus any synthetic distractors flagged as synthetic.
• Alternatives: multi-reference or acceptable-alternative labels on the evaluation split.
• Dialect: Snowflake, BigQuery, Postgres, Databricks SQL or T-SQL, since quoting and function names affect parsing.
• Splits: held-out databases, not just held-out questions, to measure transfer to unseen schemas.
• Masking: sample values and literals with personal or account data removed or replaced, with the method documented.
• License and provenance: a written license covering the schemas, queries and values, since public dataset licenses are frequently missing or wrong [8].

Grain, keys and history requirements for the underlying tables are covered in what to specify when licensing tabular data. If you need a held-out benchmark rather than training labels, see private text-to-SQL evaluation sets on enterprise databases.

Where real enterprise schemas and query logs come from

The scarce ingredient is not questions but real, wide schemas with the history of how analysts actually queried them. Companies that run BI tools on top of a warehouse hold this in query history, dbt projects, LookML or semantic-layer definitions, and dashboard SQL. Turning that into licensed training data requires the company's approval, review of what the schema and values reveal, and removal of personal details from sample values before anything leaves its environment.

SourceX sources operational datasets from US companies on request and manages the licensing, including ongoing purchases, for AI teams wherever they are based. Datasets are not held in stock, so a request describes the data needed and does not guarantee a match; every release is approved by the supplying company. Each dataset is rights-reviewed for ownership and consents and delivered under a license that defines records, uses, term and delivery. You can describe a schema linking requirement on the SourceX buyers page. For context on the wider category, start at the tabular, time-series and transactional data hub or the AI data guides.

Request schema linking data from real enterprise warehouses

SourceX works through Find, Assess, Agree, Transact and Manage, and nothing is contracted until a supplier agrees. Names, emails, phone numbers and account numbers are removed or replaced before delivery, with the method recorded and a sample checked, and delivery runs through private, access-controlled workflows after an executed agreement. Describe the schema scale, label grain and negatives you need at sourcex.si/buyers.