Skip to content

AI uses for records

RAG table retrieval training data: linking each row to its column definitions

By SourceX Editorial · Updated

9 sections

Short answer

Link a retrieved table row to its column definitions by giving each row a table ID, version and key, and each column a pointer to a versioned definition with unit, type, missing-value markers and effective dates. Each question's label then lists the rows and definitions required, and a retriever passes only when it returns both, with definitions in force for the row's period.

Retrieval

RAG table retrieval training

This page covers RAG table retrieval training data: linking each row to its column definitions.Illustrative

Key takeaways

  • A row gives the value; only the column definition gives its unit, denominator and rule.
  • Label each question with the rows and the definitions required, not rows alone.
  • Check that each definition was in effect for the period the row describes.
  • Locate rows by table version and primary key; use CSV fragments only as a convenience.
  • Plant hard negatives such as sibling columns, older table versions and superseded definitions.
  • Score row recall, definition recall and joint evidence success separately.

Give every row its own evidence unit with a stable table ID, a table version and a row key, and give every column the answer uses a pointer to a definition object: name, label, description, unit, data type, missing-value markers and the definition's own version and effective dates. A retrieval label for a question then lists both the rows and the definitions required. A system passes only if it retrieves both and the definition was in force for the period the row describes.

This matters for RAG table retrieval training data because a row without its definitions is ambiguous in ways that a reader cannot see. A cell holding 4.1 under a header called RR can be a return rate as a share of units, a share of orders, or a count in thousands. The header row is often cut off by chunking, and the meaning of a column can change between dictionary versions while the header stays the same.

The rest of this page specifies the schema, shows a valid and a failing record, lists the validator checks, describes the hard negatives to plant, and sets out how to score retrieval. The examples are illustrative and are not drawn from any company's data.

Why is the right row still not enough evidence?

A row tells you the value; only the definition tells you what the value means, in which unit, and under which rule. Questions that business users ask of tables nearly always depend on both. What was the return rate in the West region in the second quarter needs the row for West and Q2, and the definition that says how the rate was computed during that quarter.

Research on table question answering points the same way. OTT-QA, an open question answering benchmark over tables and text, contains questions that require retrieving and aggregating information from both tables and passages, and its authors report that a baseline with an iterative retriever and a BERT-based reader scored below 10% exact match. Work on dense table retrieval found that a retriever designed to handle tabular context can improve retrieval over text-oriented retrievers. Neither result is about column definitions specifically, but both show that the evidence for a table answer is often more than the table cell itself.

Three things break the link in practice. Extraction and chunking separate the header row from the values, so a chunk holds numbers with no labels. Definitions drift: a finance team changes how a rate is computed at the start of a fiscal year and keeps the column name. And restatements rewrite past rows, so the same key returns different values depending on which snapshot was indexed. A dataset that records only the gold row cannot tell these failures apart, because in each case the retriever found a row that looks right.

Where do column definitions live in business systems?

Definitions are rarely stored next to the rows they explain. In practice they sit in a separate file or system, which is exactly why a retrieval dataset has to link the two explicitly.

  • CSV on the Web describes columns with properties such as titles, datatype and null, and its examples attach a dc:description to a column; its primaryKey property names the columns that identify each row.
  • Frictionless Table Schema gives each field a name, an optional title and description, and lets missingValues declare which strings count as missing; a field-level list replaces the schema-level one.
  • dbt lets teams write a description for each column of a model in a properties YAML file, which then appears in generated documentation.
Where do column definitions live in business systems?
Where the definition livesWhat it usually holdsHow to link it to rowsCommon risk
CSV on the Web metadata fileColumn name, titles, datatype, a dc:description, null markers, primary keyTable URL plus column nameMetadata file not shipped with the extract
Frictionless Table SchemaField name, title, description, type, constraints, missingValues, primaryKeyResource path plus field nameField-level missing values override schema-level ones unnoticed
dbt model propertiesModel and column descriptions in YAML next to the transformation codeModel name plus column nameDescriptions edited without a version record
Data dictionary spreadsheetLabels, business rules, units, ownersDictionary tab and row per columnSeveral copies with different edits
BI semantic layer or report notesMetric formulas and filtersMetric ID used by the reportFormula differs from the warehouse column
Footnotes in the document itselfUnits, restatements, scope notesFootnote marker in the header or cellLost when the table is extracted from a PDF

Which fields does one table evidence record need?

This is the asset: the minimum fields for one table evidence record. Each record joins a question to the rows and column definitions that together support the answer. Example values are illustrative.

  • Use RFC 7111 fragment identifiers where the source is a CSV file. They let a URI select a row, a column or a cell, for example row=14 or cell=14,3, so a citation can point at an exact location instead of a whole file.
  • Keep the row key as the primary locator and the fragment as a convenience. Row positions shift when a file is re-exported; key values do not.
Which fields does one table evidence record need?
FieldPurposeExample value (illustrative)Validator rule
table_id and table_versionIdentify the exact table snapshotreturns_by_region, v2026-07-01Version resolves to a stored snapshot
row_locatorPoint at the row by key and by positionregion=West, quarter=2026-Q2; CSV fragment row=14Key is unique under the declared primary key
columns_usedColumns the answer depends onrr_units, units_shippedEvery column exists in the snapshot
definition_refsPointer to each column's definition objectdict v7: rr_units, units_shippedOne definition per column used
definition_effectivePeriod each definition applies torr_units v7 effective 2026-01-01 onwardCovers the period the row describes
unit, datatype and missing markersHow to read the cellPercent of units, decimal, NA means not reportedCell value parses under the declared type
access_scopeWho may see the row and the definitionRegional ops managers and financeScope is the same or narrower than the asker's
question and as_of dateThe query and the time it is asked aboutReturn rate West Q2 2026, as of 2026-10-18as_of falls within data coverage
required_evidence_setRows plus definitions needed for a correct answerRow 14 plus rr_units v7Non-empty, includes at least one definition
answer, citations and abstain flagGold answer, cited spans, and whether to abstain4.1% of units shipped; cites row 14 and rr_units v7Each cited span is in the evidence set

How does a valid record differ from a failing one (illustrative)?

The pair below shows the same question with one record that passes every check and one that fails. Both are constructed for this article.

At a glance

How does a valid record differ from a failing one (illustrative)

  • Valid record: question asks for the West region return rate in Q2 2026. Evidence set holds row 14 of returns_by_region v2026-07-01 (region West, quarter 2026-Q2, rr_units 4.1) and the definition rr_units v7 from the data dictionary, which says return rate as a percent of units shipped, effective from January 1, 2026. Gold answer: 4.1% of units shipped. Citations: row 14 and rr_units v7. Access scope: regional ops managers, matching the asker.
  • Failing record: same question and row, but definition_refs points to rr_units v6, which defined the rate as a percent of orders and stopped applying on December 31, 2025. The validator fails it on definition_effective, because the row describes Q2 2026. A model trained on this record learns to explain the number with the wrong denominator.
  • Second failure mode: the record lists the row but no definition. The validator fails it on required_evidence_set, because the column name rr_units alone does not state a unit. If no definition exists in any source, the correct gold label is an answer with a stated assumption or an abstention, not a bare number.

Which checks should a validator run on every record?

Run the checks below before any record enters a training or evaluation split. Each one is cheap and catches a failure that is invisible once the record is flattened into a prompt.

  • Locator resolves: the table version exists and the row key returns exactly one row.
  • Key integrity: the declared primary key is unique in the snapshot, since a duplicate key makes the locator ambiguous.
  • Definition coverage: every entry in columns_used has a definition reference, and every reference resolves.
  • Temporal fit: each definition was in effect for the period the row describes, not just on the date the dataset was built.
  • Type and unit fit: the cell parses under the declared datatype, and the gold answer states the unit the definition gives.
  • Missing values: a cell matching a declared missing marker is treated as missing, and the gold answer abstains or says so rather than reading the marker as a value.
  • Access fit: the evidence's access scope is no broader than the scope of the user the question is asked for.
  • Citation support: every cited span appears in the required evidence set, and nothing in the gold answer depends on evidence outside it.

Which hard negatives look like the right row or the right definition?

A table retriever trained only on easy negatives learns to match region names and quarter labels. Plant negatives that share those surface features and differ on the thing that decides the answer.

Which hard negatives look like the right row or the right definition?
Hard negativeWhat it shares with the gold evidenceWhat differsCorrect behavior
Previous table versionSame row key and columnsValues before a restatementPrefer the version in force for the as_of date
Sibling columnSimilar name, such as rr_orders next to rr_unitsDifferent denominatorRetrieve the column the definition names
Superseded definitionSame column nameOlder formula and effective datesUse the definition effective for the row's period
Same key, other tableRegion and quarter labelsDifferent business unit's returnsMatch on table identity, not labels alone
Out-of-scope rowCorrect valuesAsker lacks accessWithhold and say access is required

How should retrieval be scored when rows, definitions and access all matter?

Count a question as correctly retrieved only if the retrieved set contains the gold rows and the definitions in force for the row's period, and the asker is allowed to see both. Score those conditions separately so you can tell a retriever that misses definitions from one that finds stale ones.

  • Row recall: share of required rows retrieved.
  • Definition recall: share of required definitions retrieved, counting a superseded definition as a miss.
  • Joint evidence success: the share of questions where every required row and definition was retrieved. This is the number that predicts answer quality.
  • Citation support: share of cited spans that are in the required evidence set.
  • Abstention accuracy: on questions with no valid definition or no access, the share where the system abstains or states its assumption.
  • Break results out by failure class: missing definition, superseded definition, wrong table version and access mismatch. A retriever with a good average can still fail every question that depends on a changed definition, and that is the class business users notice first.
  • Report each metric with the number of questions behind it. No results are reported here; these are the measures the schema makes possible.

How do you request tables with their data dictionaries attached?

Public table benchmarks are useful for method, but they rarely ship the versioned data dictionaries, restatements and access rules that enterprise tables carry. Exports from real business systems, with their dictionaries, model descriptions and change history kept alongside the rows, are what let a team build records that pass the checks above.

SourceX does not hold warehouse tables in reserve; it finds them at U.S. businesses after a team files its request-data form, which asks for scope, volume, language, format, timing and licensing needs, so that form is where to say that dictionaries and their version history must travel with the rows. The supplying company approves the scope and terms and signs before anything ships, which makes the original request the place to fix what the delivery includes. Access rules matter too: if the source tables carry row-level permissions, ask for the permission scope to be described per table so that the access_scope field can be filled from evidence rather than guessed.

Frequently asked questions

What is RAG table retrieval training data?

It is a set of questions paired with the table rows and supporting context that answer them, used to train or evaluate a retriever. For business tables the supporting context should include column definitions, units and versions, not only the matching rows.

Why does a retrieved row need its column definition?

Column names are often abbreviations, and the same name can mean different things across dictionary versions. Without the definition, a model can quote the correct number with the wrong unit or denominator.

How should a row be referenced in a citation?

Use the table ID, its version and the row's primary key values. For CSV sources, an RFC 7111 fragment such as row=14 can be added as a convenience, but positions shift on re-export, so the key should stay the primary locator.

What should happen when no definition exists for a column?

The gold label should abstain or answer with an explicit assumption. Training records that answer confidently without a definition teach the model to guess units.

Which standard formats can hold column definitions?

CSV on the Web metadata and Frictionless Table Schema both describe columns with names, titles, descriptions, types and missing-value markers. Teams using dbt often already keep column descriptions in model property files.

Sources

  • The W3C CSV on the Web metadata vocabulary describes columns with titles, datatype and null properties, uses dc:description on columns in its examples, and defines primaryKey for identifying rows. Source
  • Frictionless Table Schema field descriptors carry a name, title and description; missingValues declares missing-value strings, and a field-level list replaces the schema-level one. Source
  • RFC 7111 defines URI fragment identifiers for text/csv that select rows, columns or cells, such as row=4 or cell=4,1. Source
  • dbt model properties accept a description for each column, written in YAML alongside the model. Source
  • OTT-QA contains questions requiring retrieval and aggregation over tables and text, and its baseline with an iterative retriever and BERT-based reader scored below 10% exact match. Source
  • Herzig and colleagues show that open-domain question answering over tables can be improved by a retriever designed to handle tabular context. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies