AI uses for records
RAG table retrieval training data: linking each row to its column definitions
By SourceX Editorial · Updated
9 sections
Short answer
Link a retrieved table row to its column definitions by giving each row a table ID, version and key, and each column a pointer to a versioned definition with unit, type, missing-value markers and effective dates. Each question's label then lists the rows and definitions required, and a retriever passes only when it returns both, with definitions in force for the row's period.
Retrieval
RAG table retrieval training
Key takeaways
- A row gives the value; only the column definition gives its unit, denominator and rule.
- Label each question with the rows and the definitions required, not rows alone.
- Check that each definition was in effect for the period the row describes.
- Locate rows by table version and primary key; use CSV fragments only as a convenience.
- Plant hard negatives such as sibling columns, older table versions and superseded definitions.
- Score row recall, definition recall and joint evidence success separately.
How do you link a retrieved table row to its column definitions?
Give every row its own evidence unit with a stable table ID, a table version and a row key, and give every column the answer uses a pointer to a definition object: name, label, description, unit, data type, missing-value markers and the definition's own version and effective dates. A retrieval label for a question then lists both the rows and the definitions required. A system passes only if it retrieves both and the definition was in force for the period the row describes.
This matters for RAG table retrieval training data because a row without its definitions is ambiguous in ways that a reader cannot see. A cell holding 4.1 under a header called RR can be a return rate as a share of units, a share of orders, or a count in thousands. The header row is often cut off by chunking, and the meaning of a column can change between dictionary versions while the header stays the same.
The rest of this page specifies the schema, shows a valid and a failing record, lists the validator checks, describes the hard negatives to plant, and sets out how to score retrieval. The examples are illustrative and are not drawn from any company's data.
Why is the right row still not enough evidence?
A row tells you the value; only the definition tells you what the value means, in which unit, and under which rule. Questions that business users ask of tables nearly always depend on both. What was the return rate in the West region in the second quarter needs the row for West and Q2, and the definition that says how the rate was computed during that quarter.
Research on table question answering points the same way. OTT-QA, an open question answering benchmark over tables and text, contains questions that require retrieving and aggregating information from both tables and passages, and its authors report that a baseline with an iterative retriever and a BERT-based reader scored below 10% exact match. Work on dense table retrieval found that a retriever designed to handle tabular context can improve retrieval over text-oriented retrievers. Neither result is about column definitions specifically, but both show that the evidence for a table answer is often more than the table cell itself.
Three things break the link in practice. Extraction and chunking separate the header row from the values, so a chunk holds numbers with no labels. Definitions drift: a finance team changes how a rate is computed at the start of a fiscal year and keeps the column name. And restatements rewrite past rows, so the same key returns different values depending on which snapshot was indexed. A dataset that records only the gold row cannot tell these failures apart, because in each case the retriever found a row that looks right.
Where do column definitions live in business systems?
Definitions are rarely stored next to the rows they explain. In practice they sit in a separate file or system, which is exactly why a retrieval dataset has to link the two explicitly.
- CSV on the Web describes columns with properties such as titles, datatype and null, and its examples attach a dc:description to a column; its primaryKey property names the columns that identify each row.
- Frictionless Table Schema gives each field a name, an optional title and description, and lets missingValues declare which strings count as missing; a field-level list replaces the schema-level one.
- dbt lets teams write a description for each column of a model in a properties YAML file, which then appears in generated documentation.
| Where the definition lives | What it usually holds | How to link it to rows | Common risk |
|---|---|---|---|
| CSV on the Web metadata file | Column name, titles, datatype, a dc:description, null markers, primary key | Table URL plus column name | Metadata file not shipped with the extract |
| Frictionless Table Schema | Field name, title, description, type, constraints, missingValues, primaryKey | Resource path plus field name | Field-level missing values override schema-level ones unnoticed |
| dbt model properties | Model and column descriptions in YAML next to the transformation code | Model name plus column name | Descriptions edited without a version record |
| Data dictionary spreadsheet | Labels, business rules, units, owners | Dictionary tab and row per column | Several copies with different edits |
| BI semantic layer or report notes | Metric formulas and filters | Metric ID used by the report | Formula differs from the warehouse column |
| Footnotes in the document itself | Units, restatements, scope notes | Footnote marker in the header or cell | Lost when the table is extracted from a PDF |
Which fields does one table evidence record need?
This is the asset: the minimum fields for one table evidence record. Each record joins a question to the rows and column definitions that together support the answer. Example values are illustrative.
- Use RFC 7111 fragment identifiers where the source is a CSV file. They let a URI select a row, a column or a cell, for example row=14 or cell=14,3, so a citation can point at an exact location instead of a whole file.
- Keep the row key as the primary locator and the fragment as a convenience. Row positions shift when a file is re-exported; key values do not.
| Field | Purpose | Example value (illustrative) | Validator rule |
|---|---|---|---|
| table_id and table_version | Identify the exact table snapshot | returns_by_region, v2026-07-01 | Version resolves to a stored snapshot |
| row_locator | Point at the row by key and by position | region=West, quarter=2026-Q2; CSV fragment row=14 | Key is unique under the declared primary key |
| columns_used | Columns the answer depends on | rr_units, units_shipped | Every column exists in the snapshot |
| definition_refs | Pointer to each column's definition object | dict v7: rr_units, units_shipped | One definition per column used |
| definition_effective | Period each definition applies to | rr_units v7 effective 2026-01-01 onward | Covers the period the row describes |
| unit, datatype and missing markers | How to read the cell | Percent of units, decimal, NA means not reported | Cell value parses under the declared type |
| access_scope | Who may see the row and the definition | Regional ops managers and finance | Scope is the same or narrower than the asker's |
| question and as_of date | The query and the time it is asked about | Return rate West Q2 2026, as of 2026-10-18 | as_of falls within data coverage |
| required_evidence_set | Rows plus definitions needed for a correct answer | Row 14 plus rr_units v7 | Non-empty, includes at least one definition |
| answer, citations and abstain flag | Gold answer, cited spans, and whether to abstain | 4.1% of units shipped; cites row 14 and rr_units v7 | Each cited span is in the evidence set |
How does a valid record differ from a failing one (illustrative)?
The pair below shows the same question with one record that passes every check and one that fails. Both are constructed for this article.
At a glance
How does a valid record differ from a failing one (illustrative)
- Valid record: question asks for the West region return rate in Q2 2026. Evidence set holds row 14 of returns_by_region v2026-07-01 (region West, quarter 2026-Q2, rr_units 4.1) and the definition rr_units v7 from the data dictionary, which says return rate as a percent of units shipped, effective from January 1, 2026. Gold answer: 4.1% of units shipped. Citations: row 14 and rr_units v7. Access scope: regional ops managers, matching the asker.
- Failing record: same question and row, but definition_refs points to rr_units v6, which defined the rate as a percent of orders and stopped applying on December 31, 2025. The validator fails it on definition_effective, because the row describes Q2 2026. A model trained on this record learns to explain the number with the wrong denominator.
- Second failure mode: the record lists the row but no definition. The validator fails it on required_evidence_set, because the column name rr_units alone does not state a unit. If no definition exists in any source, the correct gold label is an answer with a stated assumption or an abstention, not a bare number.
Which checks should a validator run on every record?
Run the checks below before any record enters a training or evaluation split. Each one is cheap and catches a failure that is invisible once the record is flattened into a prompt.
- Locator resolves: the table version exists and the row key returns exactly one row.
- Key integrity: the declared primary key is unique in the snapshot, since a duplicate key makes the locator ambiguous.
- Definition coverage: every entry in columns_used has a definition reference, and every reference resolves.
- Temporal fit: each definition was in effect for the period the row describes, not just on the date the dataset was built.
- Type and unit fit: the cell parses under the declared datatype, and the gold answer states the unit the definition gives.
- Missing values: a cell matching a declared missing marker is treated as missing, and the gold answer abstains or says so rather than reading the marker as a value.
- Access fit: the evidence's access scope is no broader than the scope of the user the question is asked for.
- Citation support: every cited span appears in the required evidence set, and nothing in the gold answer depends on evidence outside it.
Which hard negatives look like the right row or the right definition?
A table retriever trained only on easy negatives learns to match region names and quarter labels. Plant negatives that share those surface features and differ on the thing that decides the answer.
| Hard negative | What it shares with the gold evidence | What differs | Correct behavior |
|---|---|---|---|
| Previous table version | Same row key and columns | Values before a restatement | Prefer the version in force for the as_of date |
| Sibling column | Similar name, such as rr_orders next to rr_units | Different denominator | Retrieve the column the definition names |
| Superseded definition | Same column name | Older formula and effective dates | Use the definition effective for the row's period |
| Same key, other table | Region and quarter labels | Different business unit's returns | Match on table identity, not labels alone |
| Out-of-scope row | Correct values | Asker lacks access | Withhold and say access is required |
How should retrieval be scored when rows, definitions and access all matter?
Count a question as correctly retrieved only if the retrieved set contains the gold rows and the definitions in force for the row's period, and the asker is allowed to see both. Score those conditions separately so you can tell a retriever that misses definitions from one that finds stale ones.
- Row recall: share of required rows retrieved.
- Definition recall: share of required definitions retrieved, counting a superseded definition as a miss.
- Joint evidence success: the share of questions where every required row and definition was retrieved. This is the number that predicts answer quality.
- Citation support: share of cited spans that are in the required evidence set.
- Abstention accuracy: on questions with no valid definition or no access, the share where the system abstains or states its assumption.
- Break results out by failure class: missing definition, superseded definition, wrong table version and access mismatch. A retriever with a good average can still fail every question that depends on a changed definition, and that is the class business users notice first.
- Report each metric with the number of questions behind it. No results are reported here; these are the measures the schema makes possible.
How do you request tables with their data dictionaries attached?
Public table benchmarks are useful for method, but they rarely ship the versioned data dictionaries, restatements and access rules that enterprise tables carry. Exports from real business systems, with their dictionaries, model descriptions and change history kept alongside the rows, are what let a team build records that pass the checks above.
SourceX does not hold warehouse tables in reserve; it finds them at U.S. businesses after a team files its request-data form, which asks for scope, volume, language, format, timing and licensing needs, so that form is where to say that dictionaries and their version history must travel with the rows. The supplying company approves the scope and terms and signs before anything ships, which makes the original request the place to fix what the delivery includes. Access rules matter too: if the source tables carry row-level permissions, ask for the permission scope to be described per table so that the access_scope field can be filled from evidence rather than guessed.
Frequently asked questions
What is RAG table retrieval training data?
It is a set of questions paired with the table rows and supporting context that answer them, used to train or evaluate a retriever. For business tables the supporting context should include column definitions, units and versions, not only the matching rows.
Why does a retrieved row need its column definition?
Column names are often abbreviations, and the same name can mean different things across dictionary versions. Without the definition, a model can quote the correct number with the wrong unit or denominator.
How should a row be referenced in a citation?
Use the table ID, its version and the row's primary key values. For CSV sources, an RFC 7111 fragment such as row=14 can be added as a convenience, but positions shift on re-export, so the key should stay the primary locator.
What should happen when no definition exists for a column?
The gold label should abstain or answer with an explicit assumption. Training records that answer confidently without a definition teach the model to guess units.
Which standard formats can hold column definitions?
CSV on the Web metadata and Frictionless Table Schema both describe columns with names, titles, descriptions, types and missing-value markers. Teams using dbt often already keep column descriptions in model property files.
Sources
- The W3C CSV on the Web metadata vocabulary describes columns with titles, datatype and null properties, uses dc:description on columns in its examples, and defines primaryKey for identifying rows. Source
- Frictionless Table Schema field descriptors carry a name, title and description; missingValues declares missing-value strings, and a field-level list replaces the schema-level one. Source
- RFC 7111 defines URI fragment identifiers for text/csv that select rows, columns or cells, such as row=4 or cell=4,1. Source
- dbt model properties accept a description for each column, written in YAML alongside the model. Source
- OTT-QA contains questions requiring retrieval and aggregation over tables and text, and its baseline with an iterative retriever and BERT-based reader scored below 10% exact match. Source
- Herzig and colleagues show that open-domain question answering over tables can be improved by a retriever designed to handle tabular context. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.