Table Grain, Keys and History: What to Specify When Licensing Tabular Data

This page covers the specification elements that only tables have. For the general structure of a request (use case, rights, budget, delivery), start with the guide to writing a data request for suppliers, and see the structured data buyer's guide for the wider cluster.

Why tabular requests fail on arrival

Many failed tabular deliveries are not wrong data but under-specified data: the extract matches what was asked and still cannot be joined, aggregated or split in time. Typical failure modes are an orders table that turns out to be one row per order line, customer IDs that were re-pseudonymized per table so joins break, a status column that only holds today's value, and amounts with no currency column. Each of these comes from a question the request never asked.

Text-to-SQL research makes the same point from the model side. The BIRD benchmark was built on large, real databases precisely because correct answers depend on understanding stored values, codes and external business knowledge, not only table and column names [1]. If your request does not ask for that value-level context, you will receive a schema your model cannot interpret.

State the grain of every table first

The grain is the single sentence that says what one row represents, and it should appear in the request before any field list. "One row per invoice line, per revision" and "one row per invoice, latest state" are different datasets with different uses, row counts and leakage risks.

Write the grain together with its uniqueness rule. For example: invoice_line is unique on (invoice_id, line_no); account_daily_balance is unique on (account_id, as_of_date). Ask the supplier to confirm the grain in writing and to report any duplicate-key rows they find in the source rather than silently dropping them. A grain mismatch discovered after delivery usually means a re-extract, not a fix you can apply yourself.

Keys, joins and what de-identification does to them

Primary and foreign keys are what make a multi-table extract more than a pile of CSVs, so list them explicitly and set an expected join coverage. Relational deep learning benchmarks such as RelBench treat the database as a graph built from primary-foreign key links and row timestamps, which means broken or missing keys remove the signal those models learn from [2].

For each foreign key, state the parent table and an acceptable orphan rate, for example "fewer than 0.5% of ticket.customer_id values without a matching customer row." Ask the supplier to report the orphan rate before and after de-identification. If identifiers are pseudonymized, the same input value must map to the same token in every table and every delivery, or joins and incremental refreshes will break; the pseudonymization method itself is a separate decision covered under privacy pages. Use relational database datasets for machine learning and sampling relational extracts without breaking keys for the downstream implications.

Snapshot versus change history

Decide whether you need the state of each entity at one moment or the full sequence of changes, because suppliers cannot reconstruct history their systems overwrote. A current snapshot is cheap to extract but can leak future information into a model that predicts an earlier event, while change history supports point-in-time training sets and temporal splits.

Specify the history form you will accept. Options include slowly changing dimension Type 2 rows with valid_from, valid_to and is_current; an append-only change log with op (insert, update, delete) and changed_at; periodic snapshots with a snapshot_date; or an event log. For process data that spans several objects (orders, deliveries, invoices), the OCEL 2.0 standard defines an object-centric event log metamodel with relational SQLite, XML and JSON exchange formats [6], which gives you a precise vocabulary to request. Always ask for created_at and updated_at on every mutable table, and see business event sequence data and the target leakage audit for why timestamps matter.

Time coverage, timezones and schema drift

Time coverage should be stated as a closed window plus the field that defines it, such as "orders with created_at between 2021-01-01 and 2025-12-31 UTC, including later updates through the extract date." Without the defining field, one supplier filters on creation and another on last update, and the two extracts are not comparable.

Require that every timestamp carry an explicit offset or be documented as UTC or local, with the source timezone named. Ask how the supplier handles schema changes inside the window: renamed columns, retired codes and new mandatory fields. Table formats such as Apache Iceberg track each column by a unique ID so renames and drops do not depend on names or positions [4]; whatever format you receive, ask for a change log of schema events with effective dates.

Units, currencies, null semantics and code tables

Every measured or monetary column needs a declared unit, and every amount needs either a currency column or a documented fixed currency. Ask whether amounts are gross or net of tax and discounts, whether they are stored in minor units (cents), and how refunds and reversals are signed.

Null semantics deserve their own line in the spec. Ask the supplier to distinguish "not applicable," "unknown," "not collected before date X," and a true zero, and to document any sentinel values such as -1, 9999-12-31 or empty strings. Code columns (status_cd, reason_code, gl_account) are useless without their lookup tables, so request the code tables with descriptions and validity dates, plus a data dictionary with metric definitions. BIRD's emphasis on value meaning is the practical argument: a model that sees RSN_07 cannot learn what it means [1].

Machine-readable schema and acceptance checks

Ask for the schema in a machine-readable form so you can validate the delivery automatically rather than by reading a PDF. For JSON or JSONL exports, a JSON Schema document (as of October 2026, draft 2020-12 is the current release, with validation keywords defined in its Validation specification) lets you check required fields, types and enumerations on arrival [5]; for Parquet or warehouse shares, the embedded schema plus a dictionary file serves the same role.

Write the acceptance checks into the request: row counts per table and per month, key uniqueness, orphan rates, null rates per required field, value ranges and code coverage against the lookup tables. The structured dataset validation checks page lists these in detail, and accepted file formats covers packaging.

Worked request specification

The template below shows how a procurement lead might write the table-level part of a request for support and billing data.

Illustrative example: invented to show structure; it does not describe an available dataset.

Notice that the window, the account history and the file format are marked flexible. The SAP announcement of a real ERP research dataset was notable because real enterprise multi-table data is rarely available for research [3], so the more of your spec that is negotiable, the more suppliers can say yes.

Privacy constraints that change the schema

De-identification decisions change columns, not just values, so put the constraints you need into the spec before suppliers start extracting. Under HIPAA, the Safe Harbor method removes 18 identifier types, including all date elements more specific than the year that relate to an individual, which can collapse a daily event table into yearly granularity; Expert Determination is the alternative that may preserve more temporal detail [7].

For California consumer data, the CCPA defines deidentified information and attaches conditions to the business that holds it [8]; the CCPA deidentified data buyer obligations page covers what a buyer inherits. Ask the supplier which columns will be dropped, generalized, shifted or tokenized, and re-check your grain and join-coverage requirements against that list.

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

How SourceX handles tabular requests

SourceX sources operational datasets, including support and sales histories, engineering records and finance workflows, from US companies on request; nothing is held in stock and a request does not guarantee a match. Buyers describe the data rather than the businesses, and a specification like the one above is what lets suppliers judge fit. Each dataset is rights-reviewed and delivered under a license defining records, uses, term and delivery, with personal details such as names, emails and account numbers removed or replaced before delivery. You can draft yours with the data request builder and submit it through the SourceX buyer page.

Request tables specified by grain, keys and history

SourceX finds US businesses that hold the data you describe, assesses data and licensing permissions, and agrees pricing and allowed uses in a license before anything is transacted. Every release is approved by the supplying company and delivered through private, access-controlled workflows. Describe the tables you need.