Systems and records
How to validate that a data dictionary matches the fields delivered
By SourceX Editorial · Updated
9 sections
Short answer
To validate that a data dictionary matches the fields delivered, express the dictionary as a machine-readable schema, read the schema the files actually carry, and compare them with a script. Agree a matching rule for extra or missing fields, then check types, required fields, code lists, keys, units and removed fields, and give every mismatch a severity before the data is accepted.
Checklist
Dataset data dictionary
Key takeaways
- A dictionary drafted from the source system often describes fields the export pipeline renamed, dropped or recoded.
- Agree the matching rule first: exact, equal, subset or superset decides whether extra or missing columns fail.
- Read the delivered schema from the files themselves, the CSV header or the Parquet footer, never from a description.
- Ten checks, from file inventory to hashes, each with a default severity, make the test repeatable.
- Meaning drift such as a unit or time zone change is always blocking, even when names and types still match.
How do you validate that a data dictionary matches the fields delivered?
Turn the dictionary into a machine-readable schema, read the schema the files actually carry, and compare the two with a script rather than by eye. The comparison covers the field set, field order where it matters, declared types, required fields, allowed codes, keys and a small set of meaning checks, and every difference gets a severity before anyone opens the data.
Most acceptance problems with exported business data are not corrupt files. They are a dictionary written from the source system's admin screen while the export pipeline renamed, dropped, split or recoded fields on the way out. A reconciliation test catches that gap in minutes, and it gives the supplier a precise list of fields to fix instead of a vague complaint that the data looks wrong.
The test has two inputs and one output. The inputs are the dictionary, expressed as a schema file, and the delivered files themselves. The output is a reconciliation report: one row per finding, each naming the field, the rule it broke, the evidence and whether it blocks acceptance.
Where dictionaries drift away from the files they describe
A dictionary drifts from its delivery for predictable reasons, and knowing them tells you which checks to write first. The common one is authorship: the dictionary is drafted from the CRM or help desk field list, while the delivered columns come from an export job, a warehouse view or a redaction step that changed them.
At a glance
Where dictionaries drift away from the files they describe
The second cause is late change. A privacy review removes a free-text field or replaces names with tokens, and the dictionary is never updated. The third is silent recoding, where a status field gains new values in the source system and the code list in the dictionary still shows last year's options. Meaning drift is the hardest: a column keeps its name and type but switches units, time zone or definition between deliveries.
- Renamed in transit: the dictionary says ticket_status, the file header says status.
- Dropped after review: a notes field listed in the dictionary is absent because redaction removed it.
- Added by the pipeline: export tools often append load timestamps or row hashes that nobody documented.
- Type widened: an integer ID arrives as text because one source row had a prefix.
- Codes extended: a priority field contains a value that is not in the published code list.
- Meaning changed: a duration field moves from minutes to seconds with the same name.
Which matching rule should the field-set check use?
Pick the matching rule before you write the test, because it decides whether an extra or missing column is a failure. The Frictionless Table Schema standard names five rules in its fieldsMatch property: exact (same fields, mapped by order, the default), equal (same fields, mapped by name), subset (every declared field present, extra fields allowed), superset (only declared fields, some may be absent) and partial (at least one declared field present).
Validation libraries make the same choice in their own terms. In Great Expectations, the column-set expectation defaults to an exact match, and setting exact_match to false lets additional observed columns pass as long as the declared set is present. The W3C metadata vocabulary for CSV defines schema compatibility by column count and by matching names or titles at each position. Whatever tool you use, write the rule into the delivery specification so both sides apply the same one.
| Delivery situation | Matching rule | Why | What it lets through |
|---|---|---|---|
| Fixed-layout CSV loaded by position | Exact: same fields, same order | A loader that reads by column index breaks on any reorder | Nothing; any change fails |
| Parquet or JSON Lines read by field name | Equal: same fields, any order | Order carries no meaning when readers bind by name | Reordered columns |
| Supplier may add audit columns | Subset: all declared fields, extras allowed | Extra fields are tolerated but must be reported | Undocumented added fields, flagged as warnings |
| Fields removed by privacy review are expected | Superset: only declared fields, some may be missing | Absent fields are allowed if the removal log names them | Missing fields that the redaction record explains |
| Exploratory sample before a contract | Partial, reported in full | Useful only to see what exists, never for acceptance | Almost anything; do not use for acceptance |
The dictionary-to-schema reconciliation test, check by check
Use the following table as the test specification. Each row is one assertion with the evidence it reads and a default severity; adjust severities in the delivery specification, not in the script. Run the checks in order, because a failed file inventory or field-set check makes later results unreliable.
| Check | Assertion | Reads from the delivery | Default severity | Finding records |
|---|---|---|---|---|
| R01 File inventory | Every resource in the dictionary has a file, and every delivered file is described | Manifest paths and file listing | Block | Missing or undocumented path |
| R02 Field set | Delivered field names satisfy the agreed matching rule | CSV header row or Parquet footer schema | Block | Missing, extra or renamed field |
| R03 Order and name form | Names match case-sensitively with no trimming; order matches when the rule is exact | Header bytes and positions | Block | Case, whitespace or position difference |
| R04 Declared type | Every value casts to the declared type and format | Full column scan | Block above agreed error rate | Count and sample of failing values |
| R05 Required | Fields marked required contain no missing values | Values after missing-value markers are applied | Block | Count of missing values |
| R06 Codes | Coded fields contain only listed values | Distinct values per coded field | Warn, block if unmapped share is material | Each unlisted value with its count |
| R07 Keys | Declared primary keys are unique; foreign keys resolve | Key columns across resources | Block | Duplicate keys and orphan references |
| R08 Units and time | Ranges and formats fit the declared unit and time zone | Min, max and format profile | Warn | Observed range against expectation |
| R09 Removed and tokenized fields | Fields marked removed are absent; tokenized fields hold no raw values on a sample | Header and a value sample | Block | Field and sample row reference |
| R10 Version and hash | Dictionary version and file hashes match the manifest | Manifest and computed hashes | Block | Version string and hash mismatch |
Where each side of the comparison comes from
Read the delivered schema from the files, never from a supplier's description of them. For CSV, the header row is the only declared schema, so types have to be inferred by casting every value against the dictionary. For Parquet, the format writes file metadata after the data, and readers are expected to read that metadata first; the logical type annotations stored there tell you how each column was written.
Express the dictionary side in a standard rather than a spreadsheet. A Table Schema descriptor gives each field a name, type, format and constraints such as required and unique, plus missingValues, primaryKey and foreignKeys at table level. Croissant, the MLCommons metadata format for machine learning datasets, describes record sets whose fields each have a name, a dataType and a source pointing at a specific column of a file, and lets file objects carry a sha256 checksum. Either can be validated by software, which a word-processor dictionary cannot.
Keep the human dictionary too. Descriptions, business definitions and known caveats belong in prose, and the datasheet approach proposed by Gebru and colleagues documents motivation, composition and collection process for that reason. The point is that the machine-readable file and the prose describe the same fields, and the test checks the first against the data.
Illustrative case: one help desk export that passes and one that fails
The following is an illustrative example, not real customer data. A supplier delivers a tickets table exported from a help desk system as Parquet, with a dictionary expressed as a Table Schema descriptor using the equal matching rule. The dictionary declares the fields below.
- Passing delivery (illustrative): six fields with matching names in a different order, which equal allows; all ticket_id values unique; every channel and priority value listed; first_response_minutes integer and non-negative. Report: zero blocking findings, one note that column order differs.
- Failing delivery (illustrative): R02 finds response_time_s in place of first_response_minutes and an undocumented _loaded_at field; R06 finds the channel value social in a small share of rows; R08 finds created_at values carrying a local offset rather than UTC. Report: two blocking findings on R02, one warning each on R06 and R08.
- Resolution (illustrative): the supplier restores the documented field, drops or documents the load column, adds social to the code list with a definition, and normalizes timestamps to UTC, then ships a new dictionary version with updated hashes.
| Field | Type | Constraints | Dictionary definition |
|---|---|---|---|
| ticket_id | string | required, unique, primary key | Stable ticket identifier, tokenized from the source ID |
| created_at | datetime | required | Creation time in UTC, ISO 8601 |
| channel | string | required, codes: email, chat, phone, web | Channel through which the request arrived |
| priority | string | codes: low, normal, high, urgent | Priority at first assignment |
| first_response_minutes | integer | minimum 0 | Minutes from creation to first agent reply |
| resolution_code | string | codes listed in the code table | Final resolution category set by the agent |
Which mismatches block acceptance and which become an exception note?
Block acceptance when a mismatch changes what a field means or makes a documented promise about the field false; record an exception when the meaning is intact and the fix is documentation. A renamed field whose content changed units is a blocking defect even if a mapping would make the test pass, because a model trained on it learns the wrong quantity.
Code-list gaps are the usual grey zone. A handful of unlisted values that the supplier can define in one line belong in the exception log with a corrected code list in the next version. A large share of unmapped codes usually means the dictionary was written against a different system configuration, and the whole field needs re-documenting.
| Finding | Decision | Condition that changes it |
|---|---|---|
| Extra undocumented field | Warn and request documentation | Block if the field could contain personal data |
| Missing declared field | Block | Accept if the removal log names it and the rule is superset |
| Reordered fields | Accept | Block if consumers load by position |
| Unlisted code values | Warn and log | Block when the unmapped share is material for the task |
| Type cast failures | Block above the agreed error rate | Accept below it with failing rows listed |
| Unit or time zone change | Block | None; meaning drift is never cosmetic |
Can a buyer run the test without the supplier's internal systems?
Yes, if the package carries everything the test reads: the schema file, a manifest with relative paths and hashes, the code tables and the version string. None of those require access to the help desk, CRM or warehouse that produced the export, and a buyer who has to ask for system access to validate a delivery has received an incomplete package.
What a buyer cannot check alone is whether a definition is true to the source system. For that, ask the supplier for a short attestation per field family stating how it was derived, plus a small reference sample the supplier has traced back to source records. The reconciliation test then proves the files match the documentation, and the attestation covers the documentation matching reality.
Version the test with the dictionary. When a recurring delivery adds a field or retires a code, the supplier publishes a new dictionary version and a change note, and the buyer runs the same test against the new version. Keeping old reports alongside each version shows exactly when a field changed and which shipment introduced it.
Putting the reconciliation test into a data request
State in the request that the dictionary must arrive as a machine-readable schema, name the matching rule and the severities, and say the reconciliation report is part of acceptance. That turns a documentation preference into a testable delivery term and shortens the back and forth after the first shipment.
Because SourceX sources each dataset from U.S. businesses only after a buyer describes the need, and handles rights review and data preparation before delivery, the documentation terms can be set at the start: the request form can carry format and documentation requirements such as a Table Schema or Croissant descriptor, alongside scope, volume and timing.
Frequently asked questions
What should a dictionary-to-schema reconciliation test include?
At minimum: a file inventory check, a field-set check under an agreed matching rule, type casting, required fields, code lists, key uniqueness and references, unit and time zone checks, removed-field checks and a version and hash check against the manifest. Each finding should name the field, the rule, the evidence and a severity.
Is a JSON Schema enough to validate a tabular delivery?
It works well for JSON Lines records, where each object can be checked for required properties and types. For CSV and Parquet tables, a table-oriented format such as Table Schema or Croissant is a better fit because it also expresses keys, missing-value markers and relationships between tables.
Should column order be part of the test?
Only when a consumer loads columns by position, as some CSV pipelines do. For Parquet or any reader that binds fields by name, order carries no meaning, and an equal or subset rule avoids false failures.
How do you catch a field whose meaning changed but whose name did not?
Structural checks cannot see it, so add range and distribution checks against the declared unit and compare them with the previous delivery. A duration that jumps by a factor of sixty or timestamps that shift by a fixed number of hours are typical signals that need a supplier explanation.
Who should own failures found by the test?
The supplier fixes the data or the dictionary, and the buyer's integration owner decides whether a finding blocks acceptance under the delivery specification. Keeping that split avoids the engineer who found an issue also deciding its business impact.
Sources
- The Table Schema standard defines field descriptors with name, type, format and constraints, table-level missingValues, primaryKey and foreignKeys, and a fieldsMatch property whose values exact, equal, subset, superset and partial set how declared fields map to the data. Source
- Croissant 1.1 describes record sets whose fields have a name, a dataType and a source that can point to a specific column of a file, and lets file objects carry a sha256 checksum. Source
- The Great Expectations column-set expectation requires an exact match by default; with exact_match set to false, observed columns must include the set but additional columns pass. Source
- Parquet writes file metadata after the data, readers are expected to read that metadata first, and logical type annotations are stored in the file metadata. Source
- The W3C Metadata Vocabulary for Tabular Data treats two schemas as compatible when they have the same number of non-virtual columns and the columns at each index match by name or title. Source
- Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process and recommended uses. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.