Skip to content

Systems and records

How to validate that a data dictionary matches the fields delivered

By SourceX Editorial · Updated

9 sections

Short answer

To validate that a data dictionary matches the fields delivered, express the dictionary as a machine-readable schema, read the schema the files actually carry, and compare them with a script. Agree a matching rule for extra or missing fields, then check types, required fields, code lists, keys, units and removed fields, and give every mismatch a severity before the data is accepted.

Checklist

Dataset data dictionary

This page covers how to validate that a data dictionary matches the fields delivered.Illustrative

Key takeaways

  • A dictionary drafted from the source system often describes fields the export pipeline renamed, dropped or recoded.
  • Agree the matching rule first: exact, equal, subset or superset decides whether extra or missing columns fail.
  • Read the delivered schema from the files themselves, the CSV header or the Parquet footer, never from a description.
  • Ten checks, from file inventory to hashes, each with a default severity, make the test repeatable.
  • Meaning drift such as a unit or time zone change is always blocking, even when names and types still match.

How do you validate that a data dictionary matches the fields delivered?

Turn the dictionary into a machine-readable schema, read the schema the files actually carry, and compare the two with a script rather than by eye. The comparison covers the field set, field order where it matters, declared types, required fields, allowed codes, keys and a small set of meaning checks, and every difference gets a severity before anyone opens the data.

Most acceptance problems with exported business data are not corrupt files. They are a dictionary written from the source system's admin screen while the export pipeline renamed, dropped, split or recoded fields on the way out. A reconciliation test catches that gap in minutes, and it gives the supplier a precise list of fields to fix instead of a vague complaint that the data looks wrong.

The test has two inputs and one output. The inputs are the dictionary, expressed as a schema file, and the delivered files themselves. The output is a reconciliation report: one row per finding, each naming the field, the rule it broke, the evidence and whether it blocks acceptance.

Where dictionaries drift away from the files they describe

A dictionary drifts from its delivery for predictable reasons, and knowing them tells you which checks to write first. The common one is authorship: the dictionary is drafted from the CRM or help desk field list, while the delivered columns come from an export job, a warehouse view or a redaction step that changed them.

At a glance

Where dictionaries drift away from the files they describe

The second cause is late change. A privacy review removes a free-text field or replaces names with tokens, and the dictionary is never updated. The third is silent recoding, where a status field gains new values in the source system and the code list in the dictionary still shows last year's options. Meaning drift is the hardest: a column keeps its name and type but switches units, time zone or definition between deliveries.

  • Renamed in transit: the dictionary says ticket_status, the file header says status.
  • Dropped after review: a notes field listed in the dictionary is absent because redaction removed it.
  • Added by the pipeline: export tools often append load timestamps or row hashes that nobody documented.
  • Type widened: an integer ID arrives as text because one source row had a prefix.
  • Codes extended: a priority field contains a value that is not in the published code list.
  • Meaning changed: a duration field moves from minutes to seconds with the same name.

Which matching rule should the field-set check use?

Pick the matching rule before you write the test, because it decides whether an extra or missing column is a failure. The Frictionless Table Schema standard names five rules in its fieldsMatch property: exact (same fields, mapped by order, the default), equal (same fields, mapped by name), subset (every declared field present, extra fields allowed), superset (only declared fields, some may be absent) and partial (at least one declared field present).

Validation libraries make the same choice in their own terms. In Great Expectations, the column-set expectation defaults to an exact match, and setting exact_match to false lets additional observed columns pass as long as the declared set is present. The W3C metadata vocabulary for CSV defines schema compatibility by column count and by matching names or titles at each position. Whatever tool you use, write the rule into the delivery specification so both sides apply the same one.

Which matching rule should the field-set check use?
Delivery situationMatching ruleWhyWhat it lets through
Fixed-layout CSV loaded by positionExact: same fields, same orderA loader that reads by column index breaks on any reorderNothing; any change fails
Parquet or JSON Lines read by field nameEqual: same fields, any orderOrder carries no meaning when readers bind by nameReordered columns
Supplier may add audit columnsSubset: all declared fields, extras allowedExtra fields are tolerated but must be reportedUndocumented added fields, flagged as warnings
Fields removed by privacy review are expectedSuperset: only declared fields, some may be missingAbsent fields are allowed if the removal log names themMissing fields that the redaction record explains
Exploratory sample before a contractPartial, reported in fullUseful only to see what exists, never for acceptanceAlmost anything; do not use for acceptance

The dictionary-to-schema reconciliation test, check by check

Use the following table as the test specification. Each row is one assertion with the evidence it reads and a default severity; adjust severities in the delivery specification, not in the script. Run the checks in order, because a failed file inventory or field-set check makes later results unreliable.

The dictionary-to-schema reconciliation test, check by check
CheckAssertionReads from the deliveryDefault severityFinding records
R01 File inventoryEvery resource in the dictionary has a file, and every delivered file is describedManifest paths and file listingBlockMissing or undocumented path
R02 Field setDelivered field names satisfy the agreed matching ruleCSV header row or Parquet footer schemaBlockMissing, extra or renamed field
R03 Order and name formNames match case-sensitively with no trimming; order matches when the rule is exactHeader bytes and positionsBlockCase, whitespace or position difference
R04 Declared typeEvery value casts to the declared type and formatFull column scanBlock above agreed error rateCount and sample of failing values
R05 RequiredFields marked required contain no missing valuesValues after missing-value markers are appliedBlockCount of missing values
R06 CodesCoded fields contain only listed valuesDistinct values per coded fieldWarn, block if unmapped share is materialEach unlisted value with its count
R07 KeysDeclared primary keys are unique; foreign keys resolveKey columns across resourcesBlockDuplicate keys and orphan references
R08 Units and timeRanges and formats fit the declared unit and time zoneMin, max and format profileWarnObserved range against expectation
R09 Removed and tokenized fieldsFields marked removed are absent; tokenized fields hold no raw values on a sampleHeader and a value sampleBlockField and sample row reference
R10 Version and hashDictionary version and file hashes match the manifestManifest and computed hashesBlockVersion string and hash mismatch

Where each side of the comparison comes from

Read the delivered schema from the files, never from a supplier's description of them. For CSV, the header row is the only declared schema, so types have to be inferred by casting every value against the dictionary. For Parquet, the format writes file metadata after the data, and readers are expected to read that metadata first; the logical type annotations stored there tell you how each column was written.

Express the dictionary side in a standard rather than a spreadsheet. A Table Schema descriptor gives each field a name, type, format and constraints such as required and unique, plus missingValues, primaryKey and foreignKeys at table level. Croissant, the MLCommons metadata format for machine learning datasets, describes record sets whose fields each have a name, a dataType and a source pointing at a specific column of a file, and lets file objects carry a sha256 checksum. Either can be validated by software, which a word-processor dictionary cannot.

Keep the human dictionary too. Descriptions, business definitions and known caveats belong in prose, and the datasheet approach proposed by Gebru and colleagues documents motivation, composition and collection process for that reason. The point is that the machine-readable file and the prose describe the same fields, and the test checks the first against the data.

Illustrative case: one help desk export that passes and one that fails

The following is an illustrative example, not real customer data. A supplier delivers a tickets table exported from a help desk system as Parquet, with a dictionary expressed as a Table Schema descriptor using the equal matching rule. The dictionary declares the fields below.

  • Passing delivery (illustrative): six fields with matching names in a different order, which equal allows; all ticket_id values unique; every channel and priority value listed; first_response_minutes integer and non-negative. Report: zero blocking findings, one note that column order differs.
  • Failing delivery (illustrative): R02 finds response_time_s in place of first_response_minutes and an undocumented _loaded_at field; R06 finds the channel value social in a small share of rows; R08 finds created_at values carrying a local offset rather than UTC. Report: two blocking findings on R02, one warning each on R06 and R08.
  • Resolution (illustrative): the supplier restores the documented field, drops or documents the load column, adds social to the code list with a definition, and normalizes timestamps to UTC, then ships a new dictionary version with updated hashes.
Illustrative case: one help desk export that passes and one that fails
FieldTypeConstraintsDictionary definition
ticket_idstringrequired, unique, primary keyStable ticket identifier, tokenized from the source ID
created_atdatetimerequiredCreation time in UTC, ISO 8601
channelstringrequired, codes: email, chat, phone, webChannel through which the request arrived
prioritystringcodes: low, normal, high, urgentPriority at first assignment
first_response_minutesintegerminimum 0Minutes from creation to first agent reply
resolution_codestringcodes listed in the code tableFinal resolution category set by the agent

Which mismatches block acceptance and which become an exception note?

Block acceptance when a mismatch changes what a field means or makes a documented promise about the field false; record an exception when the meaning is intact and the fix is documentation. A renamed field whose content changed units is a blocking defect even if a mapping would make the test pass, because a model trained on it learns the wrong quantity.

Code-list gaps are the usual grey zone. A handful of unlisted values that the supplier can define in one line belong in the exception log with a corrected code list in the next version. A large share of unmapped codes usually means the dictionary was written against a different system configuration, and the whole field needs re-documenting.

Which mismatches block acceptance and which become an exception note?
FindingDecisionCondition that changes it
Extra undocumented fieldWarn and request documentationBlock if the field could contain personal data
Missing declared fieldBlockAccept if the removal log names it and the rule is superset
Reordered fieldsAcceptBlock if consumers load by position
Unlisted code valuesWarn and logBlock when the unmapped share is material for the task
Type cast failuresBlock above the agreed error rateAccept below it with failing rows listed
Unit or time zone changeBlockNone; meaning drift is never cosmetic

Can a buyer run the test without the supplier's internal systems?

Yes, if the package carries everything the test reads: the schema file, a manifest with relative paths and hashes, the code tables and the version string. None of those require access to the help desk, CRM or warehouse that produced the export, and a buyer who has to ask for system access to validate a delivery has received an incomplete package.

What a buyer cannot check alone is whether a definition is true to the source system. For that, ask the supplier for a short attestation per field family stating how it was derived, plus a small reference sample the supplier has traced back to source records. The reconciliation test then proves the files match the documentation, and the attestation covers the documentation matching reality.

Version the test with the dictionary. When a recurring delivery adds a field or retires a code, the supplier publishes a new dictionary version and a change note, and the buyer runs the same test against the new version. Keeping old reports alongside each version shows exactly when a field changed and which shipment introduced it.

Putting the reconciliation test into a data request

State in the request that the dictionary must arrive as a machine-readable schema, name the matching rule and the severities, and say the reconciliation report is part of acceptance. That turns a documentation preference into a testable delivery term and shortens the back and forth after the first shipment.

Because SourceX sources each dataset from U.S. businesses only after a buyer describes the need, and handles rights review and data preparation before delivery, the documentation terms can be set at the start: the request form can carry format and documentation requirements such as a Table Schema or Croissant descriptor, alongside scope, volume and timing.

Frequently asked questions

What should a dictionary-to-schema reconciliation test include?

At minimum: a file inventory check, a field-set check under an agreed matching rule, type casting, required fields, code lists, key uniqueness and references, unit and time zone checks, removed-field checks and a version and hash check against the manifest. Each finding should name the field, the rule, the evidence and a severity.

Is a JSON Schema enough to validate a tabular delivery?

It works well for JSON Lines records, where each object can be checked for required properties and types. For CSV and Parquet tables, a table-oriented format such as Table Schema or Croissant is a better fit because it also expresses keys, missing-value markers and relationships between tables.

Should column order be part of the test?

Only when a consumer loads columns by position, as some CSV pipelines do. For Parquet or any reader that binds fields by name, order carries no meaning, and an equal or subset rule avoids false failures.

How do you catch a field whose meaning changed but whose name did not?

Structural checks cannot see it, so add range and distribution checks against the declared unit and compare them with the previous delivery. A duration that jumps by a factor of sixty or timestamps that shift by a fixed number of hours are typical signals that need a supplier explanation.

Who should own failures found by the test?

The supplier fixes the data or the dictionary, and the buyer's integration owner decides whether a finding blocks acceptance under the delivery specification. Keeping that split avoids the engineer who found an issue also deciding its business impact.

Sources

  • The Table Schema standard defines field descriptors with name, type, format and constraints, table-level missingValues, primaryKey and foreignKeys, and a fieldsMatch property whose values exact, equal, subset, superset and partial set how declared fields map to the data. Source
  • Croissant 1.1 describes record sets whose fields have a name, a dataType and a source that can point to a specific column of a file, and lets file objects carry a sha256 checksum. Source
  • The Great Expectations column-set expectation requires an exact match by default; with exact_match set to false, observed columns must include the set but additional columns pass. Source
  • Parquet writes file metadata after the data, readers are expected to read that metadata first, and logical type annotations are stored in the file metadata. Source
  • The W3C Metadata Vocabulary for Tabular Data treats two schemas as compatible when they have the same number of non-virtual columns and the columns at each index match by name or title. Source
  • Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process and recommended uses. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies