Systems and records
Enterprise dataset schema versioning: keep schema and content on separate numbers
By SourceX Editorial · Updated
9 sections
Short answer
Track schema versions and content releases separately because they answer different questions. The schema version says how to read a record: field names, types, meanings and what a reader can safely ignore. The content version says which records and values you received. Each changes on its own schedule, so one number hides breaking changes or forces needless re-validation. Pin both in the manifest and ship a migration map.
System export
Enterprise dataset schema
Key takeaways
- The schema version tells a reader how to interpret records; the content version identifies which records were delivered.
- Use SemVer for the schema and an immutable date or sequence label for content; never change a released content version.
- Give every field a stable ID, reserve retired IDs, and treat a change of meaning as a new field rather than a redefinition.
- Ship one manifest that pairs both versions with the schema file, file hashes and, for every major bump, a migration file.
- Test intake against deliberately broken packages so mislabeled releases fail before they reach a training run.
Why should dataset schema versions be tracked separately from content releases?
Because a buyer's code depends on the schema while a buyer's results depend on the content, and the two change for different reasons. A new monthly extract of support tickets changes the content but not how a ticket is read. A decision to rename a field, split a code list or widen a type changes how every record is read, including records you already hold. A single version number has to pretend these are the same event.
The cost shows up in two directions. If the supplier bumps one number for every monthly extract, the buyer cannot tell from the number whether parsers need attention, so every release triggers a full integration review. If the supplier only bumps it for structural change, a refreshed extract with different rows carries the same label as the last one, and a training run pinned to that label is no longer reproducible.
Mature table formats already separate the two. The Apache Iceberg table specification keeps a list of schemas, each with its own schema ID, plus a current-schema-id, and every snapshot of the table's contents records the ID of the schema that was current when the snapshot was created. Content moves through snapshots; structure moves through schema IDs; each snapshot points at exactly one schema. A delivered dataset package needs the same pairing, written down in a manifest a buyer can check without access to the supplier's warehouse.
What does each version number promise, and who relies on it?
The schema version is a promise about interpretation, and the people who rely on it are whoever wrote the loader, the validator and the feature code. Semantic Versioning fits it well: the SemVer specification says the major version increments for incompatible changes, the minor version for backward compatible additions and the patch version for backward compatible fixes. Treat the schema, its field definitions and its code lists as the public API in that sense.
The content version is a promise about identity: these exact files, these row counts, these hashes. The people who rely on it are whoever trains or evaluates on the data and later has to say which data a model saw. SemVer contains a rule worth borrowing here even if you number content by date: once a versioned package has been released, its contents must not be modified, and any modification must be released as a new version.
Some public metadata standards fold both promises into one number, which is a useful reminder of the tradeoff. The Croissant 1.0 specification recommends MAJOR.MINOR.PATCH for a dataset's version, where a patch means the data is the same but serialized or packaged differently, a minor means existing data is unchanged with possible additions such as new fields or new records, and a major means existing data was edited, removed or reshuffled. For live datasets it recommends updating the version only for structural or backward-incompatible change, and relying on per-file sha256 checksums to detect content change. That works for a public dataset with one maintainer. For a licensed, recurring delivery, where the license, the acceptance test and the training run each point at a release, an explicit content version is clearer than inferring identity from checksums alone.
Which change bumps which number?
Classify every change by what a reader holding the old schema would experience, then decide the number. The table below is the decision rule we recommend writing into the delivery agreement. Schema bumps follow SemVer; content bumps happen on every release that changes any delivered byte of record data.
| Change | Schema version | Content version | Why |
|---|---|---|---|
| New monthly extract, same fields and meanings | Unchanged | New | Readers parse it as before; the records are different |
| Corrected values in records already delivered | Unchanged | New, with a correction note | Same structure, but anyone pinned to the old release must be able to tell |
| Add an optional field with a declared default | Minor | New if any file changes | Old readers ignore it; new readers fill the default for older files |
| Clarify a field description without changing meaning | Patch | Unchanged | No record changes and no reader breaks |
| Widen a type, such as int to long | Minor | Unchanged unless rewritten | Safe promotion in formats that define it; existing values still read |
| Rename a field but keep its meaning and field ID | Minor | Unchanged unless rewritten | Readers that resolve by ID are unaffected; name-based readers need the alias |
| Change a code list's meaning, drop a field or make one required | Major | New, re-rendered under the new schema | Old readers would misread or reject records |
| Same records re-exported as Parquet instead of CSV | Unchanged | Unchanged, new package build | Content identity holds; record the build separately with new file hashes |
How do field IDs stop a rename from looking like a new column?
Give every field a stable numeric ID that never changes and is never reused, and let names be labels. Then a rename is a minor schema change and a reader can map old files to the new schema mechanically. Without IDs, a renamed column is indistinguishable from a dropped column plus a new one, and a downstream join quietly loses its history.
Three widely used formats show the rule from different angles. Iceberg's specification says columns in data files are selected by field ID, that renaming a field must change the name but not the field ID, and that adding a field assigns a new ID. Protocol Buffers' proto3 guide says a deleted field's number must be added to a reserved list, because reusing the number can cause data corruption among other problems. Apache Avro resolves by name instead and provides aliases, alternative names that let a reader map a writer's old field name to its own; when a reader's field is missing from the writer's data, Avro uses the reader's default if one exists and signals an error if not.
For a delivered dataset the practical version is simple. Put a field_id next to every field in the schema file, keep a retired_ids list, and require that a field whose meaning changes gets a new ID rather than a new definition under an old one.
What belongs in a manifest that pairs both versions?
Every package should carry one manifest that names the content release, the schema version it is rendered in and the evidence that ties the two together. The fields below are the minimum. Paths are relative to the package root so the package can be moved and checked anywhere.
| Manifest field | Example value (illustrative) | Purpose | Check a buyer runs |
|---|---|---|---|
| dataset_id | support-tickets | Stable name that never encodes a date or version | Matches the licensed dataset name in the agreement |
| content_version | 2026.10 | Identity of the records and values delivered | Not previously seen with different file hashes |
| schema_version | 2.0.0 | How to read every file in this package | Schema file with this version is present and parses |
| schema_path and schema_sha256 | schema/tickets-2.0.0.json plus its hash | Ships the exact schema beside the data | Hash of the shipped schema matches |
| compatible_with | 2.0.x | Range of reader schemas that can read these files | Buyer's loader version falls inside the range |
| migration_path | migrations/1.2.0-to-2.0.0.json | Rules to map the previous major to this one | Required whenever the major version changed |
| files | relative path, row count, sha256 per file | Content identity at file level | Recompute counts and hashes |
| supersedes | support-tickets 2026.09 in schema 1.2.0 | Lineage to the prior release | Prior release is known and still retrievable or recorded as retired |
Worked migration: a support-ticket schema moves from 1.2.0 to 2.0.0 (illustrative)
This example is illustrative and not drawn from any customer's data. A supplier delivers monthly ticket records under schema 1.2.0. Field 4, priority, uses the values P1 to P4. Operations replaces priority with a severity scale that has different rules: severity is set by the impact on the customer, not by the queue, so a P2 ticket can be critical or minor. That is a change of meaning, so the supplier issues schema 2.0.0 and writes a migration file that a buyer can apply to old releases.
The migration file is the asset that makes the major bump usable. It lists, per field ID, what happens to old data, and it says honestly where old data cannot be converted.
| Field ID | In 1.2.0 | In 2.0.0 | Migration rule for older releases |
|---|---|---|---|
| 1 | ticket_id string | ticket_id string | Copy unchanged |
| 4 | priority enum P1 to P4 | Retired; ID reserved | Keep in a legacy column if the buyer wants it; never reuse ID 4 |
| 6 | first_response_minutes int | first_response_minutes long | Widen type; values unchanged |
| 7 | Not present | severity enum critical, major, minor | Set to null for releases before 2026.10; do not derive from priority |
| 8 | Not present | reopened_count int, default 0 | Fill from the ticket history file where present, else null with a flag |
| 3 | channel enum email, chat, phone | contact_channel enum email, chat, phone | Rename only; same ID, add alias channel |
What do a valid package and a failing package look like to the validator?
A valid package for the example declares content_version 2026.10 and schema_version 2.0.0, ships tickets-2.0.0.json with a matching hash, lists the migration from 1.2.0, and every file contains only declared fields. Every October record carries a severity value, because severity was assigned at intake from that month on. A loader written for 2.0.x reads it without special cases.
The failing cases are more instructive, because each one is a mistake a single version number makes easy. Run these as negative tests in your intake checks so you know they fail loudly.
| Failing package (illustrative) | What the validator reports | Root cause | Fix |
|---|---|---|---|
| Declares schema 1.2.0 but files contain a severity column | Undeclared field 7 in 3 files | Content was rendered in the new schema; manifest not updated | Declare 2.0.0 and ship its schema file |
| Schema 1.3.0 removes priority | Removal of a field without default needs a major bump | Breaking change labeled minor | Reissue as 2.0.0 with a migration file |
| Content 2026.09 re-sent with different hashes and the same versions | Known release with changed file hashes | Corrections shipped under an existing label | Issue a new content version with a correction note |
| New field uses ID 4 | Field ID 4 is in retired_ids | ID reused for a new meaning | Assign the next unused ID |
| Major bump with no migration_path | Missing migration for 1.2.0 to 2.0.0 | Supplier relied on release notes prose | Add a machine-readable migration file |
Can the package be checked and rebuilt on the buyer's side alone?
Yes, if the package is self-describing. Everything the checks above need, the schema file, its hash, the field IDs, the retired IDs, the migration file and per-file hashes and counts, travels inside the package with relative paths. The Frictionless Data Package standard follows the same idea: its version property should follow Semantic Versioning, and it advises that a package's name stay invariant across updates, leaving version distinction to the version property.
Rebuilding means two things. First, the buyer can recompute every hash and count and get the manifest's numbers. Second, the buyer can take an older release, apply the migration file and load it with the current reader, and the result loads cleanly with the documented nulls and flags. If either step needs a call to the supplier, the package is not yet portable.
For recurring feeds a compatibility policy makes the schema side enforceable. Confluent's Schema Registry documentation, for example, defines backward, forward and full compatibility modes, each with a transitive variant that checks against all previous versions instead of only the latest, and uses backward as the default. Pick one, write it into the agreement, and run the check before a delivery is accepted rather than after a pipeline breaks.
What to ask for when you license a recurring dataset
Put the two-number rule into the request and the agreement: content versions that never change after release, a SemVer schema with field IDs, a manifest carrying both versions, and a migration file for every major change. Ask for one valid and one deliberately broken sample package so your intake checks are proven before the first real release.
Because SourceX finds data for AI teams only once a request arrives, drawing on U.S. businesses instead of a stocked catalog, the versioning rules above can be written into that request next to scope, volume, language, format, timing and licensing needs. The business approves the scope and terms and signs before any data is delivered, which is the right moment to fix the versioning rules as well.
Frequently asked questions
Is a date-based content version enough, or does it need SemVer too?
A date or sequence number is enough for content, because content releases are not compatible or incompatible with each other, they are simply different. What matters is that a released content version never changes. Save SemVer for the schema, where minor and major carry real meaning for readers.
Does adding a new optional field require a new content version?
Only if delivered files change. If you add the field to the schema and re-render last month's files with an empty column, those files have new hashes, so record them as a new package build or content version. If the field first appears in next month's extract, that extract's own content version covers it.
How is a schema version different from a data contract?
A data contract usually covers the schema plus service terms such as delivery cadence, quality thresholds and change notice. The schema version is one field inside that contract. Keeping it separate from the content version is what lets the contract's change-notice rules apply to structure without firing on every routine refresh.
What should happen to old releases after a major schema change?
Keep them in the schema they were delivered in, and ship the migration file so they can be read under the new one. Do not silently re-render history in the new schema under the old content labels. If you do re-render, give the result a new package build so a training run pinned to the original can still be reproduced.
Can the validator be a simple JSON Schema check?
A JSON Schema check covers field presence, types and enums in each record. The pairing checks, such as matching hashes, retired field IDs, a migration file for each major bump and immutability of released content, sit one level up and need a small script that reads the manifest. Both belong in intake.
Sources
- Apache Iceberg keeps multiple schemas with schema IDs and a current-schema-id, records the schema ID on each snapshot, selects columns by field ID and keeps the ID on rename. Source
- Semantic Versioning 2.0.0 defines major, minor and patch increments and forbids modifying the contents of a released version. Source
- The Croissant 1.0 specification recommends semantic versions for datasets, defines what patch, minor and major mean for data, and recommends sha256 checksums per file. Source
- The proto3 language guide requires reserving deleted field numbers because reusing them can corrupt data. Source
- The Avro specification defines aliases and schema resolution: reader defaults fill missing fields, and a missing field without a default is an error. Source
- The Data Package standard says the version property should follow Semantic Versioning and the package name should stay invariant across updates. Source
- Confluent Schema Registry documents backward, forward, full and transitive compatibility types, with backward as the default. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.