Skip to content

Systems and records

Which source system metadata should accompany a business dataset?

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A business dataset should ship with the details a buyer needs to interpret every field: the source system and instance, its configuration and custom fields, how long it keeps change history, how records were extracted and transformed, the date range and known gaps, and which permission covers each source. A short source metadata questionnaire, answered once per system, captures all of it.

System export

Dataset source system

This page covers which source system metadata should accompany a business dataset.Illustrative

Key takeaways

  • Field names alone mislead; buyers need the system, instance and configuration behind them.
  • Custom fields, automations and defaults explain most interpretation errors, so record when each was added or retired.
  • Change history differs by system and by plan; record what is kept, for how long and with what source of change.
  • Describe each extraction and transformation step with run IDs, versions and field-level types such as copied or masked.
  • Keep native record IDs and release checksums so any record can be traced back to its source.

Which source system details should accompany a business dataset?

Every business dataset should travel with a description of the system it came from, written so that someone who has never logged into that system can read the fields correctly. At minimum that covers four things: which system and instance produced the records, how it was configured during the period covered, what history it keeps about changes, and what happened to the records between the system and the delivered files.

Buyers ask because field names lie. A column called status in a help desk export can mean the ticket state, an internal escalation flag or a custom field someone added in 2021 and stopped using in 2023. Without the source details, a buyer either guesses or spends weeks of back-and-forth reconstructing what your administrators already know.

The source metadata questionnaire on this page is the asset. It is split into three parts, system and configuration, history and audit trail, and extraction and release, and it is meant to be answered once per source system and kept with every release drawn from that system. Version 1.0, dated 2026-10-18.

Why do buyers need source system metadata at all?

Buyers need source system metadata to judge whether records mean what they appear to mean and whether they can be traced. Both are now part of how AI teams are expected to document their data.

For high-risk AI systems, Article 10(2)(b) of the EU AI Act requires data governance practices for training, validation and testing data to cover data collection processes and the origin of data, and for personal data the original purpose of collection. The NIST AI RMF Playbook asks how an organization has documented data provenance, including sources, origins, transformations, augmentations, labels, dependencies, constraints and metadata. A buyer that answers to either framework has to ask the supplier the same questions.

Dataset documentation practice points the same way. Datasheets for Datasets, proposed by Gebru and coauthors, organizes documentation into motivation, composition, collection process, preprocessing, distribution and maintenance. The Croissant RAI vocabulary for machine learning datasets defines properties for the data collection process, the collection type and the collection timeframe. The Data & Trust Alliance Data Provenance Standards group 22 metadata fields into Source, Provenance and Use. Source system details are the raw material for all of these.

How is source metadata different from a data dictionary or a datasheet?

Source metadata describes the system and the path the records took; a data dictionary describes the fields; a datasheet describes the dataset as a whole for its users. A buyer usually needs all three, and they answer different questions.

A data dictionary tells a reader that a column called priority holds one of four values. Source metadata tells the reader that agents set it, that customers could not, that the allowed values changed in 2023 and that the system kept a history of changes to it. A datasheet then summarizes why the dataset exists, what it contains, how it was collected and how it will be maintained.

The practical difference is who can answer. A data dictionary can often be generated from the export. Source metadata has to come from the people who administer the system, because much of it lives in configuration screens, change tickets and memory rather than in the records. That is why it is the part most often missing, and why its absence costs buyers the most time.

Part A: which system and configuration details should you record?

Record the system, the instance and the configuration that shaped the records during the covered period. Configuration matters as much as the product name: two companies on the same help desk or CRM can have entirely different fields, workflows and required values.

Ask the administrator for the configuration as it stood across the whole period, not only today. The current field list hides retired fields, renamed picklist values and automations that were switched off years ago, and each of those explains a pattern a buyer will otherwise flag as a quality problem. Where dates are unknown, say so and give the best available range.

Part A: which system and configuration details should you record?
QuestionWhy a buyer needs itIllustrative answerEvidence to attach
Which product, edition and instance produced the records?Behavior and available fields differ by product and editionHelp desk system, single production instance, EU data centerAdmin console screenshot or account settings export
What period does each instance cover?Migrations split history into sources with different rulesProduction instance from mid-2020; earlier tickets imported from spreadsheetsGo-live date record, import job logs
Which standard fields are used, and how?Same field name can carry different meanings per companyPriority set by agents, not customers; type field unusedField usage notes from the system administrator
Which custom fields exist, with definitions and dates added or retired?Custom fields drive most interpretation errorsEscalation reason added 2022, picklist of 9 values, retired 2024Field configuration export with change dates
Which workflows or automations write to records?Automated values are not human judgmentsAuto-close after 7 days without reply; auto-tagging by keywordList of active and retired automation rules
Which fields are required, defaulted or validated?Defaults inflate apparent completenessCategory required since 2023; defaults to GeneralValidation rule export

Part B: what change history does the source system keep?

Record what history the system keeps, for how long, and how it identifies the source of each change. The answer differs sharply between systems, and it decides whether a buyer can trace a value back to who set it and when.

Three widely used systems show the range. Salesforce retains field history for up to 18 months, and up to 24 months through the API, when its Field Audit Trail add-on is not enabled, so older value changes may simply be gone. HubSpot's property history shows past values for a record's properties with the date and time of each change and its source, such as an import, a workflow or an integration. In Zendesk, each ticket audit represents a single update to a ticket and can contain several events, and archived tickets are not included in the account-wide audit list, so an export that relies on that endpoint alone can miss them.

None of this is a flaw in the dataset. It is a fact about the evidence, and the questionnaire records it so that a buyer knows which values carry a change trail and which are a snapshot.

Check the retention rules for your own plan and settings rather than relying on general vendor descriptions. Add-ons, archive features and administrator choices change what is kept, and the answer for one object or field can differ from another in the same system.

Part B: what change history does the source system keep?
QuestionWhy a buyer needs itIllustrative answerEvidence to attach
Is field or property history tracked, and for which fields?Shows which values can be traced over timeHistory on status, priority and assignee onlyTracking settings export
How long is history retained?Old changes may be unrecoverableRetained per vendor default; no extended archiveVendor retention documentation, plan details
Does the history record the source of each change?Separates human edits from imports and automationsChange source shown for each property updateSample history export with source column
Were records archived, purged or deleted under a policy?Explains gaps and survivorship biasClosed tickets archived after 120 days; spam deletedRetention policy, purge job logs
Were records bulk-edited or re-imported?Bulk edits overwrite original valuesCategory remapped in bulk in 2023Change ticket or import log with date and scope

Part C: how were the records extracted, transformed and released?

Record each step between the system and the delivered files, with enough detail that a reviewer can repeat it or check it. Lineage standards give you ready-made fields for this.

OpenLineage, an open standard for lineage metadata, describes a dataset's source with a data source facet that records a name and a URI, and its column lineage facet maps each output field to the input fields used to produce it, with a transformation description and a type such as IDENTITY, an exact copy, or MASKED, where no original data remain. Using those categories in your questionnaire answers makes a later lineage export straightforward. Croissant's RAI properties for the collection process, collection type and collection timeframe fit the same answers into dataset-level metadata.

Write the answers at field level where it matters. A buyer reading that personal data were removed cannot tell whether the ticket body was redacted in place, replaced with a placeholder or dropped. A row per field, with its transformation type and the pipeline version that applied it, settles the question and makes the release reproducible if the same extraction is run again.

Part C: how were the records extracted, transformed and released?
QuestionWhy a buyer needs itIllustrative answerEvidence to attach
Which export method and endpoint produced the files?Different exports include different records and fieldsIncremental export API plus per-ticket audits for archived ticketsExtraction job configuration and run ID
What filters were applied at extraction?Filters define the populationTickets created 2020-07-01 to 2025-12-31, excluding spamQuery or filter definition
Which fields were dropped, masked or derived?Buyers must know which values are originalRequester email MASKED; ticket body IDENTITY with names redactedField-level transformation list with types
Which transformation version ran, and on which records?Mixed versions produce inconsistent recordsRedaction pipeline v2.3 on all recordsPipeline version tag and run log
Which permission reference covers each source?Access is not a licensing rightCustomer terms clause reference for tickets; employee notes under policyPermission register entry reviewed by counsel
What release ID, checksums and known gaps go with the delivery?Lets a buyer confirm files and see limitsRelease R-2026-01, checksums per file, four-month gap after importRelease manifest

Worked example: a help desk dataset with a migration exception

This example is illustrative; the company and its configuration are hypothetical. A regional software reseller prepares five years of customer support tickets from its help desk system. The first draft of its dataset documentation lists the product name, the field list and the date range, and stops there.

Answering Part A turns up two issues. The help desk went live in mid-2020, and older tickets came from a spreadsheet import, so the questionnaire records two sources with different field sets. A custom field for escalation reason existed only from 2022 to 2024, which explains why it is empty for most records; without that note, a buyer would read the blanks as missing data rather than a field that did not exist.

Part B shows that the system keeps per-ticket audits, but the account-wide audit list excludes archived tickets, which is most of the history. The extraction is changed to pull audits per ticket for archived records, and the questionnaire says so.

The exception sits in Part C. Satisfaction survey comments were stored in a separate survey tool and joined to tickets by ticket ID. The join failed for tickets merged by agents, so those comments are excluded and the count is stated. The finished questionnaire runs to two pages and answers, in advance, most of the questions a buyer's data team would otherwise send one by one.

The limits of the exercise are worth stating too. The questionnaire cannot recover history the system never kept, and it cannot confirm what a long-departed administrator intended when a field was created. Where the answer is unknown, the reseller writes unknown, gives the best available date range and leaves the decision about those fields to the buyer. A clear unknown is more useful to a buyer than a confident guess that later proves wrong, and it keeps the rest of the questionnaire credible.

Can a buyer walk any record back to the system it came from?

A reviewer can trace a claim only if every record carries identifiers that link back through the questionnaire: the source system and instance, the extraction run, the transformation version and the release. If the delivered files keep the native record ID from the source system, a reviewer can pick any record, find its source, and check its history in the system that produced it.

In practice that means four things. Keep native IDs, or a reversible mapping held by you, in every delivered record. Tag each record with the source label from Part A. Store the run ID and pipeline version from Part C in the release manifest. Record file checksums under the release ID so the files can be matched to the manifest later.

  • Test the trace before release by picking a handful of records at random and walking each back to the source system.
  • Record the test in the questionnaire with the date and who ran it.
  • Where a record cannot be traced, for example because it came from a deleted source, label it rather than drop the label.

When is a source too poorly documented to include?

Stop when you cannot name the system instance or extraction run behind a set of records; nothing downstream can be traced without them. Narrow the scope when part of the history comes from a source you cannot describe as well as the rest.

At a glance

When is a source too poorly documented to include

  • Narrow the scope to the period after a migration if the predecessor system's configuration is unknown.
  • Exclude fields whose meaning changed and cannot be dated, or deliver them with an explicit warning.
  • Request legal review when the permission reference for a source depends on customer contracts, employee policies or consent records.
  • Request privacy review when free-text fields or attachments may contain personal or sensitive information about individuals.
  • Request technical review when the extraction relies on several endpoints whose coverage overlaps or leaves gaps.
  • Recheck the questionnaire whenever the source system is reconfigured, migrated or retired, and before each new release.

How the questionnaire shortens a licensing assessment

Part A of the questionnaire also answers the first question in any licensing conversation: is there enough consistent history in one system to be worth preparing? SourceX's general guide sets the bar Part A can be checked against: buyers tend to look for two or more years of consistent history and tens of thousands of records within a single system, though rare work can qualify on smaller sets, and each case is assessed on its own.

From there, SourceX's process moves through qualification, rights review, contracting, preparation, delivery and payment, and it opens with a brief look at the company that requires no uploaded data. A system administrator who has already filled in Part A will have most of the answers that assessment asks for close at hand.

Frequently asked questions

What is source system metadata?

It is the description of the system a dataset came from: the product and instance, its configuration and custom fields, the history it keeps, and how records were extracted and transformed. It lets someone outside the company interpret the fields correctly and trace records back to their origin.

Is a data dictionary enough?

A data dictionary defines fields, which is necessary but not sufficient. It usually does not record when fields were added or retired, which values were set by automation, what change history exists or how the export was filtered, and those details change how a buyer reads the data.

Who in the company should answer the questionnaire?

The system administrator answers Part A and most of Part B, the person who ran the extraction answers Part C, and whoever holds the contracts and policies supplies the permission references. A data owner should review the whole document before it goes out.

Should source metadata include personal data?

No. The questionnaire describes systems, fields and processes, not individual records. Use illustrative values or aggregate counts, and keep sample exports free of personal information.

How often should the questionnaire be updated?

Update it whenever the source system is reconfigured, migrated or retired, and before each new release drawn from that system. Keep earlier versions with the releases they described.

Sources

  • Article 10(2)(b) of the EU AI Act requires data governance practices for high-risk AI training, validation and testing data to cover data collection processes and the origin of data, and for personal data the original purpose of collection. Source
  • The NIST AI RMF Playbook asks how an organization has documented an AI system's data provenance, including sources, origins, transformations, augmentations, labels, dependencies, constraints and metadata. Source
  • Datasheets for Datasets proposes documentation covering motivation, composition, collection process, preprocessing, distribution and maintenance, including how updates will be communicated to dataset consumers. Source
  • The Croissant RAI vocabulary defines properties including rai:dataCollection, rai:dataCollectionType, rai:dataCollectionTimeframe and rai:dataReleaseMaintenancePlan. Source
  • The Data & Trust Alliance Data Provenance Standards define 22 metadata fields grouped into three standards: Source, Provenance and Use. Source
  • Salesforce retains field history for up to 18 months, and up to 24 months through the API, when Field Audit Trail is not enabled. Source
  • HubSpot property history shows historical values for a record's properties with the date and time of each change and its source. Source
  • In Zendesk, each ticket audit represents a single update to a ticket and can contain several events; archived tickets are not included in the account-wide audit list. Source
  • OpenLineage's dataSource dataset facet records a data source by name and URI. Source
  • OpenLineage's column lineage facet maps each output field to the input fields used to produce it, with a transformation description and type such as IDENTITY or MASKED. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies