Systems and records
Dataset approved source inventory: owners, scope and approvals per build
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
An approved source inventory lists every system, export or file a dataset build may read, with its named owner, the approved scope of fields, records and dates, who approved it and on what basis, when to recheck, the exact extract used, and which dataset versions and recipients contain its records. Enforced by the build, it traces any repair, refresh or withdrawal from one source to every affected release.
Provenance trail
Dataset approved source
Key takeaways
- Keep one row per source per build, versioned rather than overwritten, with owner, scope, approval basis and recheck date.
- Record access, privacy permission and licensing rights separately; a yes on one says nothing about the others.
- Make the pipeline read its inputs from register rows and fail closed on suspended or stale approvals.
- Write each release's versions and recipients back to every source row so a change becomes a lookup.
- Match the action and notice to the kind of change, from a routine refresh to a withdrawn approval.
What should an approved source inventory contain for a dataset build?
An approved source inventory lists every system, export or file the dataset build is allowed to read. Each row names the source owner, the approved scope (record types, fields, date range and exclusions), who approved it and under what agreement or policy, when the approval expires or must be rechecked, the exact extract the build used, and the dataset versions and recipients that contain records from it.
Kept beside the build and enforced by it, the register turns three hard questions into lookups. Is this source allowed in the build? Which released examples came from it? Who needs to be told if it changes? Without it, a repair or withdrawal becomes a search through pipeline code and old emails.
Documentation standards already ask for the description half of this. The SPDX 3 dataset profile has a dataCollectionProcess property for how a dataset was collected, including the sources it came from, and a datasetUpdateMechanism property for how it is updated. Those are free-text fields on the finished dataset. The register is the structured, per-source record that lets you fill them truthfully and keep them true.
Data inventory, source register or use register: which question are you answering?
Use the register when the question is what a specific build may read and where its records went. A company data inventory answers what systems and records exist; a data map answers how data flows; a training data use register answers what a received dataset may be used for downstream. Each is useful, and none of them tracks approvals per source per build.
The test is the question someone asks you on a bad day. If a source owner says a field should never have been exported, only a source-level register tells you which versions and recipients are affected.
| Record | Question it answers | Unit of a row | Who usually keeps it |
|---|---|---|---|
| Company data inventory | What systems and records do we hold? | A system or record type | IT, data governance or operations |
| Data map | Where does data flow between systems and parties? | A flow between two points | Privacy or security team |
| Approved source inventory | What may this build read, under whose approval, and where did it go? | One approved source per build | Dataset build owner |
| Training data use register | What may we do with a dataset we received? | A dataset and its permitted uses | The receiving AI team |
| Dataset card or datasheet | What is in the released dataset and how was it made? | One released version | Dataset publisher |
The source owner scope and approval register
Use one row per source per build, and treat a change to scope, owner or extract as a new row version rather than an edit. The fields below are meant to be used as is; the illustrative entries describe an invented support ticket source.
At a glance
The source owner scope and approval register
| Field | What to record | Why it matters | Illustrative entry |
|---|---|---|---|
| Source ID and name | Stable ID plus system and object | Every derived record and log line points here | SRC-014, help desk tickets |
| Source owner | Named role and person accountable for the system's data | The person who confirms scope and answers for corrections | Head of customer support |
| Approved scope | Record types, fields, date range, filters and exclusions | Access is usually wider than approval; the build must use the narrower one | Closed tickets, 2021 to 2024, body and tags, no attachments |
| Approval reference | Who approved, when, and the agreement, policy or decision it rests on | Shows the basis for use, not just that someone said yes | Signed by owner and counsel, linked to agreement schedule |
| Rights and privacy notes | Contract terms, notices and personal data handling that limit use | Access, privacy permission and licensing right are separate checks | Customer terms reviewed; personal data de-identified before export |
| Recheck date | When the approval expires or must be confirmed again | Approvals go stale when owners, contracts or systems change | Each new release, or after 12 months |
| Extract record | Query or export job, run time, row count and a checksum | Proves which snapshot a version used | Export job run, checksum stored with the build |
| Derived outputs | Task types and example ID ranges built from this source | Lets a source change be traced to affected examples | Ticket triage and reply drafting examples |
| Released in | Dataset versions and recipients holding records from this source | Tells you who must be notified of a fix or withdrawal | Versions 1.0 and 1.1, two recipients |
| Status and change log | Approved, narrowed, suspended or withdrawn, with dated reasons | Keeps history instead of overwriting it | Approved; narrowed once to drop a tag field |
Who may approve a source, and what does the approval cover?
The source owner confirms what the data is and what the system contains; approval to use it in a dataset usually needs more than the owner. Being able to export a table is access. Being allowed to process personal data in it for this purpose is a privacy question. Being allowed to license or deliver it to someone else is a contractual and ownership question. Record each separately, because a yes on one says nothing about the others, and de-identifying records does not by itself create a right to license them.
Regulation is moving in the same direction for some AI systems. Article 10 of the EU AI Act requires training, validation and testing data for high-risk systems to be subject to data governance practices that cover, among other things, data collection processes and the origin of data, and in the case of personal data, the original purpose of the data collection. A register row with origin, collection process and original purpose filled in is a practical way to hold that information, whether or not the Act applies to a given build.
Approve the narrowest scope that serves the build. If the task needs ticket bodies and resolution codes, approve those fields and exclude attachments and internal notes explicitly. A short exclusions list prevents the most common failure: a later pipeline change that quietly widens what is read because the source allowed it.
Making the build read only from approved rows
A register that the build does not read will drift within one release. Make the pipeline take its inputs from register rows: the job looks up the approved scope, refuses to run if the row is suspended or past its recheck date, and writes the extract record and derived example IDs back to the row when it finishes.
Lineage standards give you fields to hang this on. OpenLineage defines an ownership facet that records the owners of a dataset, with an owner identifier and an optional ownership type, and a dataset version facet that records the version of a dataset. Croissant recommends the W3C PROV ontology for provenance, using wasDerivedFrom for an entity derived from another, wasGeneratedBy for the activity that produced it, and wasAttributedTo for the responsible agent. Pointing these at register IDs links the lineage graph back to the approval behind each node.
Keep the checksum with the build output. When someone asks whether version 1.1 used the same tickets as version 1.0, the answer should be a comparison of two stored values, not a rerun.
- Build inputs are register IDs, never raw connection strings
- The job fails closed on a suspended, withdrawn or stale row
- Every derived example carries the source ID it came from
- Each release writes its version and recipient list back to every source row it used
Which released examples or recipients does a source change affect?
Start from the source ID and walk forward: derived examples, the versions that contain them, and the recipients who hold those versions. The register makes that a query. Then decide the action by the kind of change, record it as an issue, and close the issue only when every affected recipient has been reconciled.
The Croissant Responsible AI vocabulary includes a dataReleaseMaintenancePlan property for versioning information such as the updating timeframe, the maintainers and deprecation policies. The register is what lets you honor that plan for one source at a time instead of reissuing everything.
| Source change | Severity default | Action on the dataset | Notice to recipients |
|---|---|---|---|
| Correction of wrong values | Medium | Repair affected examples, rerun regression checks, issue a point version | Change notice with affected example IDs |
| Routine refresh with new records | Low | New extract under the same approval; new version | Release note |
| Owner handoff | Low unless scope is disputed | New owner reconfirms scope before the next build | None unless scope changes |
| Scope narrowed | High | Remove out-of-scope fields or records from future versions; assess held versions | Notice with the removed scope and replacement version |
| Approval withdrawn or found invalid | Highest | Stop builds, remove all derived examples, follow the agreement's withdrawal terms | Formal notice under the agreement; confirm deletion or replacement |
Worked example (illustrative): a ticket export withdrawn after two releases
This example is illustrative; the company and counts are invented. A software company builds a support dataset from three approved sources: help desk tickets, a knowledge base and product release notes. Version 1.0 goes to one recipient, version 1.1 to two. After the second release, the support lead finds that one ticket queue handled a client whose contract barred sharing their tickets, and that queue was never excluded.
The register makes the scope of the problem a lookup. The ticket source row shows the approved scope, the extract records for both versions, and the example ID ranges derived from it. Filtering by the client's queue identifies 340 affected examples out of roughly 9,000 in version 1.1, and the release log shows both recipients hold them.
The row is narrowed with a dated reason, the build reruns against the narrowed scope, regression checks confirm the other two sources are unchanged, and version 1.2 ships. Each recipient gets a notice listing the removed example IDs and the steps their agreement requires. The issue closes when both recipients confirm. The exception: had the barred queue been the only source for a task type, the right move would have been to retire that task from the dataset rather than ship a thinner version that no longer represents the work.
Sources that do not fit neatly in one row
Some sources resist a single row, and forcing them into one hides risk. A shared inbox that several teams use has more than one owner, so record each owner and the part of the scope they approve. A source derived from other sources, such as a reporting table, should carry the IDs of its upstream rows so a change upstream still traces through.
Manual uploads need rows too. A spreadsheet someone attached to a ticket is still a source; give it an ID, an owner and a scope, or exclude it. When the original owner has left the company, assign an interim owner before the next build, because an approval with no one to answer for it cannot be rechecked.
The Hugging Face dataset card template asks who the source data producers are, meaning the people or systems that originally created the data. A register with owners and derivation links answers that question per source rather than with a single sentence for the whole dataset.
Limits of the register, and when to bring in counsel or privacy review
The register records approvals; it does not grant them. If a row's rights notes depend on reading a customer contract, a privacy notice or a data processing agreement, counsel or the privacy team should review that row before it is marked approved, and again when the source's terms change.
Stop and narrow the build when no one can name an owner for a source, when the approval reference points to nothing written, or when a source contains categories of personal data the approved scope never mentioned. Those are signs that the build is reading more than anyone agreed to, and the fix is to shrink the scope rather than document the gap.
The expert reviewers for this page's method are a dataset operations owner and a release or incident reviewer: the first checks that the build actually enforces the rows, the second that withdrawal and notice steps work end to end.
Taking an approved source register into a licensing review
A company that keeps this register already holds most of what a licensing review asks for: who owns each source, what was approved and on what basis. Where SourceX is involved, the company approves scope and terms and signs before any data is delivered, and very large datasets can remain in the company's own storage or ship on encrypted drives. Finding out whether your company fits begins with a brief assessment, with no records uploaded.
Frequently asked questions
What is an approved source inventory for a dataset?
It is a register of every source a dataset build may read, with the owner, the approved scope, the approval basis, a recheck date, the extract used and the versions and recipients that contain its records. It lets a team trace any fix or withdrawal from a source to everything it affected.
How is an approved source inventory different from a data inventory?
A data inventory lists what systems and records a company holds. An approved source inventory lists what one build is allowed to read, under whose approval and with what limits, and where the resulting records went. The second is narrower and tied to releases.
Who should own the source inventory for a dataset build?
The dataset build owner should keep it, because the pipeline reads it and writes back to it. Each row still needs a named source owner for the underlying system, and rights or privacy notes should be reviewed by counsel or the privacy team.
What happens to released versions when a source approval is withdrawn?
Builds from that source stop, derived examples are removed from future versions, and recipients holding earlier versions are notified under the terms of their agreement. The register's release log tells you exactly which versions and recipients are affected.
Does data provenance metadata replace a source register?
No. Standards such as PROV, OpenLineage facets and SPDX dataset fields describe lineage and collection, but they do not record who approved each source, for what scope, or when that approval must be rechecked. Use them alongside the register, pointing at register IDs.
Sources
- The SPDX 3 dataset profile's dataCollectionProcess property describes how a dataset was collected, for example the sources it was scraped from. Source
- The SPDX 3 dataset profile's datasetUpdateMechanism property describes the mechanism used to update a dataset. Source
- Article 10 of the EU AI Act requires data governance for high-risk system training data covering collection processes, data origin and, for personal data, the original purpose of collection. Source
- OpenLineage's ownership dataset facet records a dataset's owners, each with an identifier and an optional ownership type. Source
- OpenLineage's dataset version facet records the version of a dataset. Source
- Croissant recommends the W3C PROV ontology for provenance, including wasDerivedFrom, wasGeneratedBy and wasAttributedTo. Source
- The Croissant Responsible AI vocabulary defines dataReleaseMaintenancePlan for updating timeframe, maintainers and deprecation policies. Source
- The Hugging Face dataset card template asks who the source data producers are, meaning the people or systems that originally created the data. Source
See if your company qualifies
A short company assessment. No data uploads are needed.