Rights and contracts
Dataset rights engineering handoff: legal calls a pipeline can run
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A dataset rights engineering handoff works when every legal decision reaches the pipeline as a versioned log row naming the records, the requested use, the action, its conditions, the governing document, the decider and a test that proves it ran. Fedora Legal's public license records show why: of 661 decisions its generated rpmlint list treats as valid, 120 carry a category limit or usage note that the list cannot hold.
License terms
Dataset rights engineering
Key takeaways
- An engineer can act on a rights decision only when it names the records, the action, the conditions, the governing document, the decider and a test.
- Fedora's public license data holds 829 decisions; 113 allow a license only for named categories of material and 29 carry written usage notes.
- Fedora's generated rpmlint list treats 661 decisions as valid, and 120 of them lose a category limit or usage note in that translation.
- Fedora's CC0-1.0 record changed its decision content six times between July 2022 and March 2025, so a log needs versions and reruns.
- Records no decision covers should be held, not passed, and every exclusion should run before any preparation.
What does an engineer need from a rights decision?
Eight fields, not a memo. An engineer can carry out a rights decision only when it names the records it covers, the use it was decided for, the action to take, any condition on that action, the document and version that govern it, who decided and when, and the test that will prove the extract obeyed it, all under one decision ID with a version.
A sentence such as "cleared, except accounts on the reseller paper" settles a legal question and opens three engineering ones: which rows are those accounts, what happens to their attachments, and how will anyone show later that none slipped through? The dataset rights engineering handoff is the step where counsel's conclusion about which records may be licensed for a stated use becomes a filter, a transformation or a hold inside the extraction job.
A useful answer rests on four kinds of evidence: an inventory of the exact record sets and the uses requested for them, the documents that govern each set, a named decider for every open authority or scope question, and a run record showing what each rule did. Qualified counsel settles authority and scope. The log carries those decisions to the people who build the extract; it does not make them, and nothing on this page replaces that legal judgment.
Where does a dataset rights engineering handoff lose decisions?
In three places: a condition left in prose, a limited yes that turns into a plain yes, and a revision that never reaches a job that already ran. Each failure is ordinary, and each one produces an extract that looks clean.
Prose conditions fail because a pipeline cannot read them. "Exclude the reseller accounts" assumes a field that marks reseller accounts. If the help desk has no such field, an engineer improvises one from company names or email domains, and the improvised rule becomes the real decision without anyone approving it.
Flattening fails more quietly. Counsel approves support tickets for model evaluation but not training, or approves only records created before a contract changed. A catalog with one licensable flag has nowhere to put either limit, so the flag says yes and the limit lives in an email thread.
Revisions fail on timing. Rights reviews keep going after the first extract, because documents keep surfacing. If the log carries no version and the job does not record which version it applied, a narrowed decision changes nothing that was already built.
What do 829 public license decisions show about flattening?
They show that every tool reading a decision keeps only what its own format can hold. Fedora Legal records its license decisions as one TOML file per license in a public repository, a format its README says was "agreed as a simplest format for even non-technical contributors." At commit c07f8dc of 20 May 2026, the last data commit on GitLab before the project moved to Fedora's own forge, the repository held 829 decision files.
The records are not yes-or-no. Fedora's license approval guide explains that the project "applies different license approval criteria for different categories of material: code, documentation, content, fonts, and firmware." Counting the status field across all 829 files, 548 decisions allow a license for any material, 113 allow it only for named categories (52 list content, 51 documentation, 31 fonts and 1 firmware, some more than one) and 168 do not allow it. Separately, 29 records carry a written usage note and 11 name individual packages granted an exception.
The same repository generates files that tools read. Its rpmlint generator writes flat lists of valid license identifiers: 661 decisions qualify, every record whose statuses are all allowed or category-limited, and 120 of them carry a category limit or a usage note that a flat list cannot express (113 category-limited, 12 with notes, 5 with both). Its ScanCode policy generator keeps a third label, Allowed under conditions, for the 113 category-limited decisions, but not which category, and it labels the 7 fully allowed decisions that carry usage notes as plain allowed licenses.
Fedora does not hide this. Its guide to spec file license tags treats a category-limited license as acceptable only when, "in this package, it only covers that category," a judgment the packager makes and no list can. The lesson for a data team is direct: for every condition in your log, decide which tool or which person checks it, because a format that cannot hold a condition will pass the record without it.
The table shows what each format keeps of the 829 decisions:
| Decision element (count) | TOML record | ScanCode policy | rpmlint valid list |
|---|---|---|---|
| Allowed for any material (548) | Kept | Allowed License | Listed as valid |
| Allowed for named categories only (113) | Kept, with the categories | Flag only: Allowed under conditions, category not named | Lost: listed as valid, category dropped |
| Not allowed (168) | Kept | Not-allowed License | Left off the list |
| Written usage notes (29) | Kept as text | Lost; 7 on allowed records read as plain allowed | Lost; 12 sit on listed decisions |
| Named package exceptions (11) | Kept as a list | Lost | Lost |
One record, six revisions: why the log needs versions
Because a decision that was right when the extract ran can change before the next delivery. Fedora's record for CC0-1.0, the Creative Commons public domain dedication, shows how often: after the status vocabulary took its current names on 5 May 2022, the file's decision content changed six times in under three years, counted by comparing the status, the usage note and the exception list in each committed version.
At a glance
One record, six revisions: why the log needs versions
Every revision moved the line between acceptable and unacceptable files: one narrowed the decision, three added conditions under which use stays allowed, and two named individual packages as exceptions. In a data deal the equivalent is counsel narrowing a scope after a customer contract surfaces, or carving one account back in once its consent arrives.
A log row therefore needs an effective date and a version, the extraction job needs to record which version it applied, and a changed row needs its selector rerun and the result compared with what was already prepared. Whether a change touches data already delivered is a question for the signed agreement and for counsel, never for a quiet patch to the pipeline.
The six changes, by commit date:
- 27 July 2022: status narrowed from allowed for any material to allowed for content only.
- 22 August 2022: usage note added, keeping code uses that existed in Fedora packages before 1 August 2022 allowed.
- 22 January 2023: usage note extended to trivial, noncreative files that upstream projects mark with CC0-1.0 to conform to the REUSE specification.
- 12 July 2023: usage note extended to post-quantum cryptography implementations included in the liboqs library as of that date.
- 20 November 2023: first named package exception added.
- 26 March 2025: four more package exceptions added, five in total.
The rights review log: eight fields an engineer can execute
Give every decision one row with eight fields, each filled by the person best placed to fill it, and treat a row with an empty field as a hold. The table names what counsel or the data owner writes, what engineering builds from it and what goes wrong when the field is missing.
Two habits keep the log usable. Write conditions as separate fields holding lists, never as clauses buried in the action, so a cut-off date or a named account list can be tested on its own. And link the governing document instead of paraphrasing it, because the next reviewer needs to reread the clause, not a summary of it:
| Field | Written by counsel or the data owner | Built by engineering | If it is missing |
|---|---|---|---|
| Decision ID and version | R-02, version 2, with an effective date | A tag on every record set the row touches; the version logged on each run | No one can show which rule a delivery followed |
| Record selector | System, object, date window, field values or an explicit ID list | A reviewed query or filter, with the row count it returns | Engineers infer the set from prose and the guess becomes the rule |
| Requested use | The use decided, matching the signed scope | A check against the use stated in the deal documents | A decision made for one use is applied to another |
| Action | Include, exclude, transform before inclusion, or hold | A pipeline step placed before any preparation | Cleared subject to conditions is read as include |
| Conditions | Categories, cut-off dates, carve-ins and named exceptions, each as a list | One testable predicate per condition | A limited yes ships as a plain yes |
| Governing document | Contract or notice, clause and version, linked rather than quoted | A stored reference rechecked when documents change | Nobody can recheck the decision when the document is renewed |
| Decider and date | The role that decided, and when | An approval gate before the row runs | Engineers become the deciders by default |
| Test and run record | What would show the row worked | Counts in and out, absence checks on excluded IDs, stored with the delivery | An exclusion is believed rather than shown |
Illustrative: a log for one help desk export
Here is how the fields read for one fictional export: a mid-sized software company's help desk tickets from 2019 to 2025, reviewed for licensing as AI training data. The rows, documents and account counts are invented to show the format and describe no real company or deal.
Notice what the log leaves out. It never says licensable on its own, the attachment row holds instead of guessing, and the last row turns every record no decision covers into a hold, so a new ticket form or a forgotten custom field cannot reach the extract by default:
| Row | Record selector | Action and conditions | Governing document | Test |
|---|---|---|---|---|
| R-01 v1 | Tickets created 2019-01-01 to 2025-12-31 | Include, subject to the rows below | Customer terms v4 and counsel note of 12 October | Row count matches the inventory |
| R-02 v2 | Tickets from the 14 accounts on reseller paper, by account ID list | Exclude; version 2 added three accounts | Reseller agreement, confidentiality clause | None of the 14 IDs in the extract |
| R-03 v1 | All ticket attachments | Hold until third-party content is reviewed | None yet; review open | No attachment files in the extract |
| R-04 v1 | Ticket bodies containing pasted vendor knowledge-base text | Transform: remove the passage; exclude the ticket if removal fails | Vendor support terms | Reviewer checks a sample of flagged tickets |
| R-05 v1 | Internal notes written by the HR group | Exclude | Employee notice and HR policy | HR group ID absent from the extract |
| R-06 v1 | Records that match no row | Hold by default | Not applicable | Unmatched count is zero before delivery |
Why not let engineers apply the filters they already have?
Because those filters answer a different question. Admin access shows what engineers can reach, and a redaction step changes what records reveal about people; neither shows that the company may license the records for the use requested, which is the only question the log exists to carry.
The strongest form of the objection is that a careful engineer will notice the obvious exclusions anyway. Sometimes that holds, and for a dataset with one governing document and no conditions, a short log is enough. But the Fedora counts show where judgment by inspection breaks: 113 of the 829 records, about one in seven, carry a category limit, and conditions such as cut-off dates, carve-ins and named exceptions cannot be seen in the data itself. An engineer reading a ticket cannot see the reseller agreement behind it.
The opposite objection is that engineers should stay out of legal reasoning entirely. The log does not ask them in. It asks counsel to state decisions in terms of records, and it asks engineers to say when a selector cannot be built from the fields that exist, which is the most useful question an engineer can send back.
Carrying exclusions from the log into the delivery
Run the log before any preparation, record what each row did, and keep the excluded identifiers with the data owner. An exclusion applied after de-identification or sampling has already let the records into working copies that are harder to trace and harder to purge.
What to tell a recipient about the exclusions is a separate disclosure question, covered in the exclusion log guide linked on this page. The engineering duties are these:
- Apply every exclude and hold row at extraction, before de-identification, sampling or format conversion, so excluded records never enter a working copy.
- Write a run record per delivery: the log version, the rows applied, records in and out per row, and the count of records no row covered.
- Keep the identifiers of excluded records with the data owner, outside the delivered files, so a later question can be answered without reproducing the material.
- When a row changes version, rerun its selector against what was prepared and report the difference to the decider before anything else moves.
- Fail closed: a new system, field or record type enters the extract only after a row covers it.
How the log fits a SourceX assessment
It answers in advance the questions SourceX's process asks. On its trust page, SourceX says the supplier chooses which systems, records and date ranges are in scope and everything outside them stays out, and that its rights review checks whether the supplier owns the original records and may license them, customer and vendor contract limits included. The supplier confirms those rights in the agreement, and each dataset carries documentation of its source, rights, preparation and permitted use.
The exact dataset is fixed in a Dataset Deal Confirmation that both sides sign before SourceX presents the opportunity externally, so a log whose selectors match that description keeps the signed scope and the extract in agreement. Records the company has already licensed for AI training are not eligible, which makes any earlier grant one of the first rows to write. SourceX says it does not give legal advice and that uncertain questions belong with the company's own counsel before a deal proceeds.
A first step needs no data. The company assessment asks about systems, years of history and what can be exported, not for files, and it is a check of fit rather than an offer or a promise of any payout.
Decision rule, limits and how the Fedora counts were made
The decision rule: hand a row to engineering only when all eight fields are filled and its test is defined, and hold every record set without such a row. A row with an open authority or scope question stays a hold until counsel closes it, however confident the team feels.
Limits: Fedora's decisions concern public licenses applied by one review group, while business records turn on private contracts, notices and several deciders. The counts therefore illustrate how formats keep or drop conditions, not how often data teams lose them; that rate has not been measured. The help desk log is illustrative.
Method: a script read every file under data/ ending in .toml at commit c07f8dcad5b5b280ab2c0782054ee14bd7927119 of gitlab.com/fedora/legal/fedora-license-data from git objects, decoded each as UTF-8 with any byte-order mark removed (none had one) and parsed all 829 without failure. Status values were compared exactly as written, and none fell outside the six-value vocabulary. A record counts as allowed when its status list contains allowed, as category-limited when every value is a category status, and as not allowed when the list is exactly not-allowed. A usage note counts when the field is non-empty after trimming; an exception list counts when it names at least one package. The rpmlint and ScanCode figures re-apply, in the article's own script, the selection rules of tools/mkrpmlint.py and tools/generate-scancode-toolkit-policy.py at the same commit; the repository's code was not run.
The CC0-1.0 history follows the file across its 2022 move from approved/ to data/ with git's rename detection and compares the status list (the older approved field where no status field exists), the usage note with whitespace collapsed, and the exception list between consecutive versions. Changes on or before 5 May 2022, when the vocabulary was renamed, are treated as format migration, and a full-history log of the path, side branches included, shows no other content commit after that date. Decisions recorded on Fedora's new forge after 20 May 2026 are not counted.
Frequently asked questions
What is a dataset rights engineering handoff?
It is the step where counsel's decision about which records may be licensed for a stated use becomes a filter, a transformation or a hold in the extraction job. It works when each decision arrives as a log row naming the records, the action, its conditions, the governing document, the decider and a test. A memo on its own leaves engineers to guess.
Who should own the rights review log, legal or engineering?
Both write it, in different fields. Counsel or the data owner writes the decision, the conditions, the governing document and the decider; engineering writes the selector query, the pipeline step and the run record. One named owner, usually on the data owner's side, keeps the versions straight.
How specific does a record selector need to be?
Specific enough that two engineers would return the same records: a system, an object, a date window and field values or an explicit ID list. If the selector depends on a field that does not exist, the row goes back to counsel instead of being approximated with names or email domains.
What happens when counsel changes a decision after data was extracted?
The row gets a new version, its selector is rerun against everything already prepared, and the difference goes to the decider. Whether the change affects anything already delivered depends on the signed agreement, so counsel decides that, not the pipeline.
Can a license field in our data catalog replace the log?
Usually not on its own, because a single field flattens conditions. In Fedora's public license data, 120 of the 661 decisions its generated rpmlint list treats as valid carry a category limit or usage note the list cannot express. Keep the field for search and the log for decisions.
Does removing personal data make a record safe to include?
No. Redaction changes what a record reveals about people, not whether the company may license it. Customer contracts, vendor terms, third-party content and earlier licenses still need a decision, and each decision becomes its own row in the log.
Sources
- Fedora License Data held 829 license decision files under data/ at commit c07f8dc (20 May 2026); counted by the article's script: 548 allowed, 113 allowed only for named categories, 168 not allowed, 29 with usage notes, 11 with named package exceptions. Source
- Fedora License Data's README says its TOML source format was agreed as the simplest format for non-technical contributors and that the Fedora Legal team is responsible for the project. Source
- Fedora's rpmlint generator lists every license whose statuses are all allowed or category-limited in flat ValidLicenses lists, with no field for the category or the usage note. Source
- Fedora's ScanCode policy generator labels each license Allowed License, Not-allowed License or Allowed under conditions, without the category or usage note. Source
- Fedora's CC0-1.0 record was narrowed to allowed-content on 27 July 2022 and later gained usage conditions (22 August 2022, 22 January 2023, 12 July 2023) and named package exceptions (20 November 2023, 26 March 2025). Source
- Fedora's license approval guide applies different approval criteria to code, documentation, content, fonts and firmware, and keeps CC0-1.0 as allowed-content after it stopped being allowed for code. Source
- Fedora's guide to spec file license tags treats a category-limited license as acceptable in a package only when, in that package, it covers only that category of material. Source
- SourceX says the supplier decides which systems, records and date ranges are in scope, that its rights review covers ownership and customer or vendor contract limits, and that source, rights, preparation and permitted use are documented with each dataset. Source
- SourceX fixes the exact dataset in a Dataset Deal Confirmation signed before it presents the opportunity externally, and the first assessment asks about systems, history and exports without records. Source
- SourceX says it does not give legal advice and that companies should involve their own counsel for anything uncertain before a deal proceeds. Source
- SourceX's guide lists data already licensed for AI training as not eligible for a SourceX license. Source
See if your company qualifies
A short company assessment. No data uploads are needed.