Privacy and preparation
Dataset documentation quality study: testing cards in procurement reviews
By SourceX Editorial · Updated
10 sections
Short answer
A study can assess dataset cards in real procurement reviews by giving experienced license reviewers dataset packets with expert-verified facts, varying only the documentation (no card, a free-form card or a structured procurement card), and grading their verdicts and flagged issues against those facts while reviewers and graders stay blind to condition. This page is the protocol and scoring rubric; it reports no results.
Provenance trail
Dataset documentation
Key takeaways
- Measure the decision, not the document: grade reviewers' license verdicts against facts an expert panel has verified.
- Compare a structured procurement card with both no card and a free-form card, or the study cannot say which part helped.
- Keep reviewers blind to the hypothesis and graders blind to condition, and report grader agreement before adjudication.
- Treat faster, more confident reviews without higher accuracy as a warning sign, not a win.
- The study cannot show that a card is truthful or that the rights it describes exist; only the agreement establishes rights.
How could a study assess dataset cards in real procurement reviews?
Run a blinded, within-reviewer experiment: give real procurement reviewers dataset packets that contain known, expert-verified facts, vary only the documentation they receive, and grade whether their accept, condition or reject decision and the issues they flag match the verified facts. The card is useful if it raises decision accuracy and issue detection without slowing reviewers down or making them overconfident.
That framing matters because most of what is written about dataset cards measures the card itself: whether sections exist, how long they are, how often a field is filled. Those are properties of documents. A procurement review is a decision made by a person under time pressure, and a card earns its place only if it changes that decision for the better.
This page is a protocol. It describes the question, population, sampling, conditions, rubric, outcome measures, analysis plan and limits of a study that has not been run. No results are reported here, and every count in a worked example is illustrative.
Which decision should the study inform, and what would change it?
The study should inform one choice: whether a buyer or supplier should require a structured dataset card, with fixed procurement fields, as part of every licensed dataset packet, rather than accepting free-form documentation or none. If the structured card does not beat a free-form card on accuracy or detection, the extra authoring work is hard to justify.
The documentation formats being compared are not new. Datasheets for Datasets proposed that every dataset come with a datasheet covering its motivation, composition, collection process and recommended uses. Google's Data Cards Playbook describes data cards as structured summaries of essential facts that stakeholders need across a dataset's lifecycle. On the Hugging Face Hub, a dataset card is the README.md file in the dataset repository, with YAML metadata such as license, language and size at the top.
What none of these formats settles is the procurement question. A reviewer deciding whether to license a support ticket archive needs to know who may grant the rights, what personal information was removed and how, which period and systems the records cover, and what uses are excluded. The study tests whether putting those answers into a fixed card changes what the reviewer decides.
Who reviews which packets: the population and the sampling frame
The population is people who approve or reject dataset licenses as part of their job: procurement and vendor management staff, in-house counsel, privacy officers, and data or ML leads who sign off on training data. Results apply to that population only, not to students, crowdworkers or the general public.
Recruit through professional networks and partner organizations, record each reviewer's role, years of experience and number of dataset reviews in the past year, and stratify assignment by role so no condition is filled mostly by lawyers or mostly by engineers. Exclude anyone who helped write a packet or a card used in the study.
Packets are the second sampling unit. Each packet represents one dataset offered for license and includes a sample of records, the license or term sheet, a data dictionary and, depending on condition, a card. Build packets from datasets whose facts an expert panel can verify independently: rights-cleared business datasets used with the owner's permission, or public datasets whose license and composition are documented at the source. Each packet contains a set of seeded facts the panel has confirmed, some benign and some that should block or condition approval.
| Sampling unit | Inclusion rule | Recorded variables | Why it matters |
|---|---|---|---|
| Reviewer | Approved or rejected at least one dataset license in the past two years | Role, years in role, reviews per year, organization type | Role and experience are the strongest expected moderators |
| Packet | Underlying dataset facts verifiable by an independent expert panel | Record type, source systems, period covered, size band, seeded issues | Seeded facts are the gold standard the verdicts are graded against |
| Seeded issue | Confirmed by two panel members working separately | Issue class, severity, where in the packet the evidence sits | Lets the study measure detection, not just overall verdicts |
| Card | Written by authors who did not build the seeded issues | Template used, rubric score, word count | Separates the effect of card format from card quality |
The blinded reviewer protocol, from seeded packet to graded verdict
Each reviewer sees several packets, each in exactly one documentation condition, in an order balanced across reviewers. Comparing conditions within the same reviewer removes much of the person-to-person variation that would otherwise swamp the effect.
At a glance
The blinded reviewer protocol, from seeded packet to graded verdict
The three conditions are: A, the packet with no card; B, the packet with a free-form card written to the Hugging Face dataset card template; and C, the packet with a structured procurement card scored against the rubric below. The Hugging Face template matters as a realistic baseline because its default placeholder for unfilled fields is the literal text More Information Needed, so a free-form card can look complete while leaving key answers blank.
- Panel step: two experts document every packet's true facts and seeded issues, then agree on a gold verdict (accept, accept with conditions, reject) and the conditions a correct approval would attach.
- Authoring step: separate card authors, who never see the seeded-issue list, write the B and C cards from the same underlying materials.
- Assignment step: a Latin square assigns packet and condition to each reviewer so that every packet appears in every condition across the study and no reviewer sees the same packet twice.
- Review step: reviewers work in a logged environment, record a verdict, list every issue they would raise, state the conditions they would require and give a confidence rating from 0 to 100.
- Blinding step: reviewers are told the study concerns dataset review practice, not documentation formats; graders see verdicts and issue lists with the condition label removed.
- Grading step: two graders match each flagged issue to the seeded list or mark it as a false alarm; disagreements go to a third grader, and agreement is reported before adjudication.
The dataset card scoring rubric: eight procurement dimensions
The rubric scores a card on the eight things a procurement reviewer has to establish, each from 0 (absent or placeholder text) to 3 (specific, sourced and checkable). It is used two ways: to describe the cards in conditions B and C, and as an exposure variable, so the analysis can ask whether higher-scoring cards produce better decisions regardless of template.
The dimensions draw on established documentation frameworks. The Croissant RAI vocabulary, for instance, gives fields for data collection, collection time frame, use cases, limitations and personal or sensitive information. The rubric adds the rights and authority questions a license review cannot skip.
| Dimension | Scores 0 when | Scores 3 when | What the grader checks |
|---|---|---|---|
| Source and authority | No named source organization or system | Names the entity that holds the records and the person who approved release | Entity name matches the license grantor |
| Rights and license terms | License field empty or generic | States the license or agreement, permitted uses and excluded uses | Terms match the agreement in the packet |
| Collection and period | No dates or systems | Gives systems of record, start and end dates and any gaps | Dates match a sample of record timestamps |
| Composition and coverage | Only a total count | Counts by record type, field list, missing-field rates | Counts reproduce from the sample within stated rounding |
| Personal and sensitive information | Silent or says none without method | Names categories present, removal method and residual risk | Method described is visible in the sample |
| Processing and transformations | No processing described | Lists every transformation in order, with tool or rule | A grader can repeat one step on the sample |
| Intended and excluded use | No uses stated | Names intended uses and out-of-scope uses with reasons | Excluded uses align with license restrictions |
| Version and contact | No version or owner | Version identifier, change log, named contact for questions | Version matches the files in the packet |
Outcome measures and how each one is graded
The primary outcome is decision accuracy: whether the reviewer's verdict matches the panel's gold verdict. Everything else is secondary and is named before data collection so it cannot be chosen after seeing which outcome looks best.
Detection recall is the share of seeded blocking issues a reviewer flags; false alarms are issues flagged that the panel confirmed do not exist. Time is measured from opening the packet to submitting the verdict. Calibration compares stated confidence with actual accuracy, because a card that makes reviewers faster and more confident but no more accurate is a risk, not a benefit.
| Measure | Definition | Unit of analysis | Role in the study |
|---|---|---|---|
| Decision accuracy | Verdict equals the panel's gold verdict | Reviewer by packet | Primary outcome |
| Blocking-issue recall | Seeded blocking issues flagged divided by seeded blocking issues present | Reviewer by packet | Key secondary outcome |
| False alarm count | Flagged issues the panel confirmed are not present | Reviewer by packet | Secondary; checks for noise |
| Condition completeness | Required approval conditions the reviewer attached | Reviewer by packet | Secondary; tests conditional approvals |
| Time to verdict | Minutes from packet open to submission, from logs | Reviewer by packet | Secondary; cost of the card |
| Calibration gap | Mean confidence minus accuracy, by condition | Condition | Secondary; overconfidence check |
| Follow-up questions | Questions the reviewer says they would send the supplier | Reviewer by packet | Exploratory; proxy for missing information |
How many reviewers, which baselines, and how wide an interval?
A defensible result compares condition C against both baselines, reports the difference with an interval rather than a single figure, and accounts for the fact that verdicts from the same reviewer and on the same packet are not independent. A mixed-effects logistic model with random intercepts for reviewer and packet handles that structure; a cluster bootstrap that resamples reviewers is a reasonable check.
Sample size should come from a power calculation run before recruitment, using a smallest effect worth acting on that the team agrees in advance, for example a difference in accuracy large enough to justify requiring structured cards on every deal. A pilot with a handful of reviewers estimates the baseline accuracy and the reviewer-level variance the calculation needs; pilot data is not analysed as a result.
Two baselines guard against easy wins. Condition A shows what reviewers achieve from raw materials alone, and condition B shows whether any card helps. If C beats A but not B, the finding is that documentation helps, not that the structured format does. Report every pre-registered outcome, including the ones that show no difference.
For an illustration of the reporting shape only: with an illustrative 24 reviewers each judging 9 packets, the analysis would use 216 reviewer-packet verdicts, report accuracy by condition with 95 percent intervals, and show recall separately for each issue class. Those counts are not a recommendation; the power calculation decides them.
Which conclusions would this design not support?
The design would not show that dataset cards make datasets better, safer or more lawful. It measures whether reviewers decide more accurately with a given card, which depends on the card being truthful. A polished card that misstates the license would likely make reviewers worse, and the study does not test cards written to mislead.
It also would not generalize far beyond its packets and people. Results on support tickets and invoices may not hold for medical images or source code, and results from experienced reviewers may not hold for first-time buyers. Seeded issues are a simplification: real defects are often subtler and spread across documents.
Finally, a laboratory review lacks the stakes and back-and-forth of a real negotiation, where reviewers can ask the supplier questions. The follow-up question measure is a partial proxy, not a substitute. Earlier documentation research mostly analysed cards at scale; for example, a study of Hugging Face dataset cards examined how sections were used across thousands of repositories. Hugging Face's model card user study asked participants to rank card sections as readers and as authors, which captures preference rather than decision accuracy. Those corpus and preference studies and this reviewer experiment answer different questions and should not be cited as if one confirms the other.
Pre-registration checklist for a dataset card review study
Publish the following before the first reviewer opens a packet, with a version and date, and do not change it without recording the change and the reason.
At a glance
Pre-registration checklist for a dataset card review study
Checklist · 10 items
- Research question and the single primary outcome, decision accuracy against the gold verdict.
- Population definition, recruitment channels, inclusion and exclusion rules, and the role strata.
- Packet construction rules, the list of seeded issue classes, and how the expert panel confirms each one.
- Card authoring rules for conditions B and C, including the template each uses and the authors' separation from the seeded-issue list.
- Assignment design, including the Latin square and how dropouts are handled.
- Blinding procedures for reviewers and graders, and how unblinding will be detected and reported.
- Grading rules, the agreement statistic reported before adjudication, and the adjudication procedure.
- The analysis model, the smallest effect of interest, the power calculation and the planned intervals.
- Every secondary and exploratory outcome, labelled as such.
- Data handling: what reviewer data is kept, how packets containing business records are secured and destroyed, and who may see raw responses.
0 of 10 ticked. Tick each item as you go.
Scoring your own card before a licensing review
A supplier does not need to wait for the study to use its rubric. Scoring your own card before a buyer review shows which dimensions are placeholders, and the rights and authority row is usually the one that blocks approval. Answering it requires knowing who holds the records and who may grant a license, which is a different question from who can export them.
SourceX's part in licensing records directly from a company covers rights review, contracting and data preparation. A company approves scope and terms and signs before any data is delivered, so a well-scored card is part of preparation rather than a substitute for that agreement. To learn whether it qualifies, a company completes a short assessment and uploads nothing.
Frequently asked questions
What is a dataset documentation quality study?
It is a study that measures whether documentation about a dataset helps people make better decisions about it. The design here grades procurement reviewers' license verdicts against expert-verified facts, comparing packets with no card, a free-form card and a structured card.
Why blind the reviewers in a dataset card study?
Reviewers who know the study tests documentation formats may read cards more carefully than they would at work, which inflates the effect. Telling them the study concerns review practice, and removing condition labels from what graders see, reduces that bias.
Can a dataset card replace a data license agreement?
No. A card describes the data and can summarize terms, but rights come from the agreement signed by the party entitled to grant them. A reviewer still needs the agreement itself, and the rubric checks that the card matches it.
How many reviewers does a dataset card study need?
It depends on the smallest accuracy difference worth acting on and on how much reviewers vary, so the number should come from a power calculation using pilot data. Each reviewer judging several packets in different conditions makes the study far more efficient than one verdict per person.
Are there results from this study yet?
No. This page publishes the protocol only. Results would require recruited reviewers, verified packets and a pre-registered analysis, and any counts shown in examples are illustrative.
Sources
- Datasheets for Datasets proposes that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses. Source
- Google's Data Cards Playbook describes data cards as structured summaries of essential facts about ML datasets needed by stakeholders across the dataset lifecycle. Source
- Hugging Face documents that a dataset card is the README.md in a dataset repository and that YAML metadata records license, language, size and tags. Source
- The Hugging Face dataset card template's sections, and its default More Information Needed placeholder for fields left blank. Source
- The Croissant RAI specification defines responsible AI properties such as rai:dataCollection, rai:dataCollectionTimeFrame, rai:dataUseCases, rai:dataLimitations and rai:personalSensitiveInformation. Source
- Hugging Face's model card user study asked participants to rank card sections from the perspective of readers and of authors. Source
- Yang, Liang and Zou analysed dataset cards across the datasets hosted on the Hugging Face Hub as of March 2023. Source
See if your company qualifies
A short company assessment. No data uploads are needed.