Skip to content

Privacy and preparation

Dataset documentation quality study: testing cards in procurement reviews

By SourceX Editorial · Updated

10 sections

Short answer

A study can assess dataset cards in real procurement reviews by giving experienced license reviewers dataset packets with expert-verified facts, varying only the documentation (no card, a free-form card or a structured procurement card), and grading their verdicts and flagged issues against those facts while reviewers and graders stay blind to condition. This page is the protocol and scoring rubric; it reports no results.

Provenance trail

Dataset documentation

This page covers dataset documentation quality study: testing cards in procurement reviews.Illustrative

Key takeaways

  • Measure the decision, not the document: grade reviewers' license verdicts against facts an expert panel has verified.
  • Compare a structured procurement card with both no card and a free-form card, or the study cannot say which part helped.
  • Keep reviewers blind to the hypothesis and graders blind to condition, and report grader agreement before adjudication.
  • Treat faster, more confident reviews without higher accuracy as a warning sign, not a win.
  • The study cannot show that a card is truthful or that the rights it describes exist; only the agreement establishes rights.

How could a study assess dataset cards in real procurement reviews?

Run a blinded, within-reviewer experiment: give real procurement reviewers dataset packets that contain known, expert-verified facts, vary only the documentation they receive, and grade whether their accept, condition or reject decision and the issues they flag match the verified facts. The card is useful if it raises decision accuracy and issue detection without slowing reviewers down or making them overconfident.

That framing matters because most of what is written about dataset cards measures the card itself: whether sections exist, how long they are, how often a field is filled. Those are properties of documents. A procurement review is a decision made by a person under time pressure, and a card earns its place only if it changes that decision for the better.

This page is a protocol. It describes the question, population, sampling, conditions, rubric, outcome measures, analysis plan and limits of a study that has not been run. No results are reported here, and every count in a worked example is illustrative.

Which decision should the study inform, and what would change it?

The study should inform one choice: whether a buyer or supplier should require a structured dataset card, with fixed procurement fields, as part of every licensed dataset packet, rather than accepting free-form documentation or none. If the structured card does not beat a free-form card on accuracy or detection, the extra authoring work is hard to justify.

The documentation formats being compared are not new. Datasheets for Datasets proposed that every dataset come with a datasheet covering its motivation, composition, collection process and recommended uses. Google's Data Cards Playbook describes data cards as structured summaries of essential facts that stakeholders need across a dataset's lifecycle. On the Hugging Face Hub, a dataset card is the README.md file in the dataset repository, with YAML metadata such as license, language and size at the top.

What none of these formats settles is the procurement question. A reviewer deciding whether to license a support ticket archive needs to know who may grant the rights, what personal information was removed and how, which period and systems the records cover, and what uses are excluded. The study tests whether putting those answers into a fixed card changes what the reviewer decides.

Who reviews which packets: the population and the sampling frame

The population is people who approve or reject dataset licenses as part of their job: procurement and vendor management staff, in-house counsel, privacy officers, and data or ML leads who sign off on training data. Results apply to that population only, not to students, crowdworkers or the general public.

Recruit through professional networks and partner organizations, record each reviewer's role, years of experience and number of dataset reviews in the past year, and stratify assignment by role so no condition is filled mostly by lawyers or mostly by engineers. Exclude anyone who helped write a packet or a card used in the study.

Packets are the second sampling unit. Each packet represents one dataset offered for license and includes a sample of records, the license or term sheet, a data dictionary and, depending on condition, a card. Build packets from datasets whose facts an expert panel can verify independently: rights-cleared business datasets used with the owner's permission, or public datasets whose license and composition are documented at the source. Each packet contains a set of seeded facts the panel has confirmed, some benign and some that should block or condition approval.

Who reviews which packets: the population and the sampling frame
Sampling unitInclusion ruleRecorded variablesWhy it matters
ReviewerApproved or rejected at least one dataset license in the past two yearsRole, years in role, reviews per year, organization typeRole and experience are the strongest expected moderators
PacketUnderlying dataset facts verifiable by an independent expert panelRecord type, source systems, period covered, size band, seeded issuesSeeded facts are the gold standard the verdicts are graded against
Seeded issueConfirmed by two panel members working separatelyIssue class, severity, where in the packet the evidence sitsLets the study measure detection, not just overall verdicts
CardWritten by authors who did not build the seeded issuesTemplate used, rubric score, word countSeparates the effect of card format from card quality

The blinded reviewer protocol, from seeded packet to graded verdict

Each reviewer sees several packets, each in exactly one documentation condition, in an order balanced across reviewers. Comparing conditions within the same reviewer removes much of the person-to-person variation that would otherwise swamp the effect.

At a glance

The blinded reviewer protocol, from seeded packet to graded verdict

The three conditions are: A, the packet with no card; B, the packet with a free-form card written to the Hugging Face dataset card template; and C, the packet with a structured procurement card scored against the rubric below. The Hugging Face template matters as a realistic baseline because its default placeholder for unfilled fields is the literal text More Information Needed, so a free-form card can look complete while leaving key answers blank.

  • Panel step: two experts document every packet's true facts and seeded issues, then agree on a gold verdict (accept, accept with conditions, reject) and the conditions a correct approval would attach.
  • Authoring step: separate card authors, who never see the seeded-issue list, write the B and C cards from the same underlying materials.
  • Assignment step: a Latin square assigns packet and condition to each reviewer so that every packet appears in every condition across the study and no reviewer sees the same packet twice.
  • Review step: reviewers work in a logged environment, record a verdict, list every issue they would raise, state the conditions they would require and give a confidence rating from 0 to 100.
  • Blinding step: reviewers are told the study concerns dataset review practice, not documentation formats; graders see verdicts and issue lists with the condition label removed.
  • Grading step: two graders match each flagged issue to the seeded list or mark it as a false alarm; disagreements go to a third grader, and agreement is reported before adjudication.

The dataset card scoring rubric: eight procurement dimensions

The rubric scores a card on the eight things a procurement reviewer has to establish, each from 0 (absent or placeholder text) to 3 (specific, sourced and checkable). It is used two ways: to describe the cards in conditions B and C, and as an exposure variable, so the analysis can ask whether higher-scoring cards produce better decisions regardless of template.

The dimensions draw on established documentation frameworks. The Croissant RAI vocabulary, for instance, gives fields for data collection, collection time frame, use cases, limitations and personal or sensitive information. The rubric adds the rights and authority questions a license review cannot skip.

The dataset card scoring rubric: eight procurement dimensions
DimensionScores 0 whenScores 3 whenWhat the grader checks
Source and authorityNo named source organization or systemNames the entity that holds the records and the person who approved releaseEntity name matches the license grantor
Rights and license termsLicense field empty or genericStates the license or agreement, permitted uses and excluded usesTerms match the agreement in the packet
Collection and periodNo dates or systemsGives systems of record, start and end dates and any gapsDates match a sample of record timestamps
Composition and coverageOnly a total countCounts by record type, field list, missing-field ratesCounts reproduce from the sample within stated rounding
Personal and sensitive informationSilent or says none without methodNames categories present, removal method and residual riskMethod described is visible in the sample
Processing and transformationsNo processing describedLists every transformation in order, with tool or ruleA grader can repeat one step on the sample
Intended and excluded useNo uses statedNames intended uses and out-of-scope uses with reasonsExcluded uses align with license restrictions
Version and contactNo version or ownerVersion identifier, change log, named contact for questionsVersion matches the files in the packet

Outcome measures and how each one is graded

The primary outcome is decision accuracy: whether the reviewer's verdict matches the panel's gold verdict. Everything else is secondary and is named before data collection so it cannot be chosen after seeing which outcome looks best.

Detection recall is the share of seeded blocking issues a reviewer flags; false alarms are issues flagged that the panel confirmed do not exist. Time is measured from opening the packet to submitting the verdict. Calibration compares stated confidence with actual accuracy, because a card that makes reviewers faster and more confident but no more accurate is a risk, not a benefit.

Outcome measures and how each one is graded
MeasureDefinitionUnit of analysisRole in the study
Decision accuracyVerdict equals the panel's gold verdictReviewer by packetPrimary outcome
Blocking-issue recallSeeded blocking issues flagged divided by seeded blocking issues presentReviewer by packetKey secondary outcome
False alarm countFlagged issues the panel confirmed are not presentReviewer by packetSecondary; checks for noise
Condition completenessRequired approval conditions the reviewer attachedReviewer by packetSecondary; tests conditional approvals
Time to verdictMinutes from packet open to submission, from logsReviewer by packetSecondary; cost of the card
Calibration gapMean confidence minus accuracy, by conditionConditionSecondary; overconfidence check
Follow-up questionsQuestions the reviewer says they would send the supplierReviewer by packetExploratory; proxy for missing information

How many reviewers, which baselines, and how wide an interval?

A defensible result compares condition C against both baselines, reports the difference with an interval rather than a single figure, and accounts for the fact that verdicts from the same reviewer and on the same packet are not independent. A mixed-effects logistic model with random intercepts for reviewer and packet handles that structure; a cluster bootstrap that resamples reviewers is a reasonable check.

Sample size should come from a power calculation run before recruitment, using a smallest effect worth acting on that the team agrees in advance, for example a difference in accuracy large enough to justify requiring structured cards on every deal. A pilot with a handful of reviewers estimates the baseline accuracy and the reviewer-level variance the calculation needs; pilot data is not analysed as a result.

Two baselines guard against easy wins. Condition A shows what reviewers achieve from raw materials alone, and condition B shows whether any card helps. If C beats A but not B, the finding is that documentation helps, not that the structured format does. Report every pre-registered outcome, including the ones that show no difference.

For an illustration of the reporting shape only: with an illustrative 24 reviewers each judging 9 packets, the analysis would use 216 reviewer-packet verdicts, report accuracy by condition with 95 percent intervals, and show recall separately for each issue class. Those counts are not a recommendation; the power calculation decides them.

Which conclusions would this design not support?

The design would not show that dataset cards make datasets better, safer or more lawful. It measures whether reviewers decide more accurately with a given card, which depends on the card being truthful. A polished card that misstates the license would likely make reviewers worse, and the study does not test cards written to mislead.

It also would not generalize far beyond its packets and people. Results on support tickets and invoices may not hold for medical images or source code, and results from experienced reviewers may not hold for first-time buyers. Seeded issues are a simplification: real defects are often subtler and spread across documents.

Finally, a laboratory review lacks the stakes and back-and-forth of a real negotiation, where reviewers can ask the supplier questions. The follow-up question measure is a partial proxy, not a substitute. Earlier documentation research mostly analysed cards at scale; for example, a study of Hugging Face dataset cards examined how sections were used across thousands of repositories. Hugging Face's model card user study asked participants to rank card sections as readers and as authors, which captures preference rather than decision accuracy. Those corpus and preference studies and this reviewer experiment answer different questions and should not be cited as if one confirms the other.

Pre-registration checklist for a dataset card review study

Publish the following before the first reviewer opens a packet, with a version and date, and do not change it without recording the change and the reason.

At a glance

Pre-registration checklist for a dataset card review study

Checklist · 10 items

  1. Research question and the single primary outcome, decision accuracy against the gold verdict.
  2. Population definition, recruitment channels, inclusion and exclusion rules, and the role strata.
  3. Packet construction rules, the list of seeded issue classes, and how the expert panel confirms each one.
  4. Card authoring rules for conditions B and C, including the template each uses and the authors' separation from the seeded-issue list.
  5. Assignment design, including the Latin square and how dropouts are handled.
  6. Blinding procedures for reviewers and graders, and how unblinding will be detected and reported.
  7. Grading rules, the agreement statistic reported before adjudication, and the adjudication procedure.
  8. The analysis model, the smallest effect of interest, the power calculation and the planned intervals.
  9. Every secondary and exploratory outcome, labelled as such.
  10. Data handling: what reviewer data is kept, how packets containing business records are secured and destroyed, and who may see raw responses.

0 of 10 ticked. Tick each item as you go.

Scoring your own card before a licensing review

A supplier does not need to wait for the study to use its rubric. Scoring your own card before a buyer review shows which dimensions are placeholders, and the rights and authority row is usually the one that blocks approval. Answering it requires knowing who holds the records and who may grant a license, which is a different question from who can export them.

SourceX's part in licensing records directly from a company covers rights review, contracting and data preparation. A company approves scope and terms and signs before any data is delivered, so a well-scored card is part of preparation rather than a substitute for that agreement. To learn whether it qualifies, a company completes a short assessment and uploads nothing.

Frequently asked questions

What is a dataset documentation quality study?

It is a study that measures whether documentation about a dataset helps people make better decisions about it. The design here grades procurement reviewers' license verdicts against expert-verified facts, comparing packets with no card, a free-form card and a structured card.

Why blind the reviewers in a dataset card study?

Reviewers who know the study tests documentation formats may read cards more carefully than they would at work, which inflates the effect. Telling them the study concerns review practice, and removing condition labels from what graders see, reduces that bias.

Can a dataset card replace a data license agreement?

No. A card describes the data and can summarize terms, but rights come from the agreement signed by the party entitled to grant them. A reviewer still needs the agreement itself, and the rubric checks that the card matches it.

How many reviewers does a dataset card study need?

It depends on the smallest accuracy difference worth acting on and on how much reviewers vary, so the number should come from a power calculation using pilot data. Each reviewer judging several packets in different conditions makes the study far more efficient than one verdict per person.

Are there results from this study yet?

No. This page publishes the protocol only. Results would require recruited reviewers, verified packets and a pre-registered analysis, and any counts shown in examples are illustrative.

Sources

  • Datasheets for Datasets proposes that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses. Source
  • Google's Data Cards Playbook describes data cards as structured summaries of essential facts about ML datasets needed by stakeholders across the dataset lifecycle. Source
  • Hugging Face documents that a dataset card is the README.md in a dataset repository and that YAML metadata records license, language, size and tags. Source
  • The Hugging Face dataset card template's sections, and its default More Information Needed placeholder for fields left blank. Source
  • The Croissant RAI specification defines responsible AI properties such as rai:dataCollection, rai:dataCollectionTimeFrame, rai:dataUseCases, rai:dataLimitations and rai:personalSensitiveInformation. Source
  • Hugging Face's model card user study asked participants to rank card sections from the perspective of readers and of authors. Source
  • Yang, Liang and Zou analysed dataset cards across the datasets hosted on the Hugging Face Hub as of March 2023. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies