Privacy and preparation
Dataset label reliability report: measuring reviewer agreement
By SourceX Editorial · Updated
10 sections
Short answer
Measure reviewer agreement on a business dataset's labels by having two or more qualified reviewers label a stratified sample independently under one frozen guideline, then report chance-corrected agreement for each label field and class with an interval, adjudicate every disagreement with a recorded reason, and show what changed. A label reliability report packages that evidence so a buyer can check it rather than trust a single average.
License terms
Dataset label reliability
Key takeaways
- Measure agreement on a stratified, blinded sample drawn from the exact export you will deliver.
- Report agreement for each label field and class with an interval; an overall average hides weak fields.
- Every disagreement needs an adjudication entry with a reason code, and guideline fixes must reach the full dataset.
- Agreement shows consistency, not correctness, so pair it with checks against independent outcomes.
- Say plainly which fields were narrowed or left out; that makes the report more credible, not less.
How do you measure reviewer agreement on a business dataset's labels?
You measure it on a designed sample, not on the whole file and not on whatever items happened to get a second look. Pick a sample that covers every class, source system and time period, have at least two qualified reviewers label it without seeing each other's answers or the original label, and compute agreement for each label field separately.
Then treat the disagreements as the most useful output. Every one goes to an adjudicator who records the final value and why the reviewers split. The report a buyer receives shows the agreement numbers with their uncertainty, the adjudication log, the guideline version used, and exactly which delivered labels changed as a result.
One distinction shapes the whole report: agreement measures consistency, not correctness. Two reviewers who share the same misunderstanding of a category will agree perfectly and both be wrong. That is why the report pairs agreement with adjudication reasons and, where one exists, a check against an independent outcome such as a refund that was actually issued.
Which labels in an operational dataset deserve a second reviewer?
Not every field needs double review. Prioritize the labels a buyer will train or evaluate on, and the ones a person chose rather than a system computed. The origin of the label decides what agreement can tell you, so classify each label field before you sample anything.
| Label origin | Typical example | What agreement tells you | Review approach |
|---|---|---|---|
| Chosen by staff at the time of work | Ticket category or disposition code picked by a support agent in Zendesk or Salesforce | Whether the historical choice is reproducible by someone applying today's guideline | Two independent re-reviewers on a sample, original label hidden |
| Computed by a system rule | Order status set by an ERP workflow, SLA breach flag | Little; the rule either fired or not | Verify the rule and its version instead of measuring agreement |
| Added after the fact for AI use | Intent, sentiment or root-cause tags applied to past records | Whether the new label scheme is learnable and stable | Double-label a seeded sample from the start of the project |
| Derived from a later outcome | Churned, refunded, claim paid | Whether the outcome was captured consistently | Check against source documents, then sample for edge cases |
| Free-text judgment | Supervisor's quality score or written assessment | Whether scores mean the same thing across raters | Ordinal or interval agreement with weights; report rater by rater |
How should the double-review sample be drawn and sized?
Draw it from the exact export you plan to deliver, stratified so rare classes and every source system appear, and record the random seed. A purely random sample from a skewed file can contain only a handful of the rare classes a buyer cares about, which makes their agreement unknowable rather than high.
Oversample rare classes and weight them back when you report the overall figure, and say you did. Include each time period, because category definitions in a CRM often drift after a reorganization or a picklist change. Exclude nothing as too hard; difficult items are exactly where disagreement lives.
Size the sample from the precision you need on the weakest field, not from a round number. Krippendorff's bootstrapping guide shows how resampling the reliability data yields a confidence interval and the probability that agreement falls short of a chosen minimum, which tells you whether the sample was large enough to support the claim. Decide that minimum before reviewers start, write it in the report, and explain why it suits the buyer's use.
Freeze the labeling guideline and give it a version number before the sample is labeled. Reviewers should not see the original label, the other reviewer's answer, or any model prediction, and the roster should say who reviewed what and how each reviewer qualified.
Which agreement figures belong in the report, and at what level?
Report raw percent agreement and a chance-corrected statistic for each label field, never only an overall average. Chance correction matters because two reviewers who both pick the most common category most of the time will agree often by accident. The scikit-learn documentation defines Cohen's kappa as observed agreement minus expected chance agreement, divided by one minus expected agreement, and supports linear and quadratic weights for ordered labels.
When more than two reviewers label each item, or some items have missing labels, use a statistic built for that design. The statsmodels implementation of Fleiss' kappa notes that it provides no variance or hypothesis tests, so the report needs a bootstrap interval alongside it. Krippendorff's alpha, as described in Krippendorff's own computing guide, handles any number of observers and missing data, which suits real projects where not every reviewer saw every item.
Break the figure down by field. Label Studio's agreement documentation separates per-control-tag agreement from overall agreement and computes the overall score as a mean of the field scores by default, which is exactly how a strong field can hide a weak one. Show both levels, add a confusion table for each categorical field, and list the class pairs that account for most disagreements.
Choosing among kappa, Fleiss and alpha is a separate question with its own trade-offs; the decision here is what the delivered report must show regardless of which statistic you choose.
The agreement and adjudication report design
Use the table as the outline of the document that travels with the dataset. Each row is a report section; the third column says what a reviewer on the buyer side should be able to confirm from it, and the last column names the defect that most often makes that section useless.
| Report section | Fields to fill | What the buyer can confirm | Defect to avoid |
|---|---|---|---|
| Scope and version | Dataset name, release ID, export date, label fields covered, guideline version | The report describes the release being delivered | Agreement measured on an earlier export |
| Sample frame | Population count, strata, sample count per stratum, seed, weights | Rare classes and every source system were sampled | Hard or messy items dropped before sampling |
| Reviewer roster | Reviewer IDs, role, qualification, items reviewed, blinding method | Reviewers were independent and qualified | Reviewers saw the original label or each other's answers |
| Agreement by field | Raw agreement, chance-corrected statistic, statistic name, interval | Each field meets the stated minimum | One averaged score across unrelated fields |
| Agreement by class | Per-class agreement, confusion table, top confused pairs | Where the scheme is unclear | Rare classes with too few items to judge |
| Adjudication log | Item ID, field, each reviewer's value, final value, adjudicator, reason code, date | Every disagreement was resolved and explained | Final values with no reason or adjudicator |
| Changes applied | Count of delivered labels changed, by field and class, with item IDs | Which delivered labels differ from the source system | Labels overwritten silently without a flag |
| Guideline changes | Rule changed, version, items relabeled because of it | Fixes were applied to the whole dataset, not just the sample | Guideline fixed but unsampled items left as they were |
| Exceptions and limits | Fields not reviewed, inherently subjective classes, known gaps | What the report does not claim | Silence about fields that were skipped |
What should each adjudication entry record?
Each disagreement gets one row that anyone can audit later without asking the team. The reason code is the field that pays off, because a pattern of guideline gaps calls for a relabel pass while a pattern of reviewer error calls for retraining.
- Item ID and the source-system record ID it maps to, so the buyer can find the record in the delivery.
- The label field in dispute and each reviewer's value, kept even after a final value is chosen.
- The adjudicator's ID and whether that person also labeled the item, which should be avoided.
- The final value and whether it differs from the label originally stored in the source system.
- A reason code: guideline gap, ambiguous source record, missing context, reviewer slip, source system error, or genuinely ambiguous.
- Whether the case triggered a guideline change, and the version that change produced.
- For genuinely ambiguous items, whether the dataset keeps both values or a distribution instead of one forced answer.
Worked example: re-reviewing support ticket categories (illustrative)
This example is illustrative; the counts and percentages are invented to show the method, not drawn from any company. A software company plans to license three years of closed support tickets exported from its help desk, each with a category an agent chose from a twelve-value picklist.
The team draws 600 tickets stratified by category, year and channel, oversampling the four rarest categories. Two support leads who did not handle those tickets label each one under guideline version 1, with the original category hidden. Raw agreement comes out at 88 percent, which looks fine until the per-class table shows that two categories, billing dispute and refund request, account for 40 percent of all disagreements.
Adjudication finds the cause: the guideline never said which category applies when a customer disputes a charge and asks for a refund in the same ticket. The team writes a rule into guideline version 2, relabels every ticket in the full export that matches the pattern, and double-reviews a fresh sample of 150 tickets from the affected categories. The report shows both rounds, the rule, and that 6 percent of delivered labels now differ from what the help desk stored, each flagged with its item ID.
The exception is worth noting too. Agreement on a priority field stays low after adjudication because agents set priority from information outside the ticket, such as a phone call. Rather than forcing a value, the report marks priority as not reliably reconstructable from the record and leaves it out of the claimed label set.
What defect can change acceptance even when every file opens?
The most damaging defects are invisible in a file check. A dataset can load cleanly, match its schema and still fail acceptance because the report and the delivered labels describe different things.
Watch for agreement computed on a sample that excluded hard cases, reviewers who could see the original label, adjudicated values written into the sample but not propagated to matching items outside it, and a guideline revised midway without relabeling earlier work. Each of these makes the headline number describe a cleaner dataset than the one delivered.
Label errors are common even in well-known data. Northcutt and colleagues found label errors across the test sets of ten widely used benchmark datasets, which is a reason to report adjudication outcomes and changed labels rather than present the delivered labels as clean.
When should a field be narrowed, relabeled or left out?
Stop and narrow scope when a field stays below your stated minimum after one guideline revision and a second round. At that point the scheme is usually asking reviewers to infer something the record does not contain, and more rounds will not fix it.
Merge classes reviewers cannot separate, or drop a field the record cannot support, and say so in the exceptions section. For labels that are honestly subjective, such as tone, keep every reviewer's value and let the buyer decide how to aggregate. Bring in a domain specialist as adjudicator when disagreements cluster on cases that need professional judgment, such as coding a claim or interpreting a contract term, and record that person's qualification.
A report that says plainly which fields were left out is more useful to a buyer than one that claims reliability for everything.
Putting the report alongside a licensed dataset
Documentation templates already expect this evidence. The Hugging Face dataset card template asks for the annotation process, including guidelines and interannotator statistics, and for who the annotators were, while Datasheets for Datasets asks whether any preprocessing, cleaning or labeling was done. The agreement and adjudication report is the detailed answer those fields point to.
For a company considering licensing its records, the report also shows which labels came from the original system and which were revised, which matters when rights and scope are agreed. A company licensing labeled records through SourceX can hand this report over as part of preparation, where SourceX covers rights review, contracting, data preparation and delivery. Getting started takes a short company assessment rather than a sample upload, and no label leaves the company before it has approved scope and terms and signed.
Frequently asked questions
Is high percent agreement enough to call labels reliable?
No. Percent agreement ignores chance, so two reviewers who mostly pick the most common category can agree often without the scheme being clear. Report it alongside a chance-corrected statistic, per field and per class, with an interval.
How many reviewers does each sampled item need?
Two independent reviewers plus an adjudicator for disagreements is the usual minimum. Add a third reviewer for fields where you expect genuine ambiguity, and use a statistic that handles more than two reviewers and missing labels.
Should the original system label count as one of the reviewers?
Keep it separate. The original label was made under different conditions and often without a written guideline, so compare it with the adjudicated value and report the share that changed rather than folding it into the agreement figure.
What happens to labels outside the double-reviewed sample?
If adjudication leads to a guideline change, apply that rule to every matching item in the full dataset and record how many changed. Otherwise the sample is cleaner than the delivery and the report overstates reliability.
Does an agreement report prove the labels are correct?
No. It shows that the labeling is consistent and explains how disagreements were resolved. Correctness needs a separate check against independent evidence, such as a later outcome or a source document.
Sources
- The scikit-learn documentation for cohen_kappa_score defines Cohen's kappa as observed agreement minus expected chance agreement, divided by one minus expected agreement, with optional linear or quadratic weighting. Source
- The statsmodels fleiss_kappa function documents Fleiss' and Randolph's multi-rater kappa and states that it provides no variance or hypothesis tests. Source
- Krippendorff's guide to computing alpha covers reliability data from any number of observers, including data with missing values. Source
- Krippendorff's bootstrapping algorithm derives confidence intervals for alpha and the probability of failing to reach a minimally acceptable reliability. Source
- Label Studio's agreement documentation distinguishes per-control-tag agreement from overall agreement and computes overall agreement as the mean of control-tag scores by default. Source
- The Hugging Face dataset card template asks for the annotation process, including guidelines and interannotator statistics, and for who the annotators are. Source
- Datasheets for Datasets proposes questions for dataset creators, including whether any preprocessing, cleaning or labeling of the data was done. Source
- Northcutt, Athalye and Mueller identified label errors in the test sets of ten commonly used computer vision, natural language and audio datasets. Source
See if your company qualifies
A short company assessment. No data uploads are needed.