Skip to content

Leadership and readiness

Human grading and AI benchmark reliability: a recount of 51,200 grades

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Human grading strengthens an AI benchmark only when graders are independent, qualified for the dimension and measured, not assumed, to agree. A recount of SummEval's 51,200 published grades shows how fast that fails: expert and crowd grades rank 16 summarizers in nearly unrelated orders, the crowd's 0.49 agreement rests on 807 summaries with identical crowd ratings, and expert agreement runs from 0.90 to 0.45 by dimension.

Grader rubric

Human grading AI benchmark

This page covers human grading and AI benchmark reliability: a recount of 51,200 grades.Illustrative

Key takeaways

  • Human grading adds reliability only with independent graders, dimension-matched expertise, a specific rubric and measured agreement.
  • In SummEval, expert and crowd grades of the same 1,600 summaries produce system rankings with Kendall tau between -0.14 and 0.14.
  • The crowd's published 0.4920 agreement reproduces exactly, but falls to 0.06 once 807 summaries with five identical crowd rating sets are set aside.
  • The same expert slots agree at 0.90 on factual consistency and 0.45 on relevance, so reliability belongs to a dimension, not a panel.
  • Expert keys need audits: GPQA, MMLU-Redux and a Humanity's Last Exam review all found answer keys that outside evidence contradicts.
  • The eight-line grader evidence worksheet lists what to request before a human-graded score enters a memo.

When does human grading make an AI benchmark more reliable?

Human grading makes a benchmark more reliable when the graders work independently, are qualified for the specific dimension they score, and follow a rubric specific enough that two of them reach the same grade, and when that agreement is measured per dimension rather than assumed. Remove any one of those conditions and human grades add noise or copied judgments while keeping the authority of the phrase expert-graded.

When a company calls a benchmark or dataset expert-graded, it is pointing at an input that is expensive and hard to copy, and asking for credit on that basis. The phrase describes a measurement process, and measurement processes can be checked. The decision this page supports is whether a human-graded score is reliable enough to back a ranking or capability claim in an investment memo, and what evidence to request before it is.

The evidence comes from a public dataset in which experts and crowd workers graded the same outputs, recounted for this article, and from three audits that tested expert answer keys against outside evidence.

What a recount of SummEval's 51,200 grades shows

It shows that who grades, and on which dimension, can decide the ranking. SummEval's README describes 1,600 summaries, produced by 16 models from 100 CNN/DailyMail news articles, each graded for coherence, consistency, fluency and relevance by five crowd workers and three experts. Every grade in the published file is a whole number from 1 to 5, and there are 51,200 of them.

The recount first reproduced the paper. SummEval's authors report Krippendorff's alpha, a reliability coefficient for agreement among any number of raters, of 0.4920 for the crowd workers, 0.4132 for the experts' first round and 0.7127 after a second expert round. Pooling all four dimensions, the published file gives exactly 0.4920 for the crowd and 0.7187 for the experts, close to the paper's second-round figure.

Then it asked whether the two groups agree with each other. At the level of single summaries, the correlation between the expert mean and the crowd mean is close to zero on every dimension (Pearson r from 0.006 to 0.072). At the level of systems, ranking the 16 models by expert grades and by crowd grades gives orders that are close to unrelated: Kendall tau from -0.14 to 0.14. On relevance, the system the experts rank first is the one the crowd ranks last.

Even the expert ranking alone is less settled than a leaderboard suggests. Resampling the 100 source articles 2,000 times with a fixed seed, the system the experts rank first stays first in only 41 to 51 percent of resamples, depending on the dimension, and 77 to 97 of the 120 system pairs keep their order in at least 95 percent of them. The table summarizes the four dimensions:

What a recount of SummEval's 51,200 grades shows
DimensionExpert alphaCrowd alpha, all summaries / without identical rowsExpert vs crowd system ranking (Kendall tau)Experts' top system, rank under crowd grades
Coherence0.560.49 / 0.06-0.086th of 16
Consistency0.900.50 / 0.030.077th of 16
Fluency0.730.48 / 0.050.145th of 16
Relevance0.450.48 / 0.08-0.1416th of 16
Four small scatter plots, one per SummEval dimension, placing 16 summarization systems by expert rank and crowd rank; the dots scatter far from the agreement diagonal, and on relevance the experts' top system ranks last under crowd grades.
Rank of each system under expert and crowd mean grades; ties broken by model number. Source: SummEval human annotation file; SourceX recount.

Were the graders independent? The 807 identical crowd rows

Not demonstrably, and that is the first thing to check in any human-graded file. In 807 of the 1,600 summaries, all five crowd rating sets are identical on all four dimensions. All 807 belong to 12 of the 16 systems; the other four systems have none. The README describes the five crowd workers as independent.

Remove those 807 summaries and crowd agreement on the remaining 793 collapses: alpha falls to between 0.03 and 0.08 per dimension, and to 0.06 pooled. In other words, almost all of the crowd's published 0.49 comes from summaries where five supposedly separate judgments are the same down to every score.

The file does not say whether those rows record five workers who agreed exactly or one set of ratings repeated, and this recount cannot tell. For diligence it does not matter which. A reliability figure is only as good as the independence of the grades behind it, and independence is a property to verify in rater-level data, not a word to accept from a methods section.

The practical request is the raw file: one row per grade, with a pseudonymous rater ID, a timestamp and the rubric version. With that, an analyst can count identical rater vectors, see whether the same few raters graded most items, and recompute agreement without them.

Same experts, different reliability: the dimension and the rubric decide

The same three expert slots produce very different reliability depending on what they are asked to judge. Their agreement is 0.90 on consistency, 0.73 on fluency, 0.56 on coherence and 0.45 on relevance. Consistency asks whether the summary's facts match the article, which a grader can check line by line; relevance asks whether it kept the most important content, which is closer to taste.

High agreement can also hide a weak instrument. The experts gave identical consistency grades to 1,340 of the 1,600 summaries, with an average of 4.66 out of 5, and the top five systems' expert consistency means fall between 4.96 and 4.99. Graders who agree that nearly everything is fine are reliable and still barely able to separate the leaders.

Calibration history matters as much as the final figure. In SummEval's paper, expert agreement rose from 0.4132 in the first round to 0.7127 after a second round. Ask for agreement before and after each calibration round and for the rubric changes made in between; a team that reports only the post-discussion number is reporting how well its graders converged, not how well a new grader would match them.

A rubric earns the word specific when every score level has a written anchor, worked examples sit beside the anchors, the unit being graded is defined (a whole summary, a sentence, a claim), and edge cases have decision rules. A bare 1 to 5 scale with dimension names is not a rubric in that sense.

Expert answer keys are not ground truth: three public audits

Agreement measures whether graders match each other, not whether they are right, and three audits show how far expert keys can drift from outside evidence.

GPQA, a benchmark of graduate-level science questions written by domain experts, reports that experts who have or are pursuing PhDs in the relevant field reach 65 percent accuracy, or 74 percent after discounting clear mistakes they identified in retrospect, while skilled non-experts reach 34 percent despite more than 30 minutes per question with web access. Even inside the expert pool, roughly a third of first answers miss the key.

MMLU-Redux (NAACL 2025) had 14 human experts re-annotate 5,700 MMLU questions and estimated that 6.49 percent of MMLU questions contain errors, with 57 percent of the analyzed Virology questions affected. The ranking moved with the labels: Llama 3.1 405B Instruct ranked 16th on all Virology questions and first on the ones whose labels were correct.

FutureHouse estimated in July 2025 that about 29 percent (29.3 percent, plus or minus 3.7) of the text-only chemistry and biology questions in Humanity's Last Exam have answers that conflict with peer-reviewed literature, and reported that the benchmark team's own follow-up with three expert reviewers found about 18 percent problematic. The two parties disagree on the rate, but either rate is large enough to reorder close results. A human-graded benchmark therefore needs a label error audit, an independent re-check of a random sample against source evidence, alongside its agreement figures.

Could a change of graders, not a better system, explain the gain?

Often, yes, because the graders are part of the instrument. A gain reported across two versions of a human-graded benchmark can come from a better model, a better dataset, a new prompt, a larger inference budget, or a different set of graders, rubric or judge. Only a bridging study, in which a fixed sample is regraded by both the old and new grading setups, separates the last group from the rest.

Grader count is a grader change too. Ranking SummEval's 16 systems from a single expert slot instead of all three gives Kendall tau between 0.56 and 0.92 against the other two experts' ranking, so a benchmark that cuts its panel to one reviewer per item to save money changes what its rankings mean.

Another substitution to watch is the model judge. In the MT-Bench study, GPT-4 as a judge agreed with human experts 85 percent of the time with ties excluded, above the 81 percent agreement between humans, and strong judges matched human preferences more than 80 percent of the time overall. The same study documented biases, including judges favoring their own outputs: GPT-4 by about 10 percent and Claude-v1 by about 25 percent in win rate. A switch from human graders to a judge, or from one judge model to another, belongs in the evidence file with its own bridging numbers.

Grader evidence worksheet: eight lines before a human-graded score counts

Each line names what to request and the condition under which human grading strengthens or weakens the benchmark. A benchmark that clears the first five lines supports a reliability claim; the last three decide whether its rankings and version-to-version gains can be trusted.

The replication checklist is for the analyst who rebuilds the figures from raw grades; the worksheet lines come after it:

  • Recompute agreement per dimension from the rater-level file, using the statistic the paper names, and match the published figure.
  • Flag items whose rater vectors are identical, and recompute agreement and rankings without them.
  • Rebuild the system ranking from raw grades and compare it with the published leaderboard.
  • Bootstrap the ranking over source items with a fixed, published seed and report how often the leader stays first.
  • Draw a random audit sample and have an independent grader check it against source evidence.
  • Log anything that could not be reproduced as an open finding.
Grader evidence worksheet: eight lines before a human-graded score counts
LineAsk forStrengthens the benchmark whenWeakens it when
1. Rater independenceRater-level grades with pseudonymous IDs and timestampsEach item has distinct raters grading without seeing each other's scoresGrades are duplicated, consensus-only or visible to other raters
2. Qualification per dimensionRater credentials mapped to each graded dimensionExpertise matches what is graded, such as facts versus styleOne panel grades everything, or crowd workers grade expert dimensions
3. Rubric specificityThe versioned rubric, anchors and worked examplesEvery score level has a written anchor and examplesA bare numeric scale with dimension names only
4. Agreement per dimensionChance-corrected agreement with intervals, before and after calibrationReported per dimension with calibration rounds disclosedOne pooled figure, or only post-discussion agreement
5. Label error auditAn independent re-check of a random sample against source evidenceAn error rate with its method and sample sizeThe expert key is treated as truth with no audit
6. Ranking stabilityA bootstrap or rater-resampling of the rankingThe leader and key pairwise orders hold across resamplesA single ranking with no interval
7. Grader changes across versionsRater pool, rubric and judge model per release, with a bridging regradeThe same graders, or a bridging study, link the versionsReleases graded by different instruments are compared directly
8. Blinding to system identityHow outputs were anonymized and ordered for gradersGraders cannot tell which system produced an outputSystem names are visible, or one system is graded in a block

Limits of the SummEval evidence, and risks to the thesis

The recount is strong evidence about one public file and weaker evidence about business benchmarks in general. Five limits belong in any memo that leans on it:

  • SummEval grades 2020-era news summaries on a 1 to 5 scale. Business tasks with verifiable outcomes, such as a refund that posted or a claim that paid, may support much higher agreement than taste-like dimensions.
  • The identical crowd rows may come from how the file was assembled rather than how ratings were collected. The recount cannot tell, though the file is what every re-user of the dataset receives.
  • The file does not say whether each expert slot is the same person across summaries, so single-expert results assume the stored slot order means something.
  • Model judges may absorb much routine grading, as MT-Bench's agreement figures suggest, which would shrink the premium for human-graded evaluation to the hard dimensions. The opposite is also possible: trustworthy human labels become the scarce input for calibrating those judges.
  • Agreement is not validity. GPQA and MMLU-Redux show experts can share a wrong answer, so a perfect agreement score still needs a label audit.

Putting the worksheet to SourceX and other evaluation-data sellers

The worksheet applies to anyone selling human judgment as evaluation evidence, SourceX included. SourceX turns operational records that companies select into datasets licensed to AI developers, and does the work in between: qualifying the records, clearing rights, drafting the agreement, preparing the data, handing it over and paying the company. Records of that kind often carry outcomes a business recorded for its own reasons, which can replace some opinion grades with verifiable events; the status fields behind those outcomes were still set by people, and they deserve the same independence and audit questions.

SourceX has not published grader-reliability or label-audit results, and this article reports none. Investors who want its company and commercial figures receive them directly, through the materials request on the investors page, and can attach any worksheet question. Questions worth putting to any seller:

  • Can we have the rater-level grade file, with pseudonymous rater IDs, timestamps and rubric versions?
  • What was agreement per dimension before and after each calibration round, and what changed in the rubric?
  • How many items have identical grades from every rater, and what happens to agreement without them?
  • What did the last independent label audit find, on how large a random sample?
  • Did graders, rubric or judge model change between the versions you are comparing, and where is the bridging study?

Methods note: how the SummEval grades were recounted

Data: SummEval's human annotation file, model_annotations.aligned.jsonl, linked from the README at commit 81b59ad of the Yale-LILY/SummEval repository and downloaded from its Google Cloud Storage location on 2026-10-27 (5,839,062 bytes; the MD5 matches the server's ETag). Script: summeval_grader_reliability.py, kept with this article's working files for reviewers.

Rules as the script applies them. Units are the 1,600 summaries; every unit has three expert and five crowd grades per dimension. Krippendorff's alpha is computed from the coincidence matrix with the interval metric (an ordinal version is also printed), per dimension and per rater group, without assuming rater identity; alpha is 1 minus observed disagreement over the disagreement expected by chance. Pooled alpha treats each summary-dimension pair as a unit. A summary is flagged when its five crowd rating sets are exactly equal on all four dimensions. System means average each group's per-summary mean over 100 summaries; ranks sort means rounded to nine decimals, with ties broken by model number. Rank agreement is Kendall tau-b on the means. The bootstrap resamples the 100 articles with replacement, 2,000 times, with Python's random.Random(20261027); a pair counts as stable when its strict order matches the full-data order in at least 95 percent of resamples, with means compared exactly, so a resample in which the two are tied does not count.

Not measured: whether the identical rows are duplicates, rater identity across summaries, agreement intervals for each alpha, any label error audit of SummEval itself, business-task benchmarks and any SourceX data. Checked 2026-10-27.

Frequently asked questions

What level of grader agreement should a human-graded benchmark report?

There is no single cutoff that fits every dimension, so the benchmark should state its threshold before grading and report agreement per dimension with an interval. In SummEval, the same three expert slots reached 0.90 on factual consistency and 0.45 on relevance, which shows why one pooled number hides the weak dimensions.

Are experts or crowd workers better graders for AI benchmarks?

It depends on the dimension, and neither should be assumed correct. In SummEval, crowd and expert grades of the same summaries were almost unrelated, with summary-level correlations of 0.07 or less. Yet GPQA shows expert validators matching the answer key only 65 percent of the time, so expert grades still need an audit.

Can an LLM judge replace human graders?

For some tasks it comes close: in the MT-Bench study, GPT-4 agreed with human experts about as often as humans agreed with each other. The same study found judges favoring their own outputs. Use human grades to calibrate and audit a judge, and run a bridging regrade whenever the judge changes.

What raw files should an investor ask for when a company cites expert grading?

The rater-level grade file with pseudonymous rater IDs and timestamps, the versioned rubric with anchors, calibration logs with agreement before and after each round, and the most recent label audit with its sample size. Those four let an analyst rebuild the reliability claim instead of trusting a summary figure.

Why did SummEval's expert agreement rise between rounds?

The authors ran a second round of expert grading, and agreement rose from 0.4132 to 0.7127. Reporting both rounds is the useful practice, because the later figure shows how well these graders converged, while the first is closer to what a fresh grader would achieve.

Sources

  • SummEval's human annotations cover 1,600 summaries from 16 models on 100 news articles, each graded by five crowd workers and three experts on four dimensions. Source
  • The published SummEval annotation file with every expert and crowd grade, recounted for this article. Source
  • SummEval's paper reports Krippendorff's alpha of 0.4920 for crowd workers and 0.4132 then 0.7127 for the experts' two rounds. Source
  • Krippendorff's alpha is a reliability coefficient for agreement among any number of observers or raters. Source
  • GPQA's expert validators reached 65 percent accuracy (74 percent discounting clear mistakes) and skilled non-experts 34 percent. Source
  • MMLU-Redux estimated that 6.49 percent of MMLU questions contain errors, 57 percent in the analyzed Virology subset, and showed rankings changing on corrected labels. Source
  • FutureHouse estimated about 29 percent of Humanity's Last Exam chemistry and biology answers conflict with peer-reviewed literature; the benchmark team's follow-up found about 18 percent problematic. Source
  • The MT-Bench study found GPT-4 judges matching human preferences more than 80 percent of the time, with documented self-preference biases. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies