Skip to content

AI uses for records

Does the cited passage hold up? Scoring source passage grounded answer data

By SourceX Editorial · Updated

10 sections

Short answer

Score source passage grounded answer data claim by claim, against the passage each claim cites. In ExpertQA, pinned at commit 1aa7ba8, experts judged 10,338 cited claims: 69.3 percent were fully supported and 23.5 percent only partly. Just 33.1 percent of answers had every citation-worthy claim cited and fully supported, and the system with the best citation precision ranked fourth of six on that whole-answer test.

Agent run

Source passage grounded

This page covers does the cited passage hold up? Scoring source passage grounded answer data.Illustrative

Key takeaways

  • Across 10,338 cited claims in ExpertQA, experts judged 69.3 percent fully supported, 23.5 percent partly supported and 7.2 percent not applicable.
  • All 2,216 uncited claims carry the label Missing, so coverage and support must be scored as separate measures.
  • Claim-level and answer-level results disagree: 33.1 percent of answers passed a whole-answer gate, and the most precise citer ranked fourth of six.
  • Stored passage text did not raise support rates (69.6 against 68.4 percent), but it is what makes a support label repeatable.
  • Support is not authority: 30.5 percent of fully supported citations relied on sources experts rated only somewhat reliable and 3.6 percent on sources rated not reliable at all.

How often does a cited passage support the claim it is attached to?

About seven times in ten, in the public expert judgments we could pin. ExpertQA's main data file, at commit 1aa7ba8, holds 12,554 labeled claims from answers to 2,177 questions written by experts in 32 fields. Of the 10,338 claims that carry at least one citation, experts marked 7,167 (69.3%) as completely supported by the cited evidence, 2,426 (23.5%) as partial or incomplete, and 745 (7.2%) as not applicable.

That figure starts a buying decision; it does not settle one. An AI team buying source passage grounded answer data, meaning questions, answers and, for every claim, the exact passage that supports it, needs the data to separate three things a single citation score blurs: whether a claim needed a citation, whether the cited passage supports all of it, and whether the source deserves trust. This article measures each of the three in ExpertQA and turns them into a scoring sheet a buyer can hand to a supplier.

One reading rule runs throughout. A claim with no citation is a different failure from a claim with a weak one. In ExpertQA all 2,216 uncited claims carry the label Missing and no cited claim does, so the labels themselves keep coverage and support apart. A test that merges them cannot tell a system that cites carefully but rarely from one that cites everything loosely.

What ExpertQA records for each claim, and why the format suits a purchase test

It records claim-level judgments made by the people who asked the questions, which is the shape a passage-grounded dataset should take. The repository's README describes questions written by experts in their own fields, answers sampled from six system configurations, each answer split into claims, and for each claim a support label, a citation-worthiness label, a correctness label, a source reliability label and an informativeness label, with room for the expert to revise the claim and its evidence. The paper describes the same judgments on each claim and evidence pair, and concludes that attributed answers can look trustworthy while their references fall short of supporting the claims.

In the released file every answer was validated by the expert who wrote the question: all 2,177 answer records carry the same anonymized identifier as their question. That keeps domain knowledge in the loop, and it also means one reader per claim, a limit to keep in mind when reading the rates.

The six configurations differ in how citations were produced. Two cite web pages by URL only: GPT-4 prompted to answer with numbered citations to URLs, and Bing Chat. Two retrieve first and then answer from the retrieved passages, from Google search results or from the Sphere corpus. Two write the answer first and attach citations afterward, from the same two sources. A vendor's citation pipeline will usually resemble one of these designs, which is what makes the spread useful.

The authors' own correlation script treats a claim as attributable only when its support label is Complete, and skips claims marked as not worth citing. This article uses the same convention: fully supported means Complete, and partial support counts as a failure. The repository is released under the MIT License, but the web pages its passages come from remain third-party content, which matters to anyone planning to redistribute the passages.

Support rates by answer system: claim precision and answer completeness disagree

Which system looks best depends on the denominator. Retrieving search results before answering produced the highest share of fully supported citations, 84.4%, but that configuration cited only 75.3% of its claims. Attaching citations after writing, from the same search source, cited 98.6% of claims and produced more answers in which every citation-worthy claim passed: 44.1% against 31.6%.

The answer-level column is the stricter test and the closest to what a reader experiences. An answer passes when every claim the expert judged worth citing carries a citation and is fully supported. Only 686 of the 2,073 answers with at least one such claim (33.1%) pass. Answers contain several claims, so a claim-level rate in the seventies compounds into a minority of clean answers.

Ask any vendor, and any dataset, for both numbers: the share of cited claims fully supported, and the share of answers that pass a whole-answer gate. A dataset labeled for only one of them cannot rank systems reliably. Counts per system, from the released file:

Support rates by answer system: claim precision and answer completeness disagree
Answer system (ExpertQA key)Claims labeledClaims with a citationCited claims fully supportedAnswers passing the whole-answer gate
Retrieve, then answer: Google search (rr_gs_gpt4)2,48075.3%84.4%31.6%
Bing Chat, URL citations (bing_chat)2,25467.8%84.2%39.4%
Retrieve, then answer: Sphere (rr_sphere_gpt4)1,80365.2%72.1%19.3%
Answer, then cite: Google search (post_hoc_gs_gpt4)2,43998.6%71.6%44.1%
Answer, then cite: Sphere (post_hoc_sphere_gpt4)2,452100.0%55.2%33.2%
GPT-4 alone, URL citations (gpt4)1,12680.6%41.8%16.6%
A slope chart linking, for six ExpertQA answer systems, the share of cited claims fully supported (41.8 to 84.4 percent) to the share of answers passing a whole-answer gate (16.6 to 44.1 percent); the most precise citer drops to fourth.
Cited claims fully supported against answers passing the whole-answer gate, ExpertQA commit 1aa7ba8. Source: chaitanyamalaviya/ExpertQA; SourceX count.

Partial support, not a missing source, is the failure to label

Among cited claims, partial support outnumbers every other failure: 2,426 cited claims (23.5%) were judged partial or incomplete, against 745 (7.2%) not applicable. The file records two labels short of complete, Partial on 660 cited claims and Incomplete on 1,766; this article groups them because the released data does not define the difference.

Partial support is also where automatic checkers struggle most. The ALCE benchmark paper defines citation recall as whether a statement's citations, taken together, entail it, and adds a precision measure meant to flag irrelevant citations; its own human check found that the entailment-based detection of irrelevant citations had a relatively high false-positive rate because the model cannot detect partial support.

The share also moves with the citation design. After-the-fact citation from Sphere left 34.9% of cited claims partly supported, against 14.1% for Bing Chat and 14.3% for retrieve-then-answer from search. The data shows the association, not its cause, but it points a buyer at the failure worth labeling.

For a dataset, a binary supported-or-not label throws away the most common failure. Each claim needs at least three support levels, and a partial label should say what is missing: a number, a qualifier, a date, or a second fact that only another passage supplies.

Does storing the passage text change the verdict?

It barely changes how often citations hold, but it decides whether the verdict can be checked again. In ExpertQA, 7,902 cited claims came with passage text attached to their evidence and 2,436 cited a URL only; 69.6% of the first group and 68.4% of the second were fully supported.

The URL-only claims come from the two configurations that returned links, so judging them meant opening a live web page. Pages change, move and disappear, and a later reviewer may read different text from the one the expert read. A passage stored with the claim, plus the page version or retrieval date, is what makes a support label reproducible and lets a buyer audit a sample without depending on the open web.

Business records raise the stakes. A policy, price list or contract clause has versions, and the version in force on the date of the answer is the one that counts. A source passage grounded answer dataset should store the cited span, its document identifier and version, and the character offsets of the span, so a scorer can confirm the passage exists exactly as cited before judging whether it supports anything.

A supported claim can still rest on a weak source

Support and authority are separate judgments, and ExpertQA recorded both. Of the 7,167 cited claims judged completely supported, experts rated the source not reliable at all for 256 (3.6%) and only somewhat reliable for 2,189 (30.5%). A citation can match its claim closely and still point to a page an expert would not rely on.

Support is not truth either. Of the same fully supported claims, 62 (0.9%) were judged likely or definitely incorrect and 177 (2.5%) left the expert unsure: claims faithfully copied from a source that was wrong or out of date.

Combining the three judgments on citation-worthy cited claims, 74.6% are fully supported, and 71.9% are fully supported by a source not rated unreliable and not judged incorrect. The gap is small in web answers. In business records the authority question is sharper, because an expired policy or an unsigned draft can support a claim perfectly and should still fail.

Should every sentence in an answer need a citation?

No, and a rule that demands one rewards padding. Experts judged 2,458 labeled claims (19.6%) not worth citing, and 1,589 of those carried citations anyway.

A later study of citation generation argues that ALCE's metrics excessively penalize responses that either need no citations or include excessive ones, and proposes leaving statements that need no citation out of recall. The same logic belongs in a purchase test. Measure coverage on claims that need a citation, and check a citation attached to a claim that needs none for relevance instead of counting it as a success.

That requires the dataset to label citation-worthiness on every claim, as ExpertQA does. Without the label, a vendor can lift its coverage score by citing everything, and a model trained on the data learns to decorate every sentence with a reference.

The passage-level scoring sheet for a buyer's evaluation

Score each claim through five gates in order, then gate the whole answer. A claim that fails an earlier gate is not scored on later ones, so no system earns credit for a precise citation on a claim it should not have made, or for an irrelevant passage stacked onto a claim that needed none.

Each row names the field a scorer needs in the dataset and the ExpertQA field that comes closest today. The answer gate in the last row was computed on gates 2 and 3 only, because ExpertQA has no per-passage or version labels:

The passage-level scoring sheet for a buyer's evaluation
GatePass ruleWhat it stops a vendor gamingDataset field requiredExpertQA analogue
1. Citation neededA domain reviewer labels the claim citation-worthyPadding answers with cited fillerWorthiness label per claimworthiness
2. CoverageEvery citation-worthy claim cites at least one passageLeaving the hard claims uncitedClaim-to-passage linksevidence (empty means Missing)
3. Full supportCited passages together support the whole claim; partial failsCiting a passage that covers half the claimThree-level support label naming what is missingsupport: Complete, Partial, Incomplete
4. Passage necessityEach cited passage contributes; removing it weakens supportStacking irrelevant citations around one good oneSupport label per passage, not only per claimNone; judged per claim
5. Authority and versionThe source is official, in force and the right versionCiting a superseded or unofficial pageDocument version, status and date per passagereliability; no version field
Answer gateEvery citation-worthy claim passes gates 2 to 5Hiding one bad claim among good onesClaim list per answer33.1% of answers on gates 2 and 3

What a source passage grounded answer dataset must hold before you buy it

Hold the dataset to the sheet before you hold a model to it. Each record should carry the question, the answer split into claims, a citation-worthiness label per claim, the cited passages as stored text with document identifier, version and character offsets, a three-level support label per claim and per passage, an authority label for each source, and a note on every partial label saying what is missing.

Ask for agreement between two reviewers on a random sample drawn with a published seed, plus the share of items where they disagreed and how each disagreement was settled. ExpertQA relied on one expert per answer, which suits expert questions but leaves label noise unmeasured; a purchased set should report it rather than inherit the gap.

For business records, the passages come from a company's own documents, and so do the version and authority labels that make gate 5 possible. SourceX licenses business data directly from companies and licenses it to buyers in its own name, sourcing each dataset on request rather than from stock. A data request that includes the scoring sheet, the reviewer-agreement target and the acceptance test lets the search for suitable records start from the evaluation, not from whatever documents happen to exist.

Methods note: the ExpertQA file, counting rules and limits

Data: data/r2_compiled_anon.jsonl in github.com/chaitanyamalaviya/ExpertQA at commit 1aa7ba81bd4c083c12ef1aa2fee338f6cb60fa87 (14 March 2024), the file the README names as the main dataset, read by a script, expertqa_support.py, that runs no repository code. The unit is one claim in one system's answer, and fields are the distinct values of each question's field metadata. Of 12,598 claims, 44 with no support label were dropped from label shares.

Rules as the script applies them: a cited claim has at least one evidence item. An item has an attached passage when non-blank text follows its first line, which holds the citation number and URL. Labels are used as recorded, and partial means Partial or Incomplete. Citation-worthy means worthiness Yes. A weak source is reliability Not reliable at all; incorrect means correctness Definitely incorrect or Likely incorrect. The strict pass needs Complete support, a source not rated Not reliable at all and correctness not incorrect. An answer qualifies for the whole-answer gate when it has at least one citation-worthy labeled claim, and passes when every such claim is cited and Complete; 2,073 answers qualify. Percentages come from integer counts, rounded once, half up, to one decimal.

System descriptions come from the README key list, the response-collection scripts and the question-answering prompt in the same commit. Saved outputs hold counts and per-claim flags (system, labels, citation and passage flags), with no question text, claim text, URLs or annotator identifiers; the judge-identity check compares anonymized identifiers in memory and saves only the count.

Limits: the answers come from web-sourced 2023-era configurations, so the rates describe them, not current products or business records. Each claim had one judge, claims are sentence-level and often bundle several facts, and passage necessity was not labeled, so gate 4 has no measurement here. A second released file with automatic attribution labels (data/r2_compiled_autoais_anon.jsonl) was not used: matched claim by claim, its human support label differs from the main file on 2,021 of 12,585 claims (16.1%).

Frequently asked questions

What is source passage grounded answer data?

It is a set of questions and answers in which every claim in an answer links to the exact passage that supports it, stored as text with its document identifier and version. Labels record whether each claim needed a citation, whether the passage fully or partly supports it, and whether the source is authoritative. It is used to train and evaluate assistants that must show their evidence.

How often do cited passages fully support AI-generated answers?

In ExpertQA's expert judgments, 69.3 percent of 10,338 cited claims were completely supported and 23.5 percent partly supported, with wide spread across systems, from 41.8 to 84.4 percent fully supported. Those rates describe 2023-era web answer systems, not any current product. Measure your own system on your own records.

Can an automatic entailment check replace human citation labels?

Not on its own. The ALCE paper reports that its entailment-based check for irrelevant citations had a relatively high false-positive rate because it cannot detect partial support, the most common failure in ExpertQA. Use human labels on a sampled calibration set and compare the automatic scorer against them before trusting it at scale.

Should a grounded answer dataset include claims that need no citation?

Yes, labeled as such. Experts in ExpertQA judged 19.6 percent of claims not worth citing, and many carried citations anyway. A worthiness label lets a scorer measure coverage fairly and stops a vendor raising its score by citing every sentence.

Why store the cited passage instead of a link?

A link points to a page that can change or vanish, so a later reviewer may judge different text. In ExpertQA, cited claims with stored passages and those with only URLs had similar support rates, 69.6 and 68.4 percent, but only the stored passages can be re-checked exactly. For business documents, store the version and the span offsets too.

Sources

  • ExpertQA's released data at commit 1aa7ba8 holds expert labels for 12,598 claims in answers to 2,177 questions, including support, citation-worthiness, reliability and correctness labels. Source
  • The ExpertQA README names r2_compiled_anon.jsonl as the main dataset and documents the six system keys and the per-claim fields. Source
  • The ExpertQA paper describes expert judgments of claim and evidence pairs and finds that attributed answers can look trustworthy while their references fall short of supporting the claims. Source
  • ExpertQA's correlation script treats only Complete support as attributable and skips claims not worth citing. Source
  • ExpertQA's retrieve-and-read configuration retrieved passages from Google search before answering. Source
  • ExpertQA prompted GPT-4 to answer with numbered in-line citations to URLs on the web. Source
  • The ExpertQA repository is released under the MIT License. Source
  • The ALCE paper defines citation recall and a citation precision measure for irrelevant citations, and reports that its entailment-based irrelevance check cannot detect partial support. Source
  • A follow-up study argues ALCE's metrics penalize responses that need no citations or include excessive ones, and proposes excluding statements that need no citation from recall. Source
  • SourceX's request-data page says it licenses business data to buyers in its own name and sources datasets on request. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies