Skip to content

AI uses for records

Enterprise answer citation benchmark: a claim-by-claim study protocol

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

An enterprise answer citation benchmark should grade claims, not whole answers: split each response into material claims, check every claim against the exact version of the business record it cites, and score miscited, unsupported, uncited and missing-evidence cases separately. Fix the claim set, gold evidence, systems, metrics and pass bar before any run. SourceX has not run this study; this page publishes the protocol, and no results exist yet.

Evaluation split

Enterprise answer citation

This page covers enterprise answer citation benchmark: a claim-by-claim study protocol.Illustrative

Key takeaways

  • A 2023 audit of four generative search engines found only 51.5 percent of sentences fully supported by their citations.
  • The unit of grading is the material claim: any statement that would change what the person asking does next.
  • Seven ledger outcomes keep an invented figure, a true claim with the wrong citation and a correct refusal from merging into one score.
  • A citation to the right document in a superseded version is a failure, so every item carries a question date and a pinned version.
  • No system has been tested; SourceX has published no benchmark results, and the report will release every outcome count if the study runs.

What would an enterprise answer citation benchmark decide?

It would decide whether one specific AI system can be trusted to answer questions about a company's own records with citations a reviewer could rely on, one claim at a time. The system passes only when every material statement in an answer is backed by the business record it points to, in the version that was in force on the date the question was asked.

That is a narrower and more useful decision than asking whether a model hallucinates. A team choosing between two retrieval assistants for a claims desk, a policy help line or a month-end close does not need a general reliability score. It needs to know how often each system attaches a figure, a deadline, an eligibility rule or an obligation to a record that does not say it, and whether the system declines when no record answers the question.

Four kinds of evidence would make the answer useful: a fixed set of questions drawn from real work, expert-built gold evidence for each material claim, one pinned document collection shared by every system under test, and claim-level outcomes reported with their denominators and uncertainty. Without all four, a ranking cannot be reproduced, audited or defended to the people who will act on the answers.

What public studies already measured, and what they leave open

Published audits show that citations often fail to back the statements they decorate, but none measured systems answering from private business records. Nelson Liu, Tianyi Zhang and Percy Liang (arXiv 2304.09848, 2023) had people check four generative search engines and found that on average only 51.5 percent of generated sentences were fully supported by their citations, while 74.5 percent of citations supported the sentence they were attached to.

Other studies reach the same conclusion from different directions. FActScore (arXiv 2305.14251, 2023) split generated biographies into atomic facts and reported 58.3 percent of facts supported for ChatGPT and 71.5 percent for the search-augmented PerplexityAI. A Stanford RegLab and Yale study (arXiv 2405.20362, 2024) reported that legal research tools from LexisNexis and Thomson Reuters each hallucinated between 17 and 33 percent of the time. NIST's Generative AI Profile (NIST AI 600-1, July 2024) lists confabulation, false content presented with confidence, among the risks particular to generative systems.

Each figure describes the systems and question types of its own study and date, so none is a current error rate for any product. The table sets out what each study graded and the question it leaves open for a buyer working with business records.

What public studies already measured, and what they leave open
StudyWhat was gradedReported resultOpen question for business records
Liu, Zhang and Liang, 2023Sentences and citations from four generative search engines51.5% of sentences fully supported; 74.5% of citations supported their sentenceDo internal documents with versions and access limits behave like public web pages?
FActScore, 2023Atomic facts in generated biographies58.3% of facts supported for ChatGPT; 71.5% for PerplexityAIWhich facts are material to a business decision, and who sets that rule?
Stanford RegLab and Yale, 2024Answers from commercial legal research toolsEach tool hallucinated between 17% and 33% of the timeHow do results move when the collection is one company's own files?
GDPval, OpenAIExpert-written occupational tasks graded blind by professionalsAn experimental automated grader agreed with human experts 66% of the timeCan an automated grader stand in for experts when checking citations to records?

The claim ledger: seven outcomes a material claim can receive

Each material claim in a tested answer receives exactly one outcome, assigned against the expert gold evidence and the record the system actually cited. Keeping outcomes separate is what makes a result usable: an invented invoice total, a correct total cited to the wrong invoice and a sensible refusal need three different fixes, and one blended score hides which fix is needed.

The ledger starts from the three support levels in the human judgments released with the Liu, Zhang and Liang study (complete, partial and no support) and adds outcomes that appear only when documents carry versions and some questions have no answer in the collection. A claim contradicted by its cited record is scored as unsupported. Every example in the table is invented and uses one question about a distributor's returns policy.

The claim ledger: seven outcomes a material claim can receive
OutcomeAssigned whenIllustrative exampleScored as
SupportedThe cited record, in the pinned version, states the whole claimA 30-day return window cited to the policy version in force on the question datePass
Partly supportedThe cited record backs part of the claim and the rest is addedThe window is right, but a restocking condition appears nowhere in the recordFail; partial share reported separately
MiscitedThe claim matches gold evidence, but the cited record does not state itThe correct window, cited to a shipping page that never mentions returnsFail on citation, pass on content
Wrong versionThe cited document is right, but that version was not in force on the question dateA superseded 45-day window quoted for a question dated after the changeFail
UnsupportedNo record in the collection states the claim, or the cited record contradicts itAn invented exception for opened itemsFail
Uncited material claimA material claim carries no citationA refund timeline stated with no sourceFail
AbstentionThe system declines or says the records do not answerRight only when gold evidence confirms no record answers the questionPass or fail against gold
A decision tree that routes one material claim through three checks in order, cited, pinned version and supported by the cited record, ending in pass or in the ledger failures uncited, wrong version, or partly supported, miscited or unsupported.
How the protocol routes each material claim to one ledger outcome. A design, not results: no system has been tested.

Which claims count as material, and who decides?

A claim is material when changing it would change what the person asking does next: an amount, a date or deadline, an eligibility condition, a named party, an obligation or a status. Connective phrasing, restatements of the question and hedges are not graded, which keeps the denominator on statements that can cause a wrong action.

Claims are extracted the way FActScore treats a biography, as short standalone facts, but the extraction rule and the materiality rule are written and frozen before anyone sees a system's output. Two trained reviewers extract claims from a calibration sample independently, the rules are revised until the reviewers agree, and only then are the rules locked for the main run.

Gold evidence is marked by people who work with the records every day. Anthropic's engineering guidance on agent evaluations is plain about the standard: a task is good when two domain experts, grading apart from each other, would agree on pass or fail. Questions experts cannot agree on go into a disagreement file that is reported with the results, because quietly dropping them would flatter every system tested.

The preregistered protocol: what is fixed before any system runs

Everything that could be tuned after seeing results is written down and time-stamped first: the question set, the document collection and its version pin, the gold evidence, the systems and their settings, the metrics, the pass bar and the rule for publishing results that disappoint.

The protocol below is the asset this page publishes. Entries marked illustrative show the precision expected of each line; they say nothing about data SourceX holds, because SourceX sources datasets on request rather than keeping them in stock.

The preregistered protocol: what is fixed before any system runs
ElementFixed before the first runIllustrative entryWhy it is locked
Decision and pass barThe deployment decision and the bar a system must clearDeploy only if the material-claim support rate clears the preset bar in every task familyStops the bar moving to fit a favored system
Records and version pinDocument collection, version identifiers and a question date per itemPolicies, contracts and tickets frozen as of one stated date, each with a content hashLets anyone re-check a citation later
Question setSampling frame, task families and exclusionsLogged requests from three task families, with missing-evidence items kept inPrevents picking easy questions after the fact
Gold evidenceExpert-marked passages per material claim, plus no-answer flagsTwo experts per item; disagreements adjudicated and loggedDefines supported before any output exists
Systems and settingsEvery system, model version, retrieval setting and promptEach system run at its default and at its best documented configurationSeparates configuration effects from the system itself
Metrics and denominatorsPrimary and secondary metrics and how each is computedPrimary: supported material claims divided by all material claimsBlocks choosing a metric after the results
Uncertainty and minimum countsInterval method, clustering unit and minimum questions per familyStandard errors clustered by question; family results shown only above the preset countKeeps small gaps from being read as wins
Publication ruleWhat is released whatever the outcomeFull ledger counts per system, including null and negative resultsPrevents a report that carries only good news

How the results would be scored, with error bars that hold up

The primary metric is the material-claim support rate: claims scored as supported, divided by all material claims in the answers, with uncited material claims kept in the denominator. A system therefore cannot raise its score by citing less.

Three secondary metrics sit beside it. Citation precision, as Liu, Zhang and Liang define it, is the share of citations that support the statement they are attached to. Abstention accuracy is measured only on items where gold evidence confirms that no record answers the question, reported next to the rate of unnecessary refusals on answerable items. The version error rate is the share of citations that point to a superseded version.

Claims sit inside answers and answers inside questions, so they are not independent observations. Miller's paper on adding error bars to evaluations (arXiv 2411.00640, 2024) recommends clustered standard errors when questions come in related groups, and reporting the number of questions beside each estimate. The protocol clusters by question and prints both numbers in every results table.

Grading combines the three grader types Anthropic describes for agent evaluations: code checks that each citation resolves to a record identifier in the pinned collection, a model judge for support, and human experts. Anthropic advises calibrating model judges closely with human experts, and GDPval shows why: OpenAI reported that its experimental automated grader agreed with human expert graders 66 percent of the time. The protocol reports judge-expert agreement on a held-back sample before any judge score counts, and reverts to expert grading wherever agreement falls below the preset level.

Could a result reflect something other than citation quality?

Yes, and the first alternative is the one the brief names: until the study is conducted, this page is a research commission, not evidence. No system has been run, no claim graded and no result produced, so nothing here ranks any product.

At a glance

Could a result reflect something other than citation quality

One case challenges the headline metric directly. A system that refuses often can post high citation precision while failing the people who need answers, which is why abstention accuracy and unnecessary refusals appear next to the support rate and why no single number decides a deployment.

Once results exist, five rival explanations must be ruled out before any conclusion stands. Each has a planned check:

  • Question selection: compare results on the full frame with a random subsample drawn before grading began.
  • Judge preference: report expert-only scores on a fixed sample and flag any system whose rank changes.
  • Leakage: use records that were never public and log when each one was created.
  • Configuration: report default and best documented settings as separate rows, never blended.
  • Collection drift: re-hash the collection when the study closes and void items whose records changed.

Which records and permissions the benchmark needs

The benchmark needs business records the company has approved for evaluation use, plus permission to publish the redacted examples a methods report must show. Being able to export records is different from having the right to license them for testing, and de-identification alone does not create that right.

Useful collections share three traits: questions people actually asked, documents with revision history and effective dates, and enough breadth that some questions have no answer inside the collection. Policy manuals with change logs, contract files with amendments, and help desk tickets that cite knowledge articles are typical candidates; each needs a rights review and a privacy review before anything leaves the company.

This is where SourceX's process applies. SourceX does not keep evaluation collections on a shelf; it looks for matching records at U.S. businesses after a buyer asks, and its request form captures scope, volume, language, format, timing and licensing needs. Delivery follows the supplying company's sign-off, which means approving the scope and terms and signing the agreement. A team planning this study can submit the requirement with its intended use and acceptance test; doing so implies no current stock and no particular result.

Methods note: status, sources and what is not measured

Status on 2026-10-26: this is a proposed study. SourceX has not run it, holds no results and has published no benchmark. Results are not yet published and will appear only if the study is conducted under the protocol above.

Sources: the comparison table and metric definitions draw on Liu, Zhang and Liang (2023) and their released annotation files, FActScore (2023), the Stanford RegLab and Yale legal tools study (2024), Miller (2024), OpenAI's GDPval page, NIST AI 600-1 (2024) and Anthropic's engineering guidance on agent evaluations, each checked on its publisher's own site on 2026-10-26. The claim ledger, the materiality rule and the protocol are this article's own framework.

Not measured: how often people act on unsupported claims, answer latency or cost, citations to images and spreadsheets, records in languages other than English, and any system's current performance. The study as designed would not measure these either; each needs its own protocol.

Frequently asked questions

What is an enterprise answer citation benchmark?

It is a test of whether an AI system's answers about a company's own records cite those records correctly, claim by claim. Each material statement is checked against the exact version of the document the system cited, and failures are classified by type. The result shows how often a reader could verify an answer by opening its sources.

How is citation accuracy different from answer accuracy?

An answer can be right while its citation is wrong, or it can cite a real record that does not say what the answer claims. Citation accuracy asks whether the cited record supports the claim; answer accuracy asks whether the claim is true. A benchmark for business use needs both, which is why the ledger separates miscited claims from unsupported ones.

Can a language model grade citations instead of human experts?

Sometimes, but only after its agreement with experts has been measured on the same kind of records. Anthropic advises calibrating model judges closely with human experts, and OpenAI reported that GDPval's experimental automated grader agreed with expert graders 66 percent of the time. Use a model judge where agreement clears a preset bar and experts everywhere else.

Why do document versions matter when grading citations?

Business documents change: a policy, price list or contract amendment can replace the text a system retrieves. If the collection is not frozen and each question is not dated, a citation can be judged against text that did not exist when the question was asked. Pinning versions with content hashes lets anyone re-check a result later.

Are there SourceX results for this citation study yet?

No. This page publishes the study design only; no system has been tested and no results exist. If the study is run, the report will include every outcome count per system, including results that favor no one.

Sources

  • A 2023 human evaluation of four generative search engines found 51.5 percent of generated sentences fully supported by citations and 74.5 percent of citations supporting their sentence. Source
  • The study's released verifiability judgments label each statement-source pair as complete, partial or no support, with annotators copying the minimal supporting evidence. Source
  • FActScore splits a generation into atomic facts and reports the share supported by a reliable source: 58.3 percent for ChatGPT and 71.5 percent for PerplexityAI on biographies. Source
  • Legal research tools from LexisNexis and Thomson Reuters each hallucinated between 17 and 33 percent of the time in a 2024 study. Source
  • Evaluation results should use clustered standard errors when questions come in related groups and report the number of questions with each estimate. Source
  • GDPval tasks were written by experienced professionals and graded blind by experts; an experimental automated grader agreed with human experts 66 percent of the time. Source
  • NIST AI 600-1 (July 2024) names confabulation, confidently presented false content, as a generative AI risk. Source
  • Anthropic's evaluation guidance: a good task is one where two domain experts reach the same verdict; model judges should be calibrated with human experts; graders are code-based, model-based and human. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies