Skip to content

AI uses for records

Multi document business answer dataset: 7 checks run on MultiHop-RAG

By SourceX Editorial · Updated

8 sections

Short answer

A multi document business answer dataset should list every source document each answer needs and be scored on those sources, not just the final string. In MultiHop-RAG, a public benchmark pinned at commit 3dd4d4e, 1,086 of 2,255 answerable queries (48.2%) need three or four distinct articles, 59.6% have yes or no answers, and one article supports 496 queries.

Document stack

Multi document business

This page covers multi document business answer dataset: 7 checks run on MultiHop-RAG.Illustrative

Key takeaways

  • Record every required document per answer by ID, version, role and effective date, and score retrieval of all of them.
  • In MultiHop-RAG at commit 3dd4d4e, 48.2% of answerable queries need three or four articles; inference questions are the widest.
  • Final-answer accuracy is easy to inflate: 59.6% of answers are yes or no, and always answering yes scores 58.2% on them.
  • Check independence: one article supports 496 queries, and one string answers a third of inference questions.
  • Test whether one document would have been enough before calling an item multi-document, and pin the corpus with a hash.

How often does an answer need more than one document?

In MultiHop-RAG, a public benchmark built to require evidence from several documents, nearly half of the answers need three or more. Read at commit 3dd4d4e, it holds 2,556 queries over a corpus of 609 news articles; 2,255 come with evidence, and every one of those cites at least two distinct articles: 1,169 cite two, 774 cite three and 312 cite four. That puts 1,086 answers, 48.2% of those with evidence, across three or four documents.

The shape depends on the question. Inference questions, which ask the reader to identify something from facts in several reports, are the widest: 671 of 816 need three or four articles. Comparison and temporal questions mostly need two, and never more than three. The remaining 301 queries are deliberately unanswerable, with no evidence and the fixed answer 'Insufficient information.'

Treat these numbers as the shape of a well-built test, not as a measure of how often business questions span documents. MultiHop-RAG was assembled to require several sources, so it cannot say how many questions in a company's own knowledge base would. We found no published census of it for enterprise records; the way to get it for a given corpus is the single-document test described below, run on a sample of real questions.

Counted by distinct article URL, documents per answer break down as follows:

How often does an answer need more than one document?
Query typeQueries2 documents3 documents4 documents
Inference816145359312
Comparison8566781780
Temporal5833462370
Null (no evidence)301000
All queries2,5561,169774312

What MultiHop-RAG is, and why the commit has to be pinned

MultiHop-RAG is a question answering dataset for testing retrieval and reasoning across documents, released with a paper the README lists as accepted at COLM 2024. Its README describes 2,556 queries with evidence distributed across 2 to 4 documents, and states that the dataset is licensed under ODC-BY. According to the paper's abstract, the authors ran two experiments, one on how well embedding models retrieve the evidence and one on how well models including GPT-4, PaLM and Llama2-70B reason over it, and found that existing RAG methods handle multi-hop queries poorly.

The construction is documented in the repository. Its pipeline notes describe collecting news through a news API, crawling article bodies, filtering by token length, removing duplicates, keeping factual sentences, and grouping articles by keyword or entity before claims and question-answer pairs are generated. Each query keeps its evidence list: the article's title, source, URL, category, publication time and the supporting fact.

Pinning matters here in a very literal way. The dataset files were removed from the repository's main branch in commit 33d3ad0 on 16 July 2026, while the README still points readers to a Hugging Face loader. Every count in this article comes from the last commit that held the files, 3dd4d4e from 3 April 2025, read through Git LFS objects whose SHA-256 hashes match the pointers in that commit. A benchmark that can move under a link is a reminder that a buyer's own evaluation set needs a frozen snapshot and a hash.

Seven checks for a multi document business answer dataset

Run seven checks before trusting any multi-document answer set, whether it is a public benchmark or a dataset licensed from a company. Each check below names the failure it catches, what it found in MultiHop-RAG at commit 3dd4d4e, and the rule a business dataset should meet. MultiHop-RAG passes the structural checks, and its weaknesses are exactly the ones a business corpus has to fix.

The counts come from a script that reads only the two dataset files and saves no query text, answers or author names; the pass, risk and gap verdicts in the figure are editorial readings of them:

Seven checks for a multi document business answer dataset
CheckFailure it catchesMultiHop-RAG at 3dd4d4eRule for a business dataset
Count documents, not factsTwo facts from one document counted as two sources168 queries list more evidence facts than distinct articles; by article, every answer needs 2 to 4Count distinct document IDs and versions per answer
Single-document leakAn item one document can answer alone0 answerable queries rest on a single articleTest each document alone; relabel items one document answers
Guessable answersFinal-answer scores that reward guessing1,343 of 2,255 answers (59.6%) are yes or no; always saying yes scores 58.2% on themScore cited evidence, not only the answer string
Answer concentrationA few answers repeated across many items271 of 816 inference answers (33.2%) are the same string; the top two cover 59.1%Cap repeats or report accuracy per distinct answer
Evidence reuseItems that are not independentOne article supports 496 queries (22.0% of those with evidence); 6 articles support 100 or moreReport results by document cluster
Partial evidenceConfident answers when one needed document is missing301 fully unanswerable queries; no item withholds only part of the evidenceAdd variants with one required document removed
Authority and datesA stale or superseded document treated as currentPublication times on every fact, median spread 26.7 days; no status, version or supersedes fieldRequire effective dates, status and supersession links
A scorecard of seven checks run on MultiHop-RAG: distinct documents per answer 2 to 4 and single-document leaks 0 pass; yes or no answers 59.6 percent, top inference answer 33.2 percent and most reused article 496 queries are risks; partial-evidence items and authority fields are absent.
MultiHop-RAG dataset files at commit 3dd4d4e. Verdicts are SourceX's reading of the counts. Source: yixuantt/MultiHop-RAG; SourceX count.

Why a correct final answer can still be unsupported

A model can return the right final answer while having read the wrong documents, or only one of them, and a string-match score cannot tell the difference. In MultiHop-RAG, 93.3% of comparison and temporal questions have a yes or no answer, so roughly half of those can be passed by chance, and a model that answered yes to every binary question would score 58.2% on them without retrieving anything.

Inference answers are guessable in a different way. Only 72 distinct answers cover the 816 inference questions, and the most common one appears 271 times. A system that has learned the frequent entities of the corpus will look strong on accuracy while its retrieval is weak. The fix is to score what was cited: whether every required document was retrieved and whether each claim in the answer is supported by a cited one. The evidence completeness guide linked below covers set recall and citation scoring in detail, so this page does not repeat it.

Order and position deserve their own controls when three documents have to be read together. Liu and colleagues found, in multi-document question answering and key-value retrieval experiments, that models tend to use relevant information best when it sits at the start or end of a long input and worst when it sits in the middle, even for models built for long contexts. A multi-document evaluation should therefore record where each required document appeared in the context and report results with the order varied.

What business documents add that news articles lack

Business answers combine documents that play different roles and carry different authority, and news articles rarely do. A warranty decision needs the purchase, the product terms in force and the repair history; a credit question needs a contract clause, an amendment and an invoice. Each source has a job, and an answer that uses the right three types but the superseded version of one is wrong.

MultiHop-RAG's evidence has none of that structure. Its 609 articles are peers: 598 of the 2,255 answerable queries (26.5%) draw every document from a single publisher, and the rest mix publishers whose reports carry equal weight. Its evidence carries publication times, and across the 2,255 answerable queries the earliest and latest evidence sit a median of 26.7 days apart; the temporal questions depend on that ordering. In a business corpus the same spread is where trouble starts, because the later document may amend, replace or contradict the earlier one, and the dataset has to say which governs.

Three properties separate a business answer set from a news benchmark: each required document is labeled with its role and its effective date; documents that supersede others say so; and the documents come from different systems joined by real keys, such as an order number linking a contract, an invoice and a support ticket. Join keys also have to survive de-identification, or the answer can no longer be traced across its sources.

Would one document have been enough?

The strongest objection to a multi-document dataset is that a single summary document often holds the whole answer, so the multi-hop structure is an artifact of how the test was built. That objection is right often enough to test for: business corpora are full of recaps, quarterly reviews and case summaries that restate facts from several sources.

The test is simple. For each item, present each required document alone, then each pair, and ask whether the reference answer still follows. If one document suffices, relabel the item as single-document and record which document it was. Then search the corpus for derived documents, such as an email recap or a case note, that restate the facts of the required set, and decide whether they count as evidence or as a shortcut to exclude.

MultiHop-RAG would pass the first half by construction: none of its 2,255 answerable queries rests on one article. A company's corpus will not, and the share of items that survive the test is the honest answer to how often that company's questions need several documents. Report it alongside the results, with the count of items relabeled.

Where the MultiHop-RAG counts stop

The counts describe how one benchmark was built, not how business questions behave, and five limits shape how far they travel. Documents are identified by exact URL, so a story syndicated under two addresses would count twice; we did not check for syndicated copies. The yes-or-no and concentration figures describe reference answers, not any model's behavior.

The evidence lists come from the dataset's generation pipeline, which grouped factual sentences before writing questions. Another article in the corpus may also support an answer without being listed, so a document's absence from an evidence list does not prove it is irrelevant, and a scorer should accept alternative evidence where the corpus contains it rather than punish a system for finding it.

The 301 unanswerable queries all share one answer string, 'Insufficient information.', which a system can learn to emit without judging anything. A business test of abstention should vary the expected wording and, more usefully, ask the model to name the document it is missing. Finally, the median date spread of 26.7 days is a property of news coverage windows; in business records the gap between a contract and its amendment can be years, which makes supersession checks more important, not less.

Specifying a multi-document answer dataset for a supplier

Specify the dataset by the evidence structure each answer needs, then let the supplier show how its records meet it. A request that says only 'questions with multi-document answers' invites items that name several documents but need one, or that join documents no real workflow would join.

SourceX licenses business data directly from companies and sources datasets on request rather than holding stock, so a buyer can set out the document types, the joins between them, volume, format and an acceptance test such as the single-document check on the request-data form. Documents move only after the supplying company has approved what is in scope, agreed the terms and signed.

A useful specification asks for the following on every item:

  • The question, the reference answer and the answer type (entity, number, date, yes or no), so guessable types can be weighted or capped.
  • Every required document by ID and version, with its role in the answer (rule, case fact, amount, exception) and its effective date.
  • Supersession links for any document that amends or replaces another, and a flag when sources conflict.
  • The join keys that connect the documents across systems, preserved through de-identification.
  • A single-document test result: which documents, if any, answer the question alone.
  • Variants with one required document withheld, labeled with the expected behavior: say what is missing rather than fill it in.
  • Corpus snapshot date and hash, so the set can be re-run as documents change.

Frequently asked questions

What is a multi document business answer dataset?

It is a set of questions whose correct answers require facts from two or more business documents, such as a contract, an amendment and an invoice. Each item lists every required document, its role and version, so a model can be scored on whether it used all the evidence and not only on its final answer.

How many documents should a multi-document question require?

As many as the real decision needs; public multi-hop sets such as this one use two to four. In MultiHop-RAG, 1,169 answerable queries need two articles, 774 need three and 312 need four. More important than the count is that each document is necessary: test that no single document answers the question alone.

Why not just score the final answer?

Because many answers can be guessed. In MultiHop-RAG, 59.6% of answers with evidence are yes or no, and answering yes every time scores 58.2% on those. Scoring the cited documents shows whether the system actually combined the sources.

Is a public benchmark enough for an enterprise RAG evaluation?

It is a good check of structure, not of business content. Public sets such as MultiHop-RAG use news articles of equal authority, with no versions, supersession or cross-system joins. An enterprise evaluation needs documents whose roles, dates and authority match the decisions the system will support.

How do I keep a multi-document test set reproducible?

Freeze the corpus and the item file, record a hash of each, and pin the version used in every result. MultiHop-RAG's files left its main branch in 2026 while its README still linked elsewhere, which is the situation a pinned snapshot protects against.

Sources

  • MultiHop-RAG's query and corpus files at commit 3dd4d4e: 2,556 queries over 609 articles, with the evidence list for each query, counted for this article. Source
  • The MultiHop-RAG README describes 2,556 queries with evidence across 2 to 4 documents and an ODC-BY license. Source
  • The repository's pipeline notes describe how news articles were collected, filtered to factual sentences and grouped before questions were generated. Source
  • Commit 33d3ad0 removed the dataset files from MultiHop-RAG's main branch on 16 July 2026. Source
  • The MultiHop-RAG paper reports two experiments, on evidence retrieval and on model reasoning, and finds existing RAG methods weak on multi-hop queries. Source
  • Liu and colleagues found that models use relevant information best at the start or end of a long input and worst in the middle. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies