Skip to content

AI data market

Retrieval dataset sample buyer test: require confusing near matches

By SourceX Editorial · Updated

10 sections

Short answer

A retrieval dataset sample should include near matches that a real system confuses with the right source: superseded versions, sibling records for another product or customer, drafts beside approved documents, date- or region-scoped variants, permission-restricted copies and questions with no answer. Accept the sample only if each case names its controlling source and required evidence, and a baseline retriever actually ranks the near matches close to it.

Retrieval

Retrieval dataset sample

This page covers retrieval dataset sample buyer test: require confusing near matches.Illustrative

Key takeaways

  • A sample of easy queries cannot separate a strong retriever from a keyword baseline.
  • Require labeled near matches by type: old versions, sibling entities, drafts, scoped variants, restricted copies, split evidence and unanswerable look-alikes.
  • Every case needs a controlling source, dates, permissions, a required evidence set and support spans.
  • Prove hardness by checking that decoys rank near the controlling source in a baseline retriever.
  • Score abstention separately, because some tools drop queries with no relevant documents.

Which confusing near matches should a retrieval dataset sample include?

Include the near matches your production system will actually face, at a stated quota, with each one labeled. Seven kinds cover most enterprise corpora: an older version of the right document, a sibling document about a different product, customer or region, a draft next to the approved text, a variant that applies to a different date or jurisdiction, a copy the asking user may not see, an answer split across two documents where one alone looks complete, and a question the corpus cannot answer even though a document looks close.

The reason is simple. A sample built from queries that match one obvious document will score well for almost any retriever, so it tells a buyer nothing about how the full dataset will separate good systems from weak ones. The hard cases are where the purchase pays off, and they are also the cases a supplier is most tempted to leave out because they take expert time to label.

This guide is a buyer test for the sample, not a guide to building a whole corpus. It gives an acceptance plan you can send to a supplier before the sample arrives, a check that the near matches are really hard, and the judgment rules that keep the scores honest.

Why does a sample of easy queries hide the problem?

Easy queries hide it because lexical overlap alone already finds the answer. The BEIR paper, which evaluates retrievers zero-shot across heterogeneous datasets, reports BM25 as a robust baseline: a simple keyword method holds up across many domains. If a sample's queries are mostly answered by the document with the most shared words, a buyer cannot tell a strong retriever from BM25.

The Dense Passage Retrieval paper shows what hard looks like in practice. Its training data used BM25 negatives: top-ranked passages that do not contain the answer but match most question tokens. Those are exactly the near matches a sample should prove it contains, because they are what a retriever has to learn to reject.

So the first question for any sample is not how many queries it has. It is how many of its queries have a near match that a baseline ranks high, and whether the label says which document actually controls the answer.

Real enterprise corpora make this harder than public benchmarks suggest. Support articles get rewritten every release, policies carry regional variants, and teams copy text into internal notes. A sample that has none of this mess was probably built from a cleaned subset, and the full delivery will either share that cleanliness (and be less useful) or arrive messier than the sample promised.

Seven near-match types and what each one tests

Each type below tests a different way retrieval can return a plausible but wrong source. The examples are illustrative and drawn from a typical software support and policy corpus.

Seven near-match types and what each one tests
Near-match typeIllustrative exampleWrong retrieval looks likeRequired label
Superseded versionRefund policy v3 beside current v4Answer quotes the old refund windowControlling version ID and effective date
Sibling entitySetup guide for Pro plan beside Enterprise planSteps for the wrong planEntity the query is about, with its ID
Draft beside approvedUnapproved draft of a security FAQAnswer cites text never publishedApproval status of each candidate
Date or region scopeHoliday schedule for EU staff beside U.S. staffCorrect text for the wrong scopeScope fields the answer depends on
Permission-restricted copyInternal escalation note mirroring a public articleAnswer exposes text the user cannot seePermitted document set per test user
Split evidencePrice in one document, eligibility in anotherHalf-answer from one sourceFull required evidence set
Unanswerable look-alikeQuestion about a feature that was never builtConfident answer from a related pageExpected abstention and the closest decoy

The hard case and source evidence acceptance plan

Send this plan to the supplier with the sample request, so the sample is built to it rather than graded against it after the fact. Each row is a gate; a sample that fails a reject condition goes back with the row number. Version 1.0, dated 2026-10-18. Quotas are set by the buyer for its own use case; the plan only requires that they are stated in advance.

The hard case and source evidence acceptance plan
CheckThe sample must showEvidence fieldAccept ifReject if
Query intentWhat the asker needs and for which entityquery_intent, entity_idEvery query names its entity and taskIntent is inferred from the answer
Controlling sourceThe one document that governs the answercontrolling_doc_id, versionOne controlling source per answerable querySeveral marked controlling with no rule
Date and scopeWhen and where the answer holdseffective_from, effective_to, scopeDated sources carry datesDates missing on versioned documents
PermissionsWhich test users may see each documentacl_group per document and userEach restricted case has an allowed and a denied userNo permission metadata at all
Hard negativesNear matches for each case, by typedecoy_doc_ids, decoy_typeEvery stated type meets its quotaDecoys are random or off-topic
Required evidence setAll documents needed for a full answerrequired_doc_idsSplit-evidence cases list every partOnly one document per case
Citation supportWhich passage supports each answer claimsupport_spansClaims map to spans in controlling sourcesSpans point to decoys
AbstentionQueries the corpus cannot answeranswerable flag, closest decoyUnanswerable cases are present and labeledAll queries are answerable

What else has to arrive with the sample for the plan to work?

The labels are only checkable if the sample ships with the corpus around them. A list of queries and answers without the decoy documents, the version history and the permission groups lets a buyer read the cases but not test them. Ask for a self-contained package that can be loaded into a fresh index and run without the supplier's help.

The package also has to make clear what is still missing. A short sample note should say which near-match types were hard to find in the supplier's records, and which were constructed by selecting from real records versus written by a labeler. A buyer can accept a constructed case if it is labeled as such; what it cannot accept is a sample that hides how its hard cases were made.

  • The sample corpus: every controlling source and every decoy, with stable document IDs that match the labels.
  • Version history for versioned documents, with effective dates and the version each case treats as controlling.
  • Permission groups for documents and for each test user, in plain fields rather than embedded in file paths.
  • The labeling guide used, including how labelers chose the controlling source when two documents disagreed.
  • Labeler notes on disputed cases and how each dispute was resolved, with the reviewer's role.
  • A count of cases per near-match type, so the buyer can check quotas without reading every row.
  • A statement of which records were excluded for rights or privacy reasons, so gaps are not mistaken for clean coverage.

How do you prove the near matches are actually hard?

Run two retrievers on the sample corpus and look at where the decoys land. Use BM25 and one dense embedding model the buyer already trusts. For each case, record the rank of the controlling source and of each labeled decoy. A decoy that never appears in the top results for either retriever is not a near match for practical purposes, however reasonable it looks to a person.

A practical acceptance test is that, for most hard cases, at least one decoy ranks at or above the controlling source in at least one retriever. Set the share yourself and write it into the plan. If almost every controlling source ranks first for both retrievers, the sample is easy, and the full dataset probably is too.

Keep the output in a format other tools can read. The DPR repository's retriever training format stores, for each question, positive contexts plus separate pools of negative and hard negative contexts with the same structure. Asking the supplier to deliver decoys in a similar question-level structure makes the hardness check repeatable when the full dataset arrives.

Watch for false negatives while you do this. A high-ranking decoy is sometimes a second valid source that the labeler missed. Each one should be reviewed by the source authority owner and either relabeled as relevant or confirmed as wrong with a reason. Track how many decoys were relabeled; a high share suggests the labeling guide itself needs work before the full dataset is built.

Is the retrieved source permitted for this asker and valid on the question date?

Relevance is not enough; the evidence also has to be permitted for the asker and valid on the date of the question. A document can be the right topic and still be the wrong answer because it expired last quarter or because the person asking has no access to it.

Enterprise search systems already trim results by access. Microsoft's Graph search documentation, for example, describes querying OneDrive and SharePoint content to which the signed-in user has read access. A retrieval sample that drops permission metadata cannot test whether a pipeline keeps that boundary, so every restricted case in the sample should come with at least one allowed and one denied test user.

Dates work the same way. For each versioned case, the label should name the controlling version as of the question date, so that replaying an old question returns the version in force then. A buyer can test this directly by asking the same question with two different as-of dates and checking that the controlling source changes.

If the supplier cannot provide permission or date metadata because the source system never recorded it, that is a legitimate limit, not a reason to fake it. Narrow the sample to cases where the metadata exists, and note in the acceptance record that permission and date tests are out of scope for this dataset.

How should judgments handle unjudged documents and unanswerable queries?

Decide before scoring what an unjudged document and an unanswerable query count as, because common tools make silent choices. The trec_eval documentation notes that queries with no relevance information are ignored, and that a relevance value of -1 marks a pooled document that was not judged. An unanswerable query with no relevant documents can therefore drop out of a retrieval score without anyone noticing.

Score abstention separately. For each unanswerable case, check whether the system declined or flagged low confidence, and whether it cited the closest decoy. Report that rate next to the retrieval scores rather than inside them. An unanswerable case should also name its closest decoy, so that a confident wrong answer can be traced to the page that tempted it.

Generated-answer metrics need the same care. The Ragas documentation defines context precision as the degree to which relevant chunks are ranked above irrelevant ones, and context recall as the share of claims in a reference answer supported by the retrieved context; both depend on a reference. A sample whose reference answers were written from a decoy will reward retrieving the decoy, which is why the plan requires support spans in the controlling source.

Worked review (illustrative): a 50-query support knowledge base sample

This review is illustrative; the company, counts and findings are invented to show how the plan is applied. A buyer requested a 50-query sample from a supplier's support knowledge base and ticket history, built to the plan above with a quota for each near-match type.

The first pass found 50 queries with controlling sources and support spans, but only three of the seven near-match types present: superseded versions, sibling entities and split evidence. There were no permission-restricted cases and no unanswerable queries. Under the plan, that is a reject on the permissions and abstention rows, returned with row numbers.

The hardness check on the resubmitted sample showed decoys ranking at or above the controlling source in 31 of 42 hard cases for at least one retriever, which met the share the buyer had set. Four high-ranking decoys turned out to be valid second sources; the source authority owner relabeled them as relevant. Two cases were dropped because nobody could say which version had been in force on the question date.

The counterexample matters as much as the pass. One query looked hard because its decoy ranked first, but the decoy was an identical copy of the controlling article under a different URL. That is a duplicate, not a near match, and the buyer asked for duplicates to be collapsed before the full delivery rather than counted toward the quota.

The accepted plan then becomes part of the delivery specification. The buyer keeps the same gates, quotas and hardness check for the full dataset, so the sample is a preview of the acceptance test rather than a one-time impression. If the full delivery's decoys rank lower than the sample's, or a near-match type thins out, the plan already names the row to reject on.

Requesting a retrieval sample with hard cases built in

Hard cases come from real operating records: version histories, product-specific documentation, approval workflows, permission groups and questions customers actually asked. Those are hard to synthesize convincingly, which is why the acceptance plan asks the supplier to show them rather than describe them.

Retrieval corpora with version chains, permission groups and real decoys are something SourceX locates among U.S. businesses once a buyer asks, not something kept on a shelf. Attach the acceptance plan when you describe scope, volume, language, format, timing and licensing needs, so the supplier builds the sample to your gates.

Frequently asked questions

What should a hard case and source evidence acceptance plan include?

Query intent and entity, the controlling source with version and dates, permission groups per test user, labeled decoys by type, the full required evidence set, support spans for each answer claim, and labeled unanswerable queries. Each row should state an accept and a reject condition before the sample is built.

How many near matches should a retrieval sample have?

There is no universal number. Set a quota per near-match type based on how often each occurs in your production queries, write it into the plan, and check the sample meets it. What matters more than the count is that a baseline retriever actually ranks the decoys close to the controlling source.

What is the difference between a hard negative and a duplicate?

A hard negative looks relevant but does not support the answer, such as an outdated version or a page about a sibling product. A duplicate is the same content under another identifier. Duplicates should be collapsed or labeled, not counted as hard cases.

Why do unanswerable queries need their own score?

Some evaluation tools ignore queries that have no relevant documents, so they can disappear from retrieval scores. Scoring abstention separately shows whether the system declines when it should, instead of answering confidently from a close decoy.

Should the sample include permission metadata?

Yes, if the system will serve users with different access. Without access groups on documents and test users, a buyer cannot test whether retrieval respects permissions, and a correct-looking answer may expose text the asker should not see.

Sources

  • The BEIR repository describes BEIR as a heterogeneous benchmark of diverse retrieval tasks with corpus, queries and qrels for each dataset. Source
  • The BEIR paper evaluates retrieval models zero-shot on a heterogeneous benchmark and reports that BM25 is a robust baseline. Source
  • The Dense Passage Retrieval paper used BM25 negatives, top-ranked passages that do not contain the answer but match most question tokens, as hard training negatives. Source
  • The DPR repository's retriever training format stores positive contexts and separate pools of negative and hard negative contexts per question with the same structure. Source
  • The trec_eval documentation says queries with no relevance information are ignored and that a relevance value of -1 marks a pooled document that was not judged. Source
  • Ragas defines context precision as how well a retriever ranks relevant chunks above irrelevant ones in the retrieved context. Source
  • Ragas defines context recall as the share of claims in a reference answer supported by the retrieved context, which requires a reference. Source
  • Microsoft Graph search documentation describes querying OneDrive and SharePoint content to which the signed-in user has read access. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies