Systems and records
Retrieval query reformulation dataset: when user words miss archive labels
By SourceX Editorial · Updated
9 sections
Short answer
Evaluate retrieval across a user-to-archive vocabulary gap with a retrieval query reformulation dataset: real user wording paired with the archive's own label, the mapping type, the controlling document version in force on the query date, the asker's permissions, hard negatives and the expected answer or abstention. Score whether retrieval returns the current, authorized governing source with enough supporting evidence, reported per mapping type, rather than how similar the retrieved text looks.
Retrieval
Retrieval query
Key takeaways
- Vocabulary mismatch is the normal case in enterprise archives, so it needs its own test set rather than a few lucky examples.
- Each record pairs verbatim user wording with the archive's preferred label, a mapping type and the controlling source version in force on the query date.
- Relevance judgments must encode authority, version and access: stale versions score zero and restricted documents fail the run.
- Hard negatives such as superseded versions and sibling labels catch more failures than random documents.
- Report results per mapping type, and treat correct abstention as a pass, not a miss.
How do you evaluate retrieval when users and archive labels use different words?
Build a test set in which every record pairs the words a business user actually typed with the label the archive uses for the same thing, then score whether retrieval returns the controlling, current document that this user is allowed to see. Similarity between the query and a chunk is not the target. The target is the right source, in the right version, for the right person, with enough evidence to answer or a clear reason to abstain.
The gap is old and large. In a 1987 study of spontaneous word choice, Furnas, Landauer, Gomez and Dumais found that two people chose the same term for the same object with probability below 0.20, and argued that systems relying on one designer's word would fail in most attempts. Enterprise archives inherit that problem: the people who filed the records named them for filing, not for the colleague who will search next year.
A retrieval query reformulation dataset makes that gap measurable. It holds the user wording, the archive's term, the type of mapping between them and the evidence a correct answer needs. With it you can test a rewriter, a synonym layer, a hybrid retriever or an embedding model on the cases where wording differs, which are the cases that ordinary benchmarks under-sample.
What the mismatch looks like inside real business systems
The mismatch is usually between plain-language requests and a controlled vocabulary that grew around a system's configuration. In Zendesk, an agent searches "customer can't get in" while the help center article is titled for single sign-on assertion errors. In Salesforce, a case is filed under a Case Reason picklist value such as a product defect code, while the account manager asks about "the broken unit complaint". In NetSuite, a buyer asks for "the refund from the supplier" and the record is a vendor credit.
Jira components, SharePoint managed metadata and ServiceNow categories all behave the same way. Labels are short, internal and occasionally renamed; user wording is long, situational and often borrowed from another department. A reorganization or a system migration can leave two generations of labels in the same archive, which is why the version of the label scheme belongs in every test record.
The W3C SKOS model gives a useful vocabulary for describing these mappings. A concept has at most one preferred label per language, alternative labels for synonyms, acronyms and spelling variants, and hidden labels that stay out of displays but remain available to free-text search, which the SKOS Reference names as the usual home for misspellings. The protocol below borrows those distinctions so annotators classify each pair the same way.
The query to terminology mapping collection protocol
Collect one record per user query and controlling source. Each field below is required unless marked optional; a record missing a required field is held back rather than guessed. Store the protocol version and date with the dataset so later teams know which rules produced it.
| Field | What to record | Where it comes from | Acceptance rule |
|---|---|---|---|
| query_text | The user's words, verbatim, including typos and internal shorthand | Site search logs, ticket subjects, chat requests, emailed questions | No paraphrasing; personal names, emails and account numbers replaced with typed placeholders |
| asker_context | Role, team and permission group of the person who asked, plus the date asked | Directory group at query time, not today's group | Permission group resolvable to an access list as of the query date |
| archive_label | The preferred label of the concept the user meant | Taxonomy, picklist, category tree or KB title list | Label exists in the stated scheme version |
| label_scheme_version | Name and version or export date of the vocabulary | Admin export of the picklist, term store or category tree | One version per record; renamed labels keep both versions |
| mapping_type | exact, alternative, hidden variant, broader, narrower, related or no match | Two independent annotators, then adjudication | Agreement recorded; disputes resolved by the source authority owner |
| controlling_source | Document ID, version and effective dates of the record that governs the answer | Document management system or KB revision history | The version in force on the query date, not the latest |
| required_evidence | The passages or fields an answer must cite to be complete | Annotator marks spans in the controlling source | Every claim in the reference answer traces to a marked span |
| hard_negatives | Two to five near misses: superseded versions, sibling labels, restricted documents | Retriever candidates plus annotator selection | Each negative labeled with why it is wrong |
| expected_behavior | Answer, answer with caveat, or abstain, and the reason | Annotator, reviewed by the source authority owner | Abstain when no permitted, current source supports an answer |
| provenance | Collection date, protocol version, annotator IDs and de-identification log | Collection pipeline | Present on every record before release |
From search log to labeled pair, step by step
Start from real requests, because synthetic questions tend to reuse the archive's own vocabulary and hide the problem you are trying to measure. Then let the people who own each source decide what controls the answer.
Steps
From search log to labeled pair, step by step
Checklist · 8 items
- Define the population: which archive, which user groups and which period the evaluation must represent.
- Sample queries from logs stratified by user group and by the label family they eventually reached, so that rare departments are not drowned out by the help desk.
- Keep failed searches: queries with no click, an immediate reformulation or a follow-up ticket are where mismatch lives.
- Remove personal data from query text before annotation, replacing names and identifiers with typed placeholders so the wording pattern survives.
- Have two annotators map each query to a preferred label and mapping type without seeing each other's answers; send disagreements to the source authority owner.
- Resolve the controlling source as of the query date, using revision history, and record its effective dates.
- Mark the evidence spans, choose hard negatives and set the expected behavior.
- Freeze the corpus snapshot and the label scheme export together, and record both versions in the dataset card.
0 of 8 ticked. Tick each item as you go.
Scoring authority, version and permission instead of similarity alone
Score each run on whether it found the governing record that this user may see, not on whether it found text that resembles the query. Standard retrieval metrics still apply, but the relevance judgments behind them have to encode authority, version and access.
BEIR, a benchmark built to test retrieval across heterogeneous tasks, packages each dataset as a corpus, queries and qrels and reports nDCG, MAP, recall and precision at cutoffs. Use the same structure, but grade the controlling current source highest, give older versions a zero, and drop documents the asker could not open from the judged set for that query. Microsoft's SharePoint documentation describes search results as always security trimmed, so a production comparison should apply the same trimming in the test harness.
Answer-level checks sit on top. The Ragas context recall metric splits a reference answer into claims and measures the share supported by the retrieved context; map that onto the required evidence field so a run that retrieves the right document but the wrong section still fails.
| Check | Pass condition | Failure it catches | How to report |
|---|---|---|---|
| Label reach | Controlling source ranked in the top k for the user wording | Vocabulary mismatch the rewriter or embedding did not bridge | Recall at k by mapping type |
| Version | No superseded version ranked above the current one | Stale policies outranking their replacements | Share of queries with a stale document above the current one |
| Permission | No document outside the asker's access list returned | Leakage through query expansion or shared indexes | Count of violations; any violation fails the run |
| Evidence support | All claims in the reference answer supported by retrieved passages | Right document, wrong section | Context recall against required evidence spans |
| Abstention | Abstains when expected behavior is abstain | Confident answers from retired or restricted material | Precision and recall of abstention |
| Mapping slice | Each mapping type scored separately | Strong exact-match results hiding weak hidden-variant results | Per-slice table, never a single pooled number |
Worked example: an accounts payable archive (illustrative)
The records below are invented to show how filled fields look. They describe a mid-size distributor's accounts payable team searching a NetSuite-backed archive and a SharePoint policy library; no real company or dataset is described.
- The approval matrix row shows why version belongs in the qrels: both versions match the words equally well.
- The petty cash row is a correct abstention, not a retrieval failure.
- The bank change row produces two records from one query wording, one per permission group, because the right behavior differs by asker.
| User words | Archive label and mapping type | Controlling source | Expected behavior |
|---|---|---|---|
| refund from supplier for damaged pallets | Vendor credit (alternative) | AP procedure: vendor credits, version 4, effective this fiscal year | Answer, citing the credit request and approval steps |
| who signs off on big purchases | Purchase approval matrix (broader) | Approval matrix version 7; version 6 is a hard negative | Answer from version 7 only |
| PO doesn't match invoice | Three-way match exception (exact concept, informal wording) | Exception handling work instruction, current revision | Answer with the tolerance rule and escalation step |
| w9 for new vendor | Form W-9 collection (hidden variant) | Vendor onboarding checklist | Answer; the misspelled form is a hidden label, not a new concept |
| petty cash rules | Petty cash policy (exact), retired | None in force; the retired policy is a hard negative | Abstain and point to the current expense policy owner |
| vendor bank change approvals | Vendor master data change control (narrower) | Treasury controls procedure, restricted to treasury group | Answer for treasury users; abstain for AP clerks |
Choosing hard negatives that look right and are wrong
The most useful negatives are documents a careful employee might also open by mistake. Superseded versions of the controlling source, sibling labels in the same category tree, a policy from another business unit and a restricted document on the same topic catch more failures than random documents do.
Record why each negative is wrong, because the reason drives the fix. A superseded version points to metadata and freshness ranking, a sibling label points to the rewriter, and a restricted document points to access control rather than relevance.
Do not assume unjudged documents are irrelevant. The MS MARCO passage ranking documentation notes that a passage marked relevant is a true positive but an unmarked passage may not be a true negative, since not every candidate was seen by a judge. In a small private collection the same thing happens, so re-judge any unjudged document a new system ranks highly before counting it as an error.
Where a reformulation should not count as a win
A rewrite that raises recall is not automatically an improvement. If expansion reaches a restricted label, retrieves a retired policy or replaces a specific request with a broader concept that returns the wrong procedure, the score should fall even when more relevant-looking text comes back.
Ambiguous wording is the main exception to the one-query, one-label rule. "Credit" in a finance archive can mean a vendor credit, a customer credit memo or a credit limit. Record each plausible reading as its own record with the asker context that selects it, and keep a slice of genuinely ambiguous queries where the expected behavior is a clarifying question.
The protocol has limits. It measures retrieval and evidence support, not the quality of the generated prose. It ages as label schemes are renamed and policies replaced, so refresh it when the vocabulary export, the permission model or the corpus snapshot changes. A few hundred records per mapping type is a practical starting size for comparing systems, but treat small slices as directional and report them with their counts.
When the mismatch data has to come from outside your company
Teams building retrieval for a domain they do not operate in often lack the one ingredient this protocol needs: real requests from the people who use the archive. Licensed records such as support tickets with the articles that resolved them, or request logs with the documents eventually used, can supply that wording, provided the supplier holds the rights to license it and personal data is removed before delivery. Access to a system, privacy permission and the contractual right to license are separate questions, and de-identification alone does not settle the last one.
Because SourceX keeps no ready-made collections and instead looks for records at U.S. businesses on request, a team can set out the scope, volume, language, format, timing and licensing a query-to-label collection needs in its request form.
Frequently asked questions
What is a retrieval query reformulation dataset?
It is a collection of real user queries paired with the terms the target archive uses for the same need, plus the documents that should be retrieved. Good versions also record the version of the controlling source, who asked and what evidence a correct answer needs. It is used to test query rewriters, synonym layers and retrievers on vocabulary mismatch.
How is this different from a synonym list?
A synonym list says two terms can mean the same thing. A mapping dataset says what a particular user meant at a particular time, which document governed the answer and whether that user could see it. The extra context is what lets you score retrieval rather than just expand queries.
Can LLM-generated queries replace real user wording?
They help with coverage but tend to echo the archive's own vocabulary, which is exactly the overlap you are trying not to measure. Use real logged wording for the evaluation set and keep any generated queries in a separate, labeled slice. Compare results on both before trusting the generated slice.
How many annotators does the mapping step need?
Two independent annotators per record is a sensible minimum, with disagreements sent to the owner of the source. Record agreement by mapping type, since hidden variants and broader or narrower mappings usually disagree more often than exact matches. Low agreement on a slice is a signal to tighten the guidelines before scaling.
Should permission-restricted documents appear in the test set?
Yes, because leakage through expansion is one of the failures the set must catch. Include restricted documents as hard negatives for users without access and as positives for users with access. Any run that returns a document outside the asker's access list should fail outright.
Sources
- In a study of spontaneous word choice, two people favored the same term for an object with probability below 0.20, and access through one designer's word was predicted to fail 80 to 90 percent of the time. Source
- SKOS defines preferred, alternative and hidden labels; hidden labels stay out of displays but remain available to free-text search and normally hold misspellings. Source
- BEIR is a heterogeneous retrieval benchmark whose datasets load as corpus, queries and qrels and are scored with nDCG, MAP, recall and precision. Source
- MS MARCO passage ranking uses real queries sampled from anonymized Bing logs, and its labels make positives true positives while unlabeled passages may still be relevant. Source
- Ragas context recall is the share of claims in a reference answer that can be attributed to the retrieved context. Source
- SharePoint search results are always security trimmed, so users only see content they have permission to see. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.