AI uses for records
Retrieval evidence disagreement dataset: retrieve both records first
By SourceX Editorial · Updated
9 sections
Short answer
A retrieval task should name every record on each side of a disagreement, plus the record or rule that decides between them, as one required evidence set. Give retrieval credit only when the whole set is returned, then score whether the answer follows the authority rule, respects the user's permissions and surfaces or abstains when no rule applies. A right answer from one record still fails.
Retrieval
Retrieval evidence
Key takeaways
- Make the full conflict set, every side of the disagreement, the unit retrieval is scored on.
- Name the record or written rule that decides which side controls; without one, the right behavior is to surface both.
- A correct answer drawn from one record is a retrieval failure, because the system never saw the conflict.
- Derive each item's permission scope from the source system's access lists and test restricted cases.
- Pair every conflict item with a counterexample whose values only look different, such as the same date in two time zones.
How should a retrieval task require both conflicting records before an answer?
Give every item a required evidence set that names each side of the disagreement and the record that decides between them, and award retrieval credit only when the whole set comes back. An answer that happens to be right but rests on one side of the conflict is scored as a retrieval failure, because nothing in the context showed the system that a disagreement existed or why one side wins.
This mirrors a design choice made years ago in fact verification. The FEVER dataset paired each claim with annotated evidence sentences, and its main score counts a claim as correct only when the label is right and a complete evidence set was supplied. The authors reported that their baseline pipeline dropped sharply in accuracy once correct evidence was required, which is the gap this kind of task is built to expose.
For a retrieval evidence disagreement dataset built from business records, the same rule becomes three requirements per item: retrieve both conflicting records, retrieve the authority record or rule, then answer, abstain or surface the conflict as the item specifies. The rest of this page sets out the fields, the scoring and a worked case.
Concurrent disagreements, not superseded versions
This pack targets two records that are both current and both plausible, held in different systems, that state different values for the same fact. A CRM field and a signed contract disagree on a renewal date; a support ticket and the order system disagree on what shipped; a price list and a quote disagree on a discount. Neither record is an old version of the other.
Researchers describe this as inter-context conflict, one of three types in a 2024 survey of knowledge conflicts in language models, alongside conflicts between retrieved context and the model's own memory and inconsistencies inside the model. WikiContradict, a benchmark of 253 human-annotated instances built from contradictions Wikipedia editors had flagged, found that models often fail to identify and handle these real-world conflicts between passages.
Superseded versions are a different test with different fields: effective dates, supersedes chains and as-of questions. If your disagreement is old policy against new policy, the version-aware evaluation design linked below fits better. Use this pack when the question is which live record controls, not which one is newer.
Why a right answer drawn from one record still fails the item
A correct answer from one record is luck, not evidence handling. If the retriever returns only the signed amendment and the model states the amended date, the output looks perfect, yet the same system would state the CRM's date with equal confidence on the next query where ranking flips. The item exists to test whether the system saw the conflict.
Common retrieval metrics hide this. Context recall in the Ragas library, for example, breaks a reference answer into claims and measures the share supported by the retrieved context. When the reference answer states only the winning value, one retrieved record supports every claim and recall reads as complete. To catch one-sided retrieval, the reference has to include the conflict itself as a claim, or the item needs a separate check on the required evidence set.
Record which side was retrieved on every failure. A system that always finds the CRM record and misses the contract has a different defect from one that finds both and picks wrong, and the fix for each is different: indexing and chunking for the first, ranking or generation for the second.
The contradiction retrieval and authority task pack, field by field
Each item in the pack is one question, one conflict and one decision, with every field filled before the item is used. The field list below is an editorial template, version 1.0 dated 2026-10-18; example values are illustrative and do not describe any real dataset.
Keep the conflict set and the authority record as separate fields. Merging them hides the most useful diagnosis, which is whether the system found the disagreement but not the rule that settles it.
| Field | What it holds | Illustrative value | Check before use |
|---|---|---|---|
| Query and intent | The user question and the decision it serves | When does the Acme service agreement renew? | Intent is answerable from records, not opinion |
| As-of date and user role | The date the answer must hold for and who is asking | 2026-09-30; account manager | Role maps to a real permission group |
| Conflict set | Record IDs for every side of the disagreement | crm:opp-4471 renewal field; dms:amend-2 clause 3 | Each side is current and in the corpus snapshot |
| Authority record | The record or rule that decides which side controls | Contract policy: signed documents override CRM fields | Rule is written down, dated and owned |
| Hard negatives | Similar records that must not be cited | Amendment 1 draft; another account's renewal email | Negatives are close in wording, not just topic |
| Permission scope | Which conflict-set records this role may see | Both visible; pricing exhibit restricted | Scope matches the source system's access list |
| Expected behavior | Answer, surface both, or abstain | Answer with amended date and note the CRM mismatch | Behavior follows from authority and permissions |
| Required citations | Records the answer must cite | dms:amend-2; policy clause | Every material claim maps to a citation |
| Outcome evidence | Later record showing which value the business acted on | Renewal invoice dated to the amended term | Outcome postdates both conflicting records |
| Counterexample | A near-identical item with no real conflict | Same dates in different time zones | Normalization rule is recorded |
Which record decides when two live records disagree?
The deciding record is whatever the business has written down as controlling for that fact, and the item must name it. Without a named authority, the gold answer is the annotator's preference, and a system that disagrees with it is not wrong in any way the business would recognize.
At a glance
Which record decides when two live records disagree
In practice the authority comes from a small number of rule types. Teams should check that each one is a real written policy, not a habit, and record its owner and date in the item.
Have the authority assigned by someone who owns the process, such as the contracts manager or the billing lead, and have a second reviewer confirm it on a sample. Where the two disagree, the item is either rewritten with a narrower question or reclassified as having no rule. Disagreement between reviewers is itself useful information about where the business's own records are ambiguous.
- System-of-record designation: a data governance or finance policy says which system owns a field, such as billing owning invoice terms.
- Document over field: a signed agreement or approved order controls over a value typed into a CRM or ticketing tool.
- Approval level: a record approved by a role with authority for that decision overrides one entered by a role without it.
- Event over status: a dated event record, such as a carrier delivery scan, controls over a status flag that someone sets by hand.
- No rule: if the business has none, the expected behavior is to surface both records and say the conflict is unresolved, not to pick one.
Does the retrieved evidence authorize this answer for this user and date?
Only if every record the answer depends on is visible to the asking user on the as-of date. Retrieval tests often run as an administrator, which hides failures where the controlling record sits in a restricted repository and the visible record is the wrong one.
Search platforms implement this with document-level permissions. Microsoft's Azure AI Search, for example, documents a security filter pattern in which each document carries user or group identifiers and the query passes the caller's groups so results are trimmed to what that caller may see. Whatever the platform, the item's permission scope field should be derived from the source system's access lists, not assigned by the annotator.
Write the expected behavior for the restricted case explicitly. If an account manager can see the CRM field but not the signed amendment, the correct output is not the amended date, which would leak restricted content; it is a statement that the visible record may not be controlling and a pointer to who can confirm. A pack that never includes such items will reward systems that ignore permissions.
Scoring a retriever on conflict sets instead of single hits
Score the conflict set as a unit and keep retrieval and answer metrics apart. The metrics below are computed per item and reported by conflict type, source system pair and permission condition, so that one strong category cannot hide a weak one.
At a glance
Scoring a retriever on conflict sets instead of single hits
Read the metrics together. High conflict-set recall with low expected-behavior accuracy points at generation: the model saw both records and still ignored the rule. A high one-sided rate concentrated on one system usually means that system's records were chunked or indexed poorly, for example a contract PDF split so that the controlling clause lands in a chunk with no account name. Report counts alongside shares, since many packs hold only a few dozen items per conflict type and a single item moves a share noticeably.
Run a simple baseline before comparing systems: retrieve with keyword search alone and with the production retriever, on the same frozen snapshot. If the baseline already finds both sides of most conflicts, the pack is too easy and its hard negatives need work.
- Conflict-set recall at k: share of items where every record in the conflict set is in the top k results.
- One-sided rate: share where exactly one side of the conflict was retrieved, reported by which side.
- Authority recall at k: share where the authority record or rule was retrieved alongside the conflict set.
- Citation support: share of answers whose material claims each map to a cited record that supports them.
- Expected-behavior accuracy: share of items where the system answered, surfaced both or abstained as specified.
- Leakage rate: share of restricted-scope items where the answer revealed content the role could not see.
Illustrative case: a renewal date the CRM and the signed amendment disagree on
Illustrative scenario with invented details. A company's CRM shows a service agreement renewing on the original anniversary date. A signed amendment in the document management system moved the term end by three months, but nobody updated the CRM field. The company's contract policy says signed documents control over CRM entries.
The item asks an account manager's question as of a date between the two possible renewal dates. The conflict set is the CRM field and amendment clause; the authority record is the policy line; hard negatives include an unsigned draft of the amendment with a different date and a renewal email for a similarly named account. The later renewal invoice, dated to the amended term, is the outcome evidence that the business acted on the amendment.
In a test run, a system that retrieves only the amendment and gives the amended date scores on answer accuracy but fails conflict-set recall. A system that retrieves both, cites the policy and states the amended date while noting the CRM mismatch passes every check. A system that retrieves the draft and states its date fails on hard negatives even though it cited a contract document.
The paired counterexample uses the same account with a CRM date and contract date that look different only because one is stored in UTC and the other in local time. The expected behavior is to answer without flagging a conflict, and the normalization rule is recorded in the item so graders do not mark a correct answer down.
Building the pack from real company records
Real conflicts come from operational history: tickets, orders, contracts, approvals and the systems that copy fields between them. Synthetic conflicts tend to change one obvious value in one sentence, while real ones differ in format, wording and location, sit in different systems and leave an outcome trail. RAMDocs, a research dataset that mixes ambiguity, misinformation and noise across documents for one query, remained hard even for the strongest baseline model the authors tested, a reminder that conflicts mixing several kinds of disagreement are harder than single-edit ones.
Freeze the corpus snapshot with the items, keep the source system and record ID for every conflict-set member, and store the outcome evidence with its own date. Strip or replace personal details in contact fields, email headers and free text before records leave the company, and keep stable pseudonymous keys so the joins between conflicting records survive.
AI teams that want conflict sets from real business systems can request them from SourceX, which sources them from U.S. businesses instead of keeping stock; the request form asks for scope, volume, language, format, timing and licensing needs. A supplier releases nothing until it has approved the deal's scope and terms and signed.
Frequently asked questions
What is a retrieval evidence disagreement dataset?
It is a set of retrieval tasks in which the corpus holds two or more current records that state different values for the same fact. Each task names the conflicting records, the record or rule that decides between them and the expected behavior, so a system can be scored on whether it found the whole disagreement, not just one answer.
Do conflicting live records need different test items from stale versions?
Yes. Outdated-document tests check whether a system picks the version in force on a date. Disagreement tests check whether it finds two live records from different systems and applies the business's authority rule. The fields differ: version tests need effective dates and supersedes links, while disagreement tests need a conflict set and a named authority.
What should a system do when no rule decides between the records?
Surface both records, say that they disagree and name who could resolve it. Picking one side without a written authority rule teaches the system to sound certain where the business itself is not, so the item should score abstention or disclosure as the correct behavior.
How many conflicting records should an item require?
Most items need two records plus the authority record, but some facts have three or more sources, such as a price in a list, a quote and an invoice. Require every source that a careful human reviewer would check before answering, and record that set in the item.
Can standard context recall catch one-sided retrieval?
Only if the reference answer includes the conflict as a claim. When the reference states just the winning value, a single retrieved record can support every claim, so a separate conflict-set recall check is the more reliable measure.
Sources
- FEVER pairs claims with annotated evidence, and its main score credits a claim only when the label is right and a complete evidence set is provided; the baseline lost accuracy once evidence was required. Source
- WikiContradict contains 253 human-annotated instances of real-world inter-context conflicts from Wikipedia, and models often struggle to identify and manage them. Source
- A survey of knowledge conflicts in language models distinguishes context-memory, inter-context and intra-memory conflicts. Source
- RAMDocs combines ambiguity, misinformation and noise in retrieved documents for one query and remains hard for the strongest baseline model the authors tested. Source
- Ragas context recall breaks the reference answer into claims and measures the share of claims supported by the retrieved context. Source
- Azure AI Search documents a security filter pattern that stores user or group identifiers on each document and trims results to the caller's groups at query time. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.