AI uses for records
Business evidence authority benchmark: a protocol for conflicting records
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
This is a protocol, not a result. A business evidence authority benchmark gives a model several records that disagree about one fact, plus the rule that decides which record governs, and asks for the controlling answer and the source behind it. Two domain experts adjudicate every case independently. Grading checks the answer, the cited source and the rule, and rewards abstention when the rule leaves the conflict open.
Evaluation split
Business evidence authority
Key takeaways
- Source authority is a separate skill from finding the newest document or the most repeated value; a benchmark has to make recency and majority the wrong answer often enough to tell them apart.
- Each case needs the conflicting records, their provenance, the written rule that decides between them and an expert-adjudicated answer that names the controlling record.
- Two experts label every case blind to model outputs; disagreements go to a third adjudicator and genuinely open conflicts become abstention items rather than being forced to one answer.
- Hold out whole companies and whole conflict families, report clustered confidence intervals and pre-register metrics before any model is run.
- No results are reported here; the page publishes the method so a team can run it on rights-cleared cases.
How could a benchmark test source authority across conflicting business records?
Build cases in which two or more business records state different values for the same fact, supply the rule that decides which record governs, and ask the model for the governing value, the record it came from and the rule it applied. That is the whole idea; the rest of this protocol is about making the cases fair, the labels trustworthy and the scores interpretable.
A concrete case: an order form lists one payment term, the master agreement lists another, and an invoice in the ERP repeats a third. The question is not which is newest or which appears most often. It is which one the parties agreed would control, and whether the evidence is enough to say.
This page is a method. It reports no measured results, no model rankings and no error rates. Any figure below is either cited from a published source or labeled as an illustrative design target.
Why is source authority a different skill from retrieval or recency?
Retrieval finds relevant text; authority decides which relevant text wins. A survey of knowledge conflicts in language models separates conflicts between context and the model's memory, conflicts among pieces of context, and inconsistencies within memory itself. Business authority problems are mostly the second kind: several retrieved records that disagree.
Existing conflict benchmarks show the gap is real but measure it on general knowledge. ClashEval built more than 1,200 questions across six domains, perturbed the supporting content and found models often adopted the wrong retrieved value over a correct prior. WikiContradict assembled 253 human-annotated real contradictions from Wikipedia and found models often struggle to identify and handle them.
Neither setting supplies a governing rule. In business records the rule usually exists in writing: a precedence clause, a policy naming the system of record, an approval limit. A model that answers with the latest or most frequent value can look competent on general conflict sets and still be wrong whenever the written rule points elsewhere. The benchmark should isolate exactly that failure.
Which conflict families should the benchmark cover?
Cover the families where a written rule, not judgment about truth, settles the conflict, plus one family where the rule leaves the conflict open. Public contracting offers a clean example of a written rule: FAR clause 52.215-8 resolves inconsistencies in a solicitation or contract by a fixed order, starting with the Schedule (excluding specifications) and ending with the specifications. Private agreements carry similar order-of-precedence clauses, and companies write similar rules into policies about which system holds the official value.
The families below are illustrative design choices, not findings. Each should have enough cases that a per-family score has a usable interval.
| Conflict family | What disagrees | Rule that decides | Illustrative question |
|---|---|---|---|
| Contract precedence | Master agreement, order form and statement of work give different terms | The agreement's order-of-precedence clause | Which payment term applies to this order, and which document sets it? |
| System of record | CRM, ERP and a spreadsheet show different customer prices | Written policy naming the system that holds the official price | What price should this invoice use, and from which system? |
| Signed amendment vs informal message | A signed change order and a later email propose different delivery dates | Amendment clause requiring signed changes | Which delivery date is binding on the effective date shown? |
| Approval authority | Two approvals exist, one above the approver's limit | Delegation of authority matrix in force at approval time | Was this purchase validly approved, and by whom? |
| Policy version in force | Current refund policy differs from the version in force at the event date | Effective dates on policy versions | Which refund rule applied to this request on its date? |
| Regulated record vs working note | A signed record and a draft note disagree on a quantity | Rule that the signed record is the record of account | What quantity is on record, and which document controls? |
| Unresolved conflict | Records disagree and no rule in the bundle decides | None; the correct response is to abstain and name the missing rule | Can the controlling value be determined from this evidence? |
What does each expert-adjudicated authority case contain?
Every case is a bundle of records with provenance, a rule excerpt and an adjudicated answer. Provenance fields borrow the structure of the W3C PROV-O ontology, which describes records as entities produced by activities and attributed to agents, with derivation links between entities. Recording who produced each record, in which system and from what earlier record lets the grader check whether the model cited the right document for the right reason.
At a glance
What does each expert-adjudicated authority case contain
Use the schema below as written. Fields marked for the grader are hidden from the model.
| Field | Content | Grader use |
|---|---|---|
| case_id and company_group | Stable case key and a pseudonymous company identifier used for holdout splits | Split and cluster key; never shown as evidence |
| conflict_family | One of the families in the coverage table | Per-family scoring and stratification |
| question | One fact to determine, with the as-of date | Defines what a correct answer must contain |
| evidence_bundle | Each record with doc_type, source system, created and effective dates, author role, signature status and derivation link | Shown to the model; the grader checks cited record ids against it |
| rule_excerpts | Verbatim precedence clause, policy text or authority matrix in force on the as-of date | Shown to the model; the grader checks the cited rule id |
| gold_answer and gold_source | The controlling value and the record id that supplies it | Exact or normalized match for answer and source |
| gold_rule_path | The rule id and the step that excludes each losing record | Rubric check of the model's stated reasoning |
| abstain_correct | True when no rule in the bundle decides the conflict | Scores abstention as correct and any forced answer as wrong |
| adjudication_record | Both expert labels, agreement flag, adjudicator decision and ambiguity note | Agreement statistics and audit trail; hidden from the model |
| rights_status | License basis, de-identification method and release tier | Release gate; cases without clearance are excluded |
How should experts adjudicate each case?
Two qualified experts label each case independently, without seeing model outputs or each other's answers, and a third adjudicates disagreements. Experts should come from the function that owns the rule: contract managers for precedence cases, controllers for system-of-record and approval cases, operations leads for policy versions.
Label quality matters more than volume. A widely cited audit estimated an average label error rate of 3.4 percent across ten common benchmark test sets, and found that once mislabeled test examples become somewhat more common, a smaller model can outrank a larger one. Authority labels are harder than most, so the procedure has to catch errors before they enter the test set.
- Each expert records the answer, the controlling record, the rule and a confidence level from a fixed scale.
- Agreement on answer and source is required for a case to enter the test set directly; report chance-corrected agreement per family.
- Disagreements go to an adjudicator who reads both rationales and either resolves the case, rewrites the rule excerpt to remove an accidental ambiguity, or marks the case as genuinely open.
- Genuinely open cases become abstention items rather than being forced to one label.
- A random audit sample of agreed cases is re-labeled later by a different expert to estimate residual error.
- Every change to a case after adjudication is versioned, and the benchmark release names the case set version.
How should cases be sampled and held out?
The unit of sampling is the company, not the record. Cases from one company share templates, naming habits and rule wording, so a random split by case leaks those patterns into the test set. Assign whole companies to development or test, and hold out at least one conflict family entirely so the team can see whether a gain generalizes to a rule type the system never saw.
Stratify by family and by difficulty: how many records are in the bundle, whether the winning record is the oldest, and whether the winning value appears only once. An illustrative target is a balanced number of cases per family with at least a third in which the controlling record is neither the newest nor the majority value; the exact numbers belong in the pre-registration, not in a later write-up.
Disclose the population (industries, company size bands, source systems), collection dates, the denominator of candidate cases screened and why each was excluded. Keep the test set private. If any part is released, record a canary string, and consider an ordering-based contamination test of the kind proposed for black-box models, which compares the likelihood of a benchmark's canonical order against shuffled orders.
Which baselines, metrics and pre-registration steps make a result credible?
Score the answer, the source and the rule separately, and compare against simple heuristics that the benchmark is designed to defeat. If a model cannot beat 'pick the newest record' by a clear interval on the cases where the newest record loses, it has not learned authority.
Steps
Which baselines, metrics and pre-registration steps make a result credible
Baselines: newest-record heuristic, majority-value heuristic, system-of-record lookup alone, the model without rule excerpts, the model with rule excerpts, and a held-out human expert who did not label the case. Metrics: answer accuracy, controlling-source accuracy, rule accuracy, abstention precision and recall, a recency-trap rate (the share of cases where the newest record loses and the model chose it) and a majority-trap rate. Report every metric per family with bootstrap confidence intervals resampled by company, because cases within a company are correlated.
Checklist · 7 items
- Pre-register the research question and the primary metric before any model run.
- Fix the case set version, holdout companies and held-out family.
- Record model names, versions, prompts, retrieval settings and decoding parameters for every run.
- State the minimum effect that would count as a difference between conditions.
- Define how abstentions, malformed outputs and refusals are scored.
- Plan the error analysis categories in advance: wrong rule, right rule misapplied, wrong record cited, missed abstention.
- Commit to publishing negative and null results alongside any gains.
0 of 7 ticked. Tick each item as you go.
What rights, release limits and blind spots does the benchmark carry?
Every case must come from records the source company has the right to license for evaluation, with names, account numbers and other identifiers removed or replaced consistently so that cross-record links still work. Contracts and policies may contain counterparties' confidential terms; the license needs to cover those or the excerpts must be rewritten by the expert while keeping the rule's logic intact.
The benchmark measures whether a model applies a stated rule to stated records. It does not decide what a contract means in law, and a case's gold answer is an expert reading for evaluation purposes, not legal advice. Precedence rules vary by agreement and jurisdiction, and the public-contracting example above is itself under revision as the FAR is rewritten, so any rule excerpt must carry its effective date.
Release conditions belong in the pre-registration: who may run the test set, whether scores may be published, and when cases are retired because they have been exposed. A benchmark that leaks loses its value as a measure of authority reasoning.
Sourcing rights-cleared conflict cases
Real conflict bundles come from companies whose contracts, orders, invoices and approvals sit in different systems for years. A team that needs such bundles can ask SourceX to source them from U.S. businesses; nothing is held in stock, so the request itself should name the conflict families, the number of companies, the systems, the language, the delivery format and whether the use is evaluation only.
Each contributing company signs off on scope and terms before a single record is delivered, and SourceX's later use, sublicensing and distribution of the de-identified cases are bounded by that signed agreement. That sequence suits a held-out benchmark, because evaluation-only and no-redistribution terms can be written in before any case leaves the company.
The email says approved, the system says rejected: which record is ground truth?
Neither record is ground truth on its own. The outcome label comes from reconstructing four things: when each statement was made and when it was recorded, whether its author had authority for that decision on that date, whether the approval carried a condition or was later superseded, and what actually happened downstream. Only then does the case get a final label, and sometimes the honest label is undetermined.
Systems produce these conflicts on their own. Salesforce's help documentation for classic approval processes says an email reply registers only when a supported word such as approve, approved, yes, reject, rejected or no sits on the first line, and that a misspelled word or one on the wrong line is not registered. An approver can write approved in good faith while the request stays pending in the system, and a later decision there can then contradict the email.
Treat the email as a statement, the system status as a recorded state, and test both against what followed. Anthropic's guide to agent evals draws the same line: a booking agent may say the flight is booked, but the outcome is whether a reservation exists in the database. In business records the downstream state is the purchase order issued, the goods shipped or the invoice paid.
| Explanation for the conflict | What the timeline shows | Record that settles it | Supported outcome label |
|---|---|---|---|
| Approval never registered | An authorized approver replied approved; the system never captured the step | System audit trail, the approval email and the downstream action | Approved with an incomplete system record if the downstream action followed; otherwise undetermined |
| Conditional approval lapsed | The email approved subject to a check that never passed | The record of the condition, such as a legal sign-off, credit check or budget release | Rejected; the approval lapsed with its condition |
| Superseded decision | A later decision by the same or a higher authority reversed the email | Both decisions with timestamps and the approvers' roles | Rejected; the later decision controls |
| No authority | The email's author was outside the delegation limit on that date | Delegation of authority table in force on the decision date | Rejected; the email is a recommendation |
| Different object | The email approved one amount or version and the system rejected another | Quote or document versions with their identifiers | Two decisions, each labeled on its own object |
| System entry error | The rejection contradicts every later record and was corrected or disputed | Correction history and the process owner's confirmation | Approved with a wrong recorded status, only with owner confirmation |
| Evidence runs out | No record shows the condition, the authority or the downstream state | None available | Undetermined, naming the missing record |
How do you build the evidence timeline without filling in the gaps?
Write one row per event in the order events happened, and give a missing record its own row instead of inferring what it would have said. Each row carries the event time, the time the record was written, the actor, the actor's role and limit on that date, the record identifier and the single fact the record asserts. A gap row names the record that should exist, where it was searched for and the fact that it was not found.
Process owners then adjudicate from the timeline alone under the page's double-labeling procedure, and each disagreement is written against the row that caused it. A disagreement pinned to a row tells the next reviewer whether the dispute is about authority, timing or a missing document, which a bare final label cannot. The entry below is illustrative; the company, amounts and limits are invented.
| Step (illustrative) | Record | What it asserts | Authority and timing check |
|---|---|---|---|
| Day 1 | Purchase request in the procurement system for a $42,000 software renewal | The IT manager submitted the request | Delegation table version 3 in force |
| Day 2 | Email from the department head: approved, go ahead | An approval stated outside the system | Department head limit is $25,000 under version 3; not enough for this amount |
| Day 6 | Rejection in the system by the CFO: defer to next quarter's budget | The required approver rejected the request | CFO approval required above $25,000; within authority |
| Day 6 to day 9 | Gap: no record that the requester told the vendor to proceed | Procurement notes and the vendor thread searched; nothing found | Recorded as a gap, not as an event |
| Day 9 | Vendor invoice placed on hold in accounts payable | No purchase order matches the invoice | Downstream state agrees with the rejection |
| Label | Adjudicated by two process owners | Rejected; the email approval was outside authority | One owner attached a vendor-relations note to the gap row |
Does the reconstructed packet beat either record alone?
Test it before paying for reconstruction at scale. Give the same model the same held-out cases in four arms: the email alone, the system record alone, both raw records, and the reconstructed timeline with its authority map. Score each arm against the adjudicated labels with the same model version, prompt and number of trials.
Pre-register the margin the packet must clear over the both-raw-records arm. Joining more records adds linkage work and privacy exposure, and if the packet does not beat raw records by that margin, the cheaper dataset is the right purchase. No results from this design are published yet.
- Outcome accuracy against the adjudicated label, compared as paired differences on the same cases.
- Statement trap rate: cases where the model adopts the email's stated outcome and the adjudicated label differs.
- Status trap rate: cases where the model repeats a system status that the timeline shows was wrong or incomplete.
- Abstention on undetermined cases, scored correct only when the answer names the missing record.
What should a dataset brief for conflicting business instructions specify?
A brief for conflicting instructions turns one repeated failure into a task, an evidence boundary and a pass test before any records are requested. Instructions are not facts: a sales director's email saying ship the order today and a credit hold in the ERP are both directions, and the question is which one the agent may act on, not which value is true.
Two published conventions help write the rank rule. OpenAI's Model Spec treats an instruction as not applicable when it conflicts with a higher-authority instruction or is superseded by a later instruction at the same level. The instruction hierarchy paper by Wallace and colleagues built training data with aligned cases, where lower-priority instructions should be followed, and misaligned ones, where they should be ignored, while guarding against over-refusal. A business brief needs the same balance, or an agent learns to refuse every direction that does not come from the top.
Run the non-data checks first. If the agent chooses correctly once each direction's author role and limit are shown beside it, the fix is exposing those fields, not more training cases; if it cannot keep an order on hold because no tool offers that action, the gap is tooling; if qualified owners disagree on which direction controls, the policy needs a decision before any dataset can encode it. The lines that survive become the request itself: SourceX's request form asks for scope, volume, language, format, timing and licensing needs, and datasets are sourced on request rather than held in stock.
| Brief line | What to write | Illustrative entry: orders released over a credit hold | Check before collection |
|---|---|---|---|
| Repeated failure | The wrong action, how often it occurs and where that count comes from | The agent releases held orders after a sales email asks it to | Count taken from logged agent actions, not anecdotes |
| Directions in scope | Each instruction source, the system it lives in and its rank | Credit policy, credit manager release, sales director email, customer request | Ranks confirmed in writing by the process owner |
| Authority evidence per case | The record that proves rank on the decision date | Credit limit table and delegation version with effective dates | Every case carries the version in force |
| Evidence boundary | What the agent may see at decision time; later records used only for grading | Hold reason, order and emails up to the decision | No input timestamped after the decision |
| Pass test | The action, the end state and the authority cited | Order stays on hold; release requested from the credit manager | Graded on system state, not on the reply |
| Case mix | Shares of aligned, misaligned and no-rule cases | Includes valid releases by the credit manager | Aligned cases present, so refusal is not rewarded |
| Non-data checks | Tests run before any data is bought | Show role and limit beside each direction; confirm a hold action exists; measure owner agreement | Failure persists with authority shown, and owners agree |
Which conflicts does evidence settle when no written rule decides?
Status, timing, identity and quantity conflicts are settled by a later or more specific record rather than by a precedence clause or policy, so they belong beside the rule-settled families above as evidence-settled rows. They share a cause public conflict benchmarks rarely model: one system updated before another, someone changed one record and not its twin, or two records describe the same customer under different keys. Nobody published a falsehood.
Each row names the evidence that usually settles the conflict, what a defensible answer looks like and what counts as wrong. The rule-settled families stay in as controls, so a team can see whether a model that handles written rules also handles conflicts without one. The examples are illustrative.
| Conflict family | Business example | Evidence that usually settles it | Defensible answer | Wrong answer |
|---|---|---|---|---|
| Status | The CRM shows an opportunity closed-won; billing shows no active subscription | The signed order with its start date, and the provisioning log | Won but not yet live, because the start date is next month | Customer is active, with no provisioning record cited |
| Timing | A ticket shows closed before the customer's last reply, but the two systems stamp different time zones | Audit-trail timestamps converted to one zone | Order uncertain when both events fall inside the stated clock tolerance | Sequence read from display times without converting zones |
| Identity | Two customer accounts share an email address but list different billing addresses | Account history, merge log and the address on paid invoices | Possibly one customer, held for review when the history does not decide | Accounts merged, or one chosen, with no linking evidence cited |
| Quantity | A purchase order says 100 units, the receipt 96 and the invoice 100 | Receiving record, credit memo or an approved variance | Pay for 96 and hold the balance when no variance approval exists | Pay for 100 as though every unit arrived |
How do paired evidence bundles test whether the answer moves with the evidence?
Ship every case as two evidence bundles built from the same rights-cleared records: a complete bundle that contains the record settling the conflict, and an incomplete bundle with that one record, the pivot, removed. On the complete bundle the expected output is a resolved answer; on the incomplete one it is a conditional answer that names the pivot, or the type of record that would decide.
The pair tests what a single bundle cannot. A model that gives the same confident answer to both either ignored the pivot when it was present or invented it when it was absent, and on a one-bundle benchmark both failures look like competence. QuestBench, from Google DeepMind, reports that in its planning tasks models tended not to hedge even when explicitly offered a not sure option, so a benchmark that never withholds the deciding record cannot see that behavior.
Public faithfulness work points the same way. FaithEval, from Salesforce AI Research, contains 4.9K problems across unanswerable, inconsistent and counterfactual contexts, and its authors report that larger models are not necessarily more faithful to the context. RAMDocs, from Wang and colleagues, stores for each question a list of gold answers, one per legitimate reading of an ambiguous name, and a separate list of wrong answers. A business case needs both lists, so the schema gains hidden fields beside gold_answer: the variant, the pivot record id, the expected certainty (resolved, conditional on a named record, or irreducible), the defensible alternatives and the wrong answers the records contradict.
Report flip consistency beside the verdict rates: the share of pairs in which the model answers conditionally on the incomplete bundle and resolves the complete one. Split flip failures in two. An invented pivot, resolving the incomplete bundle anyway, points to overconfidence or leakage from training data; an ignored pivot, staying conditional with the deciding record present, points to retrieval or reading failures inside long bundles.
How should a reconciliation be graded when experts legitimately split?
With three verdicts, right, defensible or wrong, built on a claim-level check against the bundle. Before grading, sort every expert disagreement into one of three kinds. A slip, such as a misread date, is corrected. A disagreement that disappears once one expert sees a record the other overlooked means the bundle was really incomplete, so it becomes a new incomplete variant with that record as its pivot. A disagreement that survives full evidence is genuine, and both answers enter the defensible set, so a model giving either one earns credit; abstention stays the correct response only where nothing in the bundle decides.
Then split each answer into atomic claims and score every claim against the bundle, not against the world. FActScore, from Min and colleagues at EMNLP 2023, set the pattern: break long-form text into atomic facts and measure the share a knowledge source supports, and its released code lets a team swap in its own knowledge source. Here the knowledge source is the bundle, so a claim that happens to be true but appears in no record is still unsupported.
Citations get a separate check. ALCE, the citation benchmark from Princeton NLP, scores citation recall by asking whether the passages a sentence cites jointly entail it, and citation precision by flagging cited passages the sentence did not need; its evaluation code runs both checks with an NLI model. The same two tests transfer to records. Only load-bearing claims, the ones the conclusion cannot survive without, decide the verdict; an incidental claim can cost precision but never the verdict.
Apply the verdict rules in order and stop at the first that fires, because a correct final answer resting on an invented record is still wrong:
- Wrong if any load-bearing claim is unsupported or contradicted.
- Wrong if the answer appears on the case's wrong-answer list or falls outside the adjudicated answer set.
- Wrong if the model resolves an incomplete bundle or an irreducible case with full confidence, or gives the same confident answer to both bundles of a pair.
- Defensible if the answer is a listed alternative, or a conditional answer that names the pivot record or its type, with its load-bearing claims supported.
- Right if the answer is gold_answer, its load-bearing claims are supported and its stated certainty matches the expected certainty.
| Claim label | Test against the bundle | Illustrative example (quantity conflict) | Effect on the verdict |
|---|---|---|---|
| Supported | A cited record states it | Receipt R-2 records 96 units on the delivery date, citing R-2 | Counts toward right or defensible |
| Inferable | Follows from cited records in one step the answer writes out | So 4 units remain unreceived, citing the order and R-2 | Counts when the step is shown |
| Unsupported | No record in the bundle states or implies it | The vendor shipped the balance last week, with no record cited | Wrong if load-bearing; a precision loss if incidental |
| Contradicted | A record in the bundle states otherwise | No credit memo exists, when memo C-7 sits in the bundle | Wrong, whatever the final answer |
| Overcited | A cited record adds nothing the claim needs | Citing the vendor master file to support a unit count | Citation precision loss only |
Sources for this section
- Salesforce's help documentation says an email reply to a classic approval request registers only when a supported word (approve, approved, yes, reject, rejected or no) is on the first line, and a misspelled or misplaced word is not registered. Source
- Anthropic's engineering guide to agent evals (January 9, 2026) defines the outcome as the final state in the environment: a flight-booking agent may say the flight is booked, but the outcome is whether a reservation exists in the database. Source
- OpenAI's Model Spec treats an instruction as not applicable when it is misaligned with a higher-level instruction or superseded by a later instruction at the same level. Source
- The instruction hierarchy paper (Wallace and colleagues) generated training data for aligned instructions, which the model should follow, and misaligned ones, which it should ignore, while balancing against over-refusal. Source
- FaithEval has 4.9K problems across unanswerable, inconsistent and counterfactual contexts, and its authors report larger models are not necessarily more faithful. Source
- RAMDocs records gold answers for each disambiguated entity, a list of wrong answers, and documents typed correct, misinformation or noise. Source
- QuestBench reports that in its Planning-Q domain, models tend not to hedge even when offered a not sure option. Source
- FActScore measures factual precision over atomic facts against a knowledge source that users can replace with their own. Source
- ALCE's evaluation code scores citation recall by NLI entailment from the joint cited passages and flags unnecessary citations for precision. Source
Frequently asked questions
What is a business evidence authority benchmark?
It is an evaluation set in which business records disagree about a fact and a written rule decides which record governs. The model must give the controlling value, cite the record and name the rule, or abstain when the rule leaves the conflict open.
How is this different from testing RAG on outdated documents?
Outdated-document tests ask whether a system finds the current version of one document. Authority tests compare different record types, such as a contract, an order form and an invoice, where the newest record is often not the one that controls.
Why use two experts per case instead of one?
Authority labels depend on reading rules carefully, and a single reader's mistakes go straight into the test set. Independent double labeling with adjudication catches errors, reveals ambiguous rules and produces agreement statistics a reader can check.
Which benchmark result would hold on new companies and task families?
Only a result measured on held-out companies and at least one held-out conflict family, with confidence intervals resampled by company. Gains that appear only on companies or families seen during development usually reflect templates rather than authority reasoning.
Does this page report any model results?
No. It publishes the protocol, schema and pre-registration checklist only. Results require rights-cleared cases, adjudicated labels and runs recorded under the pre-registered settings.
Sources
- A survey of knowledge conflicts in LLMs separates context-memory, inter-context and intra-memory conflicts. Source
- ClashEval curated over 1,200 questions across six domains with perturbed context and found models often adopt incorrect retrieved content over correct prior knowledge. Source
- WikiContradict contains 253 human-annotated real inter-context conflicts from Wikipedia, and models often struggle to identify and manage them. Source
- FAR 52.215-8 resolves inconsistencies in a solicitation or contract by a fixed order of precedence, from the Schedule to the specifications. Source
- W3C PROV-O models provenance with Entity, Activity and Agent classes and relations such as wasDerivedFrom and wasAttributedTo. Source
- Ten widely used benchmark test sets carry an estimated average label error rate of 3.4 percent, and a modest rise in mislabeled test examples can reverse which model ranks higher. Source
- Contamination of a test set in a black-box model can be tested by comparing the likelihood of the canonical example order with shuffled orders. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.