AI uses for records
Enterprise question context experiment: count repairs, not replies
By SourceX Editorial · Updated
10 sections
Short answer
To measure what missing context costs an enterprise Q&A system, hold out real questions with the record that governs each answer, remove one context field at a time (effective date, scope, authority, supporting passage) and score wrong actions and the minutes a person needs to repair them, not reply similarity. A public floor test shows the stakes: without its date, at least 30.2 percent of SituatedQA's temporal items must be answered wrong.
Value drivers
Enterprise question context
Key takeaways
- Score what happens after the reply: actions a record owner would reverse, and the minutes a reviewer needs to find the governing record and fix the answer.
- Remove one context field per arm (effective date, scope, authority, supporting passage) so each cost can be attributed.
- On SituatedQA's dev and test files, no context-blind system can miss fewer than 30.2 percent of temporal items or 70.9 percent of geographic ones.
- Scoring against the union of answers across contexts marks every one of those 3,227 unavoidable errors as correct.
- Write the stop rules before the first run, including a pause when held-out questions lack an identifiable governing record.
What does missing context cost an enterprise Q&A system?
Two things: wrong actions taken on an answer that sounded right, and the human effort needed to notice and undo them. An answer about a refund window, a discount limit or an escalation path can be accurate for some customers on some dates and wrong for the person asking today. Whether it is right depends on context that sits outside the question: the date the policy took effect, the segment or region it covers, whether the document was approved or superseded, and which passage actually supports the answer.
Most evaluations of enterprise Q&A compare the reply text with a reference answer. That shows whether the system can produce a plausible answer, not what an answer costs when the context was missing. The experiment set out here moves the score downstream, to the action a person would take on the reply and the work needed to repair it, and removes one piece of context at a time so each cost can be traced to a field.
The hypothesis it tests is simple to state and easy to falsify: on questions whose correct answer depends on a context field, removing that field raises the rate of wrong actions and the minutes of repair compared with the same system given full context, and the rise differs by field. If the rates do not move, the field is not worth paying for in a licensed dataset.
Why a reply-matching score hides the cost
Because a reply can match an answer that is correct for some other context, and a scorer that accepts any listed answer counts it as right. Benchmarks built for context-dependent questions already separate the two. SituatedQA, a dataset of questions whose answers depend on when or where they are asked, stores for each item the gold answers for its own context and, separately, the union of answers across all of that question's contexts. Its evaluation script reports exact-match accuracy against the item's own answers and a separate any score against the union.
Keeping those two numbers apart is the right instinct, and an enterprise evaluation should copy it. In a company the union is large: last year's policy, this year's policy, the European version, the draft that was never approved. A reply that matches any of them reads well, passes a similarity check and still sends a refund, a quote or an escalation down the wrong path. The cost is in the action, so the score has to be too.
A public floor test: what SituatedQA loses without the context
Some of the cost of missing context can be measured without any model at all. If a system sees a question but not its date or location, it must give the same answer to every context of that question, so the most it can get right is the share of contexts that one answer covers. The remainder is a floor: errors no model improvement can remove, because the information is not in the input. SituatedQA's authors describe the dataset as questions whose answers change with when and where they are asked, and report that existing models struggle with answers that are frequently updated or tied to uncommon locations.
The floor was computed on SituatedQA's development and test files at commit 1ef8545. The temporal files hold 6,218 items from 2,054 questions; 981 questions appear with two or more dates, and 751 of those (76.6 percent) change their gold answer across dates (a question changes answer when its items carry two or more distinct normalized gold lists). A context-blind system must miss at least 1,877 temporal items: 30.2 percent of all of them, and 36.5 percent of the items whose question has more than one date. The geographic files hold 1,904 items from 480 questions, and 479 of those questions change answer across locations. There the floor is 1,350 items, 70.9 percent.
Then the scoring trap. In every one of the 3,227 floor errors, temporal and geographic together, the context-blind answer appears in the item's union-of-contexts list. Scored against the union, the floor disappears: every unavoidable error counts as correct.
These are not enterprise questions. They come from open-domain sources, the contexts are dates and places rather than policy versions and customer segments, and exact-match scoring under-credits answers phrased differently from the gold. What transfers is the mechanism: an effective date works like SituatedQA's date and a region or segment like its location, and a company's own records decide how large the floor is for its questions.
The design, fixed before the first run
Fix every element below in writing before any arm sees a held-out question, and publish the plan with the result. The thresholds in the table are illustrative values a team would replace with its own; the structure is the part to copy.
Two rules carry most of the weight. The held-out set is frozen with its gold labels before arms run, so nobody tunes a prompt on the questions that decide the comparison. And interim looks are either forbidden or done with a sequential method designed for them: Johari and colleagues show that ordinary p-values and confidence intervals become unreliable when experimenters keep checking and choose when to stop.
The stop rules, in full: the run ends when every arm has been scored on the pre-registered number of questions, set by a power calculation from the expected wrong-action rate of the full-context arm. An arm stops early if reviewers find, in its first scored batch, more irreversible wrong actions than a ceiling written into the plan, since no such arm would be deployed. The whole run pauses if more than an agreed share of held-out questions turns out to have no governing record its owner can name, because the gold is then unreliable. Nobody compares arms before the end unless the plan names a sequential method for doing so.
The design elements and the decision each one fixes:
| Element | Decision fixed in advance | Why it matters |
|---|---|---|
| Hypothesis | Removing a field raises the wrong-action rate on questions that depend on it by at least 5 points (Illustrative threshold) | A stated threshold makes the result falsifiable |
| Held-out set | Real questions from tickets and chat, each with the record its owner names as governing on the question date, frozen with gold labels | Gold that cites a record can be audited; gold written from memory cannot |
| Strata | Questions tagged by the field they depend on: date, scope, authority, passage, or none | Each arm is judged on the questions it can affect |
| Arms | Full context; minus effective date; minus scope; minus authority markers; minus supporting passage; context-blind baseline | One removal per arm attributes the cost to one field |
| Primary metric | Wrong-action rate per stratum, judged blind to arm by a reviewer holding the governing record | Measures consequence, not wording |
| Secondary metrics | Repair minutes and edits per answer; abstention quality; confident wrong answers | Separates cheap mistakes from expensive ones |
| Stop rules | Fixed sample per arm; harm stop; validity pause; no unplanned interim looks | Prevents stopping when the numbers look good |
Context omission map: one field, one arm, one failure
Each arm removes one kind of context, and the map below says what that context settles, which record proves it and how an answer fails without it. The fields are the ones a buyer would ask a supplier to keep with each question, so the map doubles as a specification for licensed knowledge records.
Two fields need care. Authority is not the same as access: a document a user may read can still be a draft, and a document they cannot read may be the one in force, so permission filters belong in a separate test. And the supporting passage arm does not delete the governing record; it replaces it with related text that does not settle the question, which is the failure retrieval systems produce most often:
| Context field | Question it settles | Record that proves it | Failure when it is missing | Arm |
|---|---|---|---|---|
| Effective date or version | Which rule was in force on the question date | Version history with effective dates, not edit dates | Current policy applied to a past transaction | Minus effective date |
| Scope | Which customers, regions or products the answer covers | Applicability fields or the owning team's scope note | Right answer for the wrong segment | Minus scope |
| Authority | Whether the source is approved and current | Approval status, owner and supersession link | Draft or retired guidance quoted as policy | Minus authority markers |
| Supporting passage | Whether the cited text actually supports the answer | The passage the owner marks as governing | Fluent answer resting on related text | Minus supporting passage |
| None (control stratum) | Questions with one answer in every context | Owner confirms the answer does not vary | No expected change; detects noise | All arms |
Scoring wrong actions and the minutes it takes to repair them
A wrong action is an action the record owner would reverse: an approval, a quote, a routing decision or an instruction that contradicts the governing record for that asker on that date. Score it on the action the reply would lead to, judged by a reviewer who holds the governing record and does not know which arm produced the reply. Grade severity by reversibility, because an incorrect internal note and an incorrect refund are not the same cost.
Repair effort gets its own measurement. Machine translation research has long scored output by the editing a person must do to make it acceptable: Snover and colleagues, presenting at AMTA in 2006, scored translation output by how much it must be edited to match a reference and defined a human-targeted variant. Borrow the idea, and record two numbers per answer: the edits needed and the minutes a timed reviewer spends finding the governing record and writing the corrected answer. Edits and minutes can disagree, which is why both are kept.
Abstention needs a guard so an arm cannot win by refusing to answer. Selective classification research, such as El-Yaniv and Wiener's work in the Journal of Machine Learning Research, treats coverage (how often a system answers) and the error rate on the answers it gives as two sides of one risk-coverage trade-off. Report wrong actions at each coverage level, and count a decline as correct only where the gold says the question should be asked back or escalated.
Evidence-grounding test: does the cited passage govern this asker on this date?
Before a held-out question enters the set, and again when a reply is graded, run the same checks on the evidence. A question whose gold answer fails them is not ready, and a reply that cites evidence failing them is wrong however well it reads:
- A governing record is named for the question, by its owner, with an identifier that resolves.
- The record was in force on the date the question was asked, by its effective date rather than its last edit.
- The record's scope covers the asker's segment, region and product.
- The record is approved and not superseded; if it was superseded later, the gold reflects the question date.
- The cited passage, not just the document, supports every claim in the answer.
- Where two records conflict and no owner has decided, the gold answer is to ask or escalate, and the conflict is kept rather than resolved by the labeler.
Could the gap come from something other than missing context?
Yes, and the design has to separate four look-alikes. The first is the alternative the brief names: the relevant text may be present but outdated, unauthorized or too thin to support a confident answer. That is a corpus problem, not a context omission, and the control is to grade full-context answers against the grounding checks; if the full-context arm fails them often, fix the set before comparing arms.
The second is retrieval failure. If the governing record is never retrieved, removing a context field changes nothing, so log whether the governing record appeared in the context for every answer and analyze the arms on questions where it did. The third is leakage through wording: questions that say 'this year' or name a region carry their own context, so tag them and report results with and without them. The fourth is the reviewer: graders who can see which arm produced an answer, or who grade arms in a fixed order, shift scores, so blind the arm and randomize order.
Records that can run the experiment, and what is not published
The experiment needs real questions joined to the records that governed their answers, with effective dates, scope fields, approval status and the passage the owner relies on. Public datasets such as SituatedQA can show the mechanism and the floor; only company records can show the cost for a company's own policies, products and customers. No such enterprise run is reported here: the design is published, the public floor test is computed, and enterprise results are not published.
SourceX licenses business data directly from the companies that hold it and licenses it onward in its own name, and its request page says datasets are sourced on request rather than kept in stock. A buyer who wants records for this experiment can describe them on the request form: choosing Workflow datasets adds fields for Business task, Input / context required, Required outcome labels and Evaluation criteria, where the context fields, the wrong-action labels and the grounding checks above fit directly. Nothing here implies that suitable records exist today or that a model trained or tested on them will improve.
Methods note: SituatedQA files, counting rules and limits
Data: data/qa_data/temp.dev.jsonl, temp.test.jsonl, geo.dev.jsonl and geo.test.jsonl from github.com/mikejqzhang/SituatedQA at commit 1ef8545 (committed 10 August 2022), extracted with git show and read on 28 October 2026 by a script that uses only the Python standard library; no repository code was run and no model was queried. The train files were not used, because the geographic train file holds one context per question.
Rules, as the script applies them: an item is one line; a question group is the set of items sharing an id and question text within one file. Answers are normalized as the repository's evaluation script does: Unicode NFD, lowercase, punctuation removed, the whole words a, an and the removed, whitespace collapsed. The best context-blind answer for a group is the normalized gold string contained in the most items' gold lists, and the floor is the group's item count minus that number. An error is hidden by union scoring when that answer is missing from the item's own gold list but present in its normalized any_answer list. Shares are exact fractions rounded once, half up, to one decimal.
Limits: exact-match gold lists miss correct answers phrased differently, so the floor is stated under the dataset's own scoring; the questions are open-domain, not enterprise; and the floor bounds context-blind systems only, saying nothing about how often a system given the context gets it right. The experiment design has not been run on any company's records.
Frequently asked questions
What is an enterprise question context experiment?
It is a controlled test that asks an enterprise Q&A system the same held-out questions with full context and with one context field removed at a time, such as the effective date, scope, authority markers or the supporting passage. It then compares wrong actions and repair effort across those arms to show what each field is worth.
Why score wrong actions instead of answer accuracy?
A reply can match an answer that is right for a different date or segment and still lead to a wrong refund, quote or escalation. Scoring the action a reviewer would reverse, and the minutes needed to repair it, captures the cost that text similarity misses.
What is a context-blind floor?
It is the smallest share of items a system must get wrong if it never sees the context, because one answer cannot fit every context of a question. On SituatedQA's development and test files the floor is 30.2 percent for date-dependent items and 70.9 percent for location-dependent ones.
How do you stop the experiment from being gamed by abstaining?
Report the error rate on answered questions at each level of coverage, and count a decline as correct only where the gold answer says the question should be asked back or escalated. An arm that declines everything then shows low coverage rather than a low error rate.
Has the enterprise experiment been run?
No. The design and the public floor test are published here; no company's records have been tested and no enterprise results are reported. Running it requires real questions joined to the records that governed their answers.
Sources
- SituatedQA's development and test files hold the temporal and geographic items, gold answers and union-of-contexts answers used for the floor test. Source
- The README describes questions whose answers depend on temporal or geographic context and defines any_answer as the union of answers across a question's contexts. Source
- The evaluation script normalizes answers DrQA-style and reports a separate any score against the union of answers. Source
- The SituatedQA paper says answers to the same question may change with when and where it is asked, and that existing models struggle with frequently updated answers and uncommon locations. Source
- Snover and colleagues (AMTA 2006) measured the editing needed to turn system output into an acceptable version, including a human-targeted variant. Source
- El-Yaniv and Wiener define coverage for selective classifiers and study the trade-off between coverage and risk on accepted samples. Source
- Johari and colleagues show that frequentist p-values and confidence intervals become unreliable when experimenters choose sample sizes by continuously monitoring tests. Source
- SourceX's request page says it licenses data directly from companies, licenses it in its own name and sources datasets on request; the form carries workflow specification fields. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.