AI uses for records
Qualified business answer dataset: score the limitation a yes needs
By SourceX Editorial · Updated
9 sections
Short answer
A qualified business answer dataset labels whether the evidence makes a limitation part of the correct answer, and scores a missing required limitation, or an unsupported one, as an error. In MDN's browser compatibility data, 668 of 49,159 supported answers (1.4 percent) need a documented limitation, rare enough that a grader checking only the yes barely notices when every one is dropped.
Agent run
Qualified business answer
Key takeaways
- Label each question's evidence state: limitation required, note offered, none, or unknown.
- In browser-compat-data, 668 of 49,159 supported answers need a stated limitation, and every partial implementation has its note written down.
- Dropping every limitation costs only 1.4 points on an answer-only grader, so headline accuracy hides the failure.
- A random 100-item test set would hold no caveat-required case about a quarter of the time; oversample the slice and report it apart.
- Charge unsupported caveats as errors too, or a system that hedges everything looks careful.
What makes an answer qualified rather than merely correct?
An answer is qualified when it carries the limitation that the governing evidence attaches to it, and only that limitation. If the source says a feature works but only partly, a bare yes is wrong in the way that matters to the person acting on it, even though a grader that compares yes with yes will mark it right.
This matters for evaluation because the usual scoring pushes the other way. Kalai and colleagues at OpenAI argue in Why Language Models Hallucinate (arXiv, 2025) that binary right-or-wrong grading rewards guessing over admitting uncertainty, and that the fix lies in changing how existing benchmarks score answers. The same logic applies to caveats: when a grader checks only the headline answer, a system that drops every limitation loses nothing, and a system that adds limitations everywhere is never charged for the noise.
A qualified business answer dataset therefore needs three labels per question, not one: the answer, whether the evidence makes a limitation required, optional or absent, and the limitation text with its source. The rest of this page measures how often the second label is required in a large public reference, and turns the result into a scoring grid.
Why a browser compatibility matrix is a fair test bed
Because its maintainers already decide, record by record, when a yes needs a qualification. MDN's browser-compat-data project describes itself as machine-readable compatibility data for web technologies, used by MDN Web Docs, CanIUse, Visual Studio Code, WebStorm and other tools. Each feature carries one support statement per browser, and the schema defines a partial_implementation flag for support that does not implement mandatory specified behavior, is inconsistent with other browsers or confuses feature detection.
The project's data guidelines make the limitation part of the record: if a contributor sets partial_implementation to true, they must write a note describing how the implementation is incomplete. A browser that recognizes an API name with no discernible behavior is also recorded as partial rather than unsupported. That is the shape of most enterprise knowledge, too: a support matrix, a known-issues page or a policy table where the headline is yes and the footnote decides whether the yes is usable.
We read the data at commit bf12f64 (October 27, 2026), counted with our own script and ran nothing from the repository. To avoid derived data, we used only Chrome, Firefox and Safari, the three browsers whose statements are never mirrored from another browser, and the first statement for each, which the schema asks to be the most relevant and most recent.
How often a correct yes needs its limitation: 49,159 answers counted
About one in 74, and the limitation is always written down. Across 20,913 features and three browsers, 49,159 feature-browser pairs are supported in a default release. Of those, 668 (1.4 percent) need a stated limitation to be correct: 577 partial implementations, every one with a note, and 91 supported only under a vendor prefix or another name. Another 1,666 supported pairs (3.4 percent) carry a note without the partial flag, context the evidence offers but does not mark as required.
Two further groups sit outside the supported count but would trap a bare yes just as surely: 520 pairs where support exists only behind a flag and 1,106 where it exists only in a preview release. The table shows every class from the first matching rule, applied in the order listed in the methods note:
| Class of the current statement | Pairs | Share of 62,739 | Correct short answer |
|---|---|---|---|
| Supported, no caveat | 46,825 | 74.6% | Yes |
| Supported, with a note | 1,666 | 2.7% | Yes; note optional |
| Partial implementation | 577 | 0.9% | Yes, with the limitation |
| Prefixed or renamed | 91 | 0.1% | Yes, under another name |
| Behind a flag or preview only | 1,626 | 2.6% | Not by default |
| Not supported or removed | 11,954 | 19.1% | No |
Where the required limitations cluster
In particular browsers and areas, which is where a test set has to look for them. Firefox accounts for 400 of the 668 caveat-required answers, 2.5 percent of its 16,085 supported pairs, against 0.9 percent for Chrome and 0.7 percent for Safari. By area, WebDriver automation features carry the highest rate at 6.3 percent of 1,012 supported pairs, web APIs 1.7 percent of 23,855, and JavaScript language features only 3 of 3,851.
Counted by feature rather than pair, 612 of the 20,298 features supported in at least one of the three browsers (3.0 percent) need a limitation in at least one of them. The notes are specific enough to change what a developer builds. Firefox's support for the Notification requireInteraction property is recorded as partial with the note that it is only supported on Windows and sits behind a flag elsewhere. Safari's support for the ::marker pseudo-element is limited to the color and font-size properties. An answer that says yes in either case sends someone to build on behavior that is not there.
What an answer-only grader cannot see
The cost of dropping every limitation, because it hides inside a small share of a large test. If a system answers yes to all 49,159 supported pairs and never mentions a limitation, a grader that checks only the yes scores it 100 percent, and a grader that requires the documented limitation scores it 98.6 percent. The 1.4-point gap is the whole failure, spread too thin to show up in a headline accuracy.
The opposite failure is just as easy to build. A system that attaches every available note to every yes includes the 577 required partial-implementation notes, but it also attaches limitation language to 46,825 pairs whose evidence carries no note at all, 95.3 percent of the supported answers. A grader that never penalizes unsupported caveats would rate that system as careful.
Random sampling makes both failures worse. Drawing supported pairs uniformly, a 200-item test set would contain about 2.7 caveat-required items on average and would contain none at all 6.4 percent of the time; at 100 items the chance of none is 25.4 percent. A test that may hold zero cases of the behavior cannot measure it, so the caveat-required slice must be sampled on purpose and reported on its own.
The caveat scoring grid: evidence state against answer behavior
Score each answer against the evidence state of its question, not against a single gold string. Take the question of whether Firefox supports the requireInteraction option for notifications. The record marks Firefox 117 support as partial, Windows only and behind a preference elsewhere, so the evidence state is limitation required.
Answer A, yes since Firefox 117, is incorrect: a developer testing on macOS gets default notification behavior instead of one that waits for the user. Answer B, yes on Windows from version 117 and otherwise only behind a preference, citing the compatibility record, is correct. Answer C, support may vary by platform, is incorrect because it names no platform and no route to the preference. A grader that compares only the word yes passes A and B and may pass C; the grid passes only B.
The grid below is the evidence-grounding test for a qualified answer set; each row is a combination a grader must handle, and a buyer can attach it to a data request as the acceptance rule:
| Evidence state | Answer behavior | Score | Reason |
|---|---|---|---|
| Limitation required | Gives the answer and the limitation, citing the passage | Correct | Matches what the evidence says a user needs |
| Limitation required | Gives the answer, omits the limitation | Incorrect | The unqualified answer leads to the wrong action |
| Limitation required | Adds a vague hedge such as may vary | Incorrect | A hedge is not the documented limitation |
| Note offered, not required | Omits or includes the note | Correct | Optional context is neither rewarded nor charged |
| No limitation in evidence | Adds a limitation | Incorrect | Unsupported caveat; counts as an error, not caution |
| Evidence silent or unknown | States that the source does not settle it | Correct | Unknown is the true answer |
Do users not just want the short answer? The strongest objection
They do, and the grid lets them have it. Nothing in it rewards length: an answer to a question with no flagged limitation should be short, and adding caveats there is marked wrong. The grid asks for the limitation only where the evidence record itself says the plain answer misleads, which in this data is 1.4 percent of supported answers.
The harder version of the objection is that the evidence can be wrong or stale. Compatibility data changes as browsers ship, notes are written by contributors with different habits, and 1,666 pairs carry notes that nobody flagged as partial. The defenses are plain: pin the version of the evidence each label was taken from, record who marked a limitation as required, and keep disputed or unflagged notes in their own class rather than forcing a verdict. A label that cannot name its source and date should not be scored at all.
Building a qualified answer set from your own knowledge base
Start from documents that already separate the headline from the footnote, and keep the authority behind each footnote. In a company knowledge base those are support matrices, known-issues pages, release notes, policy exception tables and approved answers that a specialist corrected. Each question needs the answer, the evidence state, the limitation text, the document version and effective date, and the role that made the limitation required.
Run the census before labeling. Count, in the documents you plan to use, how many approved answers carry a flagged limitation, how many carry optional notes and how many carry neither, the same three-way split measured here. That base rate sets how hard to oversample the caveat-required slice, and it shows whether your knowledge base flags limitations at all or leaves them in free text that someone must first mark up.
SourceX licenses business data directly from companies, and datasets are sourced on request rather than held in stock. A buyer building this kind of test can share the scope, volume, language, format, timing and licensing it needs through the request-data form, and attach the scoring grid above as the acceptance test. Ask for these fields in every record:
- The question as a user asked it, and the answer an expert approved.
- An evidence state for each question: limitation required, note offered, none, or unknown.
- The limitation text, quoted from the governing document, with its version and date.
- The role that decided the limitation was required, and any later correction.
- A flag for limitations that apply only to some products, plans or customers, kept apart from general ones.
- A deliberately oversampled slice of caveat-required questions, with its share of the full set stated.
Methods note: browser-compat-data rules, exclusions and limits
Data: github.com/mdn/browser-compat-data at commit bf12f642f00a, read on October 27, 2026 (CC0). Every .json file in the data folders is read; the browsers, build, docs, lint, schemas, scripts, types and utils folders, and tooling folders such as .vscode, are skipped. A feature is any object with a __compat key. Browsers are chrome, firefox and safari, checked to have no upstream field; mirrored statements would be counted and excluded, and none occurred. The current statement is the object, or the first element of an array.
Classes, first match wins: no statement; version_removed set (removed); version_added false (not supported); null (unknown); preview (preview only); flags present (behind a flag); partial_implementation true (partial); prefix or alternative_name (prefixed or renamed); notes (supported, with a note); otherwise supported, no caveat. Caveat-required means partial or prefixed or renamed. The zero-in-sample probabilities are exact hypergeometric values with no random draw. Percentages are rounded once, half up.
Not measured: whether each note is accurate today, how often limitations change between releases, other browsers and runtimes, and whether any model or product answers these questions well. The counts describe one reference's evidence, not a company's knowledge base, and the share of caveat-required answers in enterprise documents may be higher or lower.
Frequently asked questions
Should a model add caveats to every answer to be safe?
No. A caveat the evidence does not support is an error, because it can stop someone from using something that works. In the grid on this page, an unsupported caveat scores the same as omitting a required one.
How do you decide which caveats are required?
Let the evidence decide, not the grader's taste. A limitation is required when the governing document marks the plain answer as incomplete, as browser-compat-data does with its partial_implementation flag and mandatory note. Optional notes are kept as context and neither rewarded nor charged.
How large should the caveat-required slice of a test set be?
Large enough to measure on its own, which random sampling will not deliver when the base rate is low. In this data, a random 100-item set would contain no caveat-required case about a quarter of the time. Oversample the slice deliberately and report its accuracy separately from the rest.
What if the documented limitation is out of date?
Pin each label to the document version and date it came from, and re-check labels when the source changes. A grader should never score an answer against a limitation it cannot date, and disputed notes belong in their own class.
Can partial credit work for caveats?
Only for optional context. When the evidence requires a limitation, an answer without it leads to the wrong action, so partial credit would hide exactly the failure the test is meant to find.
Sources
- MDN's browser-compat-data at commit bf12f64, read on October 27, 2026, supplies every count of features, support classes, browsers and areas on this page. Source
- The project README describes browser-compat-data as machine-readable compatibility data for web technologies, used by MDN Web Docs, CanIUse, Visual Studio Code, WebStorm and other software. Source
- The compatibility data schema defines partial_implementation as support that does not implement mandatory specified behavior, is inconsistent with other browsers or confuses feature detection, and asks that arrays of statements list the most relevant, most recent entries first. Source
- The data guidelines require a note describing how an implementation is incomplete whenever partial_implementation is true, and record non-functional defined API names as partial rather than unsupported. Source
- The ::marker data records Safari support from version 11.1 as a partial implementation limited to the color and font-size properties. Source
- The Notification data records Firefox support for requireInteraction from version 117 as partial: supported only on Windows and behind a flag on other operating systems. Source
- Kalai, Nachum, Vempala and Zhang (OpenAI, 2025) argue that binary grading in training and evaluation rewards guessing over acknowledging uncertainty and propose changing how existing benchmarks are scored. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.