Privacy and preparation
Expert confidence labels for business tasks: know, infer or can't tell
By SourceX Editorial · Updated
10 sections
Short answer
Separate what an expert knows, infers and cannot tell by recording two things for every label: the basis of the judgment (stated in the record, checked against a second record, applied from a written rule, inferred, or cannot tell) and a pointer to the evidence. Add what would change the call. A single 1-to-5 confidence slider mixes these and hides the part a reviewer or buyer needs.
Redaction
Expert confidence labels
Key takeaways
- Record the basis of each expert judgment separately from how confident the expert feels.
- Every basis code needs its own evidence: record IDs for facts, a rule version for policy calls, an alternative and flip condition for inferences.
- Cannot tell is a real answer when the records are incomplete, and it teaches models to abstain.
- Disagreement between qualified experts on inferred labels is information, not noise to average away.
- Audit labels by basis code; if inferences hold as often as facts, the codes are not working.
- Expert labels add value to licensed records but do not create the right to license them.
How do you separate what an expert knows from what they infer?
Ask for the basis of each judgment, not just a confidence number. For every label, the expert picks one basis code (the answer is stated in the record, confirmed in a second record, applied from a written rule, inferred from patterns, or not determinable) and points to the record fields that support it. Inferred labels also carry the alternative the expert rejected and the fact that would flip the call.
This matters most for business tasks, where the answer is often half in the record and half in the expert's head. A senior accounts payable clerk can read a disputed invoice and say the root cause was a unit-of-measure mismatch. Sometimes the purchase order and receiving log prove it. Sometimes the clerk is recognizing a familiar pattern from a vendor they have dealt with for years. Both are valuable, but they are different kinds of data, and a model trained or evaluated on them should know which is which.
The guide below turns that distinction into labels a team can apply the same way on Monday and on Friday, across experts who have never met.
Why does a single confidence score hide the part reviewers need?
A single score blends two separate questions: how good is the evidence, and how sure is this person. An expert can be very sure of an inference that no record supports, and fairly unsure about a fact that is plainly written in the ticket because the field has been unreliable in the past. A 4 out of 5 on both tells a reviewer nothing about which items to audit.
Fields that have thought hard about this keep the two apart. The U.S. intelligence community's analytic standards directive, ICD 203, asks analysts to distinguish the underlying information from their own assumptions and judgments, and separately to express and explain the uncertainty behind major judgments. The IPCC's guidance for its lead authors builds confidence from the type, amount, quality and consistency of the evidence plus the degree of agreement, and expresses that confidence qualitatively rather than as a made-up percentage.
Annotation research points the same way. Aroyo and Welty argue that the idea of one correct answer per item is a myth for many interpretation tasks, and the disagreement survey by Uma and colleagues in the Journal of Artificial Intelligence Research reviews methods that learn from disagreement instead of averaging it away. For business records, the practical lesson is narrower: record the basis and the evidence, and let the confidence follow from them.
The evidence strength and uncertainty label guide
Use these seven codes as the basis field. Each code has an evidence requirement; a label without its required evidence is returned to the expert, not accepted with a lower score. The examples come from common business systems and are illustrative.
| Code | What it means | Evidence the expert must attach | Illustrative business example | How to use it downstream |
|---|---|---|---|---|
| K1 Stated | The answer is written in the record itself | Record ID and the field or sentence that states it | Zendesk ticket where the customer writes the order number and the agent's macro names the refund reason | Safe for training targets and evaluation keys |
| K2 Confirmed | The answer is supported by a second, independent record | Both record IDs and the matching fields | NetSuite vendor bill whose quantity matches the receiving record but not the purchase order line | Strongest class; use for gold evaluation sets |
| R Rule applied | The answer follows from a written policy or rule applied to recorded facts | Rule name, version or effective date, and the facts it was applied to | Expense report flagged because a hotel night exceeds the travel policy cap in effect that quarter | Train with the rule version attached; re-check when the rule changes |
| I1 Strong inference | Not stated, but the recorded facts point clearly one way | Supporting fields, the rejected alternative, and what would flip it | Jira bug judged a regression because it appeared the day after a release touching the same module | Usable for training with the basis kept; sample for audit |
| I2 Weak inference | A plausible reading, but another path is about as likely | The two leading readings and the missing fact that would decide | Salesforce opportunity marked lost on price, but notes also mention a competitor's integration | Keep both readings; exclude from gold sets |
| U Cannot tell | The records available do not support any answer | What is missing and where it would normally be | Service call closed with no resolution code and no technician note | Teach abstention; never force a guess |
| X Outside scope | The question needs expertise the reviewer does not claim | The expertise needed | A billing dispute that turns on a contract clause the reviewer cannot interpret | Route to a qualified reviewer; do not count as a label |
Which fields should sit next to every expert label?
Ten fields are enough for most business tasks. Fewer, and you cannot audit; many more, and experts start filling them with filler. Keep the free-text fields short and make the coded fields required.
At a glance
Which fields should sit next to every expert label
- Item ID and the task version the expert saw.
- Label value: the answer itself, in the task's controlled vocabulary.
- Basis code from the guide: K1, K2, R, I1, I2, U or X.
- Evidence pointers: record IDs and field names, not pasted text, so the evidence can be re-checked after de-identification.
- Rule reference and effective date, required when the basis is R.
- Alternative considered, required for I1 and I2.
- Flip condition: the single fact that would change the answer, required for I1 and I2.
- Missing evidence, required for U: what record or field would have settled it.
- Expert scope: the role and system experience that qualifies this person for this task type, recorded once per expert and linked.
- Review trail: second reviewer, agreement or disagreement, and the adjudicated outcome if any.
Can a labeling tool enforce the basis and flip-condition fields?
Yes, most annotation tools can make the basis code mandatory and show the extra fields only when they apply. In Label Studio, for example, the Choices tag has a required parameter that blocks submission until a choice is made, and a visibleWhen setting that, combined with whenTagName and whenChoiceValue, shows a control only when a particular choice is selected. That lets the alternative and flip-condition boxes appear only for I1 and I2, so experts are not asked for them on K1 items.
Whatever tool you use, check two things before the first batch. First, the basis code must be stored as its own field in the export, not folded into a comment. Second, the evidence pointers must survive your de-identification step; if record IDs are replaced, keep a mapping so a reviewer can still find the supporting record.
Worked example: coding the cause of a disputed vendor invoice
This example is illustrative, not drawn from a real company. The task asks an accounts payable expert to code the root cause of each invoice that was placed on hold, using the vendor bill, the purchase order, the receiving record and the hold notes. Here is how one expert's answers on four invoices map to the guide.
- Invoice A, coded quantity mismatch, basis K2. The receiving record shows fewer units than the bill, and the purchase order matches the receiving record. Evidence: the bill ID, the receipt ID and the two quantity fields. No flip condition needed.
- Invoice B, coded price variance, basis R. The bill price exceeds the purchase order price by more than the tolerance in the written matching policy in effect that month. Evidence: both prices and the policy version. If the tolerance changes later, this label is re-checked, not silently kept.
- Invoice C, coded unit-of-measure error, basis I1. The bill is in cases, the purchase order in single units, and the totals line up if a case holds twelve. Nothing in the records states the case size. Alternative considered: a genuine overbilling. Flip condition: a vendor catalog entry showing a different case size.
- Invoice D, coded cannot tell, basis U. The hold note says only that it was released after a call. Missing evidence: a call log or an email confirming what was agreed.
What evidence would change the expert's call, or make another path equally valid?
The flip condition is the most useful thing an expert can add to an inference, because it turns a private judgment into a check someone else can run. When a reviewer later finds the vendor catalog and it shows a case of twelve, invoice C can be promoted from I1 to K2. When the catalog shows a case of six, the label is wrong and the reviewer knows exactly why.
Two experts who both give I1 or I2 labels with different answers have not necessarily made a mistake. ICD 203 lists analysis of alternatives among its tradecraft standards, and the disagreement literature treats stable disagreement as information about the item. Keep both answers with their bases and flip conditions, and decide per use: an evaluation set should usually exclude items where qualified experts split, while a training set for a model that must explain its reasoning can keep them as cases with more than one defensible path.
How do you check that the labels mean what they claim?
Audit by basis code, not by expert alone. Take a sample from each code, give a second qualified reviewer the same records plus any extra records the first expert did not have, and record whether the label holds. Labels coded K2 should almost always hold; labels coded I2 should hold noticeably less often. If an expert's I1 labels hold as rarely as their I2 labels, the codes are not carrying information and the expert needs recalibration on examples.
Report the audit results per code and per expert in the dataset documentation, with sample sizes. Do not convert the codes into probabilities unless the audit data supports a mapping, and say so when it does not. The IPCC guidance makes the same distinction between qualitative confidence and probabilistic statements, which are used only when the evidence allows.
- Sample every code in every batch, including K1, so stated facts with unreliable fields are caught.
- Blind the second reviewer to the first expert's code.
- Track how often U items are resolved once more records are pulled; a high rate means the task is missing a source, not that experts are timid.
- Pay per item reviewed, never more for a confident code, so nobody is rewarded for upgrading I2 to K1.
When should cannot tell stay in the dataset?
Keep U items when the model you are training or testing will face the same gaps in real work. A support or finance agent that guesses a resolution code when the record has none is worse than one that says the record is incomplete, and U items are the only examples that teach the difference. Mark them clearly so nobody mistakes them for unlabeled data.
The exception is when U is common because the task was badly scoped. If a third of items come back U because the hold notes live in an email system nobody exported, fix the extract and relabel; do not ship a dataset that mostly teaches the model to give up. The X code works the same way: a cluster of X labels on one task type means you need a different expert, not a lower bar.
What expert labels add when records are licensed, and what they do not
Basis codes make a labeled set of business records easier to evaluate, but they do not change who may license the records. Access to a system, privacy permission for the people in it, ownership of the records and contractual rights to license them are separate questions, and de-identification alone does not create a right to license. Expert labels added by employees are part of what a company would need to cover in its terms.
Once expert-labeled records are ready to license, SourceX carries the work of qualifying the company, reviewing rights, contracting, preparing the data, delivering it and paying the business. The company signs and approves scope and terms before anything is delivered, and a short assessment with no data upload tells it whether it is a fit.
Frequently asked questions
What is the difference between expert confidence and evidence strength?
Evidence strength describes the records behind a label: stated, confirmed by a second record, derived from a written rule, or inferred. Confidence describes how sure the person is. Recording the basis and the evidence separately lets a reviewer see when an expert is confident about something no record supports.
Should experts give a percentage for each label?
Usually not. Percentages invite false precision unless you have audit data showing how often each kind of label holds. Qualitative basis codes with required evidence are easier to apply consistently, and you can attach measured hold rates later from an audit.
How many basis codes are too many?
If experts hesitate between codes on ordinary items, there are too many or the definitions overlap. Seven codes work for most business tasks because each one has a distinct evidence requirement; merge codes rather than add shades of confidence.
What should happen when two experts disagree on an inferred label?
Keep both answers with their bases and flip conditions, then decide per use. Exclude split items from gold evaluation sets, and keep them as multi-path cases where the model must reason about alternatives. Pull the record named in the flip condition when it exists; that often settles the item.
Can a model learn to abstain from cannot-tell labels?
Only if those items are labeled as deliberate abstentions with the missing evidence named, not left blank. Blank items look like unfinished work to a training pipeline. Clear cannot-tell labels also show the buyer which questions the records cannot answer.
Sources
- ICD 203 asks analysts to separate underlying information from assumptions and judgments, to express uncertainty in major judgments and to analyze alternatives. Source
- IPCC AR5 guidance builds confidence from the type, amount, quality and consistency of evidence and the degree of agreement, expressed qualitatively, and uses probabilities only where evidence allows. Source
- Aroyo and Welty argue that assuming one correct answer per item is a myth for many human annotation tasks. Source
- Uma and colleagues survey evidence of disagreement in human judgments used to train AI and methods that learn from it. Source
- Label Studio's Choices tag can require a selection and show controls only when a particular choice is selected. Source
See if your company qualifies
A short company assessment. No data uploads are needed.