AI uses for records
Expert decision rejected alternatives: annotating why an option lost
By SourceX Editorial · Updated
9 sections
Short answer
Expert annotations preserve why an option was rejected by giving every option the decider actually weighed its own row: its outcome, the one constraint that decided it, a pointer to the record proving that constraint held at decision time, the condition that would have revived it, whether a peer could defensibly choose it, and the expert's confidence. Typed reasons let a model learn the rule, not a surface preference.
Agent run
Expert decision rejected
Key takeaways
- A chosen and rejected pair records which option won; the rejection reason is what shows which facts the decision depended on.
- Give each weighed option its own row with a single primary constraint, evidence pointer, revival condition, equal-validity mark and confidence.
- Type every rejection as a hard rule, resource limit, trade-off, dominance, evidence gap or judgment call, because each teaches a learner something different.
- Do not invent alternatives for routine single-option decisions, and flag rows written from late recall.
- Test the boundary by changing the primary constraint in a copy of the case and checking that a second expert's choice changes with it.
How should expert annotations preserve the reason an option was rejected?
Record every option the expert actually weighed as its own annotation row, not just the winner. Each row names the option, its outcome (chosen, rejected, infeasible, deferred), the single constraint or criterion that decided it, a pointer to the record that proves that constraint held at decision time, the condition that would have revived it, and how sure the expert was.
The point is that the losing option often carries the most information. A dispatcher who sends a senior technician tomorrow instead of a junior one today is not expressing a general preference for seniority. The junior option failed one specific test, perhaps a missing refrigerant certification for that unit, and the decision means something different once you know which test it was.
Without that row, a model trained on the decision learns a pattern (senior, later) that is right for the wrong reason. With it, the model can learn the rule (certification required for this equipment class) and an evaluator can check whether a model's answer respects the same constraint when the facts change.
Why a chosen and rejected pair loses the reason
Standard preference formats store which output won, not why the other one lost. Hugging Face's TRL library defines a preference example as a prompt, a chosen completion and a rejected completion, and Anthropic's published HH-RLHF data stores each line as a pair of texts, one chosen and one rejected. Neither format has a slot for the reason, and for their purpose they do not need one.
Business decisions are different because the reasons are usually checkable. A credit hold release, a carrier choice or a repair path is rejected because a specific limit, policy, price, stock level or deadline ruled it out, and that fact sits in a system of record. A bare pair throws that away and leaves a learner to guess among every difference between the two options.
Software architecture teams solved a version of this problem long ago. The MADR decision record template has separate parts for decision drivers, considered options, the chosen option with its justification, and the pros and cons of each option. The template Jeff Tyree and Art Akerman published lists assumptions, constraints, positions (the viable alternatives considered) and an argument for why one position was selected, and warns that a missing alternative invites the question of whether it was ever considered. The annotation format below borrows that structure and makes each field small enough for a person to fill in during real work.
What should the alternatives and constraint annotation format include?
The format is one decision header plus one row per option. The header fixes the case: decision ID, decision time, the decider's role, the question being decided and the record snapshot the decider could see. The option rows carry the substance.
Keep the rejecting reason to one primary constraint per row, with optional secondary reasons. Experts often list four reasons when one was decisive and the rest were confirming; forcing a primary reason exposes which one actually ended the option's chances.
Store the rows in whatever form the team already uses: extra columns in a spreadsheet export, a child table keyed by decision ID, or one JSON object per option. What matters is that the option row and the decision header can be joined, and that the evidence field points to a record version rather than a live screen that will change tomorrow.
The outcome values do different jobs. Infeasible means the option could not be executed at all, so it should never appear as the losing side of a preference comparison. Deferred means the option was postponed rather than refused, which matters when the same case comes back a week later and the deferred option is chosen. Not considered is for options a reviewer or second expert raises afterward; recording them separately keeps hindsight out of the original decider's reasoning.
| Field | What it holds | Allowed values or form | Why it matters |
|---|---|---|---|
| option_id and label | A short name for one path the decider weighed | Stable ID within the decision, plain-language label | Lets later rows, reviews and counterfactual tests refer to the same option |
| outcome | What happened to the option | chosen, rejected, infeasible, deferred, not_considered | Separates options that lost a comparison from options that were never possible |
| primary_constraint | The one fact or criterion that decided this option | Free text plus a constraint type from the next section | Shows what the decision actually depended on |
| constraint_evidence | Where the constraint can be checked | Record ID, field and value as of decision time | Turns a stated reason into a verifiable one |
| secondary_reasons | Other factors that pointed the same way | Zero to three short entries | Keeps confirming reasons without inflating their weight |
| revival_condition | What would have made this option acceptable or preferred | One testable statement about a field or fact | Marks the decision boundary for evaluation and counterfactual cases |
| equal_validity | Whether another qualified expert could defensibly choose it | yes, no, unsure, with a one-line note | Stops a defensible alternative being scored as an error |
| confidence | How sure the decider was that this option was worse or impossible | high, medium, low | Low-confidence rejections are candidates for second review |
| annotator and timing | Who wrote the row and how long after the decision | Role, same-day or later recall | Late recall is weaker evidence and should be flagged |
Which kind of reason knocked the option out?
Classify each rejection by the kind of reason, because different kinds teach different things. A hard rule should become a constraint a model never crosses; a trade-off should become a weighting that shifts when costs or deadlines shift; a dominance call should be checked against the numbers.
If the annotator cannot place a rejection in one of these types, the reason is usually too vague to use and the row should go back for clarification.
| Constraint type | Test the annotator applies | Illustrative business example | What a learner should take from it |
|---|---|---|---|
| Hard rule | Would choosing it break a written policy, contract term, license or safety requirement? | Refund above a limit needs a manager approval the agent does not have | A boundary that holds whatever the other factors are |
| Resource limit | Was something required unavailable at decision time? | No certified technician free before Thursday | A condition that can change; check availability first |
| Trade-off | Was it possible but worse on a weighted criterion? | Expedited freight met the date but cost more than the margin allowed | A weighting that should move with prices, deadlines and margins |
| Dominance | Was another option at least as good on every criterion that mattered? | Two carriers same price, one with a later pickup | Easy to verify from data; useful as a sanity check |
| Evidence gap | Could the option not be verified in time? | Customer could not confirm the serial number, so a warranty claim was not filed | The missing fact, not the option, decided the case |
| Judgment call | Did experience or a soft signal decide it, with no written rule? | Dispatcher distrusts a site contact's description after past mismatches | Weak signal; needs agreement checks before training on it |
Revival conditions: the fact that would have saved the losing option
Ask for the revival condition in the expert's own terms, then convert it into a field and value that exists in the records. 'If the junior technician had held the certification' becomes technician.certifications contains the required class. A revival condition that cannot be stated against a field is a sign that the stated reason is not the real one.
Mark equal validity separately from confidence. An expert can be sure an option was worse for this customer and still accept that a peer might reasonably have chosen it. Barbara Plank's work on human label variation argues that disagreement among annotators is often plausible variation rather than error, and the survey by Uma and colleagues reviews training methods that drop the assumption of a single gold label. In decision data that means a second expert choosing the rejected option is not automatically a mistake by either.
Three kinds of evidence typically flip a decision, and each should be checked when you test the boundary:
- A changed value in the primary constraint field: a stock level, credit limit, certification or date that crosses the threshold the expert named.
- A new fact that removes an evidence gap: the serial number arrives, the approval is granted, the inspection report is filed.
- A shift in weights for trade-offs: a rush surcharge drops, a service-level penalty rises, or a customer tier changes.
- If none of these flips the choice when you vary it, the stated reason was probably not decisive; record that finding against the row.
Worked example (illustrative): a dispatcher picks the next-day senior technician
This case is illustrative; names, counts and timings are invented to show the format. A heating and cooling company receives a no-cooling call for a commercial rooftop unit on a Monday afternoon. The dispatcher weighs four options and sends a senior technician on Tuesday morning.
Read together, the rows show that the decision was driven by one hard rule and one resource limit, not by any preference for seniority. A learner trained on the pair (senior Tuesday over junior Monday) alone would miss that; the annotation lets an evaluator change the certification fact and expect the decision to change with it.
The evaluation use follows directly. Build a variant of the case in which the junior technician holds the certification and every other field stays the same; a well-behaved model should now be allowed to send the junior technician today. Build a second variant where the senior technician's commercial job finishes early; the senior-today option becomes acceptable and arguably preferred. If a model keeps choosing Tuesday in both variants, it learned the surface pattern, and the annotation is what lets you detect that.
The subcontractor row is the instructive exception. The dispatcher marked it unsure for equal validity because some peers would have called the customer and asked. That is a judgment about customer relationships, not a rule, and it belongs in the data as a legitimate alternative rather than a mistake.
| Option | Outcome and type | Primary constraint and evidence | Revival condition | Equal validity |
|---|---|---|---|---|
| Junior technician today | Rejected, hard rule | Unit uses a refrigerant class the junior is not certified for; technician profile and equipment record | Junior holds that certification | No |
| Senior technician today | Infeasible, resource limit | Senior already booked until 7 pm on a commercial job; dispatch board snapshot | Senior free before 4 pm | Yes, if free |
| Subcontractor today | Rejected, judgment call | Contract requires notice before third-party technicians, and the dispatcher judged a same-day call to the customer unlikely to succeed; contract terms field and call notes | Customer agrees to a subcontractor by phone | Unsure |
| Senior technician Tuesday 8 am | Chosen | Earliest certified slot; customer accepted next-morning arrival in call notes | Not applicable | Not applicable |
When recording losing options adds noise instead of signal
Do not force rows for options nobody weighed. If a routine purchase order went to the usual vendor without any comparison, inventing three losing vendors after the fact manufactures rationale that never existed. Mark the decision as single-option and move on; it is still useful data, just a different kind.
Be careful with late recall. An expert asked weeks later which options they considered will often produce a tidy list that includes options they only think of now. Capture rows at or near decision time where possible, and flag every row written later so reviewers can weigh it accordingly.
Stop and narrow the scope when rejection reasons depend on legal interpretation, safety judgments or regulated eligibility rules that annotators are not qualified to state. In those cases record only what the system shows (the rule cited, the approval missing) and send the interpretation to a qualified reviewer instead of asking the annotator to explain it.
Committee decisions need one more adjustment. When a pricing committee or a review board decides, the rejecting reason may differ by member. Record the reason stated in the minutes or the approval record as the primary constraint, and add individual members' reasons only where a record shows them; reconstructing each member's private reasoning is guesswork.
Review checks before the annotations feed training or evaluation
Run a short set of checks on a sample before anyone trains or evaluates on the annotations. These checks are cheap, and each one catches a failure that otherwise reaches the model unnoticed.
At a glance
Review checks before the annotations feed training or evaluation
When a check fails, fix the format before fixing the rows. A high share of rejections typed as judgment calls usually means annotators lack access to the policy or schedule that held the real reason, so give them that access. Frequent evidence mismatches usually mean the evidence field is pointing at current values, so switch it to a dated snapshot or an audit history before annotating more cases.
- Evidence check: for a random sample of rejected rows, open the cited record and confirm the constraint value was what the row says at decision time, not today.
- Type check: every hard-rule rejection cites a written rule; every trade-off names the criterion and direction; judgment calls are tagged as such.
- Boundary check: change the primary constraint value in a copy of the case and confirm a second expert now prefers the revived option.
- Agreement check: give a subset of cases to a second expert blind to the first annotation and compare chosen options, rejection types and equal-validity marks.
- Coverage check: count decisions with only one option row and confirm they were genuinely single-option rather than skipped.
- Incentive check: confirm annotators were not rewarded for row count or speed, which pushes them toward invented alternatives and copied reasons.
Rejected options inside a licensed decision dataset
Companies that already keep decision records, such as quotes with lost alternatives, dispatch boards, approval logs and exception notes, often hold much of this structure without calling it annotation. When a business wants to license selected decision records like these for AI, SourceX's part is to carry them through rights review, contracting and preparation. Whether rejected-option notes can be included depends on what the records contain and who appears in them, which the rights and privacy review settles case by case.
If your records capture the options people weighed and the constraints that ruled them out, the starting point is a short company assessment, and qualifying needs no data upload.
Frequently asked questions
What is a rejected alternative in an expert decision?
It is an option the decider actually weighed and did not choose. Recording it, with the constraint that ruled it out, shows which facts the decision depended on. Options nobody considered are not rejected alternatives and should not be added after the fact.
How is this different from preference data with chosen and rejected responses?
Preference pairs store which output won, usually without a reason. The annotation format here adds a typed reason, the record that proves it, a revival condition and an equal-validity mark for each losing option. That lets a learner pick up the rule rather than a surface pattern.
How many alternatives should an annotator record per decision?
Record the options the decider actually weighed, which in most operational decisions is two to four. A single-option decision is valid and should be marked as such. Padding the list with options nobody considered makes the data worse.
What if two experts would have chosen differently?
Record both and mark the losing option as equally valid if both choices are defensible. Treating every disagreement as an error throws away real variation in expert judgment. Persistent disagreement on a hard-rule rejection, however, usually means the rule is unclear and needs a policy owner to settle it.
Can rejection reasons be filled in from records alone?
Some can. Hard rules and resource limits often show up directly in policy tables, schedules and stock levels. Trade-offs and judgment calls usually need the decider's input, ideally captured near decision time.
Sources
- Hugging Face TRL defines a preference dataset example as a prompt, a chosen completion and a rejected completion. Source
- Anthropic's HH-RLHF data stores each line as a pair of texts, one chosen and one rejected. Source
- The MADR template records decision drivers, considered options, the chosen option with its justification, and pros and cons of each option. Source
- The Tyree and Akerman decision template lists assumptions, constraints, positions considered and the argument for the selected position. Source
- Plank (EMNLP 2022) argues that annotator disagreement is often plausible human label variation rather than error. Source
- Uma et al. (JAIR 2021) survey methods for learning from annotator disagreement without assuming a single gold label. Source
See if your company qualifies
A short company assessment. No data uploads are needed.