Skip to content

AI uses for records

Expert decision rejected alternatives: annotating why an option lost

By SourceX Editorial · Updated

9 sections

Short answer

Expert annotations preserve why an option was rejected by giving every option the decider actually weighed its own row: its outcome, the one constraint that decided it, a pointer to the record proving that constraint held at decision time, the condition that would have revived it, whether a peer could defensibly choose it, and the expert's confidence. Typed reasons let a model learn the rule, not a surface preference.

Agent run

Expert decision rejected

This page covers expert decision rejected alternatives: annotating why an option lost.Illustrative

Key takeaways

  • A chosen and rejected pair records which option won; the rejection reason is what shows which facts the decision depended on.
  • Give each weighed option its own row with a single primary constraint, evidence pointer, revival condition, equal-validity mark and confidence.
  • Type every rejection as a hard rule, resource limit, trade-off, dominance, evidence gap or judgment call, because each teaches a learner something different.
  • Do not invent alternatives for routine single-option decisions, and flag rows written from late recall.
  • Test the boundary by changing the primary constraint in a copy of the case and checking that a second expert's choice changes with it.

How should expert annotations preserve the reason an option was rejected?

Record every option the expert actually weighed as its own annotation row, not just the winner. Each row names the option, its outcome (chosen, rejected, infeasible, deferred), the single constraint or criterion that decided it, a pointer to the record that proves that constraint held at decision time, the condition that would have revived it, and how sure the expert was.

The point is that the losing option often carries the most information. A dispatcher who sends a senior technician tomorrow instead of a junior one today is not expressing a general preference for seniority. The junior option failed one specific test, perhaps a missing refrigerant certification for that unit, and the decision means something different once you know which test it was.

Without that row, a model trained on the decision learns a pattern (senior, later) that is right for the wrong reason. With it, the model can learn the rule (certification required for this equipment class) and an evaluator can check whether a model's answer respects the same constraint when the facts change.

Why a chosen and rejected pair loses the reason

Standard preference formats store which output won, not why the other one lost. Hugging Face's TRL library defines a preference example as a prompt, a chosen completion and a rejected completion, and Anthropic's published HH-RLHF data stores each line as a pair of texts, one chosen and one rejected. Neither format has a slot for the reason, and for their purpose they do not need one.

Business decisions are different because the reasons are usually checkable. A credit hold release, a carrier choice or a repair path is rejected because a specific limit, policy, price, stock level or deadline ruled it out, and that fact sits in a system of record. A bare pair throws that away and leaves a learner to guess among every difference between the two options.

Software architecture teams solved a version of this problem long ago. The MADR decision record template has separate parts for decision drivers, considered options, the chosen option with its justification, and the pros and cons of each option. The template Jeff Tyree and Art Akerman published lists assumptions, constraints, positions (the viable alternatives considered) and an argument for why one position was selected, and warns that a missing alternative invites the question of whether it was ever considered. The annotation format below borrows that structure and makes each field small enough for a person to fill in during real work.

What should the alternatives and constraint annotation format include?

The format is one decision header plus one row per option. The header fixes the case: decision ID, decision time, the decider's role, the question being decided and the record snapshot the decider could see. The option rows carry the substance.

Keep the rejecting reason to one primary constraint per row, with optional secondary reasons. Experts often list four reasons when one was decisive and the rest were confirming; forcing a primary reason exposes which one actually ended the option's chances.

Store the rows in whatever form the team already uses: extra columns in a spreadsheet export, a child table keyed by decision ID, or one JSON object per option. What matters is that the option row and the decision header can be joined, and that the evidence field points to a record version rather than a live screen that will change tomorrow.

The outcome values do different jobs. Infeasible means the option could not be executed at all, so it should never appear as the losing side of a preference comparison. Deferred means the option was postponed rather than refused, which matters when the same case comes back a week later and the deferred option is chosen. Not considered is for options a reviewer or second expert raises afterward; recording them separately keeps hindsight out of the original decider's reasoning.

What should the alternatives and constraint annotation format include?
FieldWhat it holdsAllowed values or formWhy it matters
option_id and labelA short name for one path the decider weighedStable ID within the decision, plain-language labelLets later rows, reviews and counterfactual tests refer to the same option
outcomeWhat happened to the optionchosen, rejected, infeasible, deferred, not_consideredSeparates options that lost a comparison from options that were never possible
primary_constraintThe one fact or criterion that decided this optionFree text plus a constraint type from the next sectionShows what the decision actually depended on
constraint_evidenceWhere the constraint can be checkedRecord ID, field and value as of decision timeTurns a stated reason into a verifiable one
secondary_reasonsOther factors that pointed the same wayZero to three short entriesKeeps confirming reasons without inflating their weight
revival_conditionWhat would have made this option acceptable or preferredOne testable statement about a field or factMarks the decision boundary for evaluation and counterfactual cases
equal_validityWhether another qualified expert could defensibly choose ityes, no, unsure, with a one-line noteStops a defensible alternative being scored as an error
confidenceHow sure the decider was that this option was worse or impossiblehigh, medium, lowLow-confidence rejections are candidates for second review
annotator and timingWho wrote the row and how long after the decisionRole, same-day or later recallLate recall is weaker evidence and should be flagged

Which kind of reason knocked the option out?

Classify each rejection by the kind of reason, because different kinds teach different things. A hard rule should become a constraint a model never crosses; a trade-off should become a weighting that shifts when costs or deadlines shift; a dominance call should be checked against the numbers.

If the annotator cannot place a rejection in one of these types, the reason is usually too vague to use and the row should go back for clarification.

Which kind of reason knocked the option out?
Constraint typeTest the annotator appliesIllustrative business exampleWhat a learner should take from it
Hard ruleWould choosing it break a written policy, contract term, license or safety requirement?Refund above a limit needs a manager approval the agent does not haveA boundary that holds whatever the other factors are
Resource limitWas something required unavailable at decision time?No certified technician free before ThursdayA condition that can change; check availability first
Trade-offWas it possible but worse on a weighted criterion?Expedited freight met the date but cost more than the margin allowedA weighting that should move with prices, deadlines and margins
DominanceWas another option at least as good on every criterion that mattered?Two carriers same price, one with a later pickupEasy to verify from data; useful as a sanity check
Evidence gapCould the option not be verified in time?Customer could not confirm the serial number, so a warranty claim was not filedThe missing fact, not the option, decided the case
Judgment callDid experience or a soft signal decide it, with no written rule?Dispatcher distrusts a site contact's description after past mismatchesWeak signal; needs agreement checks before training on it

Revival conditions: the fact that would have saved the losing option

Ask for the revival condition in the expert's own terms, then convert it into a field and value that exists in the records. 'If the junior technician had held the certification' becomes technician.certifications contains the required class. A revival condition that cannot be stated against a field is a sign that the stated reason is not the real one.

Mark equal validity separately from confidence. An expert can be sure an option was worse for this customer and still accept that a peer might reasonably have chosen it. Barbara Plank's work on human label variation argues that disagreement among annotators is often plausible variation rather than error, and the survey by Uma and colleagues reviews training methods that drop the assumption of a single gold label. In decision data that means a second expert choosing the rejected option is not automatically a mistake by either.

Three kinds of evidence typically flip a decision, and each should be checked when you test the boundary:

  • A changed value in the primary constraint field: a stock level, credit limit, certification or date that crosses the threshold the expert named.
  • A new fact that removes an evidence gap: the serial number arrives, the approval is granted, the inspection report is filed.
  • A shift in weights for trade-offs: a rush surcharge drops, a service-level penalty rises, or a customer tier changes.
  • If none of these flips the choice when you vary it, the stated reason was probably not decisive; record that finding against the row.

Worked example (illustrative): a dispatcher picks the next-day senior technician

This case is illustrative; names, counts and timings are invented to show the format. A heating and cooling company receives a no-cooling call for a commercial rooftop unit on a Monday afternoon. The dispatcher weighs four options and sends a senior technician on Tuesday morning.

Read together, the rows show that the decision was driven by one hard rule and one resource limit, not by any preference for seniority. A learner trained on the pair (senior Tuesday over junior Monday) alone would miss that; the annotation lets an evaluator change the certification fact and expect the decision to change with it.

The evaluation use follows directly. Build a variant of the case in which the junior technician holds the certification and every other field stays the same; a well-behaved model should now be allowed to send the junior technician today. Build a second variant where the senior technician's commercial job finishes early; the senior-today option becomes acceptable and arguably preferred. If a model keeps choosing Tuesday in both variants, it learned the surface pattern, and the annotation is what lets you detect that.

The subcontractor row is the instructive exception. The dispatcher marked it unsure for equal validity because some peers would have called the customer and asked. That is a judgment about customer relationships, not a rule, and it belongs in the data as a legitimate alternative rather than a mistake.

Worked example (illustrative): a dispatcher picks the next-day senior technician
OptionOutcome and typePrimary constraint and evidenceRevival conditionEqual validity
Junior technician todayRejected, hard ruleUnit uses a refrigerant class the junior is not certified for; technician profile and equipment recordJunior holds that certificationNo
Senior technician todayInfeasible, resource limitSenior already booked until 7 pm on a commercial job; dispatch board snapshotSenior free before 4 pmYes, if free
Subcontractor todayRejected, judgment callContract requires notice before third-party technicians, and the dispatcher judged a same-day call to the customer unlikely to succeed; contract terms field and call notesCustomer agrees to a subcontractor by phoneUnsure
Senior technician Tuesday 8 amChosenEarliest certified slot; customer accepted next-morning arrival in call notesNot applicableNot applicable

When recording losing options adds noise instead of signal

Do not force rows for options nobody weighed. If a routine purchase order went to the usual vendor without any comparison, inventing three losing vendors after the fact manufactures rationale that never existed. Mark the decision as single-option and move on; it is still useful data, just a different kind.

Be careful with late recall. An expert asked weeks later which options they considered will often produce a tidy list that includes options they only think of now. Capture rows at or near decision time where possible, and flag every row written later so reviewers can weigh it accordingly.

Stop and narrow the scope when rejection reasons depend on legal interpretation, safety judgments or regulated eligibility rules that annotators are not qualified to state. In those cases record only what the system shows (the rule cited, the approval missing) and send the interpretation to a qualified reviewer instead of asking the annotator to explain it.

Committee decisions need one more adjustment. When a pricing committee or a review board decides, the rejecting reason may differ by member. Record the reason stated in the minutes or the approval record as the primary constraint, and add individual members' reasons only where a record shows them; reconstructing each member's private reasoning is guesswork.

Review checks before the annotations feed training or evaluation

Run a short set of checks on a sample before anyone trains or evaluates on the annotations. These checks are cheap, and each one catches a failure that otherwise reaches the model unnoticed.

At a glance

Review checks before the annotations feed training or evaluation

When a check fails, fix the format before fixing the rows. A high share of rejections typed as judgment calls usually means annotators lack access to the policy or schedule that held the real reason, so give them that access. Frequent evidence mismatches usually mean the evidence field is pointing at current values, so switch it to a dated snapshot or an audit history before annotating more cases.

  • Evidence check: for a random sample of rejected rows, open the cited record and confirm the constraint value was what the row says at decision time, not today.
  • Type check: every hard-rule rejection cites a written rule; every trade-off names the criterion and direction; judgment calls are tagged as such.
  • Boundary check: change the primary constraint value in a copy of the case and confirm a second expert now prefers the revived option.
  • Agreement check: give a subset of cases to a second expert blind to the first annotation and compare chosen options, rejection types and equal-validity marks.
  • Coverage check: count decisions with only one option row and confirm they were genuinely single-option rather than skipped.
  • Incentive check: confirm annotators were not rewarded for row count or speed, which pushes them toward invented alternatives and copied reasons.

Rejected options inside a licensed decision dataset

Companies that already keep decision records, such as quotes with lost alternatives, dispatch boards, approval logs and exception notes, often hold much of this structure without calling it annotation. When a business wants to license selected decision records like these for AI, SourceX's part is to carry them through rights review, contracting and preparation. Whether rejected-option notes can be included depends on what the records contain and who appears in them, which the rights and privacy review settles case by case.

If your records capture the options people weighed and the constraints that ruled them out, the starting point is a short company assessment, and qualifying needs no data upload.

Frequently asked questions

What is a rejected alternative in an expert decision?

It is an option the decider actually weighed and did not choose. Recording it, with the constraint that ruled it out, shows which facts the decision depended on. Options nobody considered are not rejected alternatives and should not be added after the fact.

How is this different from preference data with chosen and rejected responses?

Preference pairs store which output won, usually without a reason. The annotation format here adds a typed reason, the record that proves it, a revival condition and an equal-validity mark for each losing option. That lets a learner pick up the rule rather than a surface pattern.

How many alternatives should an annotator record per decision?

Record the options the decider actually weighed, which in most operational decisions is two to four. A single-option decision is valid and should be marked as such. Padding the list with options nobody considered makes the data worse.

What if two experts would have chosen differently?

Record both and mark the losing option as equally valid if both choices are defensible. Treating every disagreement as an error throws away real variation in expert judgment. Persistent disagreement on a hard-rule rejection, however, usually means the rule is unclear and needs a policy owner to settle it.

Can rejection reasons be filled in from records alone?

Some can. Hard rules and resource limits often show up directly in policy tables, schedules and stock levels. Trade-offs and judgment calls usually need the decider's input, ideally captured near decision time.

Sources

  • Hugging Face TRL defines a preference dataset example as a prompt, a chosen completion and a rejected completion. Source
  • Anthropic's HH-RLHF data stores each line as a pair of texts, one chosen and one rejected. Source
  • The MADR template records decision drivers, considered options, the chosen option with its justification, and pros and cons of each option. Source
  • The Tyree and Akerman decision template lists assumptions, constraints, positions considered and the argument for the selected position. Source
  • Plank (EMNLP 2022) argues that annotator disagreement is often plausible human label variation rather than error. Source
  • Uma et al. (JAIR 2021) survey methods for learning from annotator disagreement without assuming a single gold label. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies