Skip to content

AI uses for records

Expert task answer equivalence: when two different answers are both right

By SourceX Editorial · Updated

10 sections

Short answer

Label multiple valid answers by writing the business requirement as acceptance checks before anyone answers, then recording every answer that passes those checks as a member of one accepted-answer set. Tag each with why it counts (same outcome, within tolerance, policy-permitted alternative or preference), the evidence, and what would change the call. Adjudicate only answers that fail a check or conflict with each other.

License terms

Expert task answer

This page covers expert task answer equivalence: when two different answers are both right.Illustrative

Key takeaways

  • Label against the requirement written as checks, not against one expert's answer.
  • Write the checks before anyone answers so they describe the requirement rather than defend a favorite.
  • Separate same outcome, within tolerance, permitted alternative, acceptable but ranked and not equivalent.
  • Record flip conditions: the fact that would make each accepted answer unacceptable.
  • Adjudicate blind and only where a check fails or answers conflict; reward accurate grading, not agreement.
  • Escalate to a specialist when a check depends on tax, legal, clinical or contested rules.

How do you label multiple valid answers that satisfy the same business requirement?

Keep every answer that meets the requirement and say why it meets it, instead of choosing one as gold and marking the rest wrong. The unit you label is the requirement, written as checks; answers are graded against the checks, and two answers are equivalent when both pass and the business would accept either.

This matters most in expert work, where the right answer is often a family of answers. Two controllers can close the same short shipment in different ways that both leave the ledger correct. Two dispatchers can build different schedules that both meet every service window. A dataset that keeps only one of each teaches a model that the other, equally sound path is an error, and an evaluation that keeps only one penalizes a model for choosing it.

The answer equivalence and adjudication protocol below has three parts: a classification of the kinds of equivalence, a step sequence for two experts and an adjudicator, and a record format that keeps the evidence, the uncertainty and the conditions that would flip the decision.

Why does a single gold answer misgrade expert work?

A single reference answer measures closeness to one expert's wording or route, not whether the requirement was met. Public benchmarks learned this early and moved to graders that accept many forms of the same result.

The pattern shows up in widely used evaluation code. The SQuAD scoring script normalizes text (lowercasing, removing punctuation and articles) and takes the maximum score over all reference answers, so any listed answer counts. Math-Verify parses answers into a symbolic form and compares them, and its own example treats the set written as a union of two sets as equal to the same four elements listed out. The text-to-SQL test suite metric executes queries against databases to estimate semantic accuracy and is the official metric for the Spider, SParC and CoSQL leaderboards, replacing a match on the query's form. SWE-bench applies a model's patch and runs the repository's tests; an instance counts as resolved when the patch makes the required tests pass, whatever the patch looks like.

Business tasks need the same move, but the checks are not tests a computer can run. They are the conditions an expert would sign off on: the balance is right, the customer was told, the policy limit was respected. Writing those conditions down is most of the work.

Which kinds of equivalence should a label distinguish?

Distinguish at least four relations, because each one carries different information for training and for evaluation. A label of equivalent with no basis cannot be audited and cannot tell a model which differences matter.

Which kinds of equivalence should a label distinguish?
RelationTestBusiness illustrationHow to label it
Same outcomeBoth answers leave the system of record in the same end stateTwo sequences of ERP postings that produce identical balances and open itemsequivalent: outcome; keep both paths
Within toleranceResults differ, but inside a stated tolerance the business acceptsTwo delivery routes with arrival times inside the same customer windowequivalent: tolerance; record the tolerance and its source
Permitted alternativeDifferent end states, each allowed by the governing policyShort-paying an invoice versus paying in full and raising a debit memoequivalent: policy; cite the policy clause
Acceptable but rankedBoth pass every check; experts prefer one for a stated reasonTwo customer replies, one resolving in fewer contactsaccepted set with a preference and its reason
Not equivalentOne answer fails a check, or they cannot both be trueOne refund breaches the approval limit; the other does notadjudicate; record the failed check

The answer equivalence and adjudication protocol

Run the protocol per task, with two qualified experts answering independently and a third person adjudicating only what the checks cannot settle. The order matters: the checks are written before any answer is seen, so they describe the requirement rather than defend a favorite answer.

A small pilot can drop the second independent expert on routine tasks, but never the written checks or the relation label. A practical split runs the full two-expert sequence below on a sample of each task type and a single-expert version on the rest; if the sample keeps producing accepted sets with more than one answer, that task type admits alternatives and deserves the full protocol, and the sample result belongs in the dataset documentation.

Version the checks either way. When a check is rewritten mid-batch, regrade the tasks already labeled or mark them with the check version they used, so that two records that look alike were judged by the same rule.

  • Step 1, requirement owner: write the requirement as three to seven acceptance checks, each observable in the record (balance, status, message, timing, approval), plus the governing policy or tolerance for each.
  • Step 2, expert A and expert B separately: produce an answer, the source records relied on, a short rationale and a confidence level, without seeing each other's work.
  • Step 3, either expert or a checker: grade each answer against every check as pass, fail or cannot tell, with the evidence for each grade.
  • Step 4, classify the relation: if both pass every check, assign same outcome, within tolerance, permitted alternative or acceptable but ranked; otherwise mark not equivalent.
  • Step 5, adjudicator, only for not equivalent or cannot tell: decide using the checks and evidence, blind to which expert wrote which answer, and record the deciding check.
  • Step 6, both experts: write the flip condition for each accepted answer, meaning the fact that, if different, would make it unacceptable.
  • Step 7, program lead: review a sample of accepted sets each batch for checks that were too loose, and rewrite the check rather than the label.

What should each equivalence record contain?

One record per task holds the requirement, every answer and its grades, and the decision. The fields below are enough for a buyer to reuse the record for training or evaluation without calling the experts.

At a glance

What should each equivalence record contain

What should each equivalence record contain?
FieldContentWhy it is needed
task_id and source_refsStable id and pointers to the de-identified source recordsLets a reviewer reopen the evidence
requirement_checksNumbered checks with the policy, tolerance or rule behind eachDefines correctness independently of any answer
answersEach answer with author role, rationale and confidenceKeeps alternative valid paths instead of discarding them
check_gradesPass, fail or cannot tell for every answer and check, with evidenceShows exactly where answers differ
relation and basisOne of the five relations, with the clause or tolerance citedMakes equivalent auditable
preferenceRanked answer and the stated reason, if anySupports preference training without losing the accepted set
flip_conditionsThe fact that would make each answer unacceptableCaptures expert knowledge that the answer alone hides
adjudicationAdjudicator role and qualification, deciding check, rationale, dateRecords who settled a conflict and on what grounds
uncertaintyChecks graded cannot tell, and whyStops weak labels from looking certain

What evidence would make another path equally valid?

A second answer becomes equally valid when the evidence shows that every check still passes on its route, usually because a policy allows a choice the first expert did not consider or a tolerance was wider than assumed. Ask each expert to name that evidence explicitly, because it is the most reusable part of their judgment.

Writing checks that admit alternatives is a skill. The HealthBench grader in OpenAI's simple-evals repository scores a response against rubric items one by one and tells the grader that when a criterion says such as, for example or including, the response need not include every listed example to meet it. It also allows criteria that describe undesirable behavior and carry negative points. Business checks benefit from the same two habits: list examples without making them mandatory, and write the failure you must avoid as its own check.

Flip conditions are the mirror image. For a permitted alternative, the flip condition is usually the policy clause that would forbid it; for a tolerance case, it is the boundary of the window; for a same-outcome case, it is any downstream report that reads the intermediate steps rather than the end state.

Worked example (illustrative): two ways to close a short shipment

In this illustrative case, a distributor receives 40 of 50 ordered units, and the supplier invoices for all 50. The requirement is written as four checks: the supplier's net payment covers only received goods; the payable and inventory balances agree at period close; the supplier is notified of the shortage; and the approval limit in the purchasing policy is respected.

Expert A short-pays the invoice for 40 units and notes the shortage on the remittance. Expert B pays the invoice as billed and raises a debit memo for the 10 missing units, netted against the next payment. Both pass all four checks if the debit memo is applied before period close, and the policy permits either method. The relation is permitted alternative, with the policy clause cited. The experts disagree on preference: A prefers short-pay because it avoids an open debit; B prefers the memo because the supplier's portal rejects partial payments. Both reasons go in the record.

The exception: if the period closes before the next payment run, B's debit memo leaves a supplier balance overstated at close, check two fails, and the relation becomes not equivalent. The flip condition on B's answer is therefore the timing of the next payment run relative to close. An adjudicator would not need to choose a favorite; the record already shows which check decides it.

How do you keep reviewers honest about equivalence?

Reward reviewers for accurate check grades, not for agreeing with each other, and hide authorship during adjudication. When pay or performance depends on agreement, reviewers learn to converge on the safest answer and to mark real alternatives as errors.

Equivalence checks are also not always symmetric. Math-Verify's documentation warns that the order of the gold answer and the candidate in its verify call matters, and business checks behave similarly: an answer that does more than required can pass every check while an answer that does the minimum fails one written around the fuller answer. Grade each answer against the checks, never against the other answer.

Calibrate before a batch and during it. Seed a few tasks whose accepted sets are already settled, including at least one with two valid answers, and watch for reviewers who mark the second answer wrong. Disagreement on those tasks points to an unclear check, which is cheaper to fix than to adjudicate one task at a time.

When can equivalence not be settled by the team itself?

Stop and bring in a specialist when the checks depend on a rule the team cannot interpret with confidence: tax treatment, revenue recognition, employment law, clinical or safety thresholds, or contract terms that are in dispute. In those cases a confident equivalence label is worse than a cannot tell with the open question written down.

Narrow the scope instead of stretching the protocol when the requirement itself is contested, for example when two departments define a correct close differently. Split the task into one requirement per definition, or exclude it. The protocol also assumes the source records are complete enough to grade; if a key document is missing, grade the affected check cannot tell rather than inferring the result.

Accepted-answer sets as part of a licensed dataset

Operational records already contain alternative valid resolutions, because different staff solved similar cases in different sound ways. Documenting which alternatives the business accepts, and why, turns that variation from apparent noise into a labeled asset.

For a company whose staff already resolve cases in several sound ways, SourceX can take those records through rights review, contracting, preparation and delivery as a licensed dataset. To begin, a company completes a short assessment without uploading data, and its approval of scope and terms precedes any delivery.

Frequently asked questions

What is answer equivalence in expert labeling?

It is a label stating that two or more different answers all satisfy the same requirement. A useful equivalence label also records the basis, such as same end state, within a stated tolerance or permitted by policy, so a reviewer can check it.

Should I still pick one gold answer for training?

Keep the full accepted set and, if experts have a reason, mark a preferred answer with that reason. A buyer can then train on the preferred answer, on all accepted answers, or on preference pairs, but cannot recover discarded alternatives later.

How is this different from resolving annotator disagreement?

Disagreement resolution assumes one label is right and looks for it. Equivalence first asks whether both answers pass the requirement checks; only answers that fail a check or contradict each other go to adjudication.

Who should write the requirement checks?

Someone who owns the business outcome, such as a controller for a close task or a service manager for a dispatch task, writes them before the experts answer. Experts can propose changes, but changes are versioned and applied to every task in the batch.

What if the experts cannot agree on whether an answer passes?

Grade the check cannot tell, record the missing fact or rule that would settle it, and send the task to an adjudicator with the right qualification. If the rule is legal or regulatory, the task waits for a specialist or is excluded.

Sources

  • The SQuAD scoring code lowercases answers, strips punctuation and articles, and takes the best score across all reference answers. Source
  • Math-Verify parses answers into a common symbolic form to check mathematical equivalence, shows a set union matching the listed elements, and notes that argument order matters. Source
  • The test suite metric estimates semantic accuracy of SQL by execution and is the official metric of the Spider, SParC and CoSQL leaderboards. Source
  • SWE-bench evaluates a patch by applying it and running the repository's tests; an instance is resolved when the patch makes the required tests pass. Source
  • The HealthBench grader scores responses rubric item by rubric item, does not require every example after such as or including, and supports undesirable criteria with negative points. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies