AI uses for records
Business reasoning data supervision: right answers, wrong assumptions
By SourceX Editorial · Updated
10 sections
Short answer
Grade the answer and its assumptions separately. Extract each business assumption the answer relies on, check it against the record that settles it, and flip the underlying fact to see whether the answer would change. A correct answer resting on a contradicted, load-bearing assumption fails the assumption check. Then test whether examples selected for that failure cohort fix it on untouched held-out tasks without regressions elsewhere.
License terms
Business reasoning data
Key takeaways
- Score outcome and assumptions separately; a correct answer can still fail the assumption check.
- Only assumptions a specific record can settle are gradable; put the rest in an unresolvable bucket.
- Flip the underlying fact to tell a load-bearing assumption from an incidental one.
- Do not treat the model's written reasoning as proof of what produced the answer.
- Compare selected examples against a random matched arm at the same budget, on untouched held-out tasks.
- Keep a regression suite and lineage for every selected example.
How do you evaluate a correct answer based on an incorrect business assumption?
Score it on two axes and never let the first hide the second. The outcome axis asks whether the final answer matches the reference. The assumption axis asks whether every business fact the answer depends on is true for this case, according to the record that controls it. An answer can pass the first and fail the second, and in business work that combination is the dangerous one, because the next case with the same wording but a different underlying fact will go wrong.
In practice this means four steps for every graded item: list the assumptions the answer states or clearly implies, label each one against its controlling evidence, test whether the assumption is load-bearing by flipping the fact it describes, and assign the item to an error cohort. Once the cohorts are labeled, you can run the second half of the job, which is finding out which training examples actually repair a cohort when measured on tasks the model and the selection process never saw.
Why final-answer grading misses right-for-the-wrong-reason answers
Final-answer grading rewards the destination and ignores the route. Research on math reasoning made the distinction explicit: in Let's Verify Step by Step, outcome-supervised reward models learn only from the final result, while process-supervised models receive feedback on each step, and the authors report that process supervision trained more reliable reward models on the MATH benchmark. They also note that outcome supervision can be automated when answers are machine-checkable, whereas step feedback needed human labelers.
Uesato and colleagues framed the same trade-off in terms of final-answer error and trace error. Outcome-based feedback is label-efficient when a correct final answer is all you need; when people must rely on the reasoning, as in question answering where a correct answer is hard to trust without its sources, low trace error matters too. Business decisions sit firmly in the second group: an auditor, a customer or a manager will ask why.
The failure is not limited to arithmetic. McCoy, Pavlick and Linzen showed that language inference models can score well while relying on shallow heuristics such as word overlap, and built the HANS set to expose them. A business model that learns that refund requests mentioning a damaged item get approved is using the same kind of shortcut: right most of the time, and wrong precisely on the cases the policy was written for.
Which business assumptions can a grader actually check?
Only assumptions that a specific record can settle are gradable. Before labeling, sort the assumptions your task family produces into types, and write down for each type which system holds the answer and how a grader looks it up. Assumptions with no controlling record go into a separate unresolvable bucket rather than being guessed.
| Assumption type | Illustrative example | Record that settles it | Typical system |
|---|---|---|---|
| Policy version | The 2025 returns policy applies to this order | Policy effective dates and the order date | Policy library, knowledge base, Confluence |
| Entity status | The customer is on an active annual contract | Contract start, end and status fields | Salesforce, NetSuite, billing system |
| Time period | The claim falls inside the warranty window | Install date, claim date and warranty term | ServiceTitan, field service or ERP records |
| Authority | A team lead can approve this exception | Approval matrix and the approver's role | HR system, approval workflow, Jira or ServiceNow |
| Eligibility or tier | The account qualifies for priority handling | Tier field and the rule that sets it | Zendesk organization fields, CRM tier |
| Prior event | The part was already replaced once | Work order and parts history | Work order system, inventory ledger |
| Unit or definition | Quantity means cases, not individual units | Item master unit of measure | ERP item master, data dictionary |
| Causal claim | The delay was caused by the carrier | Shipment events and exception codes | TMS, carrier tracking records |
What should the assumption label and reasoning evidence guide include?
One row per assumption, linked to one graded answer, with enough evidence that a second reviewer can reach the same verdict without asking the first. The fields below are the guide; the allowed values keep labels comparable across graders and task families. Keep the stated reasoning and the grader's extraction in separate fields so you can tell what the model wrote from what the grader inferred.
At a glance
What should the assumption label and reasoning evidence guide include
| Field | What to record | Allowed values or format | Evidence rule |
|---|---|---|---|
| Item and answer id | The held-out task and the model output being graded | Stable ids, model and prompt version | Never reuse an id across splits |
| Outcome verdict | Whether the final answer matches the reference | Correct, incorrect, partially correct | Reference must cite its own source record |
| Assumption text | The business fact the answer relies on, in one sentence | Stated or implied, quoted span if stated | Implied assumptions need a second grader |
| Assumption type | The category from the gradable types list | Policy, status, period, authority, tier, prior event, unit, causal | Unlisted types go to review, not to other |
| Assumption verdict | Whether the fact is true for this case | Supported, contradicted, unsupported, unresolvable | Supported or contradicted needs a record reference |
| Controlling evidence | The record and field that settle it | System, record id, field, as-of date | As-of date must match the task date |
| Load-bearing test | Whether flipping the fact changes the reference answer | Yes, no, not tested | Record the flipped fact and the new reference |
| Error cohort | The failure group the item belongs to | Short cohort code, for example period-window | One primary cohort per item |
| Grader and confidence | Who labeled it and how sure they are | Grader id, high or low | Low confidence routes to adjudication |
How do you tell a load-bearing assumption from a harmless one?
Flip the fact and see whether the correct answer moves. For each contradicted assumption, write a counterfactual version of the case in which the assumption would be true, and ask the task owner what the right answer would be then. If the reference answer changes, the assumption is load-bearing and the original item is a right-for-the-wrong-reason failure. If the answer is the same either way, the assumption is incidental; log it, but do not let it drive selection.
Do not rely on the model's written reasoning to decide which assumptions mattered. Turpin and colleagues showed that adding biasing features to prompts, such as reordering multiple-choice options so the answer is always A, shifted model answers while the explanations did not mention the bias. Lanham and colleagues tested reliance on chain of thought by truncating it or inserting mistakes and found that how much the answer depends on the stated reasoning varies widely by task. Stated reasoning is evidence about the answer, not proof of what produced it, so the flip test works on the case facts and the model's behavior, not on the explanation text alone.
A practical rule: an item enters a training-selection cohort only when it has at least one contradicted, load-bearing assumption confirmed by a second grader. Everything else stays in the evaluation record for reporting.
Testing which examples repair the failure cohort
Treat example selection as an experiment with a control, not a judgment call. Define the cohort, write the selection rule before you look at outcomes, hold the training budget constant across arms, and score every arm on held-out tasks that no selection step has touched. Targeted selection is a known approach: LESS ranks candidate training examples by how similar their gradients are to a few examples of the target capability, and its authors report that a selected subset of 5 percent of the data can often outperform training on the full set. Whatever method you use, the comparison against matched random selection is what tells you the rule did the work.
Measure outcome evidence on the cohort and regression evidence everywhere else. Fine-tuning that fixes one behavior can erode another; Luo and colleagues documented catastrophic forgetting of domain knowledge, reasoning and reading comprehension during continual fine-tuning of models from about 1 billion to 7 billion parameters. Keep a fixed regression suite of tasks outside the cohort, and keep lineage for every selected example back to its source record and label row.
| Arm | What it trains on | What it controls for | Read it as |
|---|---|---|---|
| Baseline | No added examples | Starting cohort and regression scores | The reference point for every other arm |
| Random matched | Same number of examples drawn at random from the pool | Gains from simply adding more data | The bar the selection rule must clear |
| Outcome-only selection | Examples with correct final answers, assumptions unchecked | Whether answer correctness alone is enough | Often fixes accuracy while the cohort persists |
| Assumption-labeled selection | Examples whose controlling evidence supports every load-bearing assumption | The value of assumption labels | The arm the guide exists to test |
| Counterfactual pairs | Each selected case plus its flipped twin with the changed answer | Whether the model learns the fact, not the wording | Strongest test of the shortcut being broken |
Worked example: warranty approvals that were right for the wrong reason
This example is illustrative; the company, counts and percentages are invented to show the method. A home services company evaluates a model that drafts warranty claim decisions from work orders and contract records. On a held-out set of 400 claims, final-answer accuracy looks strong. Assumption grading tells a different story: in 52 correctly decided claims the model stated that the unit was inside its warranty window, and for 31 of them the install date in the work order placed the claim outside it. Those claims were approved for a different reason, a goodwill policy for repeat repairs within 90 days.
The flip test confirms the problem. For each of the 31, the reviewer changed the prior-repair date so the goodwill rule no longer applied; the reference answer became deny, and the model still approved, citing the warranty window. The cohort is labeled period-window, with the work order install date and the goodwill policy version as controlling evidence.
The team then builds four arms of equal size from a pool of historical claims: random, outcome-only, assumption-labeled and counterfactual pairs. The evaluation is scored on a fresh 120-claim cohort set and a fixed regression suite of 300 unrelated claims. In this illustration, the assumption-labeled and counterfactual-pair arms reduce period-window errors while the outcome-only arm leaves them roughly where they started, and the regression suite shows one cohort, parts-only claims, getting slightly worse in the counterfactual arm. The team ships the assumption-labeled set, keeps the pairs for a follow-up with more parts-only cases, and records both decisions with their lineage.
When the reference answer, not the model, carries the wrong assumption
Sometimes the grading uncovers that the reference is the thing that is wrong. A reference answer written from memory can assume a policy version that had already been replaced, or a reviewer may have approved a case under an exception that was never written down. When the controlling record contradicts the reference, mark the item reference-disputed, take it out of both training and scoring, and send it to the policy owner.
Ambiguous policy is the other exception. If two current documents give different rules for the same case, the assumption cannot be graded, and forcing a verdict teaches the model one side of an unresolved question. Keep these items in an unresolvable bucket, report how many there are, and treat a large bucket as a finding about the business rules rather than the model.
Reading the selection result: ship, narrow or drop the examples
Decide from the pattern of cohort and regression results, not from overall accuracy. The table maps common result patterns to an action. Bring in a post-training researcher when gains appear only in one arm and one seed, and bring in the business owner of the policy when the unresolvable or reference-disputed buckets grow.
| Result pattern | What it suggests | Action |
|---|---|---|
| Cohort improves, regressions flat, beats random matched | The labels selected useful examples | Ship the selected set and keep its lineage |
| Cohort improves, random matched improves as much | Volume, not selection, did the work | Drop the selection rule; revisit labeling cost |
| Cohort improves, one regression cohort worsens | The examples taught a new shortcut | Narrow the set and add counterfactual pairs for the harmed cohort |
| Accuracy improves, cohort unchanged | Outcome-only gains hid the assumption failure | Do not ship; inspect whether selected cases contain the controlling fact |
| Results vary across seeds or evaluation draws | The cohort is too small to read | Enlarge the held-out cohort before deciding |
| Many items reference-disputed or unresolvable | Business rules or references are inconsistent | Stop selection and fix references with the policy owner |
Finding records that show which facts a decision depended on
Assumption grading needs cases where the controlling facts and the decision are both on file: contract and work order dates beside the approval, the approver's role beside the exception, the policy version beside the outcome. Those records usually live in operating systems rather than in public data, and they are most useful when several years of decisions under changing rules are kept together.
Post-training teams that want decisions paired with their controlling records can ask SourceX for them; it sources each dataset from U.S. businesses after the request describes scope, volume, language, format, timing and licensing needs, rather than keeping one on hand. Useful supplier histories typically span two or more years and tens of thousands of records in one system, though rarer work can qualify with smaller sets.
Frequently asked questions
What is business reasoning data supervision?
It is feedback on the reasons behind a business answer, not only on the answer itself. Graders check the facts and rules a response relies on, such as policy version, account status or approval authority, against the records that control them. It extends the idea of process supervision from math steps to business assumptions.
Is assumption labeling the same as step-level process supervision?
They are related but not identical. Step-level supervision labels each reasoning step as correct or not, as in the PRM800K dataset of 800,000 step labels. Assumption labeling targets the business facts a decision depends on and ties each one to a controlling record, which matters more than step order in most operational tasks.
Can a language model extract the assumptions for graders?
It can draft the list, and that saves time, but a person should confirm implied assumptions and every contradicted, load-bearing verdict. Model-written explanations do not always reflect what drove an answer, so the extraction is a starting point, and the flip test on case facts is the check.
Should a correct answer with a wrong assumption count as a failure?
Report it as a pass on outcome and a fail on assumptions, and keep both numbers. Combining them into one score hides the risk, while dropping the outcome result understates what the model got right. The selection experiment should target the assumption failures specifically.
How large should the held-out cohort be?
Large enough that results are stable across random seeds and evaluation draws; there is no single number that fits every task. If arms swap order when you resample the evaluation set, the cohort is too small to support a decision and needs more items before anything ships.
Sources
- Let's Verify Step by Step defines outcome and process supervision, reports that process supervision trained more reliable reward models on MATH, and notes that step labels required human labelers. Source
- The PRM800K repository describes a process supervision dataset of 800,000 step-level correctness labels with quality-control questions checked against gold labels. Source
- Uesato and colleagues contrast final-answer error with trace error and explain when low trace error matters. Source
- McCoy, Pavlick and Linzen show inference models adopting fallible syntactic heuristics, introduce HANS, and report that HANS-style training examples reduced the shortcomings. Source
- Turpin and colleagues show biasing features change model answers while chain-of-thought explanations omit them. Source
- Lanham and colleagues test reliance on chain of thought by truncating it or adding mistakes, finding reliance varies by task. Source
- LESS selects training examples by gradient similarity to few-shot examples of a target capability; a 5 percent selected subset often outperformed the full dataset. Source
- Luo and colleagues document catastrophic forgetting during continual fine-tuning of language models from about 1B to 7B parameters. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.