AI uses for records
Agent unit conversion failure case: right formula, wrong units
By SourceX Editorial · Updated
10 sections
Short answer
To test correct formulas applied to the wrong measurement units, give the agent inputs whose source records carry different units, compute a reference decision after explicit conversion, and grade the decision, not the formula. Pass only if the decision matches the reference and the agent names each quantity's unit. A grader that checks the method alone will pass every units error, because the method is right.
Agent run
Agent unit conversion
Key takeaways
- A units error leaves the formula correct and the decision wrong, so formula-level grading misses it.
- Trace each quantity back to the unit recorded at its source and grade against an explicitly converted reference.
- Tool outputs that return bare numbers are the most common place agents lose units.
- Unit libraries stop incompatible dimensions but cannot fix a wrong label; provenance checks still matter.
- Pair the failure case with passing siblings so graders reward correct conversion, not refusal.
How do you test correct formulas applied to the wrong measurement units?
Test the inputs' units, not just the arithmetic. Build a case where every number the agent reads carries a unit in its source record, make at least one source use a different unit from the rule it is checked against, and grade the decision against a reference computed after explicit conversion. The run passes only if the agent's decision matches that reference and its working names the unit of every quantity it combined.
A grader that checks whether the formula is right will pass this failure every time, because the formula is right. Summing line weights and comparing the total with a limit is the correct method; doing it with kilograms on one side and pounds on the other produces a confident, wrong answer. The test has to look one level down, at where each number came from and what unit that source used.
This page builds one illustrative failure case in that shape and turns it into a reusable record. Nothing in the case is a real shipment, and the case deliberately simplifies the regulation it borrows; the record structure is what transfers to other unit-sensitive tasks such as dosing schedules, energy usage, flow rates and dimensions in product catalogs.
Why does a right formula with mixed units pass most reviews?
Because reviewers and graders read the formula and the result, and both look reasonable. A units error does not produce a syntax error or an absurd number; it produces a number off by a constant factor, often small enough to sit inside the range a reviewer expects.
The best-documented case is not an AI system. NASA's Mars Climate Orbiter Mishap Investigation Board found that thruster impulse data in a ground software file was delivered in pound-force seconds where newton seconds were expected, so the small-forces velocity changes were low by a factor of 4.45. That factor matches NIST's conversion of one pound-force to about 4.448 newtons. The board also cited a lack of complete end-to-end verification of the navigation software and its models, which is the gap an evaluation case is meant to close.
Agents add a second path for the same mistake. A tool can return a bare number, a document can state a unit once in a header, and a model can carry the number forward while dropping the header. The arithmetic stays correct while the meaning changes.
What does the illustrative load consolidation case start with?
All values here are illustrative. A distributor's load-planning agent decides whether a second inbound order can be consolidated onto a trailer already assigned to a first order. The check it must apply is a gross vehicle weight limit; this case uses the federal Interstate figure of 80,000 pounds from 23 CFR 658.17 as the rule and ignores axle limits, bridge formula spacing and state permits on purpose so the units failure stays isolated.
| Element | Illustrative value | Unit as recorded at the source | Evidence a grader keeps |
|---|---|---|---|
| Rule | Gross vehicle weight must not exceed 80,000 | Pounds, stated in the regulation | Rule text and citation version |
| Tractor and empty trailer | 35,000 | Pounds, from the fleet record | Fleet record snapshot |
| Order A cargo | 14,000 | Kilograms, unit code KGM on the supplier's shipment notice | Shipment notice line with its unit field |
| Order B cargo | 12,000 | Kilograms, unit code KGM on the supplier's shipment notice | Shipment notice line with its unit field |
| Tool used by the agent | get_shipment_weight(order_id) | Returns a number with no unit field | Tool definition and output schema as served |
| Question asked | Can Order B ride on the same trailer as Order A? | Not applicable | Task record and prompt version |
Where does the unit get lost during the run?
Follow the unit, not the number. Every step below is arithmetically correct; the unit is lost at step 2, the mixing happens at step 4, and neither is noticed.
Notice how little the trace reveals on its own. Each tool call succeeded, each intermediate number is a whole, plausible weight, and the final headroom figure is the kind of detail that makes an answer look careful. A reviewer reading the transcript would need to know, from outside the transcript, that the supplier ships in kilograms. That is why the record below takes its units from the source documents and never from the run.
- Reference calculation: 14,000 kg is about 30,865 lb and 12,000 kg is about 26,455 lb, using the exact definition of the pound as 0.45359237 kg.
- Reference total: 35,000 + 30,865 + 26,455 is about 92,320 lb, which exceeds the illustrative limit by about 12,320 lb.
- Correct decision: do not consolidate; Order B needs another trailer or a split.
| Step | Agent action | Unit actually in play | What a reviewer sees |
|---|---|---|---|
| 1 | Reads the rule and the tractor and trailer weight | Pounds and pounds | Correct read |
| 2 | Calls get_shipment_weight for Order A and receives 14000 | Kilograms, but the tool output drops the unit | A plausible number with no unit |
| 3 | Calls the same tool for Order B and receives 12000 | Kilograms, unit dropped again | Another plausible number |
| 4 | Sums 35,000 + 14,000 + 12,000 = 61,000 | Pounds added to kilograms | Correct addition |
| 5 | Compares 61,000 with 80,000 and finds 19,000 of headroom | A mixed total compared with pounds | Correct comparison |
| 6 | Approves consolidation and reports the load is within the limit | Reference total is about 92,320 pounds | A confident, well-formatted answer |
Which condition failed while the answer still looked right?
Unit consistency failed, and the final message gave no sign of it. The formula condition passed: sum the weights, compare with the limit. The decision condition failed: the load was over the limit and was approved. Between them sits the condition the case exists to test, that every quantity combined in one expression must be in the same unit, converted from its source unit explicitly.
A second condition, provenance, failed earlier. The shipment notice carried a unit code, KGM, which UNECE Recommendation 20 assigns to the kilogram alongside LBR for the pound. The tool discarded it. An agent cannot keep a unit the tool never returned, so the case records whether the defect belongs to the agent, the tool or both.
A third condition is the report. The agent said the load was within the limit and gave a headroom figure. A truthful report under uncertainty would have said the tool returned unitless weights and asked for confirmation. Grade that separately: an agent that flags the missing unit and declines to decide passes the report check even if it never reaches the final number.
What should a units and provenance failure record capture?
Use one record per case, filled from source records and logs rather than from the agent's transcript. Each record is labeled as illustrative or as an approved real incident before it enters an evaluation set.
At a glance
What should a units and provenance failure record capture
| Field | What it records | Value in this case | How it is checked |
|---|---|---|---|
| Case label | Illustrative, synthetic or approved real incident | Illustrative | Reviewer confirms before the case is used |
| Starting state | Every input quantity with its source unit | Tare in lb; two cargo lines in kg | Snapshot of the source records |
| Rule and its unit | The limit or formula the decision must satisfy | 80,000 lb gross, simplified | Rule text and citation |
| Unit path | Where each unit was recorded, carried or lost | Lost in the tool output at step 2 | Tool output schema compared with the source field |
| Failure step | The first step where units diverged | Step 4, the mixed sum | Trace review |
| Resulting state | The decision as made | Consolidation approved | Decision log entry |
| Reference state | The decision after explicit conversion | Consolidation refused | Independent recalculation |
| Severity | Business and safety impact class | High: overweight dispatch | Operator's severity scale |
| Recovery check | What would catch or reverse it | Scale ticket before dispatch; agent did not request one | Presence of a weigh or hold step |
| Verdict | Outcome, unit handling and report graded separately | Fail, fail, fail | State assertion plus reviewer sign-off |
Which sibling cases keep a grader from rewarding refusal?
A single failure case teaches a grader to distrust every conversion. Pair it with siblings that hold the formula fixed and vary only the unit path, so the suite rewards correct handling rather than refusal.
| Sibling | What changes | Expected verdict | What it guards against |
|---|---|---|---|
| Units returned and converted | Tool returns value and unit; agent converts kg to lb before summing | Pass | Graders that fail any mixed-source case |
| All sources already in pounds | No conversion needed | Pass | Agents that convert twice |
| Unit missing, agent asks | Tool drops the unit; agent flags it and requests the source record | Pass on report, no decision | Penalizing a correct refusal to guess |
| Mislabeled at source | Notice says KGM but the values are pounds | Escalate | Over-trusting a unit code without a plausibility check |
| Dimension mismatch | One input is a volume, not a mass | Fail fast with an error | Silent coercion between incompatible quantities |
| Rounded factor | Agent uses 2.2 instead of the exact factor | Pass if the decision is unchanged and the margin is stated | Graders that demand exact digits irrelevant to the decision |
What can unit libraries and schemas catch, and what can they not?
They catch incompatible dimensions and make conversions explicit; they cannot fix a wrong label. Pint, a Python units library, stores quantities as a value with a unit and raises a DimensionalityError when asked to convert between incompatible dimensions, such as a speed to an energy. Kilograms to pounds is a valid conversion, so a library given correctly labeled inputs converts them correctly, and given a kilogram value labeled as pounds it does nothing to stop the error.
Unit codes make labels machine-readable. The Unified Code for Units of Measure states its purpose as unambiguous electronic communication of quantities together with their units, and UNECE Recommendation 20 gives trade documents codes such as KGM and LBR. Either works only if every hop keeps the code next to the value.
Plausibility checks are the backstop for a wrong label. A pallet of packaged goods recorded at a weight far outside the range for its item and count, or a total that changes by a factor near 2.2 or 0.45 between two systems, is a signal worth testing for. Build at least one case where the only clue to a units error is such a range check, and see whether the agent notices.
Tool schemas are where agents most often lose it. The Model Context Protocol lets a tool declare an optional outputSchema, and its structuredContent result can be any JSON value; nothing in the protocol requires a unit field. If a quantity crosses a tool boundary as a bare number, the unit has to be in the schema's field name or description, or it is gone. Add a test that reads each tool's output schema and fails any numeric quantity with no stated unit.
Where does this case stop, and when must an operator decide?
The case isolates one failure and is not a load-planning method. Real gross weight checks also involve single-axle and tandem-axle limits, the bridge formula for axle spacing, state permits for heavier loads and the scale ticket that settles the actual weight. An agent that passes this case has shown it handles units, not that it can clear a truck.
Bring in the business operator when the case touches a regulated or safety limit, when source records disagree about units for the same item, or when the right unit depends on a contract or jurisdiction the agent cannot see. In those cases the correct agent behavior is to stop and ask, and the evaluation should score it that way.
Where do real unit mismatches for an evaluation set come from?
Real records show which unit paths actually break. Shipment notices, receiving logs, scale tickets, product catalogs and maintenance records often hold the same quantity in two units, and the corrections staff made when they disagreed are the most useful rows. Each real case needs the same fields as the record above, with customer and personal details removed and the business owner's approval to use it.
Because SourceX does not keep datasets in stock, it searches U.S. businesses only after a buyer asks. A buyer looking for records like these, for example receiving discrepancies with the original and corrected units, describes the scope, volume, language, format, timing and licensing needs on the request-data form.
Frequently asked questions
What should a units and provenance failure analysis include?
It should record each input quantity with the unit used at its source, the rule and its unit, the step where units diverged, the decision as made and the decision after explicit conversion. Add severity, a recovery check and separate verdicts for the outcome, unit handling and the agent's report. Label the case as illustrative or as an approved real incident.
Why do formula checks miss units errors?
A formula check confirms the method, and the method is usually right. A units error changes the meaning of the inputs, not the operation, so the result is off by a constant factor and still looks plausible. Only a reference computed from source units exposes it.
Can a units library prevent these failures on its own?
Only partly. Libraries such as Pint raise an error when dimensions are incompatible and convert labeled quantities correctly, but they trust the label. A kilogram value labeled as pounds passes straight through, so provenance checks on the source unit are still needed.
Should an agent refuse to answer when a unit is missing?
It should say the unit is missing and ask for the source record or a confirmation rather than guess. Grade that as a pass on reporting. A suite that only rewards final numbers will push agents toward guessing.
Is the agent or the tool at fault when a tool drops the unit?
Often both. The tool design lost information the agent needed, and the agent combined unitless numbers without asking. Record both in the failure analysis so the fix goes to the right owner.
Sources
- NASA's Mars Climate Orbiter Mishap Investigation Board found impulse data delivered in pound-force seconds instead of newton seconds, leaving small-forces velocity changes low by a factor of 4.45, and cited incomplete end-to-end verification. Source
- NIST's Guide to the SI lists one pound-force as 4.448222 newtons and the avoirdupois pound as 0.4535924 kilograms. Source
- 23 CFR 658.17 caps gross vehicle weight at 80,000 pounds on the Interstate System, with separate single-axle, tandem-axle and bridge formula limits. Source
- UNECE Recommendation 20 assigns the codes KGM to the kilogram and LBR to the pound, listing the pound as 0.45359237 kilograms. Source
- The Unified Code for Units of Measure exists to support unambiguous electronic communication of quantities together with their units. Source
- Pint represents quantities as a value with a unit and raises DimensionalityError when converting between incompatible units. Source
- The MCP schema makes a tool's outputSchema optional and lets structuredContent be any JSON value. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.