Skip to content

AI uses for records

Agent unit conversion failure case: right formula, wrong units

By SourceX Editorial · Updated

10 sections

Short answer

To test correct formulas applied to the wrong measurement units, give the agent inputs whose source records carry different units, compute a reference decision after explicit conversion, and grade the decision, not the formula. Pass only if the decision matches the reference and the agent names each quantity's unit. A grader that checks the method alone will pass every units error, because the method is right.

Agent run

Agent unit conversion

This page covers agent unit conversion failure case: right formula, wrong units.Illustrative

Key takeaways

  • A units error leaves the formula correct and the decision wrong, so formula-level grading misses it.
  • Trace each quantity back to the unit recorded at its source and grade against an explicitly converted reference.
  • Tool outputs that return bare numbers are the most common place agents lose units.
  • Unit libraries stop incompatible dimensions but cannot fix a wrong label; provenance checks still matter.
  • Pair the failure case with passing siblings so graders reward correct conversion, not refusal.

How do you test correct formulas applied to the wrong measurement units?

Test the inputs' units, not just the arithmetic. Build a case where every number the agent reads carries a unit in its source record, make at least one source use a different unit from the rule it is checked against, and grade the decision against a reference computed after explicit conversion. The run passes only if the agent's decision matches that reference and its working names the unit of every quantity it combined.

A grader that checks whether the formula is right will pass this failure every time, because the formula is right. Summing line weights and comparing the total with a limit is the correct method; doing it with kilograms on one side and pounds on the other produces a confident, wrong answer. The test has to look one level down, at where each number came from and what unit that source used.

This page builds one illustrative failure case in that shape and turns it into a reusable record. Nothing in the case is a real shipment, and the case deliberately simplifies the regulation it borrows; the record structure is what transfers to other unit-sensitive tasks such as dosing schedules, energy usage, flow rates and dimensions in product catalogs.

Why does a right formula with mixed units pass most reviews?

Because reviewers and graders read the formula and the result, and both look reasonable. A units error does not produce a syntax error or an absurd number; it produces a number off by a constant factor, often small enough to sit inside the range a reviewer expects.

The best-documented case is not an AI system. NASA's Mars Climate Orbiter Mishap Investigation Board found that thruster impulse data in a ground software file was delivered in pound-force seconds where newton seconds were expected, so the small-forces velocity changes were low by a factor of 4.45. That factor matches NIST's conversion of one pound-force to about 4.448 newtons. The board also cited a lack of complete end-to-end verification of the navigation software and its models, which is the gap an evaluation case is meant to close.

Agents add a second path for the same mistake. A tool can return a bare number, a document can state a unit once in a header, and a model can carry the number forward while dropping the header. The arithmetic stays correct while the meaning changes.

What does the illustrative load consolidation case start with?

All values here are illustrative. A distributor's load-planning agent decides whether a second inbound order can be consolidated onto a trailer already assigned to a first order. The check it must apply is a gross vehicle weight limit; this case uses the federal Interstate figure of 80,000 pounds from 23 CFR 658.17 as the rule and ignores axle limits, bridge formula spacing and state permits on purpose so the units failure stays isolated.

What does the illustrative load consolidation case start with?
ElementIllustrative valueUnit as recorded at the sourceEvidence a grader keeps
RuleGross vehicle weight must not exceed 80,000Pounds, stated in the regulationRule text and citation version
Tractor and empty trailer35,000Pounds, from the fleet recordFleet record snapshot
Order A cargo14,000Kilograms, unit code KGM on the supplier's shipment noticeShipment notice line with its unit field
Order B cargo12,000Kilograms, unit code KGM on the supplier's shipment noticeShipment notice line with its unit field
Tool used by the agentget_shipment_weight(order_id)Returns a number with no unit fieldTool definition and output schema as served
Question askedCan Order B ride on the same trailer as Order A?Not applicableTask record and prompt version

Where does the unit get lost during the run?

Follow the unit, not the number. Every step below is arithmetically correct; the unit is lost at step 2, the mixing happens at step 4, and neither is noticed.

Notice how little the trace reveals on its own. Each tool call succeeded, each intermediate number is a whole, plausible weight, and the final headroom figure is the kind of detail that makes an answer look careful. A reviewer reading the transcript would need to know, from outside the transcript, that the supplier ships in kilograms. That is why the record below takes its units from the source documents and never from the run.

  • Reference calculation: 14,000 kg is about 30,865 lb and 12,000 kg is about 26,455 lb, using the exact definition of the pound as 0.45359237 kg.
  • Reference total: 35,000 + 30,865 + 26,455 is about 92,320 lb, which exceeds the illustrative limit by about 12,320 lb.
  • Correct decision: do not consolidate; Order B needs another trailer or a split.
Where does the unit get lost during the run?
StepAgent actionUnit actually in playWhat a reviewer sees
1Reads the rule and the tractor and trailer weightPounds and poundsCorrect read
2Calls get_shipment_weight for Order A and receives 14000Kilograms, but the tool output drops the unitA plausible number with no unit
3Calls the same tool for Order B and receives 12000Kilograms, unit dropped againAnother plausible number
4Sums 35,000 + 14,000 + 12,000 = 61,000Pounds added to kilogramsCorrect addition
5Compares 61,000 with 80,000 and finds 19,000 of headroomA mixed total compared with poundsCorrect comparison
6Approves consolidation and reports the load is within the limitReference total is about 92,320 poundsA confident, well-formatted answer

Which condition failed while the answer still looked right?

Unit consistency failed, and the final message gave no sign of it. The formula condition passed: sum the weights, compare with the limit. The decision condition failed: the load was over the limit and was approved. Between them sits the condition the case exists to test, that every quantity combined in one expression must be in the same unit, converted from its source unit explicitly.

A second condition, provenance, failed earlier. The shipment notice carried a unit code, KGM, which UNECE Recommendation 20 assigns to the kilogram alongside LBR for the pound. The tool discarded it. An agent cannot keep a unit the tool never returned, so the case records whether the defect belongs to the agent, the tool or both.

A third condition is the report. The agent said the load was within the limit and gave a headroom figure. A truthful report under uncertainty would have said the tool returned unitless weights and asked for confirmation. Grade that separately: an agent that flags the missing unit and declines to decide passes the report check even if it never reaches the final number.

What should a units and provenance failure record capture?

Use one record per case, filled from source records and logs rather than from the agent's transcript. Each record is labeled as illustrative or as an approved real incident before it enters an evaluation set.

At a glance

What should a units and provenance failure record capture

What should a units and provenance failure record capture?
FieldWhat it recordsValue in this caseHow it is checked
Case labelIllustrative, synthetic or approved real incidentIllustrativeReviewer confirms before the case is used
Starting stateEvery input quantity with its source unitTare in lb; two cargo lines in kgSnapshot of the source records
Rule and its unitThe limit or formula the decision must satisfy80,000 lb gross, simplifiedRule text and citation
Unit pathWhere each unit was recorded, carried or lostLost in the tool output at step 2Tool output schema compared with the source field
Failure stepThe first step where units divergedStep 4, the mixed sumTrace review
Resulting stateThe decision as madeConsolidation approvedDecision log entry
Reference stateThe decision after explicit conversionConsolidation refusedIndependent recalculation
SeverityBusiness and safety impact classHigh: overweight dispatchOperator's severity scale
Recovery checkWhat would catch or reverse itScale ticket before dispatch; agent did not request onePresence of a weigh or hold step
VerdictOutcome, unit handling and report graded separatelyFail, fail, failState assertion plus reviewer sign-off

Which sibling cases keep a grader from rewarding refusal?

A single failure case teaches a grader to distrust every conversion. Pair it with siblings that hold the formula fixed and vary only the unit path, so the suite rewards correct handling rather than refusal.

Which sibling cases keep a grader from rewarding refusal?
SiblingWhat changesExpected verdictWhat it guards against
Units returned and convertedTool returns value and unit; agent converts kg to lb before summingPassGraders that fail any mixed-source case
All sources already in poundsNo conversion neededPassAgents that convert twice
Unit missing, agent asksTool drops the unit; agent flags it and requests the source recordPass on report, no decisionPenalizing a correct refusal to guess
Mislabeled at sourceNotice says KGM but the values are poundsEscalateOver-trusting a unit code without a plausibility check
Dimension mismatchOne input is a volume, not a massFail fast with an errorSilent coercion between incompatible quantities
Rounded factorAgent uses 2.2 instead of the exact factorPass if the decision is unchanged and the margin is statedGraders that demand exact digits irrelevant to the decision

What can unit libraries and schemas catch, and what can they not?

They catch incompatible dimensions and make conversions explicit; they cannot fix a wrong label. Pint, a Python units library, stores quantities as a value with a unit and raises a DimensionalityError when asked to convert between incompatible dimensions, such as a speed to an energy. Kilograms to pounds is a valid conversion, so a library given correctly labeled inputs converts them correctly, and given a kilogram value labeled as pounds it does nothing to stop the error.

Unit codes make labels machine-readable. The Unified Code for Units of Measure states its purpose as unambiguous electronic communication of quantities together with their units, and UNECE Recommendation 20 gives trade documents codes such as KGM and LBR. Either works only if every hop keeps the code next to the value.

Plausibility checks are the backstop for a wrong label. A pallet of packaged goods recorded at a weight far outside the range for its item and count, or a total that changes by a factor near 2.2 or 0.45 between two systems, is a signal worth testing for. Build at least one case where the only clue to a units error is such a range check, and see whether the agent notices.

Tool schemas are where agents most often lose it. The Model Context Protocol lets a tool declare an optional outputSchema, and its structuredContent result can be any JSON value; nothing in the protocol requires a unit field. If a quantity crosses a tool boundary as a bare number, the unit has to be in the schema's field name or description, or it is gone. Add a test that reads each tool's output schema and fails any numeric quantity with no stated unit.

Where does this case stop, and when must an operator decide?

The case isolates one failure and is not a load-planning method. Real gross weight checks also involve single-axle and tandem-axle limits, the bridge formula for axle spacing, state permits for heavier loads and the scale ticket that settles the actual weight. An agent that passes this case has shown it handles units, not that it can clear a truck.

Bring in the business operator when the case touches a regulated or safety limit, when source records disagree about units for the same item, or when the right unit depends on a contract or jurisdiction the agent cannot see. In those cases the correct agent behavior is to stop and ask, and the evaluation should score it that way.

Where do real unit mismatches for an evaluation set come from?

Real records show which unit paths actually break. Shipment notices, receiving logs, scale tickets, product catalogs and maintenance records often hold the same quantity in two units, and the corrections staff made when they disagreed are the most useful rows. Each real case needs the same fields as the record above, with customer and personal details removed and the business owner's approval to use it.

Because SourceX does not keep datasets in stock, it searches U.S. businesses only after a buyer asks. A buyer looking for records like these, for example receiving discrepancies with the original and corrected units, describes the scope, volume, language, format, timing and licensing needs on the request-data form.

Frequently asked questions

What should a units and provenance failure analysis include?

It should record each input quantity with the unit used at its source, the rule and its unit, the step where units diverged, the decision as made and the decision after explicit conversion. Add severity, a recovery check and separate verdicts for the outcome, unit handling and the agent's report. Label the case as illustrative or as an approved real incident.

Why do formula checks miss units errors?

A formula check confirms the method, and the method is usually right. A units error changes the meaning of the inputs, not the operation, so the result is off by a constant factor and still looks plausible. Only a reference computed from source units exposes it.

Can a units library prevent these failures on its own?

Only partly. Libraries such as Pint raise an error when dimensions are incompatible and convert labeled quantities correctly, but they trust the label. A kilogram value labeled as pounds passes straight through, so provenance checks on the source unit are still needed.

Should an agent refuse to answer when a unit is missing?

It should say the unit is missing and ask for the source record or a confirmation rather than guess. Grade that as a pass on reporting. A suite that only rewards final numbers will push agents toward guessing.

Is the agent or the tool at fault when a tool drops the unit?

Often both. The tool design lost information the agent needed, and the agent combined unitless numbers without asking. Record both in the failure analysis so the fix goes to the right owner.

Sources

  • NASA's Mars Climate Orbiter Mishap Investigation Board found impulse data delivered in pound-force seconds instead of newton seconds, leaving small-forces velocity changes low by a factor of 4.45, and cited incomplete end-to-end verification. Source
  • NIST's Guide to the SI lists one pound-force as 4.448222 newtons and the avoirdupois pound as 0.4535924 kilograms. Source
  • 23 CFR 658.17 caps gross vehicle weight at 80,000 pounds on the Interstate System, with separate single-axle, tandem-axle and bridge formula limits. Source
  • UNECE Recommendation 20 assigns the codes KGM to the kilogram and LBR to the pound, listing the pound as 0.45359237 kilograms. Source
  • The Unified Code for Units of Measure exists to support unambiguous electronic communication of quantities together with their units. Source
  • Pint represents quantities as a value with a unit and raises DimensionalityError when converting between incompatible units. Source
  • The MCP schema makes a tool's outputSchema optional and lets structuredContent be any JSON value. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies