Skip to content

AI uses for records

How to require an agent judge to cite the evidence behind its verdict

By SourceX Editorial · Updated

9 sections

Short answer

Require an AI judge to return one verdict per criterion, each with evidence anchors: the artifact relied on, an exact locator such as a record field or trace event, and the quoted value found there. Check every anchor in code, reject the agent's own claims as sole support, and record any verdict with failed anchors as unsupported rather than passed. Keep expert disagreement and a cannot-determine verdict.

Agent run

Agent judge evidence

This page covers how to require an agent judge to cite the evidence behind its verdict.Illustrative

Key takeaways

  • A judge's reasoning paragraph is a narrative, not evidence; require locators and quoted values for every verdict.
  • Verify anchors with deterministic code: resolve the locator, match the quote exactly, check admissibility and timing.
  • The agent's own messages can never be the only support for a met verdict.
  • Test the judge on adversarial runs such as self-report, superseded state and padding before it grades real work.
  • Keep a cannot-determine verdict and record expert disagreement per criterion instead of averaging it away.
  • Anchors prove presence, not sufficiency; criteria about authority or adequacy still need expert review.

How do you require an AI judge to cite evidence for its verdict?

Change the judge's output contract. Instead of one verdict and a paragraph of reasoning, the judge returns one verdict per criterion, and each verdict must carry evidence anchors: the artifact it relied on, an exact locator such as a record ID and field or an event ID in the trace, and the quoted text or value found there. Code then checks every anchor, and a verdict whose anchors fail is recorded as unsupported, never as a pass.

That contract does three jobs at once. It makes a verdict auditable by a person who never saw the run, it gives you a cheap automated test of the judge itself, and it exposes the most common failure in agent grading: a judge that believed the agent's own description of what it did.

The rest of this page gives the report format field by field, the mechanical checks that sit between the judge and the score, the shortcuts the format is designed to catch, and a worked example from an accounts payable control.

Why the judge's written explanation is not the evidence

A rationale is a story about the verdict, not a pointer to the facts behind it. Research on chain-of-thought prompting found that model explanations can systematically misrepresent the real reason for a prediction, for example when biasing features such as reordered answer options shift the answer without being mentioned in the explanation. A judge's reasoning paragraph inherits that weakness.

Judges also carry known biases. The MT-Bench and Chatbot Arena study of LLM judges examined position bias, verbosity bias and self-enhancement bias, and measured how often a judge changed its preference when two answers were swapped. A long, confident agent transcript is exactly the input that invites a verbosity-driven pass.

Common judge templates ask for reasoning but not for locations. The pairwise prompt in FastChat's LLM judge asks the model to compare the responses, provide a short explanation and then output a bracketed verdict. Inspect's default model-graded template asks the grader to reason step by step and finish with a grade letter, and the scorer stores the grader's full completion as the score's explanation. Both are reasonable defaults for short answers. Neither forces the judge to say where in a multi-step agent run it found the proof, so an explanation can read well while resting on nothing a reviewer could open.

What the evidence anchored grading report format includes

The report has one row per criterion per run, and each row must stand on its own. A reviewer should be able to open the row, follow the anchors and reach the same verdict without rereading the transcript. The fields below are the minimum; add domain fields such as a policy clause number where a criterion depends on one.

What the evidence anchored grading report format includes
FieldWhat it holdsCheck appliedIf the check fails
Criterion ID and versionThe requirement being graded, with the rubric version in force for this runMust match the frozen rubric for the evaluationRow rejected; regrade under the right version
VerdictMet, not met, or cannot determineMust be one of the three; no free textRecorded as grader failure, not as a fail
Evidence anchorsOne or more pointers, each with source type, locator and quoted valueEach anchor resolved and quote matched by codeVerdict downgraded to unsupported
Source typeSystem of record after the run, tool call result, file produced, policy text, or agent messageAgent messages cannot be the only support for metVerdict downgraded to unsupported
LocatorRecord ID plus field, trace event ID, file path plus page or character rangeLocator must exist in the run's snapshotAnchor marked missing
Quoted valueThe exact text or value at the locatorExact match against the snapshot, not fuzzyAnchor marked mismatched
Counter-evidence searchedWhere the judge looked for contrary facts, such as a later reversal or a superseded recordAt least one location listed for criteria with known reversalsFlag for human review
Judge identityModel, prompt version and temperature usedMust match the registered judge configurationRun excluded from comparison
AdjudicationExpert verdict, expert anchors and any disagreement noteFilled for the audit sample and for every disputed rowRow stays open; not counted
SeverityBusiness impact if the verdict is wrong, on the team's own scaleRequired for every not met and every unsupported rowTreated as highest severity until set

Checking every anchor with code before the verdict counts

Run the anchor checks in code between the judge's output and the scoreboard, so a judge cannot pass a run by asserting an anchor exists. The checks are deterministic, cheap and run on every row, which is what makes them worth more than a second judge.

At a glance

Checking every anchor with code before the verdict counts

Exact matching matters because paraphrase is where fabricated support hides. Citation features in model APIs show what a checkable citation looks like: Anthropic's documentation describes plain-text document citations that return the cited text together with zero-indexed start and end character positions. Whatever tool you use, the standard is the same: a quoted value plus a position that code can read back from the stored artifact.

Anchors must point at state captured after the run, not at what the agent said about the state. Agent-as-a-Judge, a published framework that has an agent gather evidence while judging, was demonstrated on DevAI, a benchmark of 55 AI development tasks with 365 hierarchical user requirements; its repository includes an example of the evidence-collection process. Collecting evidence is the right instinct; the report format adds the rule that evidence is only counted after code confirms it exists where the judge says it does.

  • Resolve: every locator opens a real object in the run's snapshot (record, event, file page).
  • Match: the quoted value equals the stored value at that locator, character for character after agreed normalization such as whitespace.
  • Admissibility: at least one anchor for a met verdict comes from the system of record or a tool result, not from the agent's own messages.
  • Timing: the anchored event falls inside the run window and in the order the criterion requires.
  • Coverage: every criterion has a row; a missing row is a grader failure, not a silent pass.

Which shortcuts could raise the score without completing the business requirement?

The shortcuts that matter are the ones where both the agent and the judge look busy while the requirement goes unmet. Build one adversarial test run for each before the judge grades anything real, and require the judge to return not met or cannot determine on each. A judge that passes any of them is not ready.

Checklist · 7 items

  1. Self-report: the agent's final message says the step was done and the system of record shows nothing. Expected verdict: not met.
  2. Prompt echo: the judge quotes the task instructions as if they were proof of completion. Expected: anchor rejected as inadmissible.
  3. Right record, wrong field: the anchor points at the vendor record but quotes a field the criterion does not concern. Expected: flagged on review sample; add a field allowlist per criterion.
  4. Superseded state: the agent made the change and a later event reversed it. Expected: not met, with the reversal cited as counter-evidence.
  5. Padding: the agent writes a long, confident summary with many plausible references. Expected: same verdict as the terse version of the run.
  6. Absence treated as success: a criterion forbids an action and the judge passes it without citing where it looked. Expected: cannot determine unless the searched locations are listed.
  7. Order swap: the required approval exists but was recorded after the action. Expected: not met on the timing check.

0 of 7 ticked. Tick each item as you go.

Worked example (illustrative): a vendor bank-detail change

This example is illustrative and uses no real company data. An accounts payable agent handles a supplier's request to change its bank details. The criterion, taken from the team's payment control, reads: the new details are verified by a callback to a phone number already on file, and the callback is logged before the change is saved in the ERP.

On its first pass, the judge returned met and cited one anchor: the agent's closing message, which said it had confirmed the change by phone. The admissibility check rejected that anchor because an agent message cannot be the only support for met, and the row was downgraded to unsupported.

Regraded with the rule in force, the judge cited the two system records below. Both resolved and both quotes matched, but the timing check failed: the bank-detail change was saved before the callback was logged. The final verdict was not met at the highest severity on the team's scale, because a change saved before verification is the exact exposure the control exists to prevent.

Worked example (illustrative): a vendor bank-detail change
AnchorLocator (illustrative)Quoted valueCheck result
Agent closing messageTrace event 41Confirmed new bank details with supplier by phoneRejected: agent message cannot stand alone
ERP vendor change logVendor V-0193, change entry 7, field bank_accountChanged 14:02, user svc-ap-agentResolved and matched
Call log entryCall record C-5521, field logged_at14:19, number on file, outcome verifiedResolved and matched
Timing ruleCriterion AP-07 version 3Callback logged before change savedFailed: change preceded callback by 17 minutes

Keeping expert disagreement visible instead of averaging it

Disagreement between experts is information about the criterion, so record it rather than resolving it by vote and discarding the minority view. When two domain adjudicators read the same anchors and reach different verdicts, the usual cause is an under-specified criterion, and the fix belongs in the rubric, not in the judge.

Keep a third verdict, cannot determine, for rows where the evidence genuinely does not settle the question. Forcing those rows into met or not met teaches the judge to guess, and it hides exactly the cases a buyer needs to see when deciding whether an agent is ready.

Compare the judge with adjudicators criterion by criterion, not overall. A judge can agree with experts on simple presence checks and diverge on ordering or sufficiency, and a single agreement number blends those together. For each criterion, track how often the judge's anchors matched the anchors the expert chose; a judge that reaches the right verdict from the wrong evidence will fail when the shortcut it relied on disappears.

Where citation rules stop helping, and who should take over

Anchors prove that something exists at a location; they do not prove that it is sufficient. A judge can cite a real approval that was granted by someone without authority, and every mechanical check will pass. Criteria about authority, adequacy or judgment need an expert-reviewed sample on top of anchor checks.

Some criteria have nothing to anchor. Tone, clarity and helpfulness are judgments about the whole response; citing a sentence does not make the verdict more reliable. Grade those separately, label them as subjective, and keep them out of the evidence-anchored score so they cannot dilute it.

Stop and bring in a specialist when a criterion depends on regulation, legal interpretation or financial controls where a wrong pass has real consequences. The evaluation lead owns the report format; a domain adjudicator, such as a controller for payment controls, should sign off the criterion wording and the admissible source list before the judge is trusted at scale. Also give any judge that gathers its own evidence read-only access: a judge that can write to the systems it inspects can change the state it is grading.

Real records give the anchors something to point at

An evidence-anchored judge is only as good as the records it can point to. Synthetic environments often lack the change logs, call records and reversals that make anchors meaningful, which is why teams building agent evaluations look for real operational histories where actions and their outcomes are linked.

Evaluation leads who need change logs, call records and reversals to anchor verdicts can ask SourceX to find them. There is no catalog behind that request: SourceX searches U.S. businesses against the scope, volume, language, format, timing and licensing needs it describes, and the business whose records the anchors would cite signs off on scope and terms before anything is delivered.

Frequently asked questions

Does asking a judge to explain its reasoning make it cite evidence?

No. Reasoning prompts produce a narrative, and research on chain-of-thought explanations shows they can misstate why a model reached its answer. Require explicit anchors with locators and quoted values, then check them in code; keep the reasoning only as a reading aid.

What should an AI judge output when the evidence is missing?

It should return cannot determine and list where it looked. Treating missing evidence as a pass rewards agents that do nothing, and treating it as a fail hides judge errors. A separate verdict keeps those rows visible for review.

Can an agent-as-a-judge check its own citations?

It can gather evidence, but the verification step should be deterministic code outside the judge. A judge that confirms its own anchors can repeat its own mistake, while a resolver that opens the locator and compares the stored value cannot be persuaded.

How many evidence anchors should each criterion require?

Set the number by what the criterion asserts. A presence check needs one admissible anchor, an ordering rule needs at least two with timestamps, and a prohibition needs the locations searched. Write the requirement into the criterion so the judge cannot choose.

Is evidence checking the same as calibrating an LLM judge?

No. Calibration compares the judge's verdicts with expert labels to measure agreement. Evidence checking tests whether each verdict rests on facts that exist. A well-calibrated judge can still cite the wrong record, so run both.

Sources

  • The MT-Bench and Chatbot Arena study of LLM-as-a-judge examined position bias, verbosity bias and self-enhancement bias, measuring consistency when two answers are swapped. Source
  • Chain-of-thought explanations can systematically misrepresent the true reason for a model's prediction, and biasing features such as reordered answer options can shift answers without being mentioned. Source
  • FastChat's pairwise LLM judge prompt asks the judge to compare responses, provide a short explanation, avoid position and length bias, and output a bracketed verdict. Source
  • Inspect's default model-graded template asks the grader to reason step by step and end with a grade letter; the scorer stores the grader's completion as the explanation and marks output without a grade as unscored. Source
  • Agent-as-a-Judge was applied to DevAI, a benchmark of 55 realistic AI development tasks with 365 hierarchical user requirements, and the repository shows how the judge collects evidence. Source
  • Anthropic's citations feature returns cited text with zero-indexed start and end character positions for plain-text documents, and page locations for PDFs. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies