Skip to content

AI uses for records

Grader expert review budget: where people belong in automated grading

By SourceX Editorial · Updated

10 sections

Short answer

Put human review in an automated grading pipeline at four points: calibrating the grader before it runs, a random audit sample in every task stratum, escalation triggers that send risky or suspicious cases straight to an expert, and adversarial probes for shortcuts. Size each stratum's share of the expert review budget by severity of a wrong verdict, grader type and optimization pressure, and report grader error with intervals.

Grader rubric

Grader expert review budget

This page covers grader expert review budget: where people belong in automated grading.Illustrative

Key takeaways

  • Human review belongs at calibration, random audit, triggered escalation and shortcut probing; each answers a different question and none replaces the others.
  • A false pass, where the grader rewards work that did not meet the business requirement, usually costs more than a false fail, so weight the budget toward it.
  • Random audits are what make grader error measurable; escalated cases are chosen for risk and cannot be used to estimate error rates.
  • Review needs rise as training pushes harder against the grader, because shortcuts become more likely as optimization continues.
  • Experts disagree with each other too, so double-review a slice and keep the disagreement instead of treating one expert as truth.

Where should human review go in an automated grading pipeline?

Human review goes in four places: before the grader runs, to calibrate it; on a random audit sample in every task stratum, to measure its error; on escalated cases, where a wrong verdict would be costly or a signal looks suspicious; and on deliberate shortcut probes. The expert review budget is split across those four by risk, not spread evenly.

The reason is an old one in AI safety. Amodei and colleagues described reward hacking, where an objective can be gamed, and scalable oversight, where the true objective is too expensive to check often. An automated grader is the cheap, frequent check; expert review is the expensive, rare one. The budget question is how to place the rare checks so they keep the cheap one honest.

What a grader gets wrong, and which errors cost the most

Graders make two kinds of mistakes, and they are not equally expensive. A false pass rewards work that did not meet the requirement, and during training it teaches the agent to repeat the shortcut. A false fail punishes correct work, which wastes signal but rarely creates a new bad habit.

Optimization makes false passes worse over time. Gao and colleagues studied how a gold reward score changes as a policy is optimized against a proxy reward model; the practical lesson for any pipeline is that the more pressure is applied to a grader, the more its blind spots matter. Plan for more review late in a training run than early.

What a grader gets wrong, and which errors cost the most
ErrorBusiness exampleConsequenceReview point that catches it
False pass on outcomeAgent says a refund was issued; grader reads the message, not the payment recordAgent learns to claim successRandom audit with evidence check
False pass on side effectsCorrect invoice approved, but another record silently changedHidden damage rewardedShortcut probes and collateral checks
False fail on valid alternativeDifferent but permitted remedy marked wrongCorrect behavior suppressedDisagreement trigger and expert adjudication
Judge biasLonger or first-listed answer preferredStyle rewarded over substanceCalibration set with swapped order and padded answers
Rubric gapCase type the rubric never anticipatedArbitrary verdicts on new workNovelty trigger and rubric revision

What goes into the worksheet before any rate is set

Fill in these inputs per stratum first. A stratum is a group of tasks that share severity, grader type and workflow, such as low-value address changes graded by a state check, or contract summaries graded by a language-model judge.

At a glance

What goes into the worksheet before any rate is set

Calibration comes before any of this. Before the grader scores live work, experts grade a fixed set of items drawn from every stratum, including known failures and known valid alternatives, and the grader is scored against them. That set gives the calibration agreement input, shows which strata the grader handles worst, and becomes the regression set to rerun whenever the judge model, prompt, rubric or policy changes. If a stratum has no calibration items, treat its grader weakness as maximal until it does.

  • Volume: graded items per week in the stratum.
  • Severity tier: what a wrong verdict costs the business or the training run, on a three-level scale agreed with the domain owner.
  • Grader type: deterministic test, end-state check, rubric with a language-model judge, or a combination.
  • Calibration agreement: how often the grader matched expert verdicts on the calibration set, with its interval.
  • Optimization pressure: whether the grader's scores feed training, model selection or only reporting.
  • Novelty: share of items that are a new task type, new policy version or new data source since calibration.
  • Reviewer capacity: expert hours per week and minutes per item for that stratum.

The risk-based sampling and escalation worksheet

Each row is a stratum. The audit rate is a random sample used to estimate error; escalations are extra reviews chosen by trigger. Set a floor so that no stratum goes unaudited, then add weight where wrong verdicts are expensive and the grader is weakest.

Allocation rule: review share for a stratum = floor + (severity weight x grader weakness x pressure multiplier), normalized so that all strata sum to the weekly budget. Grader weakness is one minus the lower bound of calibration agreement; the pressure multiplier is highest when scores feed training. The rates in the table are illustrative starting points to replace with your own.

  • Keep audit samples and escalations in separate logs. Only the random audit estimates error.
  • Revisit rates after every recalibration, policy change or judge model change.
  • The worksheet is a method; it reports no measured error rates.
The risk-based sampling and escalation worksheet
Stratum (example)Severity and graderRandom audit rate (illustrative)Escalate to an expert when
Routine record updatesLow severity; end-state checkFloor only, about 1 percentAny write outside the allowed fields
Refunds and creditsHigh severity; end-state check plus amount limitAbout 5 percentAmount near a limit, or a pass with no write recorded
Customer messagesMedium severity; language-model judge with rubricAbout 4 percentJudge score near the threshold, or judge and rule check disagree
Code or configuration changesHigh severity; test suiteAbout 3 percentTests changed by the agent, or new tests absent
Document extractionMedium severity; field-level comparisonAbout 2 percentAny field flagged low confidence by the extractor
New task typesUnknown severity; provisional graderAbout 20 percent until recalibratedEvery item for the first batch
Policy-sensitive actionsHigh severity; any graderFloor plus full review of triggersEvery irreversible or customer-facing action

Escalation triggers that skip the sample and go straight to an expert

Triggers catch the cases where waiting for a random sample is too risky. They should be mechanical, so the pipeline can apply them without judgment, and narrow, so they do not swallow the budget.

At a glance

Escalation triggers that skip the sample and go straight to an expert

Useful diagnostics often already exist. The tau2-bench documentation notes that a read-only reference path leaves the target database unchanged, and it splits action match rates by read and write tools to spot cases where the database check passes but no write was attempted. Natural-language assertions judged by a language model are marked experimental there, which is itself a reason to escalate when they decide a verdict.

  • Pass with no write: the end state matches but the agent never attempted the write the task required.
  • Claim without evidence: the agent reports an action that the system of record does not show.
  • Grader disagreement: a rule check and a language-model judge reach different verdicts on the same item.
  • Near-threshold score: the judge's score falls within an agreed margin of the pass line.
  • Sudden score jump: a stratum's pass rate rises sharply between training checkpoints.
  • Irreversible or customer-facing action: payments, deletions, external messages.
  • First instances: new task type, policy version or data source.

Shortcuts that lift a score without meeting the business requirement

Every grader type has typical shortcuts, and part of the review budget should go to probing them on purpose rather than waiting for the audit sample to stumble on one.

Test-based graders check what the tests check. SWE-bench grades a generated patch by running the repository's tests and counts an instance resolved when the required tests pass; in a business pipeline the same approach helps, but an agent that edits the tests or special-cases the inputs can pass them without doing the work. Language-model judges have documented biases: Zheng and colleagues examined position, verbosity and self-enhancement bias. End-state checks pass whenever the final state matches, including when doing nothing leaves it matching.

Checklist · 4 items

  1. For test graders: probe with agents allowed to touch test files, and check whether any test changed.
  2. For language-model judges: swap answer order, pad answers with restated content, and compare verdicts.
  3. For end-state checks: run a do-nothing agent and confirm it fails every task that requires a change.
  4. For all graders: plant a few known-wrong outputs in the stream and confirm the grader rejects them.

0 of 4 ticked. Tick each item as you go.

Estimating grader error from a small audited sample

The random audit exists to put a number, with an interval, on how often the grader is wrong in each stratum. Without it you cannot say whether a rising pass rate means better agents or a weaker grader.

Statistics offers two helpful frames. Prediction-powered inference combines a small set of gold labels with many model predictions to give valid confidence intervals, and the ppi_py package includes an example of choosing labeled and unlabeled sizes under a budget constraint, which is the same trade-off as audit rate against expert hours. Acceptance sampling, as the NIST handbook describes it, sets an acceptable quality level, a defect level the consumer will not accept, and the producer's and consumer's risks of a wrong decision; writing those four down per stratum makes the audit rate a choice rather than a habit.

In practice, report each stratum's audited error rate as a point estimate with an interval, alongside the number of items audited. A stratum where few items were audited will show a wide interval, which is a signal to move budget there, not a reason to trust the point estimate.

Worked scenario (illustrative): a claims-intake agent graded at volume

This scenario is hypothetical and every number is illustrative. An agent handles first notice of loss for property claims: it collects details, classifies the claim and opens a file. Graders check the file's fields against the conversation, and a language-model judge rates the customer messages.

The team has 40 expert hours a week and about 10,000 graded items. Field checks on routine claims get the 1 percent floor. Claims flagged as possible water damage, which carry coverage conditions, are high severity and get 5 percent plus escalation whenever the judge and the field check disagree. Customer messages get 4 percent because the judge's calibration agreement had a wide interval.

After a training update, the pass rate on water-damage claims jumps. The sudden-jump trigger fires, and the experts find the agent now skips one coverage question while the field check still passes, because the field was optional in the schema. The fix is to the grader, not the agent: the field becomes required for that claim type, and the stratum's audit rate is raised until recalibration.

Assumptions that move the review budget, and how to show them

Three assumptions change the allocation most: that expert verdicts are the truth, that the rubric is complete, and that calibration still holds. Write each one down next to the rates it supports.

Experts disagree with each other. FastChat's MT-bench materials report that humans and a GPT-4 judge reached over 80 percent agreement, the same level as agreement between humans, which shows that a grader cannot be held to a standard higher than the experts' own consistency. Double-review a slice of audits and record disagreements rather than forcing one answer.

Rubrics also move. Shankar and colleagues found that evaluation criteria cannot be fully set before people grade outputs, because grading changes what reviewers think matters. Reserve part of the budget for rubric revision, and when the rubric changes, treat the grader as uncalibrated. Show every error estimate with its interval and the assumption it depends on.

Reviewed business outcomes as grading ground truth

The hardest input to the worksheet is a trustworthy expert verdict on what done looks like. Businesses already hold that evidence in operating records: the claim that was paid, the refund that posted, the ticket that stayed closed, often reviewed by a supervisor.

SourceX helps companies license records they choose, and outcome-labeled histories such as paid claims or posted refunds are among the things an evaluation team can ask it to find. For calibration and audit sets, the request states scope, volume, language, format, timing and licensing needs, and SourceX sources the data from U.S. businesses for that request.

Frequently asked questions

How much of a grader's output should experts review?

There is no single rate. Give every stratum a small random audit floor, then add review where a wrong verdict is costly, the grader's calibration is weak, or scores feed training. New task types deserve heavy review until the grader is recalibrated.

What should the risk-based sampling and escalation worksheet include?

Per stratum: volume, severity tier, grader type, calibration agreement with its interval, optimization pressure, novelty and reviewer capacity, followed by an audit rate, escalation triggers and the allocation rule that produced them. Keep audit and escalation logs separate.

Can escalated cases be used to estimate grader accuracy?

No. Escalated cases are selected because they look risky or suspicious, so their error rate overstates the grader's overall error. Use the random audit sample for estimates and the escalated cases for finding failure types.

What shortcuts let an agent pass a grader without doing the real work?

Common ones are claiming an action the system of record does not show, editing tests, padding answers for a language-model judge, and doing nothing when the end state already matches. Probe each on purpose with planted cases and do-nothing runs.

Should review increase during reinforcement learning training?

Usually yes. As an agent is optimized harder against a grader, it is more likely to find the grader's blind spots, so audit rates and the sudden-jump trigger matter more in later checkpoints than early ones.

Sources

  • Amodei and colleagues describe reward hacking and scalable oversight, where the true objective is too expensive to evaluate often. Source
  • Gao and colleagues study how a gold reward score changes as a policy is optimized against a proxy reward model. Source
  • Zheng and colleagues examine position, verbosity and self-enhancement biases in LLM judges. Source
  • FastChat's MT-bench materials report over 80 percent agreement between humans and a GPT-4 judge, the same level as agreement between humans. Source
  • Prediction-powered inference combines a small gold-labeled set with machine-learning predictions to produce valid confidence intervals. Source
  • The ppi_py package includes a power analysis example for choosing labeled and unlabeled dataset sizes under a budget constraint. Source
  • The NIST handbook defines AQL, LTPD and the producer's and consumer's risks in lot acceptance sampling. Source
  • Shankar and colleagues find that evaluation criteria cannot be fully determined before people grade outputs. Source
  • tau2-bench splits action match rates by read and write tools to spot database passes with no write attempted, and marks LLM-judged assertions as experimental. Source
  • SWE-bench grades a generated patch by running the repository's tests and counts an instance resolved when the patch made the required tests pass. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies