Skip to content

AI uses for records

Partial credit agent evaluation: when the rubric pays for skipping the hard part

By SourceX Editorial · Updated

10 sections

Short answer

To detect partial-credit rules that reward unfinished work, list every criterion with its evidence and prerequisites, mark the steps that define success, and compute the highest score an attempt can earn while failing each one. If that ceiling reaches your pass mark or rivals a complete but imperfect attempt, the rule pays for skipping the hard part. Confirm with deliberately incomplete counterexample attempts, then gate or reweight.

Evaluation split

Partial credit agent

This page covers partial credit agent evaluation: when the rubric pays for skipping the hard part.Illustrative

Key takeaways

  • Partial credit is useful for learning signal and diagnosis, but a flat weighted checklist lets an agent bank easy criteria and skip the one that matters.
  • The core test is a ceiling: the best score achievable while a critical criterion fails. Compare it with the pass mark.
  • Criteria that check artifacts which exist whether or not the work was done, such as a status field or a drafted message, are where shortcuts hide.
  • Run counterexample attempts through the real grader; a matrix on paper can miss how the grader actually reads evidence.
  • Always report strict completion beside the partial score so a rise in one cannot hide a fall in the other.

How do you detect partial-credit rules that reward unfinished work?

Work out the highest score an attempt can earn while leaving each critical requirement unfinished, and compare it with the score you treat as success. If the two are close, the rule rewards unfinished work. A criterion dependency matrix does the arithmetic, and a small set of deliberately incomplete attempts confirms it against the real grader.

The failure has a long history under other names. Amodei and colleagues described reward hacking as an objective with a clever easy solution that formally maximizes it while undermining the designer's intent, and DeepMind defines specification gaming as behavior that satisfies the literal specification of an objective without achieving the intended outcome. A partial-credit rubric is a specification. If it can be satisfied mostly by the easy parts, an agent under optimization pressure will find that out, and an evaluation will report progress that is not there.

The test does not require giving up partial credit. Partial scores help diagnose where agents fail and give training a denser signal than a single pass or fail. The aim is to make sure partial credit is earned for real progress toward the requirement, not for activity around it.

Why does a weighted checklist pay for skipping the hard step?

Because a weighted sum treats criteria as independent, while business tasks are chains in which later steps only have value if an earlier, harder one succeeded. Points for the easy links stay on the board even when the critical link fails.

Tree-shaped rubrics show the pattern clearly. PaperBench grades leaf requirements as pass or fail and scores each parent node as the weighted average of its children. Its documentation also describes a Code-Dev variant that checks only whether code was written for each requirement and skips checking that it runs or reproduces the paper's results, and the authors call that variant convenient and cheaper but less rigorous. That is the trade made explicit: grading what is easy to inspect gives credit for work whose hardest part may be missing.

Agent tasks drawn from business processes have the same shape. Creating a vendor record is easy; confirming the bank details belong to the vendor is the hard part. Drafting a reply is easy; resolving the customer's issue is the hard part. If each earns a share of the points independently, the arithmetic favors doing the easy parts well and the hard part not at all.

When partial credit earns its place in an agent benchmark

Keep partial credit when it measures real progress toward the requirement and drop or gate it when it only measures activity around it. The distinction is visible in the matrix: progress criteria sit on the path to the critical step, activity criteria can pass whether or not that path is followed.

Partial scores are worth having for three jobs. They show where in a task agents fail, which helps engineers decide what to fix. They give reinforcement learning a denser signal than a single result at the end of a long episode. And they separate a near miss from a complete failure when comparing two agents that both fail the strict test.

Each job needs a different degree of protection. Diagnosis can tolerate ungated criteria, since nobody acts on the total. Training needs gating, because an optimizer will find any ceiling that sits above its current score. Published comparisons need both gating and a strict completion rate beside the partial score.

The criterion dependency matrix: columns and how to fill them

Build one row per criterion and fill six columns. The last column is computed, not judged, and it is the one that answers the question. Fill the evidence column with the record the grader actually reads, not the criterion's intent.

The ceiling calculation follows directly from the prerequisite column. Run it for each critical row in turn, using the four steps below, and keep the arithmetic in the worksheet so a reviewer can check it.

Checklist · 7 items

  1. Step 1: for each critical criterion, assume it fails.
  2. Step 2: remove every criterion that lists it as a prerequisite, directly or through another criterion, because those cannot have business value without it.
  3. Step 3: add up the weights of the criteria that remain and can still pass on their own evidence; that sum is the ceiling without it.
  4. Step 4: if a gate or cap applies, apply it last, so the ceiling reflects the rule as the grader implements it.
  5. Decision rule: if the ceiling without any critical criterion is at or above the pass mark, the rule rewards unfinished work and must change.
  6. Warning rule: if that ceiling is above the score a complete but imperfect attempt typically earns, the ranking of attempts is distorted even when the pass mark holds.
  7. Evidence rule: any criterion whose evidence can be produced without doing the work, such as a status value, a message claiming completion or a file that exists but is not checked, needs stronger evidence or a prerequisite.

0 of 7 ticked. Tick each item as you go.

The criterion dependency matrix: columns and how to fill them
ColumnWhat to enterWhy it matters
CriterionOne checkable requirement, phrased as a verifiable statement about the final state or recordVague criteria get graded on appearance
Evidence the grader readsThe field, file, log line or record the grader inspects to decide pass or failShows whether the evidence can exist without the work
PrerequisitesCriteria that must be true for this one to have business valueReveals points earned on top of a failed foundation
WeightPoints or share of total scoreShows how much the easy criteria are worth together
Critical flagYes if failing it means the business requirement is not met, whatever else passesDefines what counts as unfinished work
Ceiling without itFor critical rows: the maximum score achievable when this criterion fails, counting only criteria that can still passCompared with the pass mark, this is the decision

Which shortcut could raise the score without completing the business requirement?

Five shortcuts account for most unearned partial credit. Write one counterexample attempt for each, run it through the real grader, and record the score. Each attempt is constructed by hand or by editing a real trajectory, and each one leaves a critical requirement unmet.

Build the counterexamples from real material where possible. Take a successful trajectory and edit it: delete the critical step, replace the final action with a message, or point every action at a neighboring record. Edited real attempts look like what an agent would produce, while invented ones tend to be too obviously broken to test the grader's weak points.

Score every counterexample with the same grader configuration used for real runs, including the same judge model, prompt and evidence access. Store the attempts as a fixed regression set and rerun them whenever the rubric or grader changes, so a fix that reopens an old shortcut is caught before it reaches reported results.

Which shortcut could raise the score without completing the business requirement?
Counterexample attemptWhat it doesWhat a sound grader should give it
Skip the hard stepCompletes every criterion except the critical oneA score below the pass mark, ideally capped low
Claim completionWrites a confident final message or sets a done status without doing the workNo credit for criteria whose evidence is only the claim
Stop one step shortDoes all the work but omits the final commit, submission or sendLow credit if the business outcome depends on that step
Act on the wrong objectCompletes all steps on a similar but wrong recordNear zero, because no criterion is met for the right object
Break something elseCompletes the task but damages unrelated statePenalty or cap, because collateral damage fails the requirement

Illustrative scenario: vendor onboarding before and after gating

This scenario is illustrative. The task, criteria and percentages are invented to show the method and are not measurements. An agent onboards a new supplier in a procurement system. The business requirement is a usable vendor record whose payment details are verified against the supplier through an independent channel, because paying to bank details nobody confirmed with the supplier is the risk this control exists to prevent.

The original rubric has five equally weighted criteria worth 20 percent each: tax form collected, sanctions screen run, bank details verified by callback, vendor record created, requester notified. The pass mark is 70 percent. The bank verification is critical. The other four criteria can all pass when it fails, so the ceiling without it is 80 percent, above the pass mark. The skip-the-hard-step counterexample confirms it: the grader gives it 80 percent and a pass.

Two changes fix it. First, the vendor record and the notification are given bank verification as a prerequisite, so they earn nothing if it fails. Second, any attempt that fails a critical criterion is capped at 30 percent. Rerun, the skip-the-hard-step attempt scores 30 percent, the claim-completion attempt that sets a verified flag without a callback record scores 30 percent, and a complete attempt with a minor notification error scores 80 percent. The ranking now matches the business view of which attempts did the job.

The fix has a cost worth stating. With gating, a strong attempt that verifies the bank details but makes small errors elsewhere still scores well, while an attempt that does everything else perfectly scores poorly. That is the intended behavior, but it should be explained to anyone comparing results across rubric versions.

Report strict completion beside the partial score

Always publish a strict completion rate next to any partial score, so that progress on easy criteria cannot hide a flat or falling rate of finished tasks. Several established benchmarks already do some version of this.

TheAgentCompany defines its partial completion score as half the share of checkpoint points earned plus half a full-completion indicator, so an agent that earns 4 of 8 points without finishing scores 25 percent while a finished task scores 100 percent. Its task code also offers alternative strategies, such as awarding full points when the final checkpoint passes, which is a form of gating on the outcome. SWE-bench records a PARTIAL status when some target tests pass, but only a FULL status, with all target tests and all previously passing tests passing, counts as resolved. The tau-bench reference grader goes further and is all or nothing: the reward is zero if the final data does not match the gold state or a required output is missing.

There is a further lesson in the SWE-bench grading code. Its comments explain that a test runner which starts and immediately fails can leave an empty result map, and under one evaluation type that would score every target test as passing unless the harness requires evidence that the suite actually ran. A grader that does not check its own evidence is the purest form of unearned credit, and the counterexample set should include an attempt that breaks the grader's inputs.

Critical flags and pass marks: the judgments that swing the verdict

The decision is most sensitive to which criteria are marked critical and where the pass mark sits. Both are judgments, so show how the conclusion changes under the alternatives rather than presenting one score as fact.

Ask the domain adjudicator, not only the evaluation lead, to mark critical criteria, and record disagreements instead of averaging them. If two adjudicators disagree about whether a criterion is critical, compute the ceiling both ways. If the rule rewards unfinished work under either view, fix it; if only under one, report the score with both readings and the adjudicators' reasons.

Show uncertainty from the evidence as well. Where a criterion is graded by a model judge rather than a deterministic check, report how often the judge agrees with expert adjudication on a held-out sample, and treat the criterion as weaker evidence in the matrix until that agreement is known.

Signs the rubric needs a domain adjudicator before more scoring

Stop scoring and bring in a domain adjudicator when the matrix cannot be filled with confidence. That usually means the critical step is disputed, the evidence for it lives in a system the grader cannot read, or experts disagree about whether a counterexample attempt actually met the requirement.

  • The critical criterion's evidence is a free-text note or a model-written summary rather than a system record.
  • The business outcome arrives after the episode ends, such as a payment clearing or a ticket staying closed, and the grader has no access to it.
  • Counterexample scores change materially when the judge model or prompt is changed.
  • Two plausible pass marks lead to opposite conclusions about the same set of attempts.

Grounding criteria in business evidence

The strongest fix for unearned partial credit is evidence that only exists when the work was done: the callback log, the cleared payment, the closed ticket that stayed closed. Rubrics built on real business records can use that evidence, but only if the records that prove outcomes are selected with the task records, not added later.

SourceX sources licensed datasets of records from U.S. businesses for evaluation teams on request rather than from an existing catalog, so a request can name the outcome records and verification evidence a grader needs, along with scope, volume, format, timing and licensing requirements.

Frequently asked questions

What is partial credit in agent evaluation?

Partial credit gives an agent a share of the score for meeting some of a task's criteria, rather than scoring only full success or failure. It gives a richer signal for diagnosis and training. It becomes a problem when the criteria an agent can meet without finishing the task are worth enough to look like success.

What should the criterion dependency matrix and counterexamples include?

The matrix needs each criterion, the evidence the grader reads, its prerequisites, its weight, a critical flag and the computed ceiling score when each critical criterion fails. The counterexamples should cover skipping the hard step, claiming completion, stopping one step short, acting on the wrong object and breaking unrelated state.

How is partial credit related to reward hacking?

Reward hacking is when an agent maximizes a reward in a way the designer did not intend. A partial-credit rule that pays well for easy criteria creates exactly that opportunity, because an optimizing agent can raise its score by doing the easy parts and skipping the hard one.

Should agent evaluations use binary pass or fail instead?

Not necessarily. Binary scoring is harder to game but gives a sparse signal and hides how close attempts came. A common compromise is to gate partial credit on critical criteria and always report a strict completion rate alongside the partial score.

How often should the dependency test be rerun?

Rerun it whenever criteria, weights, the pass mark, the judge model or the task environment change, and whenever agents' partial scores rise without a matching rise in strict completion. That divergence is often the first sign of a new shortcut.

Sources

  • DeepMind defines specification gaming as behavior that satisfies the literal specification of an objective without achieving the intended outcome. Source
  • Amodei and colleagues describe reward hacking as an objective admitting an easy solution that formally maximizes it while undermining the designer's intent. Source
  • TheAgentCompany defines its partial completion score as half the share of checkpoint points plus half a full-completion indicator. Source
  • TheAgentCompany task scoring sums checkpoint points by default and offers strategies that award full points when the final checkpoint passes. Source
  • SWE-bench distinguishes FULL, PARTIAL and NO resolution, counts only FULL as resolved, and requires evidence that a test suite ran. Source
  • PaperBench scores a parent rubric node as the weighted average of its graded children. Source
  • PaperBench Code-Dev grades only code development requirements, skipping execution and result checks, and is described as less rigorous. Source
  • PaperBench rubrics decompose each task into a tree of sub-tasks with weighted scoring. Source
  • The tau-bench grader gives zero reward if the final data differs from the gold state or a required output is missing. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies