Skip to content

AI uses for records

Healthcare administrative task benchmark: an agent test built from approved cases

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Approved nonclinical records support an administrative agent test when each closed case becomes a task with a de-identified starting state, an instruction, allowed tools and the correct end state from how it was actually resolved under the payer rules then in force. Score end states over repeated trials, exclude cases needing clinical judgment, and require privacy and domain sign-off on a written brief first.

Evaluation split

Healthcare administrative

This page covers healthcare administrative task benchmark: an agent test built from approved cases.Illustrative

Key takeaways

  • An administrative test checks gathering, routing, submitting and tracking; the agent never decides care or medical necessity.
  • Write and sign the benchmark brief before pulling records, so fields are approved rather than inspected after export.
  • Grade the end state of the records against a gold state, and run each task several times to report pass^k.
  • Pin each task to a payer and policy version, and retire tasks when the rules change.
  • Exclude clinical-judgment cases, identity errors and rare paths; test rare patterns with labeled constructed tasks.

How can approved nonclinical records support a healthcare administrative agent test?

Closed administrative cases become test tasks when each one is turned into four parts: a de-identified starting state, an instruction, the tools the agent may use, and the correct end state taken from how the case was actually resolved under the rules in force at the time. The agent is scored on whether it reaches that end state, not on how its messages read, and every task carries a privacy sign-off and a domain sign-off before it enters the test.

Real cases matter because administrative work is full of local detail that invented tasks miss: a payer that wants a specific attachment, a scheduling template that blocks double booking, a denial reason that is usually fixed by a corrected modifier rather than an appeal. A test built from your own resolved cases measures whether an agent can do your work, which a public benchmark cannot tell you.

This page gives a benchmark brief that a team can fill in before building anything, a task family table, scoring rules, an illustrative task and the cases that should never become tasks. It is a method guide. It reports no benchmark results and makes no claim about how any agent performs.

What separates an administrative agent test from a clinical benchmark?

The boundary is the decision the agent is allowed to make. In an administrative test the agent gathers, checks, routes, submits and tracks; it never decides what care a patient should receive or whether care is medically necessary. Tasks whose correct answer depends on clinical judgment belong to a different kind of evaluation with clinical reviewers.

Published work shows both sides. MedAgentBench, from Stanford researchers, places agents in a virtual EHR environment with tasks written by physicians, which makes it a clinical benchmark. HealthAdminBench targets administration instead: its authors describe four GUI environments, an EHR, two payer portals and a fax system, and 135 expert-defined tasks across prior authorization, appeals and denials management, and durable medical equipment orders, each broken into verifiable subtasks.

Neither replaces a private test. Public tasks are built on simulated environments and someone else's workflows, and once published they can leak into training data. A private administrative test built from approved cases complements them by asking whether an agent can follow your payers, your templates and your escalation rules.

Which administrative task families make good test items?

Pick task families where the correct end state is recorded in a system and can be checked without a judgment call. If two experienced staff members would resolve the case differently and both be right, the task needs a rubric and a reviewer, which makes it slower and noisier to score; keep those for a second phase.

Sample cases on purpose rather than taking the most recent month. Stratify by payer category, outcome and difficulty so that the test is not dominated by the easiest, most common path, and write the sampling rule into the brief. A test made of routine eligibility checks will flatter any agent; the cases that cost staff time are the ones with missing documents, conflicting payer responses and second submissions.

Which administrative task families make good test items?
Task familyStarting stateCorrect end stateHow it is checked
Eligibility and benefits checkScheduled visit with coverage on fileCoverage status recorded and visit flagged or clearedField values compared with the resolved case
Reschedule under template rulesCancellation request and a provider templateNew slot that respects template and visit typeSlot validity check against template rules
Prior authorization packet completenessOrder and payer requirements listRequest marked complete or missing items listedRequired item list compared with payer checklist version
Denial triageRemittance with adjustment reason codesCorrect work queue and next actionQueue and action compared with the resolved case
Corrected claim versus appealDenied claim and denial reasonChoice of corrected claim, appeal or write-offChoice compared; harmful choices scored as hard fails
Statement dispute responsePatient billing question and account historyAccount note and response categoryCategory and account changes compared
Referral routingIncoming referral with administrative fieldsRouted to the right queue with missing items requestedQueue and request list compared
Claim status follow-upAged claim and payer status responseStatus recorded and follow-up date setStatus code and follow-up interval rule

What should the benchmark brief requiring privacy and domain sign-off include?

Ten sections, each with a named owner and a release rule. The brief is written before any record is pulled, so that the privacy reviewer approves the fields the test needs rather than inspecting whatever an engineer happened to export. A test whose brief has an unsigned section does not run against any vendor or model.

At a glance

What should the benchmark brief requiring privacy and domain sign-off include

Two sign-offs are mandatory. The privacy reviewer confirms that source permissions, field necessity and de-identification are adequate for the population. The domain reviewer, an experienced administrative lead for the workflow, confirms that each gold end state is right under the rules in force and that no task requires clinical judgment.

Treat the finished test as confidential material. Limit who can read the tasks and gold answers, never paste them into a vendor's tools during a trial, and run vendor evaluations in an environment you control where possible. A private test that leaks into a vendor's training or prompt library stops measuring anything, and it may also carry obligations from the source agreements.

What should the benchmark brief requiring privacy and domain sign-off include?
Brief sectionWhat to writeSigns offBlocks release if missing
Purpose and decisionWhich decision the results will inform, such as piloting an agent on denial triageBusiness ownerYes
Scope and nonclinical boundaryTask families included, decisions the agent may not makeDomain reviewerYes
Source records and permissionsSystems, date range, and the agreements or policies that permit this usePrivacy reviewer and counselYes
Field necessityEach field the starting state needs and why; fields deliberately excludedPrivacy reviewerYes
De-identification methodSafe Harbor or Expert Determination, token policy, date handlingPrivacy reviewerYes
Payer and policy versionWhich payer rules, checklists and internal procedures each gold answer assumes, with datesDomain reviewerYes
Environment and toolsSystems or simulated tools the agent may call, and what each returnsTechnical ownerYes
Gold end state and graderHow the correct end state was derived and how it is comparedDomain reviewer and technical ownerYes
Reliability and trialsTrials per task, the reliability metric and how uncertainty is reportedTechnical ownerNo, but results are provisional
Holdout and change logWho may see tasks, refresh rule, and every change since the last versionTechnical ownerNo, but results cannot be compared across versions

How should the grader score an administrative task?

Compare the end state of the records with the gold end state, and treat the conversation as secondary. The tau-bench benchmark for tool-using agents takes this approach: its authors describe comparing the database state at the end of a conversation with an annotated goal state, and the public code replays the gold actions on a fresh copy of the data and checks whether the two states match, then checks that any required outputs appear in the agent's replies.

Run each task more than once. The same tau-bench paper introduces pass^k, the probability that an agent succeeds on all of k independent trials of a task, because a workflow agent that solves a task on one run and fails it on the next is not reliable enough to deploy. Report the number of trials and the metric in the brief before any run.

Add subtask credit for diagnosis, not for the headline. HealthAdminBench breaks each task into verifiable subtasks, which shows where an agent fails; a team can do the same, but the deployment decision should rest on whole-task success and on hard fails. Define hard fails in advance: submitting to the wrong payer, writing off a balance that should have been appealed, changing an account outside the task, or exposing data to a channel the task did not authorize.

Report uncertainty with every number. A private test usually has tens or a few hundred tasks per family, not thousands, so a difference of a few tasks between two agents can be noise. Give the task count beside each rate, show an interval rather than a single figure where the team has the statistical support to compute one, and do not rank agents on differences the interval cannot separate.

Why does each task need a payer and policy version?

Because the correct answer moves when the rules move. A prior authorization task resolved under one payer checklist can have a different correct end state a year later, and a test that ignores this will mark a correct agent wrong. Record, for each task, which payer rules, internal procedures and code lists the gold answer assumed, and the date range they were in force.

Federal rules are part of that version. CMS says its Interoperability and Prior Authorization final rule requires impacted payers to send prior authorization decisions for medical items and services, excluding drugs, within 72 hours for expedited requests and seven calendar days for standard requests, starting January 1, 2026. A follow-up task built on a case from before that change may assume different waiting periods, so the brief should note which regime applies.

Set a refresh trigger. When a payer changes a checklist, a code list is updated, or an internal procedure changes, retire or re-label the affected tasks rather than letting them silently test an outdated rule.

Version the code lists too. Adjustment reason codes, remark codes and internal queue names change over time, and a gold answer that routes on a retired code is no longer a fair test. Store the list version with each task so a later reviewer can tell whether a failure reflects the agent or the reference data.

Illustrative task: triaging a denied claim

The task below is illustrative and invented for this page; it does not come from any real organization, case or patient. It shows the level of detail a task record needs to be gradable.

  • Starting state (illustrative): a de-identified case with a case token, payer category commercial, service category outpatient follow-up, a claim submitted at offset 0 and a remittance at offset 21 days carrying a denial with an adjustment group code and reason code, plus the practice's denial work queues.
  • Instruction (illustrative): review the remittance for this case and route it to the correct work queue with the next action recorded.
  • Tools (illustrative): read remittance detail, read claim lines without diagnosis fields, read the payer checklist version named in the brief, write work queue and next action.
  • Gold end state (illustrative): case in the coding correction queue with next action corrected claim, derived from how the real case was resolved and confirmed by the domain reviewer.
  • Hard fails (illustrative): routing to write-off, filing an appeal, or reading any field outside the listed tools.
  • Trials and metric (illustrative): five trials per task, pass^k reported at k equals 1 and k equals 5, with the count of hard fails listed separately.
  • Sign-offs (illustrative): privacy reviewer approved the field list and token policy; domain reviewer approved the gold end state under the stated checklist version.

Which real cases should never become tasks?

Exclude cases whose correct resolution depended on clinical judgment, cases where the problem was the patient's identity (a wrong merge or a misdirected statement), and cases so rare that the combination of payer, service and timing could single someone out. HHS guidance asks for no actual knowledge that remaining information could identify a person under Safe Harbor, and rare administrative paths are where that knowledge tends to exist.

Rare but important situations can still be tested. A domain reviewer can write a constructed task that reproduces the administrative pattern without copying the case, labeled as constructed in the task record. Keep constructed tasks as a separate slice so results on real cases are never mixed with results on invented ones.

Keep the starting state lean. HHS describes the minimum necessary standard as reasonable efforts to limit protected health information to what the intended purpose needs; for a test the purpose is the administrative decision, so the agent should see the fields the task requires and nothing else. A starting state that includes the full chart because it was easier to export fails the field necessity section of the brief.

What can results from this test claim, and when should the team stop?

Results can claim how an agent performed on these task families, under these policy versions, in this environment, over this many trials. They cannot claim clinical safety, performance on other payers or workflows, or performance after the rules change. State those limits next to any number the team reports.

Stop and escalate when the brief cannot be completed honestly: when the source agreements are unclear, when a task family keeps needing clinical judgment to grade, when the domain reviewer and the recorded outcome disagree on the gold end state for more than a few tasks, or when the test would need exact dates or free text to work. Each of those means the test design, not the agent, needs work first.

Sourcing approved cases for an administrative test

An organization that resolves administrative cases every day may hold more closed cases than one test needs. The benchmark brief above doubles as a scope document for licensing such records, because it already names the fields, the permissions and the sign-offs.

With SourceX, the organization is assessed briefly first, and no case records are uploaded for that. Records move only after the organization has approved the scope and terms and signed, and the de-identification that follows is done by SourceX through approved processing routes, as the agreement provides.

Frequently asked questions

What is a healthcare administrative task benchmark?

It is a set of administrative tasks, such as eligibility checks, denial triage or referral routing, each with a starting state, tools and a correct end state, used to test whether an AI agent can do the work. It excludes decisions about care. Scoring compares the end state the agent produces with the gold end state.

Can de-identified real cases be used to test an agent?

Yes, when the source agreements permit the use, the fields are limited to what the task needs, and a privacy reviewer approves the de-identification. A domain reviewer should confirm each gold answer under the rules in force at the time.

How many times should each task be run?

More than once. The tau-bench authors introduced pass^k, the chance an agent succeeds on all k trials, because single runs overstate reliability. State the number of trials and the metric in the brief before running anything.

How is this different from HealthAdminBench or MedAgentBench?

MedAgentBench tests clinical tasks in a virtual EHR, and HealthAdminBench tests administrative tasks in simulated EHR, payer portal and fax environments. A private test built from your own approved cases measures performance on your payers, templates and procedures, and stays unpublished.

Who should sign off on the test?

At minimum a qualified privacy reviewer, for permissions, fields and de-identification, and an experienced administrative lead, for scope and gold answers. Counsel reviews the source agreements, and a technical owner signs the grader and environment.

Sources

  • The HealthAdminBench authors describe four GUI environments (an EHR, two payer portals and a fax system) and 135 expert-defined tasks across prior authorization, appeals and denials, and equipment orders, each decomposed into verifiable subtasks. Source
  • MedAgentBench is a virtual EHR environment with physician-written tasks for benchmarking medical LLM agents. Source
  • The tau-bench paper evaluates agents by comparing the database state at the end of a conversation with an annotated goal state and introduces the pass^k reliability metric. Source
  • The tau-bench reward code replays the gold actions on fresh data, compares the resulting state with the agent's state, and checks that required outputs appear in the agent's responses. Source
  • CMS says the Interoperability and Prior Authorization final rule requires impacted payers to send prior authorization decisions for medical items and services, excluding drugs, within 72 hours for expedited and seven calendar days for standard requests, beginning January 1, 2026. Source
  • HHS de-identification guidance describes the Safe Harbor method, including the requirement of no actual knowledge that remaining information could identify an individual. Source
  • HHS describes the minimum necessary standard as reasonable efforts to limit PHI to the minimum necessary to accomplish the intended purpose. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies