AI uses for records
Healthcare referral routing dataset: an AI test built on a misrouted referral
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Build the test as a packet: freeze the referral at the moment it reached the wrong work queue, show only administrative evidence such as service code, specialty, location, network status and attachment flags, include the routing rule in force, and hide the correct decision and the linked outcome for scoring. Proof comes from acceptance, a linked appointment or closure, not a rerouted status, and every packet needs a counterexample twin.
License terms
Healthcare referral routing
Key takeaways
- A misrouted referral makes a strong administrative test when it is frozen at the moment it reached the wrong queue.
- Route on service, specialty, location, network status, urgency category and attachment flags; clinical narrative stays out.
- A queue change is not proof; acceptance, a linked appointment or a closure back to the referrer is.
- Pair every packet with a near twin that was correctly placed, so moving everything unusual does not score well.
- When routing depends on clinical content, the correct answer is escalation to triage, and the packet should score it that way.
How do you build an administrative AI test around a misrouted referral?
Build it as a packet with five parts: the frozen starting state when the referral entered the wrong work queue, the administrative evidence a scheduler could see, the routing rule in force that day, the correct decision, and a linked record that shows what actually happened next. The model reads the starting state and evidence, proposes an action, and is scored against the decision and the observed outcome.
The test is administrative by design. It asks whether an agent can tell that a referral sits with the wrong team, name the right team or the missing item, and move it without clinical judgment. It does not ask whether the referral was clinically appropriate, how urgent the condition is, or what care the patient needs; those questions belong to clinicians.
A healthcare referral routing dataset built this way is small and dense. Each packet takes real effort to assemble, but one well-linked case teaches more than thousands of referrals that only show a final queue name.
Where is the nonclinical line for a routing task?
Keep the fields a scheduler or referral coordinator uses to route work, and leave out clinical narrative. Routing usually depends on the requested service, specialty, location, payer network status, required attachments and the urgency category the referring office selected, not on the reasons behind them.
That boundary also matches how HHS describes the minimum necessary standard: for routine and recurring disclosures, a covered entity's policies may use standard protocols that limit the information to the minimum necessary for that type of disclosure, and when an entire medical record is needed the policies must say so explicitly with a justification. A routing test never needs the entire record. The table is a starting point for the field review a privacy officer will do.
| Field group | Keep, transform or exclude | Reason | Typical source |
|---|---|---|---|
| Requested service code and specialty | Keep | Primary routing input | Referral order or intake form |
| Receiving location and queue | Keep, pseudonymize queue names if they reveal staff | Defines the routing target | Work queue configuration |
| Payer and plan network status | Keep as category (in network, out of network, unknown) | Often decides which location can accept | Eligibility check result |
| Urgency category chosen by the referrer | Keep as stated, never re-derived | Affects queue priority, not clinical assessment | Referral form |
| Attachment checklist (present or missing) | Keep as flags | Missing items cause send-backs | Document index |
| Diagnosis narrative, notes, results | Exclude | Clinical content outside the task | Clinical documents |
| Patient name, contact details, exact dates | Exclude or transform | Identifiers; dates become intervals | Registration and audit logs |
What starting state should the packet freeze?
Freeze the referral as it stood at the moment it reached the wrong queue: the queue name, who or what assigned it, the status, the routing rule version, and the work-queue history up to that point. Anything that happened later is hidden from the model and kept for the answer key.
The HL7 FHIR Task resource is a useful vocabulary even when the source system is not FHIR based. Its specification describes a Task as tracking the state of completion of an activity, with an owner defined as the party responsible for managing task execution and a business status for nuances of the business state. It separates focus, the request being fulfilled, from basedOn, a higher-level authorization. In a routing packet, focus is the referral, owner is the queue, and business status carries labels such as awaiting records or pending eligibility.
Capture who made the first assignment as well. An interface rule, an intake clerk and a self-scheduling portal can fail in different ways: an interface rule may fail consistently on one code range, staff on unfamiliar services, and a portal when patients pick the closest-sounding option. A test set that mixes all three without a label cannot tell an AI team which failure it is learning to fix.
Record the routing rule as text plus a version or effective date. Rules change when a clinic opens, a provider leaves or a payer contract ends, and a packet that pairs an old referral with today's rule produces a false error.
The administrative routing and correction task packet
Each packet below is self-contained and labeled real or illustrative. Field names are suggestions to map onto your own referral management or EHR work queue export.
| Packet part | Contents | Visible to the model | Acceptance check |
|---|---|---|---|
| Header | Packet ID, real or illustrative label, source system, routing rule version, review sign-off | Rule version only | Privacy and administrative reviewers signed |
| Starting state | Current queue, assigning actor (rule, staff role, interface), status, time since receipt as an interval | Yes | Matches the work-queue history at that moment |
| Administrative evidence | Service code, specialty, location requested, network category, urgency category, attachment flags | Yes | Every field passes the boundary table |
| Applicable rule | Routing rule text that decides the correct queue or the send-back reason | Yes, as policy text | Rule was in force on the referral date |
| Expected decision | Reroute to named queue, return for missing item, or escalate to clinical triage | No | Agreed by two administrative reviewers |
| Outcome link | Later work-queue status, acceptance or decline, appointment status or closure | No | Observed in the record, not inferred |
| Counterexample pair | A similar packet where the starting queue was correct | Yes, scored separately | Differs from the main packet by one rule-relevant field |
What proves the reroute worked, beyond a changed queue name?
The proof is a later record showing that the receiving team accepted the referral and acted on it, not a status field that says rerouted. A queue change proves only that someone moved the referral; it does not prove the new queue was right.
Several records can serve, in rising order of strength. The FHIR Task status set includes rejected, where a potential performer decides not to execute the task before any action, and completed. The FHIR Appointment resource has a basedOn element for the request an appointment is allocated to assess, such as an incoming referral, which is the cleanest link from referral to booking. The ONC 360X closed-loop referral project requires the referral recipient to send either an accept or a decline, and its referral outcome concept covers care that was not rendered or ended early as well as completed results.
| Evidence | What it proves | What it does not prove | Use in the packet |
|---|---|---|---|
| Queue change in the work-queue log | Someone moved the referral | That the destination was correct | Context only |
| Acceptance by the receiving team | The destination took ownership | That the patient was seen | Minimum outcome link |
| Decline or send-back with reason | The destination refused and why | Where it should have gone | Negative outcome; keep the reason code |
| Appointment linked to the referral | Booking happened against this request | That the visit occurred | Strong outcome link |
| Appointment completed or closure sent to referrer | The loop closed administratively | Anything clinical about the visit | Strongest administrative outcome |
Worked case: an orthopedic referral lands in the general surgery queue
This case is illustrative and invented to show the packet; it does not describe a real provider or patient. A primary care office sends an electronic referral for a knee evaluation. An interface rule maps the service code to general surgery because an older code table grouped it there.
| Packet part | Recorded content | Expected model output | Scoring note |
|---|---|---|---|
| Starting state | General surgery intake queue, assigned by interface rule, status received, interval since receipt two business days | Detect that the service code points to orthopedics under the current rule | Rule text is visible; code table version is in the header |
| Evidence | Service code for joint evaluation, specialty orthopedics, in-network category, routine urgency, imaging report flagged present | Use only these fields | Any request for clinical notes is overreach |
| Rule | Joint evaluation codes route to the orthopedics referral queue at the requested location | Cite the rule | Citation required for full credit |
| Expected decision | Reroute to orthopedics intake; no send-back needed | Reroute with a one-line reason | Wrong queue or added send-back fails |
| Outcome link | Orthopedics accepted the next business day; appointment booked against the referral | Hidden from the model | Confirms the answer key |
The counterexample: when the receiving queue was right after all
Every main packet needs a near twin where the referral looked misrouted but was not. Without it, a model can score well by moving everything that looks unusual.
In the illustrative twin, the same knee referral sits in a different location's orthopedics queue than the one requested. It looks wrong, but the patient's plan has no network agreement at the requested site, and the routing rule sends out-of-network requests to the nearest in-network location with a notice to the referrer. The correct action is to leave it where it is and confirm the notice went out.
A second kind of counterexample marks the edge of the task. If the correct queue depends on reading clinical notes, for example choosing between two subspecialty clinics by symptom, the expected decision is escalation to clinical triage. The model earns credit for recognizing that the routing question is clinical and handing it on, and loses credit for answering it.
- Twin packets differ from the main packet by exactly one rule-relevant field.
- Escalation packets name the point where administrative evidence runs out.
- Report scores on main packets, twins and escalation packets separately.
How should a routing agent be scored on these packets?
Score the action, the reason and the restraint. An agent that reroutes correctly but cites the wrong rule will fail on the next rule change, and one that pulls clinical detail into an administrative decision fails the boundary even when its queue choice is right.
Run the packets in a fixed order with no memory between them, and freeze the rule text the model sees so a later edit to a rule cannot change an old score. When a rule does change, add new packets under the new version rather than editing old ones; the history of both versions is itself useful test material.
Keep the scoring at packet level and publish counts alongside any rate, because packet sets are small and a few cases move a percentage a long way.
| Measure | Pass condition | Fails when | Applies to |
|---|---|---|---|
| Detection | Flags a misrouted packet as needing action | Leaves a misrouted referral in place | Main packets |
| Restraint | Leaves a correctly placed twin alone | Moves a twin that was right | Twin packets |
| Destination | Names the queue in the answer key | Picks a different queue or invents one | Main packets |
| Rule citation | Quotes or names the applicable rule | Gives no rule or the wrong one | All packets |
| Boundary | Uses only visible administrative fields | Requests or infers clinical detail | All packets |
| Escalation | Hands clinical routing questions to triage | Answers a clinical routing question itself | Escalation packets |
What has to be settled before packets leave the provider?
Three questions come first: whether the organization may use these records for building a test at all, which fields the task needs, and how the released packets are de-identified. A privacy officer and counsel answer the first; the field table helps with the second; the third needs a documented method.
HHS guidance describes two de-identification methods under the HIPAA Privacy Rule. Under Safe Harbor, all elements of dates other than the year that relate to an individual are removed, which is why the packet uses intervals such as business days since receipt instead of timestamps. Expert Determination lets a qualified expert assess risk for a specific dataset and recipient. Free-text routing notes are the usual leak, because staff write names and clinical hints into them; review or remove them before any packet leaves.
Stop, narrow the scope or ask for specialist review when routing depends on behavioral health or substance use program status, when the referral involves a minor or a sensitive service, when the only proof of outcome sits in clinical notes, or when the rule version cannot be reconstructed. Those packets are better left out than half built.
Turning routing histories into a licensed dataset
The value of referral routing histories with work-queue logs, rule versions and linked outcomes sits in the corrections, the send-backs and the declines, not in the clean referrals.
Rights review, contracting, preparation, delivery and payment are handled by SourceX when a provider or vendor licenses selected routing records to AI developers. Records move only once the business has approved the scope and terms and signed, and a short company assessment with no upload shows whether an organization qualifies.
Frequently asked questions
What is a healthcare referral routing dataset?
It is a set of administrative records showing how referrals were assigned to work queues, moved, accepted, declined or sent back, linked to the rule in force and the eventual administrative outcome. Used as a test, it checks whether an agent can route referrals and correct mistakes without clinical judgment.
What can a routing packet show the agent without crossing into clinical detail?
Service code, specialty, location, network category, the referrer's urgency category and attachment flags are usually enough to route. Diagnosis narrative, clinical notes and results stay out; if routing truly depends on them, the packet becomes an escalation case for clinical triage.
Why not use the final queue name as the label?
The final queue only shows where a referral ended up, which may itself be wrong or the result of several moves. Acceptance by the receiving team, a linked appointment or a closure back to the referrer is the evidence that the routing worked.
Can the test use illustrative packets instead of real ones?
Yes, as long as each one is labeled illustrative and built from the organization's real rules and field structure. Illustrative packets are useful for edge cases that are rare in the record, but scores on them should be reported apart from scores on real packets.
Does a routing test need HIPAA review if it excludes clinical notes?
Routing records still describe a patient's care request, so the privacy officer and counsel should review the use and the de-identification method. Excluding clinical notes narrows the review; it does not remove it.
Sources
- HHS guidance on the minimum necessary standard: routine and recurring disclosures may follow standard protocols limited to the minimum necessary, and policies must explicitly justify use of an entire medical record. Source
- The HL7 FHIR Task resource describes an activity that can be performed and tracks the state of completion of that activity. Source
- In the FHIR Task definition, owner is the party responsible for managing task execution, businessStatus holds business-specific nuances, focus is the request being fulfilled and basedOn is a distinct higher-level authorization. Source
- The FHIR Task status code system defines rejected as a potential performer deciding not to execute the task before any action, and completed as the task having been completed. Source
- FHIR Appointment.basedOn is the request this appointment is allocated to assess, for example an incoming referral. Source
- In the ONC 360X closed-loop referral workflow, the referral recipient must transmit an accept or a decline, and referral outcome covers care not rendered or ended before completion. Source
- HHS de-identification guidance: under Safe Harbor, elements of dates other than year directly related to an individual are removed; Expert Determination is the alternative method. Source
See if your company qualifies
A short company assessment. No data uploads are needed.