Definitions and comparisons
Surge expert rubrics versus company history: how they fit together
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Expert rubrics and company history do different jobs in an AI dataset. Rubrics, like the expert-written criteria Surge AI describes for its benchmarks, supply a precise supervision signal; licensed company records supply real tasks, real messiness and real outcomes. Combine them by grading real cases against expert criteria and using each recorded outcome to audit the rubric, and treat every provider description as a claim until evidence supports it.
Grader rubric
Surge expert rubrics
Key takeaways
- Rubrics supply the judgment of quality; company records supply realism and recorded outcomes.
- Surge's public pages describe expert-written rubrics and simulated work environments, which are the provider's claims as of October 18, 2026.
- A paper by the same team adds detail to a provider claim but not independent confirmation.
- Grade real cases against expert criteria written without sight of the outcome, then use recorded outcomes to find missing criteria.
- Each half of a combined dataset has its own rights chain, and both need written terms.
How can expert rubrics complement historical company records in an AI dataset?
Expert rubrics supply the supervision signal and company history supplies the realism, so the strongest datasets use both: real cases drawn from a company's records, each graded against criteria written by practitioners, with the case's recorded outcome kept as a separate check on the rubric itself. Neither route substitutes for the other, because a rubric says what a good answer must contain while a record shows what the work actually looked like and how it ended.
The primary query names Surge AI, so this page uses the company's own public pages, read on October 18, 2026, to describe that route, and treats everything they say as the provider's claim. It does not assess Surge's services, pricing or terms, and it compares two sourcing routes rather than two vendors.
The practical output is a supervision and realism comparison you can fill in for any dataset proposal, a claim ledger that separates what a provider says from what has been substantiated, and three ways to join rubric work to licensed records without double counting either.
What Surge says its expert-rubric benchmarks contain
As of October 18, 2026, Surge's blog and repositories describe several benchmarks built around expert-written criteria. The company says every prompt and evaluation rubric in AdvancedIF, an instruction-following benchmark, was written by human experts rather than generated by a language model. The AdvancedIF paper on arXiv makes the same statement, that prompts and rubrics were created manually by human experts. For ComplexConstraints, the company's post describes 75 expert-crafted prompts with 1,559 evaluation rubrics.
HANDBOOK.md is the closest of these to company records. According to the company's blog, it holds 65 agentic tasks across five enterprise domains: finance, medical billing, insurance, logistics and HR. The public repository README says the policies are written by experts adapting real industry guidelines, that the team created 10 base handbooks and that each task modifies its base into a distinct document by changing specific rules and thresholds. The README describes agents working through a cluttered inbox, a multi-channel Slack workspace, Jira queues and a stack of spreadsheets and PDFs.
Two details from the repository matter for this comparison. It contains a sample of three tasks, with the full set available on request, and the services in each task are mock services built into the environment. In the sample HR task, each entry in the rubric file pairs a plain-language criterion with verifier code that checks the resulting files, for example that specific rows exist in an employee roster spreadsheet. That is a realistic simulation of company work, designed and graded by experts, not a record of work a company performed.
The supervision and realism comparison
Use this table to compare any proposal that leans on one route. Each row is a property a buyer should be able to state for the dataset, and the last column names the evidence that settles it. A blank in that column is an unknown, and it should stay written down as one rather than be filled with a provider's description.
| Property | Expert rubric route | Company history route | Evidence that settles it |
|---|---|---|---|
| Where the task comes from | Written or adapted by experts, often adversarially, for the benchmark | Occurred in real operations, with real customers, constraints and dead ends | Task provenance field per item and a sample of raw items |
| What supplies the supervision signal | Atomic criteria per task, often checked by code or a model grader | Recorded outcome, later corrections, approvals and reopen history | Criterion list or outcome field definitions with example values |
| Realism of inputs | Simulated inboxes, tickets and files designed to look messy | Actual messiness, including missing fields and inconsistent notes | Field fill rates and a blind review of 20 to 50 items |
| Coverage of rare cases | Can be written on demand for cases that never happened | Limited to what happened, weighted toward common work | Distribution of case types against the target task mix |
| Repeatability | Another expert panel can write more of the same | Cannot be recreated; history happened once | Whether the supplier can extend the set and on what basis |
| Main failure mode | Criteria that reward the look of compliance, or reflect one author's view | Records that encode past bad practice or incomplete outcomes | Agreement studies, rubric audits and outcome audits |
| Rights questions | Who owns the prompts, criteria and environment code | Ownership, privacy permission and a license for AI use | Signed terms covering the specific use, not marketing copy |
| Main unknown to record | How criteria were validated and by how many experts | How outcomes were recorded and whether they were later corrected | Methods notes dated by the supplier |
Which provider claims are substantiated, and by what
A provider's own description of its data is a claim until something outside that description supports it. Most rubric benchmarks are described first in the provider's blog, then in a paper by the same team, so the paper adds detail but not independence. Separate the two when you write up a sourcing decision.
The method itself has outside support. OpenAI's HealthBench announcement describes 5,000 conversations, each with a conversation-specific rubric written by physicians, and the Rubrics as Rewards paper shows rubric criteria used as reward signals for reinforcement learning in domains without a single verifiable answer. Those sources substantiate that rubric supervision is an established technique; they say nothing about any one vendor's rubrics.
| Claim | Where it is stated | Status on October 18, 2026 | What would substantiate it further |
|---|---|---|---|
| AdvancedIF prompts and rubrics were written by human experts | Surge blog and the AdvancedIF arXiv paper | Provider claim, repeated in a paper the provider contributed to | Annotator guidelines, authorship logs or an outside audit |
| ComplexConstraints has 75 prompts and 1,559 rubrics | Surge blog | Provider claim | The released item files, counted |
| HANDBOOK.md policies adapt real industry guidelines | Repository README | Provider claim; the sample policies can be read | Side-by-side comparison with the guidelines named |
| Each rubric criterion is checked by code | Sample task rubric file in the repository | Observed in the three-task sample only | The same check across the full task set |
| Rubrics work as training rewards | Rubrics as Rewards paper; other published methods | Supported by independent research as a technique | A pilot on your own model and task mix |
Questions that make the two routes comparable before a pilot
Ask both kinds of supplier the same questions, in writing, and date the answers. The point is not to catch anyone out; it is to turn two different sales conversations into rows you can line up in the comparison table. Where a supplier declines to answer, record the question as unknown rather than inferring an answer from its marketing.
At a glance
Questions that make the two routes comparable before a pilot
Then run a small pilot before any volume commitment. Score a few hundred items from each route with the same model and the same grader, and look at where they disagree. Disagreement is the useful output: it shows which cases the rubric route over-rewards and which the records route under-labels.
- Provenance: for each item, was the task written, adapted from a guideline, or taken from an operating record, and can the supplier mark which?
- Authorship: how many people wrote or reviewed each criterion or outcome field, and with what domain experience?
- Validation: what agreement was measured between reviewers, on how many items, and what happened to items where they disagreed?
- Grading: is each criterion checked by a person, a model grader or code, and has the grader been compared with people on a sample?
- Outcome quality: for records, how was the outcome recorded, by whom, and how often was it later corrected or reversed?
- Coverage: what share of items falls in each case type you care about, and which types are missing entirely?
- Versioning: are criteria and records frozen at a dated version, and how are later fixes communicated?
- Rights: which written terms cover training, evaluation and redistribution of each part, and who signed them?
Three ways to join rubric work to licensed records
The two routes combine well when each does the job it is good at. These three patterns cover most of what labs try, and each has a specific check that stops it from quietly degrading.
- Records supply the cases, experts supply the criteria. Draw tasks from licensed history, such as resolved support threads or closed claims, then have practitioners write atomic criteria for each case. Check: experts write criteria without seeing the recorded resolution, so the rubric is not just a restatement of what one employee did.
- The recorded outcome audits the rubric. Keep the historical outcome, such as whether a ticket reopened or a claim was later adjusted, as a held-back field. Where a response that scores well on the rubric matches a path that failed in the record, the rubric is missing a criterion. Check: report the agreement between rubric verdicts and recorded outcomes as its own number.
- Records calibrate a simulated environment. Expert-built environments such as policy handbooks and mock inboxes can be compared with real procedures and inbox samples to find what the simulation leaves out, such as forwarded threads, stale attachments or approvals given in chat. Check: list the realism gaps found and whether each was added to the simulation or left as a known limit.
Illustrative: a support-escalation dataset built both ways
This case is invented to show the comparison in use; the counts are illustrative and describe no real company or vendor. An AI lab wants to train an agent that decides when a business-software support ticket should go to engineering. It considers an expert-written set of 300 scenarios with 10 to 20 criteria each, and a licensed sample of 2,000 resolved tickets from a help desk such as Zendesk, with reopen flags and engineering links.
Filled in, the comparison shows the expert set covers rare security escalations the ticket history barely contains, while the ticket history shows how often customers send partial logs, switch topics mid-thread and get escalated twice. Neither alone covers the target mix. The lab samples 400 tickets, has two support leads write criteria for each without seeing the outcome, and keeps the reopen flag held back.
On review, roughly 15% of tickets where a response scored well on the rubric had reopened in the record, mostly because the criteria never asked whether the customer's version number was confirmed. One criterion is added and the rubric is versioned. A second pass shows the opposite problem in a smaller group: tickets that closed without a reopen but where the support agent had promised a fix that engineering never shipped. The recorded outcome looked good and was wrong, so those tickets lose their outcome label and keep only the rubric verdict. The expert-written scenarios stay in the set for the rare security cases, labeled as written rather than observed, so evaluation can report results on real and written items separately.
Where the comparison stops helping
The pairing assumes a body of records exists, can lawfully be used and reflects good practice. When any of those fails, the comparison points to one route only. Tasks that have never happened, such as a new product's support flow or deliberately adversarial prompts, can only come from experts. Fields dominated by regulated personal data may be impractical to license at all, which is one reason expert-written scenarios exist.
The opposite exception also holds. Where the value lies in what people actually did over years, such as how a dispatch board was rebalanced or how estimates were revised after site visits, a rubric cannot supply it, because the experts writing criteria were not there. A simulated environment can reproduce the shape of a busy inbox, but not the particular habits, workarounds and customer histories that make one company's records different from another's.
A third limit is drift. Rubrics age as products, policies and regulations change, and so do records. A criterion written against last year's refund policy will mark today's correct answer as wrong. Date every criterion and every record extract, and plan a review cycle for both rather than assuming either stays valid.
Treat three signals as reasons to narrow scope before buying: the supplier cannot say how outcomes or criteria were validated, the records show a practice the lab would not want a model to learn, or the rights to either part of the dataset rest on assumptions nobody has put in writing. A research editor and a procurement reviewer with domain knowledge should read the comparison before it drives a contract.
The rights question differs for each half of a combined dataset
Combining routes means two separate rights chains. For the rubric half, the questions are who owns the prompts, criteria, environment code and expert output, and what the license lets a lab do with them. For the records half, holding system access, having privacy permission to process the records, owning them and having a contractual right to license them for AI are four separate questions, and de-identification alone does not create a right to license.
Document both halves the same way. Datasheets for Datasets proposes that every dataset ship with documentation of its motivation, composition, collection process and recommended uses, and a combined dataset should have one datasheet that names which items came from which route. Keep the route label on every item through training and evaluation, so results can be reported separately for written and observed cases and a later rights question about one half does not force a lab to discard the other. This is general information, not legal advice; counsel should review the specific agreements.
Requesting the company-history half
Labs that need the records side of this pairing can ask SourceX to find it. There is no shelf of ready-made history: SourceX looks for the records at U.S. businesses once a lab's request has described the scope, volume, language, format, timing and licensing it needs. The business holding those records signs an agreement setting the scope and terms before anything is handed over, so the records half arrives with a documented rights chain that the rubric work can be built on.
Frequently asked questions
What is an expert rubric in AI training data?
It is a list of specific criteria, written by people with domain experience, that a response must meet to count as good. Each criterion is usually atomic so a person, a model grader or code can check it. Rubrics are used both to evaluate models and as reward signals in training.
Are expert rubrics better than real company records?
Neither is better in general; they supply different things. Rubrics supply a precise judgment of quality, and records supply real tasks, real messiness and real outcomes. The strongest datasets grade real cases against expert criteria and use recorded outcomes to audit the criteria.
Does Surge AI use real company data in its benchmarks?
Its public pages, read on October 18, 2026, describe expert-written policies adapted from real industry guidelines and simulated environments with mock services, not records from a company's operations. For anything beyond those pages, ask the company directly and record the answer as its claim.
How do you check a provider's claim about how its rubrics were written?
Ask for annotator guidelines, the number of experts per item, agreement statistics and a sample of raw items, then count and read them yourself. A paper written by the same team adds detail but is not independent confirmation.
Can a historical outcome be used as a label on its own?
Sometimes, when the outcome is recorded consistently and reflects good work. Outcomes such as reopen flags or later adjustments are often better used to audit a rubric than as the only label, because past practice is not always the behavior a lab wants to teach.
Sources
- Surge's HANDBOOK.md post describes 65 agentic tasks across finance, medical billing, insurance, logistics and HR, graded by deterministic criteria. Source
- The HANDBOOK.md repository says its policies are written by experts adapting real industry guidelines from 10 base handbooks, runs agents against mock inbox, Slack, Jira and file services, and publishes a sample of three tasks whose rubric entries pair criteria with verifier code. Source
- Surge says every prompt and evaluation rubric in AdvancedIF was written by human experts rather than generated by a language model. Source
- Surge's ComplexConstraints post describes 75 expert-crafted prompts with 1,559 evaluation rubrics. Source
- The AdvancedIF paper states that its prompts and rubrics were created manually by human experts. Source
- Rubrics as Rewards uses rubric criteria as reward signals for reinforcement learning in domains without a single verifiable answer. Source
- OpenAI's HealthBench pairs 5,000 conversations with conversation-specific rubrics written by physicians. Source
- Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process and recommended uses. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.