AI uses for records
Grader reliability across model families: how to test for style bias
By SourceX Editorial · Updated
10 sections
Short answer
To test whether an AI judge favors one model's writing style, cross content with style: rewrite each model's answer in another family's style and in a neutral style, hide the source, and compare the judge's scores with verified task completion from business records. If grades follow the style rather than the work, the judge is unreliable across model families. What follows is an experimental design only; no findings are reported.
Grader rubric
Grader reliability
Key takeaways
- A judge comparison is only fair if the judge scores the work, so anchor every item to a completion check made from records, not to another model's opinion.
- Swap style while holding content fixed; that crossover is what separates a preference for a model family's writing from a preference for correct answers.
- Known judge biases include position, length and markdown formatting, and research links self-preference to self-recognition and low perplexity; control the first three before testing the last.
- Report the style effect and the judge's agreement with expert adjudication separately, with intervals clustered by task, and keep expert disagreement as data.
- If the judge fails, switch to swapped-order grading, a panel drawn from different model families, or human adjudication for the affected task families.
How do you test whether an AI judge favors one model's writing style?
Hold the substance of each answer fixed and change only its style, then see whether the judge's grade moves. The cleanest version takes answers from two or more model families on tasks whose correct outcome can be checked against business records, produces style-swapped and style-neutral rewrites of each answer, hides which model wrote what, and asks the judge to grade every version.
If the judge's grades track verified completion and barely move when style changes, it is behaving like a grader of work. If grades rise whenever an answer reads like the judge's own family, or like whichever family writes longer and more formatted answers, the judge is rewarding style, and comparisons it produces across model families are suspect.
This page sets out that experiment as a protocol: the design, the blinding, the shortcut probes, the analysis plan and a pre-registration checklist. It reports no results, and every count it mentions is illustrative.
Which judge biases are documented, and which are still open?
Position, length and formatting effects are well documented; self-preference is real in some settings but its cause is still argued. The experiment has to control the first group so it can measure the second.
Zheng and colleagues, in the MT-Bench and Chatbot Arena paper, examined position, verbosity and self-enhancement biases and found the self-enhancement evidence inconclusive with the data they had. Panickssery, Bowman and Feng later reported that models can recognize their own outputs at better than chance and that the strength of self-recognition correlates with self-preference. Wataoka and colleagues found that judges rate lower-perplexity text more highly whether or not they wrote it, which points to familiarity rather than authorship. The AlpacaEval maintainers warn that automatic evaluators may favor longer outputs, outputs from models fine-tuned on the evaluator's own model, and style over factuality.
| Bias | What the literature reports | What the experiment must do about it |
|---|---|---|
| Position | Verdicts can change when the two answers are swapped | Grade every pair in both orders and count flips |
| Length and verbosity | Judges can reward longer or padded answers | Record length; include rewrites that change length without changing content |
| Formatting | Arena rankings shifted after controlling for headers, bold text and lists | Strip or equalize markdown in the neutral condition |
| Self-preference | Linked to self-recognition in one study and to low perplexity in another | Cross content and style so authorship and style vary independently |
| Lineage | Evaluators may favor models fine-tuned on the evaluator's own model | Record base model lineage for every candidate, not just the vendor name |
Why the anchor must be verified completion, not another model's opinion
A style effect is only an error if it pulls grades away from the truth, so the experiment needs an independent truth. Use tasks whose correct outcome can be checked from records: a bill that should match a purchase order and receipt in NetSuite, a Zendesk ticket whose resolution is confirmed by the customer's next contact, a Jira bug whose fix is confirmed by a merged change and a passing test.
For each task, a domain adjudicator writes down what completion means and what evidence proves it before any model runs. That definition becomes the verified completion label. Where two experts reasonably disagree about whether an answer completes the task, keep both judgments and mark the item contested rather than forcing one label; a judge should not be penalized for siding with either expert on a contested item.
Open-ended writing tasks with no checkable outcome can still be included, but analyze them separately. On those items the experiment can show that style moves the grade, but not whether the movement is wrong.
The blinded cross model grading experiment
The core design is a 2x2 crossover for each pair of model families: the content comes from family A or family B, and the style is A's native style or B's. A neutral, style-normalized version of each answer is added as a further condition. The judge never sees model names, and every pairwise comparison runs in both orders.
The same design works with single-answer grading or pairwise grading; the MT-Bench judge tooling, for example, supports both. Pick the mode the production judge will use, because results in one mode do not transfer automatically to the other.
| Design element | Specification | Why it matters |
|---|---|---|
| Research question | Does the judge's grade depend on the writing style of a model family once verified completion is held fixed? | Separates a style preference from a quality preference |
| Population | Business tasks from defined task families with a record-checkable outcome; candidate answers from at least three model families, including the judge's own | Self-preference can only be seen if the judge's family is among the candidates |
| Sampling | Stratified by task family and by verified outcome so complete and incomplete answers are both well represented | A set of mostly correct answers cannot show whether style rescues wrong ones |
| Conditions | Native, style-swapped (A content in B style, B content in A style) and neutral rewrites of each answer | Lets authorship, style and content be estimated separately |
| Blinding | Model names, system fingerprints and provider phrasing removed; order randomized and swapped | Removes explicit cues so any remaining effect comes from the text itself |
| Primary outcome | Judge grade or pairwise preference for each condition | The quantity whose shift is being tested |
| Reference | Verified completion label and expert adjudication, with contested items flagged | Defines what a correct grade is |
| Comparison | The same items graded by a judge from another family and by a panel of judges | Shows whether the effect belongs to one judge or to judges in general |
How do you blind a response when the style is the signal?
You cannot fully blind style, so make the style a controlled variable instead of a hidden one. Removing names is easy; the harder cues are phrasing habits, formatting conventions and refusal or hedging patterns that a model can recognize as its own or as familiar.
The rewrites are where the experiment can quietly fail. A rewrite that drops a figure, softens a caveat or fixes an error changes the content, and then the crossover no longer isolates style. Every rewrite therefore needs a content-equivalence check by a person who compares it with the original against the task's completion definition.
- Strip identifiers: model names, self-references, provider-specific disclaimers and signature phrases.
- Normalize the surface: same markdown policy, same units and date formats, and a recorded token length for every version.
- Produce style-swapped rewrites with a model from a family not under test, prompted to keep every fact, number, step and caveat.
- Produce a neutral rewrite in plain prose with no headers, bold text or lists.
- Have a reviewer mark each rewrite as content-equivalent or not; exclude non-equivalent rewrites rather than repairing them by hand.
- Randomize answer order and run every pair in both orders; log the position of each answer with its grade.
Which shortcut could raise a grade without completing the business requirement?
Include deliberately incomplete answers that look complete, because a judge that rewards these will reward any model that learns to produce them. These probes test whether the grader rewards shortcuts or mislabels failure, which is a separate question from family preference but uses the same items.
Each probe should be built from a real task where the verified outcome is known to be incomplete, then dressed in the surface features a judge might reward. Score the judge on how often it grades the probe as complete.
| Probe | What it looks like (illustrative) | What a reliable judge should do |
|---|---|---|
| Confident summary of unfinished work | States that the invoice was matched and approved when the receipt is missing | Grade incomplete because the completion evidence is absent |
| Rubric echo | Restates the grading criteria as headings without doing the steps | Grade on the steps performed, not the headings |
| Claimed tool success | Says a refund was issued but the order record shows no refund | Check the claim against the provided record |
| Polished formatting, wrong answer | Well-structured reply that applies the wrong return policy window | Grade the policy error, regardless of presentation |
| Plain but correct | Terse, unformatted reply that completes the task | Grade complete; brevity is not a defect |
Measuring the gap: metrics, sample and uncertainty
The headline metric is the style effect: how much the judge's grade changes when only style changes, holding content and verified completion fixed. Estimate it separately for the judge's own family style and for each other family's style, so a general preference for one popular style is not mistaken for self-preference.
A regression makes this explicit. Chatbot Arena's style control added answer length and counts of markdown headers, bold elements and lists as features in its Bradley-Terry model, and length-controlled AlpacaEval fits a generalized linear model on the length difference and predicts what the preference would be with that difference set to zero. The same approach works here with three extra terms: content source, style source and verified completion.
Report, for each judge: agreement with verified completion on uncontested items; the style effect with an interval; the self-family effect with an interval; position flip rate; and the probe pass rate. Compute intervals with a bootstrap that resamples tasks, not individual grades, because the versions of one task are not independent. Size the item set with a power calculation for the smallest style effect that would change a model selection decision, and state that threshold before running.
Two limits belong in every write-up. A rewrite model brings its own style, so a neutral condition is only neutral relative to it. And a result holds for the task families and model versions tested; a new judge version needs a new run.
Pre-registration checklist before any judge is run
Write these down and date them before the first grade is collected, so the analysis cannot drift toward a convenient finding. Nothing on this list is a result; it is the plan that makes a later result checkable.
At a glance
Pre-registration checklist before any judge is run
Checklist · 10 items
- The research question and the smallest style effect that would matter for a decision.
- Task families, inclusion rules and the record evidence that defines completion for each.
- Candidate model families and versions, the judge model and version, and their known lineage.
- Rewrite model, rewrite prompts and the content-equivalence rule for keeping a rewrite.
- Grading mode (single answer or pairwise), the judge prompt and decoding settings.
- Stratification and item counts, with the power calculation behind them.
- Primary metric, secondary metrics and the regression specification.
- How contested items, judge refusals and parse failures are handled.
- The interval method, the clustering unit and any multiple-comparison adjustment.
- What will be published, including the items or a description of them, and what will not.
0 of 10 ticked. Tick each item as you go.
If the judge fails the test, what changes in the grading setup?
A measurable style or self-family effect does not mean the judge must be discarded, but it does mean its cross-family comparisons cannot stand alone. Choose a fix that matches the failure, then rerun the same experiment to confirm it helped.
For position effects, grade both orders and average, as AlpacaEval's evaluator randomizes order for the same reason. For length and formatting effects, report style-controlled scores alongside raw ones. For self-family effects, Verga and colleagues found that a panel of smaller judges from disjoint model families showed less intra-model bias than a single large judge, which makes a panel a reasonable next test. For high-stakes task families or items where experts disagree, route grading to human adjudication and use the judge only for triage.
Where verified business tasks for the test come from
The experiment depends on tasks with a checkable outcome, which usually means closed business cases: tickets with confirmed resolutions, matched transactions, merged fixes. Public chat prompts rarely carry that evidence.
For a grading experiment, SourceX can source closed cases with checkable outcomes on request from U.S. businesses rather than from inventory; the buyer's request sets the scope, volume, language, format, timing and licensing terms. On the supplying side it runs rights review, contracting, data preparation and delivery, so the evidence that a task was completed can travel with the task itself.
Frequently asked questions
What is self-preference bias in an LLM judge?
It is a judge's tendency to grade answers from its own model, or text that resembles its own writing, more favorably than equally good answers from other models. Research has linked it to a model's ability to recognize its own outputs and to a preference for low-perplexity text, and the cause is still debated.
Is removing model names enough to blind a judge?
No. Phrasing, formatting and hedging habits can identify a model family even without its name. That is why this protocol treats style as a controlled variable through rewrites rather than assuming names were the only cue.
Does a panel of judges remove the bias?
It can reduce bias that belongs to one judge, and one study found less intra-model bias in a panel drawn from different families. It does not remove biases that all the panel members share, such as a common preference for formatted answers, so the panel should be tested with the same experiment.
How many items does this experiment need?
That depends on the smallest style effect that would change a decision and on how many task families you test. Run a power calculation before collecting grades, and cluster by task in the analysis, since the rewrites of one task are not independent observations.
Are there published results for this grading experiment?
No. This page describes the design, blinding, probes and analysis plan only. Results would need rights-cleared tasks with verified outcomes, content-checked rewrites and a dated pre-registration.
Sources
- Zheng and colleagues examined position, verbosity and self-enhancement biases in LLM judges and found the self-enhancement evidence inconclusive with the data available. Source
- LLM evaluators can distinguish their own outputs from other models' at better than chance, and self-recognition strength correlates linearly with self-preference. Source
- LLM judges assign higher evaluations to lower-perplexity texts regardless of whether the texts were self-generated. Source
- A panel of LLM evaluators from disjoint model families showed less intra-model bias than a single large judge. Source
- Length-controlled AlpacaEval fits a generalized linear model on the length difference and other features, then predicts preferences with the length difference set to zero. Source
- AlpacaEval warns that automatic evaluators may favor longer outputs, models fine-tuned on the evaluator's model and style over factuality, and its evaluator randomizes output order to avoid position bias. Source
- Chatbot Arena style control adds answer token length and counts of markdown headers, bold elements and lists as features in its Bradley-Terry regression. Source
- The MT-Bench judge tooling supports single-answer grading, pairwise comparison against a baseline and pairwise comparison across all model pairs. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.