AI uses for records
Dataset rubric version control: which rule changes break old scores
By SourceX Editorial · Updated
10 sections
Short answer
Dataset rubric version control means every grade names the rubric version that produced it, and every rubric change is classified by what it does to earlier scores. In EleutherAI's evaluation harness, 53 logged task changes split into scoring, prompt, composition, limit, answer-key and unversioned edits, and only 8 said how scores would move. Re-score, bridge or split the series according to the class, and pin rubrics by content hash.
Grader rubric
Dataset rubric version
Key takeaways
- Two scores can be compared only when both trace to known rubric content, so each grade needs a rubric version, a content hash and effective dates.
- In lm-evaluation-harness, 47 of 221 task folders carry a version of 2.0 or higher, and only 8 of 53 coded changelog entries say what the change did to scores.
- Scoring-rule changes can be re-applied to stored per-criterion marks; changes to what graders see cannot, and start a new series.
- Version labels fail: one MMLU-Pro changelog lists two different changes under the same 2.1 to 3 transition, so pin rubrics by hash.
- Before pooling across a wording change, double-grade a bridge sample under both versions and report agreement, mean shift and pass-line crossings.
What does dataset rubric version control have to record?
Every grade has to name the exact rubric that produced it, and every change to that rubric has to say what it does to grades already given. Dataset rubric version control applies that discipline to the grading criteria behind a labeled or scored dataset: QA scorecards in a support center, claim audit checklists, code review gates, the criteria an expert used to call an answer acceptable. When the rubric changes partway through the record window and the dataset keeps a single score column, a buyer cannot tell a better process from a stricter grader.
Three things travel with each score: a rubric identifier and version, a content hash or stored copy of the rubric text, and the dates during which that version governed grading. Three things travel with each change: what kind of change it was, whether earlier scores stay comparable, and the evidence behind that verdict. The second set matters more than the version number, as the change history of a public, open-source evaluation harness shows. This page measures that history, then turns it into a change-impact table, a bridge test and a data card for graded business datasets.
How often do grading rules change? 53 logged changes in one public harness
Often, and the change notes rarely say what became of earlier scores. EleutherAI's lm-evaluation-harness defines each benchmark task in configuration files, asks contributors to raise a task's version number by 1 whenever a breaking change is made, and suggests a changelog entry in the task's README. At commit d6de816 of September 14, 2026, 204 of its 221 top-level task folders declare a version in at least one configuration file, and 47 carry a version of 2.0 or higher. Eight sit at 4.0 or above, among them the configurations for GSM8K, IFEval, BBH and HumanEval at 4.0 and LongBench at 6.0.
We read every task README with a Changelog heading, 43 files of which 31 contain entries, and coded 53 change entries by what they altered, leaving out two notes that describe a first release. Scoring and answer-parsing changes lead with 17 entries, followed by prompt and few-shot presentation with 13, composition and naming with 10, generation limits and stop rules with 7, answer keys and targets with 3, and changes logged with no version number with 3.
Only 8 of the 53 entries state what the change does to scores. The FDA, SQuAD completion and SWDE notes say results are not comparable to v0. KorMedMCQA's version 3.0 note, which added an answer-extraction filter, tells readers to expect higher scores for models that do not answer with a bare letter, Minerva Math's version 3.1 note says scores shift slightly relative to 3.0, and mlqa's v0.1 note, which fixed the regex that removes Arabic articles when answers are normalized, says the fix affects scores for all Arabic subtasks. The other 45 describe the edit and leave the reader to work out whether an earlier score still means the same thing. The table gives the counts by class with one example each.
| Change class | Entries | State a score effect | Example from the changelogs |
|---|---|---|---|
| Scoring or answer parsing | 17 | 7 | GPQA 2.1 narrowed a regex that had stripped bracketed math from answer choices |
| Prompt or few-shot presentation | 13 | 0 | RACE 3.0 fixed how fill-in-the-blank questions render in few-shot examples |
| Composition or naming | 10 | 0 | Catalan Bench versions 2.0, 2.1, 2.2, 2.4 and 2.5 each added a task |
| Generation limits or stop rules | 7 | 0 | MMLU-Pro 2.1 raised maximum generated tokens from 256 to 2048 |
| Answer key or targets | 3 | 0 | PAWS-X v1 corrected the order of answer labels |
| Logged without a version number | 3 | 1 | BBH switched its dataset source and said not to expect any change in results |
Six kinds of rubric change, and the verdict each one forces
Each class of change carries a default verdict for earlier scores, and the verdict turns on one question: can the earlier work be graded again under the new rule? The table translates the harness classes into the changes a business rubric goes through, such as a support QA scorecard, a claims audit checklist or an expert review form, and names what must happen before old and new scores share a column. Business examples are illustrative.
Change-impact verdicts by class:
| Rubric change | Business example (illustrative) | Can earlier items be re-graded? | Action before pooling | Verdict for earlier scores |
|---|---|---|---|---|
| Scoring rule or threshold | A pass now needs 4 of 5 criteria instead of 3 | Yes, if per-criterion marks were kept | Re-score the stored marks under the new rule | Comparable after re-scoring |
| Answer key or reference correction | The reference refund amount for one policy was wrong | Yes | Re-grade against the corrected key and retire the old numbers | Superseded |
| Criterion wording or grader instructions | Courteous tone now requires stating the next step | Only by grading the work again | Double-grade a bridge sample under both versions | Comparable only if the bridge shows no shift |
| What the grader sees | Graders now read the whole thread, not the last reply | No | Start a new series and keep the old one labeled | Not comparable |
| Composition | Two criteria added and one retired | Partly | Compare per criterion and recompute totals on shared criteria | Comparable per shared criterion |
| Unversioned edit | Checklist file replaced with no new version | Unknown | Diff against stored copies and treat as suspect until matched | Quarantined |
One version label, two different rubrics
A version number identifies a rubric only if every change produces a new one, and the public record shows how easily that breaks. MMLU-Pro's changelog in the harness lists two different changes under the same transition, version 2.1 to version 3: a stop-sequence change dated May 20, 2025 and a trim of whitespace from answer options dated January 16, 2026. A score labeled version 3 could come from either configuration, depending on when it was produced. Three of the 53 coded entries, including the BBH dataset switch, carry no version number at all.
Pin the content, not just the label. The harness writes each task's full configuration into its results output next to the version, so a score can be traced to the settings that produced it. Hugging Face's datasets library lets a loader fix a dataset to a revision, meaning a tag, branch or commit hash, which also pins a rubric stored in the same repository as the data. For a business dataset the equivalent is a content hash of the rubric text stamped on every grade, with the version label kept as a readable alias.
Should the rubric live inside each record or beside the dataset?
Inside each record when criteria differ by item; beside the dataset, with a hash on every grade, when one rubric covers the whole set. OpenAI's HealthBench reference code shows the inside pattern: each example carries its own list of rubric items, each with a criterion, a point value and tags, and an example's score is the points earned divided by the positive points available. Editing one criterion changes that example's denominator, and because the rubric ships inside the record, the edit is visible in the data file itself. The same code loads its data from files whose names carry a timestamp, which gives each release an identity.
The beside pattern suits operational QA, where one scorecard grades thousands of cases. Store the rubric once per version with its effective dates and hash, and stamp each grade with that hash. In both patterns, keep per-criterion marks rather than only totals. Without them a scoring-rule change cannot be re-applied to old grades, and every change collapses into the costliest verdict, a new series.
Bridge before pooling: grade one sample under both versions
When earlier work cannot simply be re-scored, grade a random sample of it under both rubric versions and let the difference decide. Testing programs have used this design for decades. When the National Assessment of Educational Progress revised its long-term trend assessment in 2004, it ran a bridge study in which students were randomly assigned to the original format or the revised one, so results could be linked across the change and the trend line could continue.
A rubric bridge for a graded business dataset is smaller and cheaper, because the work already exists. Draw items from before the change, stratified by the criteria the change touched, and have graders mark each item under both versions without seeing the other mark. Report three numbers: agreement between versions, the mean shift in score and the count of items that crossed the pass line. Agreement as high as two graders reach on one version, with a shift near zero, supports pooling. A consistent shift supports an adjustment published beside both series. Items crossing the pass line in both directions mean the series should split.
Keep both numbers visible while users adjust. When AlpacaEval moved to length-controlled win rates, which its README says raised the leaderboard's correlation with Chatbot Arena to 0.98, it made the new measure the default and kept raw win rates on its website and command line. A graded dataset can do the same: publish old-rubric and new-rubric scores side by side for the bridged period, then retire the old column with a dated note.
Did the rubric move the scores, or did the records change?
Separate the two before blaming either, because newer records can be harder, easier or simply different. A bridge holds the records fixed and varies the rubric. To hold the rubric fixed, grade a sample of new records under the old version. If old-rubric scores on new records match old-rubric scores on old records, the population held still and the rubric explains the shift. If they differ, part of the shift is the work itself, and any historical comparison needs a stated break whatever the rubric did.
Corrections deserve their own reading. In the current version of their paper, Gema and colleagues' re-annotation of MMLU, published as MMLU-Redux, estimates that 6.49 percent of MMLU questions contain errors, finds errors in 57 percent of the Virology questions analyzed, and reports significant discrepancies with model scores published earlier. When a key correction moves scores, the new numbers are not drift to be smoothed away. The old ones were measured against wrong answers and should be retired, not averaged in.
A data card for a graded dataset whose rubric changed
This card belongs in the dataset's documentation and answers, for any score, which rules produced it and whether it may be compared with another. Fill one set of rows per rubric version.
Card fields, illustrative values and the check each must pass:
| Field | Holds | Illustrative value | Check |
|---|---|---|---|
| Rubric ID and version label | Stable name plus a readable version | support-qa, v4 | Unique per content hash |
| Rubric content hash | Hash of the full rubric text and scoring rule | SHA-256 of the v4 file | Recomputes from the stored copy |
| Effective from and to | Dates the version governed grading | 2025-03-01 to 2026-01-31 | No gaps or overlaps between versions |
| Change class from the prior version | One of the six classes | Criterion wording | Matches a diff of the two texts |
| Comparability verdict | Pool, re-score, adjust, split or quarantine | Comparable after re-scoring | Backed by a bridge or re-score record |
| Bridge record | Sample size, agreement, mean shift and pass-line crossings | 200 items, all four numbers reported | Sample drawn at random within strata |
| Per-criterion marks kept | Whether item-level marks survive | Yes | Required for any re-scoring verdict |
| Grader cohort | Who graded under this version | Six reviewers on one team | Changes noted, since cohorts shift scores too |
Methods note: harness commit, files and coding rules
Data: EleutherAI's lm-evaluation-harness, cloned from GitHub on 2026-10-27 at commit d6de816 of September 14, 2026. Version counts scan every file under lm_eval/tasks whose name ends in yaml or .yml, 14,443 files, for a version key inside a top-level metadata block, and roll results up to the 221 first-level folders under lm_eval/tasks. Tasks defined in Python rather than configuration files are missed, so the 17 folders with no declared version include some, such as FDA, whose READMEs still log versions.
Changelog coding: every README.md under lm_eval/tasks with a heading named Changelog, 43 files with 31 holding entries. Two further READMEs, for acpbench and glianorex, use a two-word Change Log heading and were not counted, so the 43-file count is of the one-word heading. One entry is one version transition, or one dated note where no version is given; two first-release notes were excluded, leaving 53. Each entry received one class by precedence: scoring or answer parsing, then answer key or targets, then prompt or few-shot presentation, then generation limits or stop rules, then composition or naming, with unversioned notes as their own class. Another coder could move a few borderline entries, such as a target-delimiter change coded here as presentation. An entry states a score effect when it says scores will rise, fall, shift, stay the same or stop being comparable. The verdict table, the bridge rules and the data card are editorial frameworks with illustrative values, and no SourceX dataset was measured.
Asking a supplier for rubric history
Put the rubric into the request itself. Ask for every version of the grading criteria used in the window, their effective dates, per-criterion marks and a change note per version naming its class and verdict. Then make the acceptance test concrete: every score resolves to a stored rubric version by hash, and any series pooled across a change arrives with a bridge or re-score record.
SourceX takes requirements like these through its data request form, which asks for scope, volume, language, format, timing and licensing needs, and then looks for a business that holds matching records instead of drawing on stock. Scorecards and checklists can be scoped together with the graded records, and the company holding them approves the scope and terms and signs before anything is delivered to SourceX. Whether a particular company kept its earlier rubric versions stays an open question until its records are reviewed.
Frequently asked questions
What is rubric versioning in a labeled dataset?
Each label or score records the identifier, version and content hash of the grading criteria that produced it, plus the dates that version applied. Each change to the criteria is logged with its type and a verdict on whether earlier scores stay comparable, backed by a re-score or bridge record.
Can scores graded under two rubric versions be averaged together?
Only after a check. If only the scoring rule changed and per-criterion marks were kept, re-score the old items and pool them. If the criterion wording or grader instructions changed, double-grade a bridge sample first, and if the work the grader saw changed, start a new series.
Is a version number enough to identify a rubric?
No. The MMLU-Pro changelog in EleutherAI's harness lists two different changes under the same version 2.1 to 3 transition, and 3 of 53 coded changes carry no version at all. Stamp a content hash of the rubric text on every grade and keep the version as a readable alias.
How large should a rubric bridge sample be?
Large enough to estimate agreement and mean shift with useful precision on the criteria the change touched. Stratify by those criteria rather than sampling evenly, report intervals rather than single numbers, and enlarge the sample if items cross the pass line in both directions.
What if earlier rubric versions were never saved?
Rebuild what you can from grader training material, calibration notes or tool configuration history, and date each rebuilt version. Where a version cannot be recovered, mark its scores as unversioned and use them only within their own period, never in a pooled comparison.
Sources
- The lm-evaluation-harness task guide asks contributors to bump a task's version by 1 for each breaking change and to log the change in the task README. Source
- The harness task folders at commit d6de816 are the data behind the version counts and the coded changelog entries (computed by SourceX). Source
- The MMLU-Pro task changelog lists two different changes under the same version 2.1 to 3 transition and a 2.1 change raising generation tokens. Source
- The FDA task changelog says results after its v1 scoring fix are not comparable to v0. Source
- The KorMedMCQA changelog says its v3.0 answer-extraction filter should raise scores for models that do not answer with a bare letter. Source
- The harness writes each task's configuration and version into its results output. Source
- Hugging Face's datasets library loads a specific dataset version through a revision given as a tag, branch or commit hash. Source
- HealthBench's reference code stores rubric items with points inside each example and scores examples as earned over available positive points. Source
- NCES describes the 2004 NAEP long-term trend bridge study that administered original and revised formats to randomly assigned students. Source
- The AlpacaEval README reports that length-controlled win rates reach 0.98 correlation with Chatbot Arena and that raw win rates remain visible. Source
- MMLU-Redux estimates that 6.49 percent of MMLU questions contain errors and finds errors in 57 percent of analyzed Virology questions. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.