Privacy and preparation
How to run an expert annotation fatigue audit: a study protocol
By SourceX Editorial · Updated
10 sections
Short answer
Detect reviewer fatigue by measuring, within each reviewer, whether label quality on seeded gold and re-reviewed items changes with time on task and position since the last break, while item order is randomized so difficulty cannot masquerade as fatigue. This page is a study protocol for an expert annotation fatigue audit: design, variables, metrics and a pre-registration checklist. It reports no results.
Checklist
Expert annotation fatigue
Key takeaways
- Fatigue is a within-reviewer change over a session, so a dataset-wide accuracy number cannot reveal it.
- Randomized item order is the single design choice that separates fatigue from queue order and item difficulty.
- Seed gold items at fixed positions and log open, submit and break times separately; tool lead time alone includes idle time.
- Model quality by session position with reviewer as a random effect, and report intervals, not a single verdict.
- Capture the expert's alternatives and confidence, so legitimate disagreement is not scored as tiredness.
- This page publishes the method only; no fatigue rates or effect sizes are claimed.
How do you detect reviewer fatigue affecting dataset labels?
You detect it by comparing each reviewer with themselves. Seed items with known answers through every session, randomize the order of the remaining items, and test whether accuracy on the seeded items, agreement with an independent re-review and decision behavior shift as time on task grows and recover after breaks. A dataset-level error rate cannot answer the question, because fatigue is a pattern inside sessions, not a property of the whole dataset.
This page is a protocol, not a finding. It sets out the question, population, sampling, variables, comparisons, metrics, uncertainty and limits of a batch quality and review timing study that a labeling team, a data supplier or a buyer's evaluation lead can pre-register and run. No fatigue rates, effect sizes or results are reported here, and none should be inferred from the examples, which are illustrative.
The audit matters most for expert labels: claims coded by adjusters, contracts tagged by paralegals, tickets triaged by senior support staff. Expert time is expensive, sessions run long, and the people doing the work are often the only ones who could spot their own drift.
Why is fatigue so hard to see in a finished dataset?
Because the signal is confounded with everything else that changes during a session. Items that arrive late in a queue are rarely a random sample of the work: hard cases get deferred, escalations get batched, and some queues sort by age or by customer. A drop in accuracy late in the day can mean tired reviewers or harder items, and a finished dataset usually cannot tell you which.
The best-known example of the problem comes from outside data labeling. Danziger, Levav and Avnaim-Pesso reported in PNAS in 2011 that the share of favorable parole rulings fell across each decision session and returned after a food break. Weinshall-Margel and Shapard replied that case ordering was not random and that prisoners without lawyers usually went last, which could produce the same pattern; the original authors answered that their results persisted with representation included. For an annotation audit, the lesson is not who was right but that observational order data invites this argument.
Designs with random timing are more persuasive. A PNAS study of Danish standardized tests found that scores declined for tests taken later in the day and that a break improved them, and the authors leaned on the fact that test timing was effectively random for each student. An annotation fatigue audit should copy that logic: make position in the session as close to random as the workflow allows, so any quality trend over position has fewer competing explanations.
What exactly is the research question?
State fatigue as a measurable change, not a feeling. In this protocol, fatigue means a within-reviewer decline in label quality, or a shift toward low-effort decisions, that grows with continuous time on task or items completed since the last break and partly recovers after a break, after holding item difficulty constant through randomization and stratification.
Write the hypotheses down before collecting data. A typical set is below; the team should keep, drop or reword them, but not add new confirmatory hypotheses after seeing results.
Decide the unit of exposure before choosing between the hypotheses. Items completed since the last break suits queues with similar item lengths; active minutes since the last break suits mixed work, where one contract can take as long as twenty short tickets. Pick one as primary, report the other as a sensitivity check, and say which was chosen in the registration.
- H1: accuracy on seeded gold items is lower in the last position bin of a session than in the first, within the same reviewer.
- H2: disagreement with an independent blind re-review rises with items completed since the last break.
- H3: the share of low-effort decisions (default class, first listed option, blank rationale, skipped items) rises with time on task.
- H4: measures from H1 to H3 partly recover in the first bin after a break of the length the team defines.
- H5 (exploratory, labeled as such): time per item falls late in sessions without a matching change in difficulty.
Who and what is sampled, and in what order?
The population is the reviewers whose labels will enter the dataset and the items they label during the study window. Include every reviewer working in that window rather than volunteers, because people who opt in to a fatigue study may be the least affected. Record each reviewer's qualification and tenure, but analyze pseudonymous IDs only.
Randomize item order within each reviewer's queue. Some tools make this a project setting: Label Studio, for example, offers sequential sampling, which shows tasks in Data Manager order, or random sampling, which shows them in random order. Where the business workflow forces an order, such as oldest ticket first, randomize within blocks and record the block so the analysis can adjust for it.
Seed gold items at fixed intervals, for example one in every block of a set size, drawn by stratified random sampling across difficulty tiers so every position bin gets the same mix. Gold items must look like ordinary work and must not be skippable; in Label Studio, ground truth tasks can be imported with allow_skip set to false, and they are not limited by the project's overlap setting. Separately, send a random fraction of non-gold items to a blind second reviewer who is fresh, so you can measure disagreement on real work rather than only on gold.
The per-label logging table
Every label in the study needs the same fields, captured at the moment of the decision. The table is the core of the batch quality and review timing study plan; a team can copy it into its tool's export schema as is.
One caution on timing: built-in time fields often measure something other than effort. Label Studio documents that its lead time runs from task open to submit or update, includes idle time and is cumulative across sessions. Log open, first interaction and submit timestamps yourself, plus break start and end, so active time and idle time can be separated.
Two more items belong in the log but outside the per-label row. Record every skip, deferral and escalation as its own event with a timestamp, since a reviewer who skips hard items late in a session changes the mix of what they label. And collect a short workload rating at the end of each session, so self-reported strain can be compared with what the labels show.
| Field | Definition | Source in the workflow | Why the audit needs it |
|---|---|---|---|
| Reviewer ID | Pseudonymous stable ID; no names in the analysis file | Tool user table mapped through a key held by one person | All comparisons are within reviewer |
| Session ID and session start | A continuous working block, ended by a break longer than the defined threshold | Derived from timestamps and break log | Defines position and time on task |
| Position since last break | Count of items completed since the session or break began | Derived | Primary exposure variable |
| Active time on item | First interaction to submit, excluding idle gaps above a set threshold | Client event log | Separates speed-up from idle time |
| Item type and gold flag | Gold, re-review sample or ordinary item | Seeding plan | Identifies items with a reference answer |
| Difficulty tier | Pre-assigned tier from a pilot or expert rating, set before the study | Item metadata | Lets analysis hold difficulty constant |
| Label and options shown | Chosen value plus the order options were displayed in | Annotation export | Detects first-option and default-class drift |
| Confidence and alternatives | Reviewer's stated confidence and any second plausible label | Required form fields | Separates honest ambiguity from fatigue |
| Rationale and evidence cited | Short reason and the source fields or documents relied on | Required form fields | Shows whether evidence was actually read |
| Reference outcome | Gold answer or blind re-review label, after adjudication | Gold set and second-review queue | Defines correct and incorrect |
Which metrics, comparisons and uncertainty make a finding defensible?
Use several measures, each with a comparison and an interval, because no single metric captures fatigue. Accuracy on gold is the cleanest but the rarest; disagreement with re-review covers real work but includes honest ambiguity; behavioral signals are plentiful but indirect. A defensible finding is one where the measures point the same way within reviewers.
Fit the main model at the label level: correctness on gold or re-review items as the outcome, position bin and break recovery as predictors, difficulty tier and item block as covariates, and reviewer as a random effect. Report intervals from a bootstrap that resamples reviewers, not labels, since labels from one person are not independent. Fix the number of position bins, the break threshold and the minimum effect worth acting on before data collection, and compute in advance how many gold items per bin the study needs to detect that effect.
For workload self-report, NASA's Task Load Index is a long-established instrument that rates six subscales: mental demand, physical demand, temporal demand, performance, effort and frustration. Treat it as a secondary measure; self-report and behavior can disagree, and that disagreement is itself worth reporting. Tool dashboards can help with bookkeeping; Label Studio, for instance, can compare annotators against ground truth annotations, predictions and other annotators, but the within-session comparison has to be built from the exported timestamps.
| Metric | How it is computed | Comparison | Uncertainty to report |
|---|---|---|---|
| Gold accuracy by position | Share of seeded gold items labeled correctly in each position bin | Last bin vs first bin, within reviewer | Reviewer-clustered bootstrap interval; model estimate with interval |
| Re-review disagreement | Share of sampled items where a fresh blind reviewer disagrees after adjudication | Trend across bins since last break | Interval plus the adjudicated share judged genuinely ambiguous |
| Low-effort decision rate | Share of default-class, first-option, blank-rationale or skipped decisions | Late bins vs early bins, by difficulty tier | Interval per tier; flag tiers with too few items |
| Active time per item | Median active seconds per item, idle excluded | Trend across bins with difficulty held constant | Interquartile range and bootstrap interval of the median |
| Break recovery | Change in the measures above from the last pre-break bin to the first post-break bin | Post-break vs pre-break, within reviewer | Interval; note how many breaks met the threshold |
| Workload rating | Session-end questionnaire score | Long vs short sessions, same reviewer | Descriptive only unless pre-registered |
What evidence would change the expert's decision, or make another label equally valid?
Ask for it at the time of labeling, because it is the only way to tell fatigue from disagreement. An expert who picks label B late in a session, where gold says A, may be tired, or may have seen something in the record that makes B defensible. If the form captured a stated alternative, a confidence level and the evidence relied on, the adjudicator can decide which.
Adjudicate every disagreement on gold and re-review items before scoring, and allow the outcome that the gold answer itself is wrong or that two labels are equally valid. Remove or relabel gold items that the adjudication overturns, and report how many there were. A fatigue audit that treats every departure from gold as an error will overstate fatigue on exactly the items experts find most interesting.
Record reviewer incentives too. Per-item pay, daily quotas and leaderboards change pace, and pace changes position effects. A study run under a quota measures fatigue under that quota, and should say so.
- Alternative label the reviewer considered, if any.
- Confidence on a fixed scale defined in the guideline.
- Evidence relied on: which fields, attachments or documents were opened.
- Guideline section applied, with the guideline version.
- Escalation flag for cases outside the reviewer's qualification.
- Incentive context for the session: quota, pay basis or none.
Pre-registration checklist for a fatigue audit
Register the plan before the first study session, with a date and version. The Center for Open Science describes preregistration as separating hypothesis-generating, exploratory research from hypothesis-testing, confirmatory research; in a fatigue audit that separation is what stops a team from finding a pattern after the fact and calling it fatigue.
At a glance
Pre-registration checklist for a fatigue audit
Checklist · 10 items
- Research question and hypotheses H1 to H5, with H5 marked exploratory.
- Population, study window and the rule for including reviewers.
- Randomization method and any forced-order blocks.
- Gold seeding interval, difficulty tiers and how gold answers were set.
- Re-review sampling rate and the independence rules for second reviewers.
- Definitions of session, break threshold, position bins and active time.
- Primary model, covariates, random effects and the bootstrap unit.
- Minimum effect worth acting on and the planned number of gold items per bin.
- Adjudication procedure and how overturned gold items are handled.
- Data protection for reviewer-level data: pseudonymization, access and retention.
0 of 10 ticked. Tick each item as you go.
What this protocol cannot show, and when a signal should pause labeling
A within-session decline shows that something changes with time on task; it does not prove the cause is mental fatigue rather than boredom, interruptions, shift handovers or time of day. Results from one task, one reviewer pool and one incentive scheme do not transfer to another. And a study with few reviewers can show individual differences much larger than any average trend, so report the spread across reviewers, not only the mean.
Decide in advance what operational step a confirmed signal triggers: shorter sessions, scheduled breaks, more gold late in sessions or targeted re-review of late-position labels already in the dataset. Some tools can automate a guardrail; Label Studio can pause annotators who fall below a ground truth or acceptance score threshold. Use such switches for data quality, not as a hidden performance review, and tell reviewers how their timing data is used. Reviewer timestamps are personal data in many settings, so involve whoever owns employee privacy before the study starts.
Running the audit on records you may license
Labels made inside operating companies, such as claim codes, ticket dispositions and contract tags, carry the same fatigue risk as vendor annotation, and a buyer evaluating them will ask how they were produced. Running this protocol on a sample, or documenting why session structure was not a risk, gives that answer in a form a technical reviewer can check. A quality study of this kind says nothing about who may license the records: access, privacy permission, ownership and contractual rights are separate questions.
Businesses that license selected records directly to SourceX can rely on SourceX for rights review, contracting, data preparation and delivery. Labels and records reach SourceX only after the company approves scope and terms and signs, and the first check of whether a company qualifies is a short assessment with no data upload.
Frequently asked questions
What is an expert annotation fatigue audit?
It is a study of whether experts' labels get worse, or lazier, as time on task grows within a session. It compares each reviewer with themselves using seeded gold items, blind re-review and decision behavior by session position, with item order randomized so difficulty is not mistaken for fatigue.
Can I detect fatigue from an existing dataset without a new study?
Only weakly. If the export has reliable timestamps and some items were re-reviewed, you can look for within-reviewer trends, but queue order is rarely random, so harder items arriving late can produce the same pattern. Treat such an analysis as exploratory and confirm it with a randomized, pre-registered study.
How many gold items does a fatigue audit need?
Enough to detect, in the last position bin, the smallest drop the team would act on. Fix that minimum effect first, then compute the gold items per bin and per reviewer it requires; a statistician should check the calculation. This page deliberately gives no number, because it depends on the task and the reviewer pool.
Is time per item a good fatigue measure on its own?
No. Faster work late in a session can mean fatigue, learning or easier items, and built-in timers often include idle time. Use active time with difficulty held constant, and read it alongside accuracy and low-effort decision rates.
Should fatigue results be used to rate individual reviewers?
The audit is designed to fix workflow, such as session length, breaks and gold placement, not to rank people. Reviewers should know what timing data is collected and why, and reviewer-level data needs the same privacy care as other employee records.
Sources
- Danziger, Levav and Avnaim-Pesso reported that favorable parole rulings fell within each decision session and returned after a break. Source
- Weinshall-Margel and Shapard argued the parole case order was not random, with unrepresented prisoners usually heard last. Source
- Danish test scores declined later in the day and improved after a break, with test timing effectively random for each student. Source
- Label Studio offers sequential task sampling in Data Manager order or random task sampling. Source
- Label Studio can make ground truth tasks unskippable, exempts them from overlap, and can pause annotators below a ground truth or acceptance threshold. Source
- Label Studio lead time runs from task open to submit, includes idle time and is cumulative across sessions. Source
- Label Studio evaluates annotator performance against ground truth annotations, predictions and other annotators. Source
- The NASA Task Load Index rates workload on six subscales: mental, physical and temporal demand, performance, effort and frustration. Source
- Preregistration separates exploratory, hypothesis-generating research from confirmatory, hypothesis-testing research. Source
- Danziger, Levav and Avnaim-Pesso replied that their parole results persisted when legal representation was added to the analysis. Source
See if your company qualifies
A short company assessment. No data uploads are needed.