Relevance assessment guidelines and graded scales for domain experts

Why the task framing comes before the scale

The single most useful line in an assessor guideline is the scenario, because it tells experts what "relevant" is for. The TREC 2022 NeuCLIR track asked assessors to judge as if they were gathering material for a report on the topic, then used four categories from Very Valuable to Not Relevant that were mapped to graded gains [2]. That framing turns an abstract judgment into a concrete question: would I put this passage in my report, and how prominently?

For a domain supplier, write the scenario in the vocabulary of the downstream system. A claims-adjuster copilot, a field-service RAG assistant and a contract-search tool each imply a different reader, a different tolerance for background material and a different penalty for stale content. If the scenario is missing, a senior engineer and a junior technician will both be "right" while disagreeing on a third of the pool.

State the unit of judgment explicitly: the whole document, a fixed-length passage, or a chunk exactly as your retriever indexes it. Judging full documents and then scoring chunk-level retrieval silently inflates or deflates recall, depending on where the useful span sits.

Binary vs graded relevance judgments: choosing the scale

Use binary labels when you only need recall-style metrics or a clean positive set, and graded labels when ranking order among relevant items matters. TREC's classic English collections were judged binary relevant or not relevant [1], and those qrels still support MAP and recall comparisons. Graded labels pay off for nDCG, for listwise reranker training and for distillation targets, where the gap between "answers the question" and "mentions the topic" is the signal you care about.

Keep the scale short. Four levels are common in recent TREC-style work: the NeuCLIR categories [2] and the 0-3 scale applied by the UMBRELA LLM assessor [5] both stop at four. Each additional level adds definitional boundaries that experts must agree on, and boundary disagreements, not outright errors, are what usually drive agreement down.

Illustrative example: invented to show structure; it does not describe an available dataset.

The last row is the practical tiebreaker: a graded label can always be collapsed to binary, but a binary pool cannot be upgraded without a second pass.

A graded scale definition domain experts can apply

Each grade needs a one-sentence definition, a test question the assessor can ask, and at least two worked examples drawn from your own corpus. Definitions written as adjectives ("highly relevant") fail; definitions written as observable properties of the text hold up across assessors.

Illustrative example: invented to show structure; it does not describe an available dataset.

scale_id: field-service-rag-v1
scenario: >
  You are a field technician preparing to fix the reported fault on site.
  Judge each passage by how much you would rely on it for that job.
unit: passage            # as chunked by the production retriever
grades:
  3_essential:
    test: "Could I perform the fix using this passage alone?"
    rule: Contains the specific procedure, value or part number for this model and fault.
  2_useful:
    test: "Would I keep this open while working?"
    rule: Correct model family and fault; partial steps or needs one other source.
  1_related:
    test: "Is it on topic but not something I would act on?"
    rule: Same subsystem or general background; wrong model, or symptom without remedy.
  0_not_relevant:
    test: "Would reading this waste my time?"
    rule: Different subsystem, boilerplate, navigation text, or empty chunk.
edge_cases:
  partial_answer: Grade by the most useful span; never average across the passage.
  superseded_doc: Max grade 1 if a newer revision exists in the corpus; flag 'outdated'.
  duplicate: Grade identically to its twin; flag 'dup_of' with doc_id.
  unreadable_ocr: Grade 0 and flag 'ocr_fail'; do not guess content.
  query_ambiguous: Grade against the stated scenario; flag 'ambiguous_query'.
flags: [outdated, dup_of, ocr_fail, ambiguous_query, needs_adjudication]
fields_per_judgment: [query_id, doc_id, passage_id, grade, flags, assessor_id, seconds_spent, rationale_short]

Two design choices in this sketch carry most of the weight. The outdated-document cap prevents assessors from rewarding a superseded procedure that a production system must never surface. The short rationale field costs a few seconds per item but makes adjudication and later LLM-judge alignment far cheaper.

Assessor instructions for retrieval evaluation: what the document must cover

An assessor instruction document is a contract with the supplier, so it should be versioned, testable and complete enough that a new expert can start from it alone. Research on LLM relevance labeling found that the wording and detail of the labeling instructions materially affect label quality [4], and the same holds for human experts: vague instructions shift work into individual interpretation.

Illustrative example: invented to show structure; it does not describe an available dataset.

Assessor instruction checklist

• Scenario and user persona, in one paragraph.
• Unit of judgment (document, passage, chunk) and how to handle chunks cut mid-sentence.
• Grade definitions with test questions, plus two or more worked examples per grade from the real corpus.
• Edge-case rules: partial answers, outdated or superseded revisions, near-duplicates, tables and images, non-English or mixed-language text, OCR failures.
• What not to use: outside knowledge beyond the stated persona, web search, or the rank position shown.
• Presentation rules: randomized order within a topic, no system names, no retrieval scores visible.
• Flags vocabulary and when to escalate to adjudication.
• Time guidance per item and a rule to skip and flag rather than guess.
• Guideline version ID recorded on every judgment row.

Hide rank and system identity during judging. When assessors see that a passage came first, they tend to grade it up, which biases exactly the comparison the qrels exist to make.

Inter-assessor agreement: calibrate before you scale

Measure agreement on a calibration batch, revise the guideline, and only then release the full pool. A practical pattern is to have two or three experts independently judge the same 50 to 100 query-passage pairs, compute agreement, review every disagreement in a call, and turn the recurring ones into new worked examples. DocLayNet used the same idea in another annotation domain, labeling a subset of pages two or three times to measure inter-annotator agreement and using it as a reference ceiling for model scores [7].

Report agreement in a form that fits the scale. For binary labels, Cohen's kappa (two assessors) or Fleiss' kappa (more) is standard; for ordinal grades, weighted kappa or Krippendorff's alpha with an ordinal distance credits near-misses between adjacent grades. Also publish the confusion matrix: a 1-vs-2 confusion calls for a sharper boundary definition, while a 0-vs-3 confusion usually signals a scenario misunderstanding or a bad query.

Keep a fraction of double-judged items in production, not only in calibration. Drift appears when assessors speed up or when new topic areas arrive, and a continuing overlap sample catches it before a whole batch has to be re-judged. If you manage evaluation data under the NIST AI RMF, agreement statistics and guideline versions are natural evidence for the MEASURE function [8].

Domain expert relevance assessors vs generalist and LLM labelers

Domain experts are worth their cost when relevance depends on knowledge a generalist cannot fake, such as whether a torque value applies to a given model revision or whether a clause is enforceable as drafted. A large study of machine-translation evaluation found that ratings from expert assessors using an explicit error framework diverged from crowd ratings enough to change system rankings [6]; relevance judging in specialist corpora has the same exposure.

LLM assessors are now a realistic complement. Microsoft researchers reported that carefully prompted LLMs can label relevance in line with searcher preferences [4], and UMBRELA offers an open reproduction applying a 0-3 scale [5]. The defensible pattern is to let experts write and calibrate the guideline, use their labels as gold, and treat any LLM labels as a separately validated layer; the tradeoffs are covered in LLM relevance labels vs human assessors.

For contracting terms, rates and IP with expert annotators, see contracting domain experts for post-training data. For rubric design aimed at generated answers rather than retrieved passages, see designing evaluation rubrics with domain experts.

Planning for unjudged documents and future systems

Write the guideline assuming new retrievers will surface documents your pool never judged, and decide now how those will be handled. A study that re-judged documents surfaced by neural runs on an older TREC collection found that newer runs retrieved many previously unseen documents, but most of them turned out not to be relevant when judged [3]. That supports two choices: score unjudged documents as non-relevant by default for routine comparisons, and keep the guideline and assessor pool available for periodic top-up judging when a new system changes the ranking substantially.

Record what you need to make top-ups consistent: the guideline version, the corpus snapshot ID, the pool depth per topic and which runs contributed. Without those, a top-up batch judged a year later under a revised guideline is not comparable with the original qrels. The broader collection design (corpus snapshot, topics, qrels and nugget links) is covered in building a private domain retrieval test collection.

Handing the spec to a supplier

A spec package for an external supplier should contain the instruction document, the scale definition, the calibration set with your own gold labels, the agreement threshold that gates production, and the judgment file format. Asking suppliers for their assessor qualifications, their adjudication process and a sample of rationales is standard buyer diligence; procurement scope, pooling and pricing questions are covered in buying relevance judgments (qrels), and the cluster overview is the retrieval and RAG data buyer's guide.

Sometimes the hard part is not the labels but the corpus: real operational documents and queries from the domain you are evaluating. SourceX sources operational datasets from US companies on request, including support and sales histories, engineering records and documents, and every release is approved by the supplying company; you can describe the data your evaluation needs. For label-focused requests, see license expert annotations and labels and the data annotation glossary entry. Ready-made linked labels are another option, discussed in support tickets linked to knowledge articles.

Sourcing a domain corpus for relevance assessment

If your relevance program needs real domain documents or queries to judge, SourceX can look for US businesses that hold the data you describe. Each dataset is rights-reviewed and delivered under a license that defines records, uses, term and delivery, with personal details removed or replaced before delivery. Describe your relevance assessment data needs to SourceX.