Skip to content

AI uses for records

Expert task difficulty annotation: rating dependencies and ambiguity

By SourceX Editorial · Updated

10 sections

Short answer

Experts document task difficulty best by rating separate dimensions against evidence, not giving one score. Rate dependencies (records, systems, people, step order) and ambiguity (request, rules, conflicting facts) on anchored levels, cite the record behind each level, and add an effort estimate, the alternative paths a qualified peer could defend, the evidence that would settle the task and a confidence level.

Grader rubric

Expert task difficulty

This page covers expert task difficulty annotation: rating dependencies and ambiguity.Illustrative

Key takeaways

  • A single difficulty score blends missing information, conflicting sources, judgment calls and volume, which need different handling in a dataset.
  • Dependencies and ambiguity explain most of the hardness in business tasks and can be rated on anchored levels.
  • Every rating should point to the record that justifies it so a reviewer can check it without the rater.
  • Alternative valid paths and decision-changing evidence are what make a difficult task gradable.
  • When qualified experts disagree on the same evidence, keep both ratings; the split is information about ambiguity.

How should an expert document what makes a business task hard?

Have the expert rate difficulty on separate dimensions and tie every rating to evidence in the task record. The two dimensions that explain most of the hardness in business work are dependencies, meaning how many inputs, systems, people and ordered steps the task needs, and ambiguity, meaning how many defensible readings of the request, rules or facts exist. Add a time estimate, the alternative paths the expert considered, and how confident they are.

A note that says hard, 4 out of 5 is close to useless to anyone building training or evaluation data. A note that says this task depends on three documents in two systems, one of which contradicts the other, and that a second qualified reviewer could reasonably approve it with a condition, tells a model developer exactly what skill the task exercises and how to grade it.

This guide gives a working rubric, the fields of an annotation record, a way to handle disagreement, and an illustrative example. It is written for operations leaders and domain experts at companies whose records capture skilled work, and for the annotation leads who would use those records.

Why a single difficulty score hides the useful information

Benchmark builders have already learned that difficulty has parts. When OpenAI built SWE-bench Verified, it had 93 experienced Python developers screen samples, flag issue descriptions that were underspecified, and estimate how long a fix would take in time buckets, and it published easy and hard subsets based on those estimates. Underspecification and effort were recorded as different things, because a short fix to a vague issue and a long fix to a clear one are hard in different ways.

GPQA defined difficulty another way: by the gap between people. Its question writers were domain experts, experts with or pursuing PhDs in the field answered correctly about 65 percent of the time, and highly skilled non-experts with unrestricted web access and over 30 minutes per question reached about 34 percent. METR, in turn, measures task length by how long human professionals take. Each approach captures something real, and none of them alone explains why a task is hard.

For business tasks, a single score blends at least four causes that need different fixes in a dataset: missing information, conflicting sources, real judgment calls and plain volume of work. Splitting them is what lets a buyer select tasks that test reasoning under ambiguity rather than tasks that are merely long.

The dependency and ambiguity difficulty rubric

Rate each dimension on three levels and cite the evidence that put it there. Levels are anchored by what a qualified person would have to do, not by how the rater felt, which keeps ratings comparable across experts.

Read the table as a profile, not a sum. A task rated 3 on rule ambiguity and 1 everywhere else tests a different skill from one rated 3 on input and cross-system dependencies, and a buyer looking for judgment tasks wants the first, not the second. Adding the levels into a total would hide exactly that difference.

Adapt the anchors to the trade before rating. In accounts payable, a level 3 cross-system dependency might be matching a receipt in the warehouse system to an invoice in the ERP with no shared purchase order number; in field service, it might be reconciling a technician's notes with a parts order and a warranty record. Write two or three concrete examples per level for your own work so raters anchor on the same picture.

The dependency and ambiguity difficulty rubric
DimensionLevel 1Level 2Level 3Evidence the expert cites
Input dependenciesOne record holds everything neededTwo to four records must be combinedFive or more records, or one record that must be found firstIDs of every record opened
Cross-system dependenciesOne systemTwo systems with a shared keySeveral systems with no shared key, matched by handSystem names and the join used
People dependenciesNo one else neededOne confirmation from a named roleSeveral approvals or a negotiationMessages or approvals in the record
Step orderSteps can run in any orderSome steps must precede othersOrder changes the outcome or a step cannot be undoneThe sequence actually followed
Request ambiguityOne reasonable readingTwo readings, context decidesSeveral readings and the requester must be askedThe words that allow each reading
Rule or policy ambiguityRule applies cleanlyRule needs interpretation at an edgeRules conflict or are silentClause, policy or standard cited
Information gaps and conflictsFacts complete and consistentA fact is missing but inferableFacts conflict and one must be chosenThe conflicting values and their sources
Consequence of errorEasily reversedReversible with rework or a creditHard to reverse or affects a third partyWhat happened when it went wrong, if recorded

What each difficulty annotation record must hold

The record wraps the rubric ratings with who rated, why, and how sure they are. Every field below exists to let a later reviewer, or a model developer, check the rating without asking the expert again.

At a glance

What each difficulty annotation record must hold

Write the rationale before the ratings. An explanation written first gives the levels something to be checked against; levels picked first invite a rationale written to defend the numbers. The order costs nothing and makes the record easier to audit.

What each difficulty annotation record must hold
FieldWhat to writeIllustrative entry
Rater qualificationRole, years in role and the specific competence the task needsProject manager, 9 years, commercial fit-out contracts
Task referenceStable ID of the task and the records it coversCO-0412 with linked RFI and drawing revision
Dimension ratingsLevel for each rubric rowInputs 3, systems 2, people 2, order 2, request 2, rules 3, gaps 3, consequence 2
Evidence per ratingPointer to the record that justifies each levelSpec section and RFI response disagree on finish grade
RationaleTwo or three sentences on the hardest partScope hinges on whether the RFI answer amended the spec
Alternative valid pathsOther decisions a qualified peer could defendApprove at reduced price, or reject and issue a clarification
Decision-changing evidenceFacts that would flip or settle the decisionA signed addendum adopting the RFI response
ConfidenceLow, medium or high, with the reasonMedium: addendum log incomplete
Effort estimateTime bucket for a qualified person with contextOne to four hours
Rubric version and dateVersion of the rubric usedv1.0, rated on the version frozen for this batch

What evidence would change the expert's call, or make another path equally valid?

Ask the expert to name the evidence that would settle the task, and the other answers a qualified peer could defend without it. These two fields turn a private judgment into something a grader can use: a task with several defensible outcomes needs a rubric that accepts each of them, and a task that would be easy given one missing document is testing information gathering, not expertise.

Ambiguity is common even in tasks that look simple. The AmbigQA study found that over half of the questions in an existing open-domain question set were ambiguous, with several plausible answers depending on which event or entity was meant. Business requests are often worse, because the requester assumes context the record does not show.

Record alternatives in the expert's own terms. Approve with a condition, approve at a different amount, or escalate are distinct paths, and each should carry a short note on when it would be the better call. Where only one path is acceptable, say why the others fail, citing the rule or record that rules them out.

Decision-changing evidence also tells a dataset builder how to vary the task. If a signed addendum would settle a dispute, the same task can be offered twice, once with the addendum and once without, to test whether a model notices the gap and asks for the missing document instead of guessing. That pairing is only possible when the expert wrote down what the missing piece was.

Keep the expert's uncertainty honest. A confidence level should come with its reason, such as an incomplete log or an unusual contract form, so a reviewer can tell a doubtful rating from a doubtful task.

Two qualified experts rate the same task differently: now what?

Keep both ratings and the reasons, then decide whether the disagreement is error or genuine ambiguity. Research on learning from disagreement surveys methods that drop the assumption of a single gold label and reports that, with many high-quality judgments per item, training on the distribution of labels can outperform training on an aggregated one. For a business dataset, that means a split rating is information, not noise to average away.

A practical routine has three steps. First, compare evidence: if one expert missed a record the other used, that is an error and the record is corrected. Second, if both saw the same evidence and still differ, mark the dimension as contested and keep both levels. Third, run short calibration rounds on a shared set of tasks whenever the rater pool or rubric changes, and track agreement per dimension rather than overall.

Ratings of the task are different from how demanding the work felt. NASA's Task Load Index asks people to rate mental, physical and temporal demand, performance, effort and frustration, and some teams collect a short version of that alongside the rubric. Keep the two apart: workload describes the rater's experience, while the rubric describes the task's structure and should hold across raters.

Piloting the rubric before rating a whole batch

Pilot on a small, varied batch before anyone rates at scale. A rubric that reads clearly on paper often has one dimension that raters interpret differently, and it is far cheaper to find that on a few dozen tasks than on a few thousand.

Choose pilot tasks deliberately: some the team agrees are routine, some everyone remembers as hard, and some where the outcome was later reversed or disputed. If the rubric cannot separate those groups, the anchors need work.

  • Have two qualified raters score the same pilot tasks independently, without discussing them first.
  • Compare ratings dimension by dimension and list every pair that differs by more than one level.
  • For each difference, check whether the raters opened the same records; fix the evidence before arguing about levels.
  • Rewrite any anchor that produced repeated disagreement on the same evidence, then re-rate those tasks.
  • Freeze the rubric version and date once agreement is stable, and record that version on every later annotation.

Illustrative: rating a construction change order

This example is illustrative and invented. A general contractor's project manager reviews a subcontractor's change order for upgraded wall finishes. Approving it requires the subcontract, the finish specification, a request for information answered by the architect, a drawing revision and the schedule.

The expert rates input dependencies at level 3 because five records are needed and the RFI response had to be found in a separate log. Cross-system dependencies are level 2: the documents live in two systems that share a project number. Rule ambiguity is level 3 because the specification and the RFI response give different finish grades and the subcontract does not say which governs. Consequence of error is level 2, since an overpayment could be recovered as a credit.

The expert's alternatives are approve at a reduced amount reflecting only the incremental grade, or reject and issue a clarification. The evidence that would settle it is a signed addendum adopting the RFI response. Confidence is medium. A second project manager, given the same records, rates rule ambiguity at level 2 because in her practice an RFI response binds once acknowledged. Both ratings are kept, the dimension is marked contested, and the disagreement itself becomes a useful test item.

The exception worth noting: a task can score low on every dependency and still be hard. A two-line reply to a long-standing client may require knowing a history that appears nowhere in the record. When the expert's rationale points to knowledge outside the record, flag the task as context-dependent rather than forcing a rubric level, because a model cannot be fairly graded on information it was never given.

Where difficulty ratings stop being trustworthy

Difficulty ratings are relative to the rater, the evidence they saw and the incentives around them. Effort estimates in particular depend on assumed context: SWE-bench Verified asked for the time an experienced engineer would need, and a newcomer would need more. State the assumed person in every record.

Incentives shape ratings. If raters are paid per task, they have a reason to rate quickly and cluster in the middle; if hard tasks are paid more, ratings drift upward. Pay for time or for calibrated accuracy, and audit a sample against a second rater.

Stop and narrow the scope when the rubric keeps producing contested ratings on one dimension, when raters lack the specific competence the task needs, or when the hardest part of most tasks is legal or regulatory interpretation. Those cases need a specialist reviewer, a narrower task definition, or both. Never use difficulty ratings to rank the employees whose work appears in the records; they describe tasks, not people.

From difficulty notes to a dataset others can use

Records where qualified people explain why a decision was hard are among the most useful material for training and evaluating AI on real work. The rationale, alternatives and decision-changing evidence are exactly what a grader needs and what generic data lacks. Before any of it leaves the company, the records still need rights, privacy and confidentiality review, and documented difficulty does not by itself create a right to license them.

Companies whose experts already write down why work was hard can license those records through SourceX, which manages rights review, contracting, preparation and delivery. Eligibility comes from a brief company assessment, and no annotated records are needed for it.

Frequently asked questions

What is expert task difficulty annotation?

It is a record, written by a qualified person, of what makes a task hard to do well: the inputs and systems it depends on, where it is ambiguous, what could go wrong and how long it takes. Each rating points to evidence in the task's records so others can check it.

Is time to complete a good measure of difficulty?

It is one useful dimension, and benchmarks such as SWE-bench Verified record it in time buckets. Time alone misses ambiguity and judgment, so a long but routine task can look harder than a short one that turns on a contested rule.

How many experts should rate each task?

One qualified rater is enough to start, with a second rater on a sample to measure agreement per dimension. Tasks where raters disagree on the same evidence should keep both ratings, because the disagreement shows genuine ambiguity.

Should non-experts rate difficulty too?

Their results are useful as a comparison, not as the rating itself. GPQA used the gap between expert and skilled non-expert accuracy to show its questions required expertise; the same contrast can show which business tasks need a qualified person.

Can difficulty ratings be used to evaluate employees?

They should not be. The rubric describes the structure of a task, and using it to judge the people who did the work would change how raters behave and raise employment and privacy questions.

Sources

  • For SWE-bench Verified, 93 experienced Python developers screened samples, flagged underspecified issues and estimated fix time in buckets used to form easy and hard subsets. Source
  • GPQA questions were written by domain experts; experts reached about 65 percent accuracy while skilled non-experts with web access and over 30 minutes reached about 34 percent. Source
  • METR expresses task length as the time human professionals take to complete a task. Source
  • AmbigQA found that over half of the questions in NQ-open are ambiguous, with sources such as event and entity references. Source
  • The JAIR survey Learning from Disagreement reviews methods that drop the single-gold-label assumption and reports soft-label training can outperform aggregated labels when items have many high-quality judgments. Source
  • NASA's Task Load Index rates workload on six subscales: mental, physical and temporal demand, performance, effort and frustration. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies