Skip to content

AI uses for records

Synthetic data expert review design: challenge tasks, not approve them

By SourceX Editorial · Updated

10 sections

Short answer

Experts should try to break synthetic business tasks, not sign them off. Good synthetic data expert review design mixes generated tasks with untouched real ones, hides which is which, has two experts independently challenge each item with fixed reason codes, sends disagreements to an adjudicator, plants known-bad controls to test the reviewers, and feeds every rejection back to the generation recipe with the review results disclosed.

Provenance trail

Synthetic data expert review

This page covers synthetic data expert review design: challenge tasks, not approve them.Illustrative

Key takeaways

  • Approval review asks whether a task looks plausible; challenge review asks how a real practitioner would reject it, which is the question that finds errors.
  • Untouched real tasks, mixed in blind, are the yardstick: if experts cannot tell real from generated, mismatch is low; if they flag real tasks as often, the challenge is weak.
  • Fixed challenge codes such as rule inconsistency, outdated policy and missing exception make rejections countable and fixable.
  • Two independent challengers plus an adjudicator for disagreements, with a written reason, follows the verification pattern used in adversarial datasets.
  • Planted negative controls measure whether reviewers are actually reading, not rubber-stamping.
  • Disclose the generation recipe, grounding sources, selection steps and review results alongside the tasks.

How should experts challenge the validity of synthetic business tasks?

Experts should be asked to find the reason a real practitioner would reject each generated task, working blind against untouched real tasks, rather than to approve tasks that look reasonable. The shift from approve to challenge changes what reviewers look for, how their work is recorded and what the generator learns from it.

In practice that means five design choices. Mix a held-out sample of real tasks into the review queue without labels. Give each item to two experts who work independently. Require a challenge code and a written reason for every rejection. Send disagreements to a third, senior adjudicator. Plant a small number of deliberately broken tasks to check that reviewers catch them. The output is a review record per task, not a pass rate.

Why approve-or-reject review misses the failures that matter

Generated business tasks usually fail by being plausible and wrong, and a reviewer asked to approve tends to check plausibility. An invoice exception that cites a payment term the company never used, or a claims task resolved under last year's policy, reads fluently and passes a quick look.

The failures also cluster where they hurt most. Research on training models on model-generated content found that indiscriminate use causes defects in which the tails of the original distribution disappear. For business tasks, the tail is the rare exception: the partial shipment, the duplicate vendor, the policy carve-out. A generator that underproduces those, and a review that only checks what is present, will not notice what is missing.

A review of synthetic data practice for language models makes the same point from the producer side: factuality, fidelity and freedom from bias have to be checked, and synthetic data can amplify existing biases or introduce new ones if it is not carefully designed and validated. Expert challenge is one of the few checks that tests factuality against how the work is really done.

What adversarial data collection already teaches about expert challenge

Adversarial dataset projects already run the human side of this, with the roles reversed. In Adversarial NLI, annotators wrote examples meant to fool a model and explained why the example was difficult for the system; every development and test example was then checked by two verifiers, with a third added when the first two disagreed. Dynabench generalized the idea: collect data with humans and models in the loop because models that score well on static benchmarks still fail on simple challenge examples.

Synthetic task review borrows three habits from that work. Reviewers are given an adversarial goal rather than a checklist. Every judgment carries a written reason. Disagreement triggers a further independent reader rather than a vote. NIST's Generative AI Profile also lists AI red-teaming among the structured exercises for measuring risk, which gives a governance hook for treating expert challenge as a testing step rather than a courtesy sign-off.

The adversarial expert review and adjudication protocol

The protocol below is the asset. Run it on a sample before generating at scale, then on every new generator version. Each stage names who does it and what record it leaves, so the review can be audited later.

Stage 9 is where the work becomes useful to anyone else. The Datasheets for Datasets proposal asks that every dataset document its motivation, composition, collection process and recommended uses; for synthetic tasks, the collection process is the generation recipe and the review.

At minimum, disclose the generator and version, prompts or templates, grounding sources and their dates, any filtering or selection after generation, the size and source of the real holdout, the challenge codes, the readout by origin and what changed between recipe versions. A buyer who receives synthetic tasks without that record cannot tell a challenged and revised set from an approved first draft.

The adversarial expert review and adjudication protocol
StageWhat happensWhoRecord kept
1. Freeze inputsLock the generator version, prompts, grounding sources, policy effective date and the real holdoutData leadRecipe file with versions and dates
2. Build the queueMix generated tasks with untouched real tasks and planted broken controls; strip labels and formatting tellsReview coordinatorHidden key of item origin
3. Brief challengersGive the challenge codes and the goal: find why a practitioner would reject the taskReview coordinatorBriefing version and date
4. Independent challengeTwo experts review each item separately, choose accept or a challenge code and write a reasonTwo domain expertsPer-item code, reason, time spent
5. Origin guessEach expert also guesses real or generatedSame expertsGuess per item
6. AdjudicateItems where the two disagree go to a senior expert who sees both reasonsAdjudicatorFinal decision and deciding reason
7. Unblind and scoreReveal origins; compute challenge rates by origin, control catch rate and origin-guess accuracyData leadReview readout
8. Feed backMap each challenge code to a recipe change or a grounding fix and regenerateData lead with generator ownerChange log linked to codes
9. DisclosePublish recipe, grounding, selection steps and review readout with the tasksData leadDatasheet section

Which challenge codes should reviewers use?

Fixed codes make rejections countable and point each one at a fix. Keep the list short, define each with a test a reviewer can apply, and allow a free-text reason alongside the code. The eight below cover most failures in generated business tasks.

At a glance

Which challenge codes should reviewers use

Ask challengers to pick the single code that best explains a rejection and to note a second only when two independent problems exist. Two codes per item on average usually means the codes overlap and need sharper definitions. Record accepts with the same care: an accept with no reason is fine, but the time spent per item tells you whether an expert actually worked the task, since a reviewer who accepts a multi-document reconciliation in a few seconds has not checked it.

  • Rule inconsistency: the task's answer contradicts a rule stated elsewhere in the same task or in the grounding documents.
  • Outdated or wrong policy: the task applies a policy, rate table or threshold that was not in force on the stated date.
  • Impossible state: the records describe something that cannot happen in the system, such as a payment posted before its invoice exists.
  • Missing exception: the task presents the standard path where a practitioner would expect an exception to apply.
  • Template tell: the task repeats a structure or phrasing so closely that a model could learn the template rather than the work.
  • Wrong difficulty: the task is trivially easy or requires information it does not provide.
  • Grounding gap: a fact in the task cannot be traced to any grounding source.
  • Privacy leak: the task reproduces identifiable details from a real record used for grounding.

How blinding with real tasks and planted controls keeps the review honest

Untouched real tasks are the yardstick, and they only work if reviewers cannot see which items they are. Strip origin labels, normalize formatting and remove details that only one source would contain. The real tasks must be held out from the generator entirely, or the comparison measures memory rather than realism.

Unblinding gives three readings. If experts challenge generated tasks far more often than real ones, the codes show where the generator falls short. If they challenge real tasks nearly as often, the challenge is too harsh or the codes are unclear, and the briefing needs work. If experts guess origin much better than chance, the generated tasks have a recognizable style, which is a distribution mismatch even when each task is individually correct.

Planted controls test the reviewers. Insert a few generated tasks with a known defect, one per challenge code, and track whether each is caught. A reviewer who passes planted defects is approving rather than challenging, and that reviewer's other decisions should be re-read.

When challengers disagree, who decides and what gets recorded?

Disagreement between two independent experts is information, not noise to be averaged away. Send the item to an adjudicator who sees both codes and both reasons, decides, and writes the deciding reason. Do not resolve by majority with a third quick vote, because the point is to learn which reading of the rules is correct.

Track agreement between the two challengers by code. Low agreement on one code usually means the code is ambiguous or the underlying rule is genuinely contested in the business, and either finding belongs in the review readout. When the adjudicator decides that the rule itself is ambiguous, mark the task as contested rather than forcing a label, and keep it out of evaluation sets where a single correct answer is assumed.

The adjudicator should not be the person who wrote the generation prompts or chose the grounding documents, because that person has a stake in the recipe being right. Rotate adjudicators across batches when the set is large, and have them re-adjudicate a small sample of each other's decisions so their own drift becomes visible in the readout.

Worked example: an illustrative accounts payable exception set

The readout below is illustrative and does not describe any real company or vendor. A team generated accounts payable exception tasks, such as matching invoices to purchase orders and receipts, and reviewed a queue of 400 generated tasks, 100 untouched real tasks and 16 planted controls with two challengers each.

The readout shows the generator handled price and quantity mismatches well but underproduced partial receipts and applied an old approval threshold. Experts guessed origin correctly well above chance, mainly from uniform vendor names, a template tell that the next recipe version fixed.

Worked example: an illustrative accounts payable exception set
Measure (illustrative)Generated tasksReal tasksReading
Items challenged after adjudication22%5%Clear gap; codes point to the fixes
Most common codeOutdated policyWrong difficultyGenerator grounded on a superseded approval rule
Missing exception code9% of items1% of itemsPartial receipts and credit memos underproduced
Origin guessed correctly81%77%Recognizable style; vendor names too uniform
Planted controls caught15 of 16Not applicableReviewers engaged; the missed control was re-briefed

Which real exceptions does the synthetic mixture fail to represent, and when should you stop?

After unblinding, list the exception types that appear in the real holdout but rarely or never in the generated set. That list, not the overall challenge rate, decides whether the synthetic tasks can carry a training or evaluation job alone.

There is an honest exception to this protocol: for genuinely new tasks with no real counterpart, there is nothing to blind against. In that case, have experts write a small gold set first and review generated tasks against it, and say in the disclosure that no real holdout existed.

Stop and narrow the scope when challenge rates on generated tasks stay well above real tasks after two recipe revisions, when experts cannot agree on the rules themselves, or when privacy leak codes appear at all. Each of those means more generation will not fix the problem; more real records or a narrower task definition will.

Finding untouched real tasks to review against

The protocol depends on real tasks the generator never saw, and those are often the hardest part to obtain. A buyer can ask SourceX to find such a holdout, describing its scope, volume, format, timing and licensing terms; SourceX then approaches U.S. businesses for it rather than drawing on a stock of datasets.

Because the contributing business approves the scope and terms and signs the agreement before any record changes hands, the buyer can document where the real holdout came from in the same disclosure as the synthetic recipe.

Frequently asked questions

How many experts does a synthetic task review need?

At least two independent challengers per item and one senior adjudicator for disagreements. Adversarial datasets such as ANLI used two verifiers and added a third only when the first two disagreed. More reviewers help when the rules are contested, but independence matters more than headcount.

Can an LLM judge replace the expert challengers?

It can pre-screen for obvious defects, but it cannot be the yardstick, because it may share the generator's blind spots. Keep human experts on the blinded comparison with real tasks and use planted controls to check any automated screen as well.

What if experts reject real tasks too?

That is expected at a low rate, since real work contains errors and ambiguous cases. If real tasks are challenged nearly as often as generated ones, the codes are too broad or the briefing too harsh, and the review should be recalibrated before its results are trusted.

Should rejected synthetic tasks be fixed by hand?

Fix the recipe, not the individual task, whenever a challenge code repeats. Hand edits hide the generator's weakness and make the disclosed recipe inaccurate. One-off hand fixes are acceptable when they are logged as such.

How large should the real holdout be?

Large enough to contain the exception types the generated tasks are meant to cover, with several examples of each. Size it by exception coverage rather than by a fixed ratio to the synthetic set, and keep it entirely separate from any records used to ground the generator.

Sources

  • Indiscriminate training on model-generated content causes defects in which the tails of the original distribution disappear. Source
  • Synthetic data practice requires checking factuality, fidelity and unbiasedness, and synthetic data can amplify or introduce biases. Source
  • ANLI development and test examples were verified by two verifiers, or three when the first two disagreed, and annotators wrote reasons. Source
  • Dynabench collects data with humans and models in the loop because models that do well on benchmarks still fail simple challenge examples. Source
  • Datasheets for Datasets proposes documenting a dataset's motivation, composition, collection process and recommended uses. Source
  • NIST AI 600-1 is the generative AI profile of the AI RMF and names AI red-teaming among structured exercises for measuring risk. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies