Skip to content

Definitions and comparisons

Expert demonstration versus correction data: which should you buy?

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Buy expert demonstrations when the model cannot produce a usable first attempt at a task, and expert corrections when its drafts are mostly right but fail in specific, repeatable ways. Decide on evidence rather than habit: name the failure cohort, buy a small matched batch of each at the same expert-hour budget, train both the same way, and compare gains on an untouched evaluation set while checking for regressions elsewhere.

Process

Expert demonstration

This page covers expert demonstration versus correction data: which should you buy.Illustrative

Key takeaways

  • A demonstration is an expert's complete answer written from scratch; a correction is an expert's edit of a model draft, ideally with the reason for the change.
  • Demonstrations fit failures of approach or structure; corrections fit failures that are local, repeatable and visible in the draft.
  • One correction record can feed supervised fine-tuning and preference training, but only if the draft, the edit and the model version are all kept.
  • Edits tend to keep the shape of the draft, so corrections are weak when the draft's structure is the problem.
  • Compare the two at equal expert hours, on the same untouched failure cohort, with paired differences and a regression suite.

Which failure are you buying data to fix?

Start with the failure, because it decides the data type more than any vendor preference does. If the model cannot produce a usable first attempt, such as a coherent claims-denial letter that cites the right policy clause, you need expert demonstrations. If the model's drafts are mostly right but go wrong in the same few places, you need expert corrections.

The two terms are often used loosely, so define them before a purchase order exists. An expert demonstration is a prompt paired with a complete response that a qualified person wrote from scratch. This is the classic supervised fine-tuning input: the InstructGPT work collected labeler-written demonstrations and fine-tuned GPT-3 on them with supervised learning before any preference training.

An expert correction is a model draft paired with the expert's edited version of it, and ideally a short statement of what was wrong. The DeFacto dataset is a published example of the richer form: it pairs human demonstrations with corrective instructions, edited summaries and explanations, and its authors report that fine-tuned models used the data to improve factual consistency.

Write the failure down as a cohort before choosing. A cohort is a set of real inputs where the model fails in the same way, for example support tickets that mention two products, or invoices with a credit memo attached. Without a cohort you cannot measure whether either purchase worked, and the decision defaults to whichever data type the team bought last time.

What each purchase gives your training pipeline

A demonstration gives you one training target per prompt, while a correction can give you a target, a contrast and a reason. That difference matters because the training formats you plan to use determine how much value each record carries.

Hugging Face's TRL library documents the formats plainly. A prompt-completion dataset pairs a prompt with a completion for supervised fine-tuning. A preference dataset holds a prompt, a chosen completion and a rejected completion. An unpaired preference dataset holds a single completion with a label saying whether it is preferred.

A correction record maps onto all three if you keep its parts: the edited answer is a completion, the draft and edit form a rejected and chosen pair, and the draft alone can be labelled as not preferred. Research on learning from language feedback follows the same idea: the ILF method generates refinements of an initial output using written feedback, then fine-tunes the model on the chosen refinement.

There is a catch that buyers often miss. A preference pair built from a correction teaches the most when the draft came from the model you are training, or one close to it. Drafts from an older or different model describe someone else's mistakes, so check which model produced them before treating the pairs as on-policy data.

Mixing is a legitimate answer. Some teams buy demonstrations for the prompts where drafts are unusable and corrections for the rest, which uses expert time where each format is strongest. If you mix, keep a field that records which format each record came from, so a later regression can be traced to one part of the purchase.

What each purchase gives your training pipeline
Data typeWhat one record holdsTraining formats it can feedMain weakness
Expert demonstrationPrompt and a complete expert-written responsePrompt-completion for supervised fine-tuningNo contrast with the model's own mistakes
Light correctionDraft and minimally edited answerPrompt-completion; chosen and rejected pair if the draft is keptKeeps most of the draft's structure and wording
Full correction with rationaleDraft, edited answer and a written reason for each changePrompt-completion, preference pair, unpaired label, critique trainingSlower to produce and depends on the draft's model version
Correction without the draftEdited answer only, draft discardedPrompt-completion onlyLoses the contrast that made it a correction

Why corrections inherit the shape of the draft

Edited output tends to stay close to the draft it started from, so corrections teach the model to repair its answers rather than to rethink them. The clearest evidence comes from machine translation, where human post-editing of model output has been studied for years.

Antonio Toral's study of post-editese compared post-edited machine translations with human translations and found the post-edited versions simpler, more normalised and more influenced by the source language. Castilho and colleagues found post-editese features especially in light post-editing and in certain text domains. Text an expert corrects is not the same text the expert would have written.

For a buyer, that finding becomes a rule of thumb. When the failure is a wrong fact, a missing field or a misapplied rule inside an otherwise sound answer, anchoring on the draft is harmless and corrections are efficient. When the failure is the approach itself, such as the wrong structure for a legal memo or the wrong troubleshooting order for a field-service ticket, editors will tend to patch the draft and the model will keep the flawed approach.

You can also check this in your own pilot. Measure how much of each corrected answer survives from the draft, for example with a character-level edit distance. If most corrections in the failure cohort are near-total rewrites, you are paying for demonstrations in a slower format.

The matched acquisition decision matrix

The matrix turns the choice into eight observable inputs, each measured on a small sample from the failure cohort before you commit budget. Score each input, then read the pattern: most rows pointing one way is a decision; a split means run the matched pilot in the next section.

Measure the inputs on 50 to 100 cohort items rather than on general traffic, because the right answer for the failing slice is often different from the answer for the model overall. Record the measurement method beside each score so a reviewer can repeat it.

The matched acquisition decision matrix
Decision inputHow to measure itPoints to demonstrations whenPoints to corrections when
Draft usabilityShare of cohort drafts an expert would accept after edits rather than discardMost drafts would be discardedMost drafts are salvageable
Error localityShare of each draft an expert changes, by edit distanceChanges span most of the answerChanges touch a few sentences or fields
Failure typeExpert tags a sample as approach, fact, rule, format or tone errorsApproach or structure errors dominateFact, rule or format errors dominate
Expert minutes per usable itemTimed trial of both tasks on the same promptsWriting is nearly as fast as editingEditing is clearly faster than writing
Need for a contrast signalWhether preference or critique training is plannedOnly supervised fine-tuning is plannedPreference or critique training is planned
Model update cadenceHow often the policy model checkpoint changesCheckpoints change faster than data arrivesCheckpoint stays stable through collection and training
Regression exposureOverlap between cohort skills and skills the model already does wellFix requires new behavior outside current skillsFix is narrow and adjacent skills are strong
Lineage availableWhether draft, model version, editor and timestamps can be keptDrafts cannot be stored or attributedFull draft-to-edit lineage can be kept

Running the matched pilot at equal expert hours

When the matrix splits, buy a small batch of each data type at the same number of expert hours and let the held-out cohort decide. Matching on hours rather than item counts is the point: corrections usually yield more items per hour, and the question is which hour of expert time buys more improvement.

Keep everything else identical so the only difference is the data. Evan Miller's work on adding error bars to evals recommends comparing two models on a shared question set with paired differences, which is exactly the comparison this pilot needs.

Treat expert qualification as part of the match. If senior specialists write the demonstrations while junior reviewers make the corrections, the pilot measures seniority, not data type. Use the same people in both arms where you can, and log each expert's role and years in the domain so a reviewer can see the arms were comparable.

  • Selection rule: draw pilot prompts from the failure cohort by a written rule, such as every third case by date, and never from the evaluation split.
  • Matched budget: give each arm the same expert hours from the same pool of experts, with the assignment of experts to arms randomized or rotated.
  • Same starting point: train both arms from the same base checkpoint, with the same recipe, step count and data mixture apart from the purchased batch.
  • Untouched evaluation: hold out cohort items the experts never saw, plus a general regression suite covering what the model already does well.
  • Paired scoring: score both trained models on the same held-out items and report the per-item difference with a confidence interval, not two separate averages.
  • Lineage: store the draft, the model version that produced it, the editor's pseudonymous ID, timestamps and any rationale for every correction record.
  • Stop rule: write down before training what difference counts as worthwhile, so the result cannot be reinterpreted after the fact.

Which failure cohort improved, and did anything regress?

A purchase is justified only if the target cohort improved and nothing the model already did well got worse. Report both numbers together, because a gain on the failing slice can hide a loss on adjacent tasks.

Forgetting is a documented risk, not a hypothetical one. A study of continual fine-tuning observed catastrophic forgetting across language models from 1B to 7B parameters, with the severity growing as model scale increased. A narrow batch of corrections that over-teaches one fix can produce the same pattern on a smaller scale.

Read the results with the table below. It treats the pilot as a decision, not a leaderboard: the question is which purchase to scale, how to scale it, or whether to stop.

Which failure cohort improved, and did anything regress?
Pilot outcomeWhat it usually meansDecision
Corrections improve the cohort more, no regressionThe failure is local and the drafts were good starting pointsScale corrections and keep full lineage
Demonstrations improve the cohort more, no regressionThe model needed a new approach, not repairsScale demonstrations for this cohort
Both improve similarlyEither works; cost per expert hour decidesBuy the cheaper-per-hour type, or mix and retest
Either improves the cohort but regresses elsewhereThe batch over-teaches one behaviorMix in data covering the regressed skills and rerun
Neither improves beyond the intervalThe cohort is mislabeled, too small, or not a data problemRe-examine the failure before buying more

How draft usability and other assumptions can flip the choice

The assumption that most often flips the decision is draft usability, because it drives both the speed advantage of corrections and their anchoring risk. Show the decision under a pessimistic and an optimistic value of it, not only the point estimate.

Worked scenario (illustrative, counts and hours only). A team's model misroutes support tickets that mention two products. The held-out cohort has 400 tickets, and the general regression suite has 1,500. Each arm gets 60 expert hours. In the timed trial, experts write a demonstration in about 12 minutes and correct a draft in about 5, so the arms yield roughly 300 demonstrations and 720 corrections.

Suppose the paired comparison shows the corrections arm ahead on the cohort, but the 95% interval on the difference runs from slightly negative to clearly positive. That is not yet a decision. The team checks the edit-distance log, finds that about a third of corrections were near-total rewrites, and reruns the comparison on the remaining two thirds of cohort categories, where drafts were usable.

Three other assumptions deserve an explicit line in the write-up: whether the policy checkpoint will change before the purchased data is used, whether the cohort sample reflects production frequency, and whether the experts in both arms were equally qualified. If any of these moves, rerun the matrix rather than defending the old result.

Lineage and rights to settle before either purchase

Both data types need clear provenance, but corrections carry an extra layer: the draft came from a model, and the edit came from a person working under someone's instructions. Ask who generated each draft, under which model terms, and who owns the edited work product before the batch arrives.

Keep four questions separate. Access means the supplier can export the records; privacy permission means personal data in them may be used for this purpose; ownership means who holds rights in the drafts and edits; and contractual licensing rights mean the supplier is allowed to license them to you. De-identification addresses the second question only, and does not by itself create a right to license.

Corrections that come from real business work, such as reviewers fixing AI-drafted replies in a help desk or analysts amending extracted invoice fields, may also contain client details and internal policy text. Those need the same rights and privacy review as any other business record, and counsel should review the specific contracts.

Ask the supplier to state in writing how drafts were generated, whether any customer content was sent to a third-party model to produce them, and whether editors worked under confidentiality terms. Those answers belong in the purchase file next to the pilot results.

Asking for correction trails that already exist

Many companies already hold correction trails, because people edit drafts every day in ticketing, document and finance systems, but the draft is often overwritten when the final version is saved. When you ask for these records, request the draft, the final, the version history and the reason codes, and confirm the draft survives export.

SourceX helps businesses license selected operating records, such as edit histories in help desks or finance tools, and handles sourcing, rights review, contracting, preparation and delivery. Requests are filled from U.S. businesses rather than an existing catalog, so naming the failure cohort, the draft and edit fields you need, the volume and the licensing terms is the best opening.

Frequently asked questions

What is the difference between expert demonstration data and expert correction data?

Demonstration data pairs a prompt with a complete answer an expert wrote from scratch. Correction data pairs a model's draft with the expert's edited version, ideally with the reason for each change. Demonstrations teach a new approach; corrections teach the model to fix specific mistakes it already makes.

Can one correction record be used for both supervised fine-tuning and preference training?

Yes, if the draft and the edit are both kept. The edited answer can serve as a supervised target, and the draft and edit can form a rejected and chosen pair. The pair is most useful when the draft came from the model being trained or one close to it.

Are expert corrections always cheaper than demonstrations?

Not per unit of improvement. Editing is usually faster per item, but when drafts need near-total rewrites the time advantage disappears and the edits still carry the draft's structure. Time both tasks on the same prompts and compare improvement per expert hour.

How large should the matched pilot be?

Large enough that the confidence interval on the paired difference can rule out a gain too small to matter. There is no universal number; the cohort's size, how variable the scores are and the smallest worthwhile gain decide it. Plan the evaluation size before buying the pilot batches.

Do corrections of an older model's drafts still help?

They can still serve as supervised targets, but their value as contrast data drops when the model changes. Drafts from an older checkpoint show mistakes the new model may no longer make. Record the model version on every draft so you can filter or weight stale pairs.

Sources

  • InstructGPT collected labeler-written demonstrations and used them to fine-tune GPT-3 with supervised learning before preference training. Source
  • The DeFacto dataset pairs human demonstrations with corrective instructions, edited summaries and explanations, and fine-tuned models used it to improve factual consistency. Source
  • Imitation learning from language feedback fine-tunes a model on chosen refinements of its initial outputs written with human feedback. Source
  • TRL documents prompt-completion, preference (chosen and rejected) and unpaired preference (completion and label) dataset types. Source
  • Post-edited machine translations are simpler, more normalised and more influenced by the source language than human translations. Source
  • Post-editese features appear especially in light post-editing and in certain domains. Source
  • Evaluations can be analyzed as experiments, including measuring the difference between two models on a shared question set. Source
  • Catastrophic forgetting was observed in language models from 1B to 7B parameters during continual fine-tuning. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies