Skip to content

AI uses for records

Synthetic data hard negative design: start from real business confusions

By SourceX Editorial · Updated

10 sections

Short answer

Build hard negatives from confusions your records already resolved: tickets moved between queues, invoices held as duplicates and then released, merged or reopened records. Name the one attribute that decided the right answer, generate or edit negatives that differ only in that attribute, label each with a written guide, and test the result against a locked sample of real confusions that no generator or prompt has touched.

Agent run

Synthetic data hard negative

This page covers synthetic data hard negative design: start from real business confusions.Illustrative

Key takeaways

  • The best source of hard negatives is the correction trail in business systems: reassignments, reclassifications, duplicate holds, merges and reviewer fixes.
  • Every negative needs a named deciding attribute; if more than one thing differs, it is not a clean hard negative.
  • The hardest mined candidates are often unlabeled positives, so filter and label them rather than trusting similarity alone.
  • Use generation to vary real confusions, never to invent the confusions themselves, and keep template and ungrounded outputs out of evaluation.
  • Lock a stratified real holdout before generation and judge the synthetic negatives by their effect on it, per confusion type.
  • Disclose each negative's derivation class, generator, grounding source and policy date so reusers know what they are training on.

How do you construct hard negatives from realistic business confusions?

Start from mistakes that people or systems already made in real records, not from what a generator thinks is similar. Collect pairs that were actually confused, such as a ticket routed to the wrong queue and then moved, or an invoice flagged as a duplicate and then approved. Name the single attribute that decided the correct answer, then create negatives that match the anchor on everything except that attribute, and check each one with a human label.

Synthetic generation is useful at the third step, producing many controlled variations of a real confusion. It is risky at the first, because a generator left to imagine confusions will produce the ones that are easy to describe rather than the ones that happen. And every generated set needs to be scored against a sample of untouched real cases that no generator, prompt or annotator guideline has seen.

This guide is for AI teams building retrieval, classification, routing or extraction models on business data, and for the domain reviewers who label their negatives. The annotation guide and validation design below can be copied into a labeling project as they stand.

What makes a negative hard, and when is it secretly a positive?

A hard negative is an example that looks like a match to a model or a quick reader but is wrong for a specific, statable reason. In retrieval research the standard source has been lexical overlap: the Dense Passage Retrieval paper used BM25 negatives, top passages that match most of the question's tokens but do not contain the answer, and reported that adding one such negative per question improved results while a second did not help further.

The danger is that the hardest candidates are often not negatives at all. The RocketQA authors manually checked top-retrieved passages that were not labeled positive in a common benchmark and found that about 70% were in fact positives or highly relevant, so they used a cross-encoder to filter likely false negatives before training. Business corpora have the same trap in a sharper form: two near-identical policies, a superseded and current contract, or the same customer under two account IDs.

Mining tools now expose controls for exactly this risk. The hard negative mining utility in the Sentence Transformers library lets users skip the top-ranked candidates with range_min to avoid labeling true positives as negatives, cap similarity with max_score, and require a negative to sit a fixed or relative margin below the positive. Those settings reduce false negatives but cannot tell you which attribute makes a candidate wrong; only a domain label can.

Where real confusions leave a trail in business systems

Most business systems record corrections, and every correction is a confusion someone already resolved. The table lists common signals and the pair each yields. The system names are examples of where such histories usually live; check what your own instance actually records before relying on it.

Where real confusions leave a trail in business systems
Confusion signalWhere it is usually recordedPair it yieldsDeciding attribute to name
Ticket moved to a different queue or teamHelp desk assignment history, for example in Zendesk or ServiceNowOriginal routing versus final routing for the same ticketProduct line, contract tier or issue type that the first router missed
Case or claim type changed after intakeCRM or claims case field history, for example in SalesforceTicket text under the wrong type versus the right oneA phrase, date or amount that changes the category
Invoice held as possible duplicate, then releasedAP workflow in an ERP such as NetSuiteTwo invoices from one vendor that look alikeService period, PO line or invoice number pattern
Records merged as duplicates, or a merge reversedCRM merge logs, for example in HubSpot or SalesforceTwo contact or account recordsLegal entity, location or role that separates them
Issue closed as duplicate, then reopenedIssue tracker history, for example in JiraTwo bug reports with similar symptomsVersion, component or reproduction step
Answer corrected by a reviewer or supervisorQA scorecards and coaching notesAgent's first answer versus corrected answerPolicy clause or effective date that was misapplied
Search or knowledge article opened, then abandoned for anotherKnowledge base or search logsQuery with a near-miss article and the article that solved itProduct version, region or customer type

The hard negative annotation guide, field by field

Each negative in the set gets one record with these fields. The first four come from the confusion itself; the rest are filled during generation and labeling. Annotators should see the deciding attribute only after they have given their own label, so the field guides adjudication rather than leading the label.

The hard negative annotation guide, field by field
FieldWhat it holdsAllowed values or formatInstruction to the annotator
anchor_id and positive_idThe query, ticket or record and its correct matchIDs from the source system, pseudonymizedConfirm the positive is correct before judging any negative
confusion_sourceThe real event that showed these two were confusedReassignment, reclassification, duplicate hold, merge, reopen, reviewer correctionDo not label a negative that has no recorded confusion as real-mined
deciding_attributeThe one fact that makes the negative wrongShort phrase naming a field or clauseIf more than one attribute differs, mark the negative as multi-difference
derivationHow the negative was producedReal-mined, real-edited, generated-grounded, generated-ungroundedNever change this field; it is set by the pipeline
edit_descriptionWhat changed between source record and negativeFree text, one sentenceCheck the edit changed only the deciding attribute
labelAnnotator judgmentTrue negative, false negative, ambiguous, invalid recordUse ambiguous when the source data cannot settle it; use invalid for nonsense or broken text
policy_or_rule_dateThe rule version that makes the negative wrongDate or version stringJudge against this version, not current policy
privacy_screenWhether names, account numbers and free-text personal details were removed or replacedPassed, failed, not applicableStop and flag any record that still identifies a person
adjudicationSecond label and final decisionAgree, disagree with resolution, escalatedWrite the reason when you overturn a first label

Choosing a generation recipe for each confusion

Different confusions call for different recipes, and the recipe decides how much the result can be trusted. Contrast sets, introduced by Gardner and colleagues, are small edits that annotators make to test examples so that the gold label changes, built without a model in the loop; they are the cleanest template for real-edited negatives. CheckList, a behavioral testing method by Ribeiro and colleagues, adds minimum functionality tests, invariance tests that should not change the prediction and directional tests where the output should move a known way, which suit template-based variations of a rule.

Choosing a generation recipe for each confusion
RecipeUse it whenMain riskControl
Mine real pairs from the confusion trailThe system already recorded enough resolved confusionsFew examples for rare confusion typesReport counts per type; do not pad with other recipes silently
Edit a real record so the deciding attribute flipsYou need more examples of one confusion with full realismEdits leave artifacts a model can spot, such as unusual phrasingHave a second annotator try to tell edited from real records
Template variations of a ruleThe confusion is driven by a clear rule such as a date or thresholdTemplates are narrower than real languageKeep template negatives out of the evaluation set
Generate with a model, grounded on real recordsYou need volume and can cite the source record for each outputGenerator drifts from the source or invents attributesStore the grounding record ID and check the deciding attribute exists in it
Generate with a model, ungroundedOnly for brainstorming confusion types to look forProduces plausible confusions that never happenNever use as training or evaluation data without a real match

Designing the validation sample against untouched real cases

The validation sample answers one question: do models trained or tested with the synthetic negatives behave better on real confusions they have never seen? It must be drawn before generation starts, from records the generator, the prompts and the annotation guide examples never touched, and it should stay locked until the final comparison. Sample sizes below are illustrative.

Stratify the holdout by confusion type so that rare types are represented, for example a fixed minimum such as 30 real pairs per type where the records allow, and report types that fall short rather than filling them with generated cases. Have two annotators label every holdout pair independently with the same guide, and resolve disagreements before scoring.

Plant controls in every labeling batch: a few pairs that are known positives presented as candidate negatives, and a few obviously unrelated pairs. If annotators label planted positives as negatives, the guide or the deciding attributes are unclear and the batch needs review. Finally, compare models trained with and without the synthetic negatives on the real holdout, and borrow the contrast set idea of consistency by checking whether a model gets an anchor, its positive and all of its edited negatives right together.

  • Fix acceptance criteria before generation: the minimum annotator agreement and the minimum holdout improvement that would justify keeping the synthetic negatives.
  • Keep a log of every decision that touched the holdout, including any record removed for privacy.
  • Report results per confusion type, not only overall, because a gain on common types can hide a loss on rare ones.
  • Re-draw part of the holdout when the source policy, product catalog or routing rules change.

Business confusions a grounded generator still tends to miss

Even a grounded generator tends to miss the confusions that depend on context it never sees. Check the real holdout for these before trusting a mixture, and say in the dataset card which ones are thin.

  • Confusions created by timing: a request that was correct under last quarter's policy and wrong under this one.
  • Multi-record confusions where the deciding fact sits in a related record, such as a contract amendment rather than the contract.
  • Formatting and channel artifacts: scanned PDFs, forwarded email chains and pasted screenshots that generators render too cleanly.
  • Confusions between customers or entities with near-identical names, which privacy screening can erase if it replaces both with the same placeholder.
  • Long-tail categories that the confusion trail shows only a handful of times; synthetic volume there reflects the generator, not the business.
  • Cases a human never resolved: if no one corrected the confusion, the trail does not record it, and neither will negatives mined from it.

Illustrative walk-through: duplicate invoices versus legitimate repeats

The numbers here are illustrative. An accounts payable team wants a model that flags duplicate invoices without blocking legitimate repeat charges, such as monthly service fees from the same vendor for the same amount. Its AP workflow shows a few hundred invoices held as possible duplicates in a year, about half of which were released as legitimate.

The released holds are the real hard negatives: same vendor, same amount, different service period. The team names the deciding attribute for each, mostly service period, sometimes PO line, and sets aside a stratified holdout of released and confirmed duplicate pairs before doing anything else. It then edits real invoices so that only the service period differs, and generates grounded variations of invoice layouts with each output tied to its source invoice.

Annotation catches two problems. Some edited invoices changed both the period and the invoice number, so they are relabeled multi-difference and excluded. And annotators repeatedly mark planted known duplicates as negatives when the vendor reissued an invoice with a corrected date; the guide is updated to say a reissue with the same service period is a duplicate.

The exception: one vendor bills partial shipments against a single PO line with identical amounts. Neither the trail nor the generator distinguishes these from duplicates, so the team excludes that vendor's invoices from the synthetic set and routes them to a rule instead of the model.

Disclosing how each negative was made, and when to stop generating

Anyone reusing the set needs to know how each negative was produced. Datasheets for Datasets, by Gebru and colleagues, asks dataset creators how data was acquired, whether it is a sample and how it was drawn, what preprocessing and labeling were done, and whether the raw data was kept. For a hard negative set, answer those questions per derivation class, and add the generator, prompt version, grounding sources, policy date and the share of negatives in each class.

Stop generating, or narrow the set, when any of these appear: annotators cannot agree on the deciding attribute for a confusion type; the real holdout shows no gain or a loss for a type; edited records are reliably spotted as edited; privacy screening is removing the very attribute that decides the label; or the confusion trail for a type is too small to ground anything. At that point the fix is more real records, a domain expert's review of the guide, or dropping that type, not more generation.

Sourcing the real confusions a generator cannot invent

The hard part of this method is the confusion trail itself: years of reassignments, holds, merges and reviewer corrections, with the context to label them. Those records sit inside operating companies, not public corpora.

A confusion trail is something SourceX can be asked to find: it sources records from U.S. businesses on request, not from a catalog. Its request form collects the scope, volume, language, format, timing and licensing terms a team needs, after which SourceX takes care of qualification, rights review, contracting, preparation and delivery with the record holders.

Frequently asked questions

What is a hard negative in synthetic data?

It is an example built to look like a correct match while being wrong for one specific reason, such as a different service period or policy version. In synthetic data, the negative is generated or edited rather than found, so its derivation and the attribute that makes it wrong should be recorded.

How many hard negatives per example should a training set include?

There is no general number. The Dense Passage Retrieval paper found one BM25 negative per question helped and a second did not in its setting, but business tasks differ, so decide by testing on a locked real holdout rather than by a fixed ratio.

How do you avoid false negatives when mining hard negatives?

Skip the very top-ranked candidates, require a margin below the positive, rescore with a stronger model and, most importantly, have a domain annotator confirm each negative against a named deciding attribute. Planted known positives in each batch show whether annotators are catching them.

Can a language model generate realistic business confusions on its own?

It can suggest confusion types worth looking for, but its outputs reflect what is easy to imagine, not what happens in a given business. Use ungrounded output only as a checklist and build training and evaluation negatives from real records or grounded edits of them.

Should synthetic hard negatives be used in the evaluation set?

Keep evaluation on real, untouched confusions wherever possible. Edited real records can supplement a thin type if they are labeled as edited, but template and ungrounded generated negatives make evaluation measure the generator rather than the task.

Sources

  • The Dense Passage Retrieval paper defines BM25 negatives as top passages that match most question tokens but lack the answer, and reports one such negative helped while two did not help further. Source
  • RocketQA reports that about 70% of top-retrieved passages not labeled positive in a manual check were actually positives or highly relevant, and uses a cross-encoder to remove likely false negatives. Source
  • The Sentence Transformers hard negative mining utility offers range_min to skip top candidates that may be positives, max_score, and absolute or relative margins below the positive. Source
  • Contrast sets are small annotator edits to test instances that change the gold label, made without a model in the loop, with a consistency metric across each set. Source
  • CheckList defines minimum functionality, invariance and directional expectation tests for behavioral testing of NLP models. Source
  • Datasheets for Datasets asks how data was acquired, whether it is a sample and how it was drawn, what preprocessing and labeling were done, and whether raw data was kept. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies