AI uses for records
Synthetic data hard negative design: start from real business confusions
By SourceX Editorial · Updated
10 sections
Short answer
Build hard negatives from confusions your records already resolved: tickets moved between queues, invoices held as duplicates and then released, merged or reopened records. Name the one attribute that decided the right answer, generate or edit negatives that differ only in that attribute, label each with a written guide, and test the result against a locked sample of real confusions that no generator or prompt has touched.
Agent run
Synthetic data hard negative
Key takeaways
- The best source of hard negatives is the correction trail in business systems: reassignments, reclassifications, duplicate holds, merges and reviewer fixes.
- Every negative needs a named deciding attribute; if more than one thing differs, it is not a clean hard negative.
- The hardest mined candidates are often unlabeled positives, so filter and label them rather than trusting similarity alone.
- Use generation to vary real confusions, never to invent the confusions themselves, and keep template and ungrounded outputs out of evaluation.
- Lock a stratified real holdout before generation and judge the synthetic negatives by their effect on it, per confusion type.
- Disclose each negative's derivation class, generator, grounding source and policy date so reusers know what they are training on.
How do you construct hard negatives from realistic business confusions?
Start from mistakes that people or systems already made in real records, not from what a generator thinks is similar. Collect pairs that were actually confused, such as a ticket routed to the wrong queue and then moved, or an invoice flagged as a duplicate and then approved. Name the single attribute that decided the correct answer, then create negatives that match the anchor on everything except that attribute, and check each one with a human label.
Synthetic generation is useful at the third step, producing many controlled variations of a real confusion. It is risky at the first, because a generator left to imagine confusions will produce the ones that are easy to describe rather than the ones that happen. And every generated set needs to be scored against a sample of untouched real cases that no generator, prompt or annotator guideline has seen.
This guide is for AI teams building retrieval, classification, routing or extraction models on business data, and for the domain reviewers who label their negatives. The annotation guide and validation design below can be copied into a labeling project as they stand.
What makes a negative hard, and when is it secretly a positive?
A hard negative is an example that looks like a match to a model or a quick reader but is wrong for a specific, statable reason. In retrieval research the standard source has been lexical overlap: the Dense Passage Retrieval paper used BM25 negatives, top passages that match most of the question's tokens but do not contain the answer, and reported that adding one such negative per question improved results while a second did not help further.
The danger is that the hardest candidates are often not negatives at all. The RocketQA authors manually checked top-retrieved passages that were not labeled positive in a common benchmark and found that about 70% were in fact positives or highly relevant, so they used a cross-encoder to filter likely false negatives before training. Business corpora have the same trap in a sharper form: two near-identical policies, a superseded and current contract, or the same customer under two account IDs.
Mining tools now expose controls for exactly this risk. The hard negative mining utility in the Sentence Transformers library lets users skip the top-ranked candidates with range_min to avoid labeling true positives as negatives, cap similarity with max_score, and require a negative to sit a fixed or relative margin below the positive. Those settings reduce false negatives but cannot tell you which attribute makes a candidate wrong; only a domain label can.
Where real confusions leave a trail in business systems
Most business systems record corrections, and every correction is a confusion someone already resolved. The table lists common signals and the pair each yields. The system names are examples of where such histories usually live; check what your own instance actually records before relying on it.
| Confusion signal | Where it is usually recorded | Pair it yields | Deciding attribute to name |
|---|---|---|---|
| Ticket moved to a different queue or team | Help desk assignment history, for example in Zendesk or ServiceNow | Original routing versus final routing for the same ticket | Product line, contract tier or issue type that the first router missed |
| Case or claim type changed after intake | CRM or claims case field history, for example in Salesforce | Ticket text under the wrong type versus the right one | A phrase, date or amount that changes the category |
| Invoice held as possible duplicate, then released | AP workflow in an ERP such as NetSuite | Two invoices from one vendor that look alike | Service period, PO line or invoice number pattern |
| Records merged as duplicates, or a merge reversed | CRM merge logs, for example in HubSpot or Salesforce | Two contact or account records | Legal entity, location or role that separates them |
| Issue closed as duplicate, then reopened | Issue tracker history, for example in Jira | Two bug reports with similar symptoms | Version, component or reproduction step |
| Answer corrected by a reviewer or supervisor | QA scorecards and coaching notes | Agent's first answer versus corrected answer | Policy clause or effective date that was misapplied |
| Search or knowledge article opened, then abandoned for another | Knowledge base or search logs | Query with a near-miss article and the article that solved it | Product version, region or customer type |
The hard negative annotation guide, field by field
Each negative in the set gets one record with these fields. The first four come from the confusion itself; the rest are filled during generation and labeling. Annotators should see the deciding attribute only after they have given their own label, so the field guides adjudication rather than leading the label.
| Field | What it holds | Allowed values or format | Instruction to the annotator |
|---|---|---|---|
| anchor_id and positive_id | The query, ticket or record and its correct match | IDs from the source system, pseudonymized | Confirm the positive is correct before judging any negative |
| confusion_source | The real event that showed these two were confused | Reassignment, reclassification, duplicate hold, merge, reopen, reviewer correction | Do not label a negative that has no recorded confusion as real-mined |
| deciding_attribute | The one fact that makes the negative wrong | Short phrase naming a field or clause | If more than one attribute differs, mark the negative as multi-difference |
| derivation | How the negative was produced | Real-mined, real-edited, generated-grounded, generated-ungrounded | Never change this field; it is set by the pipeline |
| edit_description | What changed between source record and negative | Free text, one sentence | Check the edit changed only the deciding attribute |
| label | Annotator judgment | True negative, false negative, ambiguous, invalid record | Use ambiguous when the source data cannot settle it; use invalid for nonsense or broken text |
| policy_or_rule_date | The rule version that makes the negative wrong | Date or version string | Judge against this version, not current policy |
| privacy_screen | Whether names, account numbers and free-text personal details were removed or replaced | Passed, failed, not applicable | Stop and flag any record that still identifies a person |
| adjudication | Second label and final decision | Agree, disagree with resolution, escalated | Write the reason when you overturn a first label |
Choosing a generation recipe for each confusion
Different confusions call for different recipes, and the recipe decides how much the result can be trusted. Contrast sets, introduced by Gardner and colleagues, are small edits that annotators make to test examples so that the gold label changes, built without a model in the loop; they are the cleanest template for real-edited negatives. CheckList, a behavioral testing method by Ribeiro and colleagues, adds minimum functionality tests, invariance tests that should not change the prediction and directional tests where the output should move a known way, which suit template-based variations of a rule.
| Recipe | Use it when | Main risk | Control |
|---|---|---|---|
| Mine real pairs from the confusion trail | The system already recorded enough resolved confusions | Few examples for rare confusion types | Report counts per type; do not pad with other recipes silently |
| Edit a real record so the deciding attribute flips | You need more examples of one confusion with full realism | Edits leave artifacts a model can spot, such as unusual phrasing | Have a second annotator try to tell edited from real records |
| Template variations of a rule | The confusion is driven by a clear rule such as a date or threshold | Templates are narrower than real language | Keep template negatives out of the evaluation set |
| Generate with a model, grounded on real records | You need volume and can cite the source record for each output | Generator drifts from the source or invents attributes | Store the grounding record ID and check the deciding attribute exists in it |
| Generate with a model, ungrounded | Only for brainstorming confusion types to look for | Produces plausible confusions that never happen | Never use as training or evaluation data without a real match |
Designing the validation sample against untouched real cases
The validation sample answers one question: do models trained or tested with the synthetic negatives behave better on real confusions they have never seen? It must be drawn before generation starts, from records the generator, the prompts and the annotation guide examples never touched, and it should stay locked until the final comparison. Sample sizes below are illustrative.
Stratify the holdout by confusion type so that rare types are represented, for example a fixed minimum such as 30 real pairs per type where the records allow, and report types that fall short rather than filling them with generated cases. Have two annotators label every holdout pair independently with the same guide, and resolve disagreements before scoring.
Plant controls in every labeling batch: a few pairs that are known positives presented as candidate negatives, and a few obviously unrelated pairs. If annotators label planted positives as negatives, the guide or the deciding attributes are unclear and the batch needs review. Finally, compare models trained with and without the synthetic negatives on the real holdout, and borrow the contrast set idea of consistency by checking whether a model gets an anchor, its positive and all of its edited negatives right together.
- Fix acceptance criteria before generation: the minimum annotator agreement and the minimum holdout improvement that would justify keeping the synthetic negatives.
- Keep a log of every decision that touched the holdout, including any record removed for privacy.
- Report results per confusion type, not only overall, because a gain on common types can hide a loss on rare ones.
- Re-draw part of the holdout when the source policy, product catalog or routing rules change.
Business confusions a grounded generator still tends to miss
Even a grounded generator tends to miss the confusions that depend on context it never sees. Check the real holdout for these before trusting a mixture, and say in the dataset card which ones are thin.
- Confusions created by timing: a request that was correct under last quarter's policy and wrong under this one.
- Multi-record confusions where the deciding fact sits in a related record, such as a contract amendment rather than the contract.
- Formatting and channel artifacts: scanned PDFs, forwarded email chains and pasted screenshots that generators render too cleanly.
- Confusions between customers or entities with near-identical names, which privacy screening can erase if it replaces both with the same placeholder.
- Long-tail categories that the confusion trail shows only a handful of times; synthetic volume there reflects the generator, not the business.
- Cases a human never resolved: if no one corrected the confusion, the trail does not record it, and neither will negatives mined from it.
Illustrative walk-through: duplicate invoices versus legitimate repeats
The numbers here are illustrative. An accounts payable team wants a model that flags duplicate invoices without blocking legitimate repeat charges, such as monthly service fees from the same vendor for the same amount. Its AP workflow shows a few hundred invoices held as possible duplicates in a year, about half of which were released as legitimate.
The released holds are the real hard negatives: same vendor, same amount, different service period. The team names the deciding attribute for each, mostly service period, sometimes PO line, and sets aside a stratified holdout of released and confirmed duplicate pairs before doing anything else. It then edits real invoices so that only the service period differs, and generates grounded variations of invoice layouts with each output tied to its source invoice.
Annotation catches two problems. Some edited invoices changed both the period and the invoice number, so they are relabeled multi-difference and excluded. And annotators repeatedly mark planted known duplicates as negatives when the vendor reissued an invoice with a corrected date; the guide is updated to say a reissue with the same service period is a duplicate.
The exception: one vendor bills partial shipments against a single PO line with identical amounts. Neither the trail nor the generator distinguishes these from duplicates, so the team excludes that vendor's invoices from the synthetic set and routes them to a rule instead of the model.
Disclosing how each negative was made, and when to stop generating
Anyone reusing the set needs to know how each negative was produced. Datasheets for Datasets, by Gebru and colleagues, asks dataset creators how data was acquired, whether it is a sample and how it was drawn, what preprocessing and labeling were done, and whether the raw data was kept. For a hard negative set, answer those questions per derivation class, and add the generator, prompt version, grounding sources, policy date and the share of negatives in each class.
Stop generating, or narrow the set, when any of these appear: annotators cannot agree on the deciding attribute for a confusion type; the real holdout shows no gain or a loss for a type; edited records are reliably spotted as edited; privacy screening is removing the very attribute that decides the label; or the confusion trail for a type is too small to ground anything. At that point the fix is more real records, a domain expert's review of the guide, or dropping that type, not more generation.
Sourcing the real confusions a generator cannot invent
The hard part of this method is the confusion trail itself: years of reassignments, holds, merges and reviewer corrections, with the context to label them. Those records sit inside operating companies, not public corpora.
A confusion trail is something SourceX can be asked to find: it sources records from U.S. businesses on request, not from a catalog. Its request form collects the scope, volume, language, format, timing and licensing terms a team needs, after which SourceX takes care of qualification, rights review, contracting, preparation and delivery with the record holders.
Frequently asked questions
What is a hard negative in synthetic data?
It is an example built to look like a correct match while being wrong for one specific reason, such as a different service period or policy version. In synthetic data, the negative is generated or edited rather than found, so its derivation and the attribute that makes it wrong should be recorded.
How many hard negatives per example should a training set include?
There is no general number. The Dense Passage Retrieval paper found one BM25 negative per question helped and a second did not in its setting, but business tasks differ, so decide by testing on a locked real holdout rather than by a fixed ratio.
How do you avoid false negatives when mining hard negatives?
Skip the very top-ranked candidates, require a margin below the positive, rescore with a stronger model and, most importantly, have a domain annotator confirm each negative against a named deciding attribute. Planted known positives in each batch show whether annotators are catching them.
Can a language model generate realistic business confusions on its own?
It can suggest confusion types worth looking for, but its outputs reflect what is easy to imagine, not what happens in a given business. Use ungrounded output only as a checklist and build training and evaluation negatives from real records or grounded edits of them.
Should synthetic hard negatives be used in the evaluation set?
Keep evaluation on real, untouched confusions wherever possible. Edited real records can supplement a thin type if they are labeled as edited, but template and ungrounded generated negatives make evaluation measure the generator rather than the task.
Sources
- The Dense Passage Retrieval paper defines BM25 negatives as top passages that match most question tokens but lack the answer, and reports one such negative helped while two did not help further. Source
- RocketQA reports that about 70% of top-retrieved passages not labeled positive in a manual check were actually positives or highly relevant, and uses a cross-encoder to remove likely false negatives. Source
- The Sentence Transformers hard negative mining utility offers range_min to skip top candidates that may be positives, max_score, and absolute or relative margins below the positive. Source
- Contrast sets are small annotator edits to test instances that change the gold label, made without a model in the loop, with a consistency metric across each set. Source
- CheckList defines minimum functionality, invariance and directional expectation tests for behavioral testing of NLP models. Source
- Datasheets for Datasets asks how data was acquired, whether it is a sample and how it was drawn, what preprocessing and labeling were done, and whether raw data was kept. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.