AI uses for records
Multimodal document negative examples that look relevant but are not
By SourceX Editorial · Updated
10 sections
Short answer
Build multimodal document negative examples by starting from the exact region that answers the query, such as a table cell, footnote, stamp or revision block, then finding pages that match on layout, template and wording but differ in that region. A person confirms the region does not answer, records why, and screens out near-copies that secretly contain the answer before the pair enters training or evaluation.
Document stack
Multimodal document negative
Key takeaways
- Start each hard negative from the exact region that answers the query, not from the retriever's mistakes alone.
- Name confusion families such as same template, superseded revision, reversal document and footnote exclusion, and cover each one.
- Pre-filter candidates scoring close to the positive, then have a person compare the two regions at full resolution.
- Store region references, deciding features, OCR confidence and cross-file links so the set can be re-audited.
- Negatives whose difference lives in a linked record belong in a separate family and in link-aware evaluation.
How do you build retrieval negatives from documents that appear relevant?
Work outward from the evidence, not inward from the retriever's mistakes. For each query, first mark the page and the region that actually answers it. Then look for pages that share everything a model might latch onto, the template, headers, vendor name, layout and vocabulary, but whose corresponding region gives a different answer or none. Those are the hard negatives worth keeping.
Each candidate is checked by a person looking at the rendered page, not only the extracted text, because the deciding difference is often visual: a void stamp, a struck-through line, a different unit in a column header, a footnote that excludes the case. The reviewer records which region differs and how, and rejects any candidate that turns out to answer the query after all.
The output is a set of query, positive region and negative region triples, each with a reason code. That reason code is what makes the set useful later: it tells you which kinds of visual confusion your retriever has learned to resist and which it has not.
Why page-image retrieval needs negatives defined at the region level
Retrievers increasingly read pages as images. ColPali embeds document page images directly with a vision language model and scores them against the query with late interaction, and its ViDoRe benchmark asks whether the right page is retrieved across text, figures, tables and infographics. A negative for such a model is a page image, and two page images from the same template are nearly identical except where the answer lives.
Page-level labels hide that detail. The MMDocIR benchmark separates page-level retrieval from layout-level retrieval, where the target is a specific paragraph, figure or table marked with a bounding box. Its authors also reviewed the page labels it inherited from the earlier MMLongBench-Doc benchmark and reported that 21.3% needed correction, a reminder that even curated page labels are noisy.
If you only know that page 14 is positive and page 15 is negative, you cannot tell whether page 15 is hard because of its table, its header or its photo. Region-level negatives let you train and evaluate on the distinction that matters, and they let a reviewer audit the label in seconds by looking at two boxes side by side.
Which visual confusions make a business document look relevant?
Most hard negatives in business document sets fall into a small number of confusion families. Name them before mining so that the set covers each one deliberately rather than inheriting whatever the current retriever happens to confuse.
| Confusion family | Why it looks relevant | Region that decides | Business example |
|---|---|---|---|
| Same template, different period | Identical layout and headers | Date field or period column | Two monthly statements from one vendor |
| Draft versus executed copy | Same text and clause numbering | Signature block, watermark or stamp | Unsigned and signed versions of a service order |
| Superseded revision | Same drawing or form number | Revision block or version footer | Rev C and Rev D of an equipment drawing |
| Same table, different unit or basis | Same row labels and figures | Column header or table note | Rate sheet per pallet versus per pound |
| Footnote or exclusion | Main text answers the query | Footnote, asterisk note or rider | Coverage schedule with an excluded location |
| Reversal or offset document | Same parties, amounts and references | Document type label or sign of amounts | Invoice and the credit memo that cancels it |
| Chart with different series | Same chart type and title | Legend, axis label or data label | Two quarterly volume charts by region |
| Degraded copy | Same document, poor scan | Region where OCR confidence is low | Faxed copy of a bill of lading |
Pinning the evidence: page, region, table cell or linked file
Record it as precisely as you would record a citation. A region reference should let anyone reopen the file, go to the page and see the same pixels the labeler saw. Region types can follow an established layout scheme: DocLayNet annotates pages with bounding boxes in 11 classes, including Caption, Footnote, Page-header, Picture, Table and Section-header, which is a practical starting vocabulary for business documents.
Tables need more than a box. Record the header path to the cell, for example section, column header and row label, because the same number can sit under two different headers on near-identical pages. For charts, record the legend entry and axis. For stamps and signatures, record presence, text and position.
Some answers depend on a second file. An invoice line only makes sense with the purchase order it references; an email answer may depend on its attachment. When the evidence spans files, store the link as part of the positive, and make sure a negative is not penalized for lacking a link that the positive needed.
The hard negative selection protocol
Lexical matching has long been the default source of hard negatives: Dense Passage Retrieval trained with BM25 passages that match most of the question's tokens but do not contain the answer. For page images that pool is too narrow, because the confusing pages often share layout and template more than words, so the protocol draws on three pools and makes the deciding region explicit.
Run these steps per query. The record fields underneath are the minimum to store per negative; keep them with the pair so the set can be re-audited after a model, OCR engine or file version changes.
- 1. Mark the positive: file ID, file version or hash, page number, bounding box, region class and, for tables, the header path to the answering cell.
- 2. Describe the deciding feature in one sentence, such as 'period end date in header' or 'per-pound unit in column note'.
- 3. Generate candidates from three pools: pages from the same template or form number, pages ranked highly by the current retriever, and pages from the same account, vendor or matter.
- 4. Pre-filter candidates whose retriever score sits too close to the positive's score, since they are disproportionately unlabeled positives.
- 5. Show the reviewer the positive region and the corresponding candidate region side by side at full resolution, with the extracted text and its OCR confidence beneath.
- 6. Label each candidate as hard negative, easy negative, unlabeled positive or unclear, and assign a confusion family from the table above.
- 7. Reject unlabeled positives and promote them to additional positives; send unclear cases to a records expert.
- 8. Balance the final set across confusion families so no single family dominates, and cap negatives per query so one long binder does not swamp the set.
- 9. Split by template, account and time so the same form never appears as a negative in training and a positive in evaluation.
| Field | What to record | Why it matters |
|---|---|---|
| Region reference | File ID, version hash, page, bounding box, region class | Lets an auditor see the exact evidence |
| Deciding feature | One-sentence difference from the positive | Explains why the pair is hard |
| Confusion family | One value from the agreed family list | Lets you measure coverage and per-family errors |
| Candidate pool | Template, retriever-ranked or same-account | Shows whether negatives depend on one retriever |
| OCR confidence | Engine, version and confidence in the deciding region | Separates visual hardness from OCR failure |
| Reviewer label | Hard, easy, unlabeled positive or unclear, with reviewer ID | Supports agreement checks and later correction |
| Cross-file links | Referenced files needed to interpret the positive | Prevents unfair negatives that lack required context |
Keeping false negatives out of a set built to be confusing
The harder a negative looks, the more likely it is a positive nobody labeled. The RocketQA authors manually reviewed top-ranked passages not marked positive in MS MARCO and found that about 70% were actually positives or highly relevant, and their training degraded when such hard negatives were used without denoising. Document sets have the same problem in sharper form, because duplicate scans, reprints and copies attached to multiple emails put the same answer on many pages.
Two filters help before any person looks. NV-Retriever describes positive-aware mining, which discards candidates whose teacher score exceeds a margin or percentage of the positive's score. The sentence-transformers mining utility exposes the same idea through absolute and relative margin settings, a maximum score, a rank range and optional cross-encoder rescoring.
Neither filter understands layout. A duplicate scan of the positive can score below the threshold because the image is skewed or faded, and a page with a different answer in one cell can score above it. That is why step 5 of the protocol puts a person in front of the two regions, and why exact or perceptual duplicates of the positive should be removed by hash before mining starts.
Measure the review itself. Have a second reviewer label a sample of candidates independently and track how often the two agree on hard negative versus unlabeled positive, broken down by confusion family. Families with frequent disagreement usually have an unclear deciding feature, and fixing the one-sentence description in step 2 tends to help more than adding reviewers. Keep the rejected unlabeled positives too: they are the most informative additions to the positive set, because they show every place the same answer appears in the collection.
Worked example (illustrative): a freight rate question across a tariff binder
In this illustrative case the query asks for the fuel surcharge that applied to a lane in a given quarter. The positive is one table cell on page 22 of a carrier's rate binder, under the column header for that quarter, with a footnote saying the surcharge is per hundredweight.
Candidate mining returns 40 pages: 25 from the same binder template, 10 ranked highly by the current page retriever and 5 from the same carrier's other files. The hash filter removes 3 rescanned copies of page 22. The score margin filter drops 4 more as likely positives.
Reviewers label the remaining 33. Twelve become hard negatives across four families: other quarters on the same template, a superseded edition with a revision footer, a table with the same rows priced per pallet, and a page whose footnote excludes the lane. Two candidates turn out to answer the query, because an amendment page repeats the rate, and are promoted to positives. The rest are easy negatives or unclear, and three unclear pages with low OCR confidence go to a records expert.
All counts here are illustrative. The point is the shape of the result: a small, labeled, family-balanced set in which every negative's difference can be pointed to on the page.
When the best negative has no visual difference at all
Sometimes the deciding difference is not on the page. Two contracts can be pixel-identical in the region that matters while one has been terminated by a separate notice. A rate sheet can be correct in itself but superseded by an email. In these cases the hard negative is defined by a linked record, and the protocol has to store that link as the deciding feature instead of a bounding box.
Treat these pairs as a separate family and keep them out of region-only training objectives, where they would teach the model that identical regions mean different things. They belong in evaluation of systems that can read links between files, and in retrieval sets where the query explicitly asks for the current or controlling document.
The protocol also has limits. It costs reviewer time per candidate, it depends on clean file versioning to make region references stable, and it says nothing about whether the documents may be used at all. Access to a file, privacy clearance for its contents and the right to license it are separate questions to settle before any page enters a dataset.
Signals that region labels need a records expert
Any one of these is a reason to narrow the query set rather than push labels through. A smaller set with confirmed regions is worth more than a large one whose hardest pairs are guesses.
- Reviewers disagree on whether a candidate answers the query more than occasionally: the query or the deciding feature is underspecified.
- The deciding region is handwriting, a stamp or a low-confidence OCR zone: a records expert who knows the document type should confirm it.
- The answer depends on business rules not visible on the page, such as which edition of a tariff governs a shipment date.
- Pages carry personal data, privileged material or another party's confidential terms: stop and route to the privacy or legal owner before labeling continues.
- File versions cannot be pinned, so a region reference may point to different pixels tomorrow.
Sourcing document sets with real near-misses
Hard negatives of this kind come from real document histories: repeated templates, revisions, reversals and amendments accumulated over years of operations. Public document collections rarely contain enough versions of the same form to supply them.
The companies that hold such document histories can license chosen records for AI through SourceX, which carries out the rights review, the contract, the preparation and the delivery. AI teams can describe the document types, volume, format, timing and licensing needs they have in a data request; datasets are sourced on request from U.S. businesses rather than held in stock.
Frequently asked questions
What is a hard negative in document retrieval?
It is a document or page that looks like a good match for the query, often sharing template, wording and layout with the correct page, but does not contain the answer. Training and evaluating on these pairs tests whether a retriever notices the detail that actually decides relevance.
Why not just use the retriever's top wrong results as negatives?
Many of them are unlabeled positives, especially in document sets full of duplicate scans and repeated pages. Top results also reflect one retriever's blind spots, so mix in template-based and same-account candidates and have a person confirm each one.
Should multimodal negatives be labeled at the page or region level?
Label at the region level whenever the deciding difference is a specific cell, footnote, stamp or revision block. Page labels can still be derived from region labels, but the reverse is not possible, and region labels make audits much faster.
How many hard negatives should each query have?
There is no fixed number; cap them per query so one long binder does not dominate, and balance them across confusion families. A few confirmed hard negatives per query usually teach more than many unreviewed ones.
Can OCR errors create fake hard negatives?
Yes. If the deciding region was misread, a page can look different in text while being identical on the image, or the reverse. Record OCR confidence for the deciding region and have a reviewer check the rendered page whenever confidence is low.
Sources
- ColPali produces multi-vector embeddings from document page images with late interaction matching, and its ViDoRe benchmark evaluates page retrieval across text, figures, tables and infographics. Source
- MMDocIR defines page-level and layout-level retrieval, with layout labels as bounding boxes around evidence, and found 21.3% of inherited page labels needed correction. Source
- DocLayNet annotates document pages with bounding boxes in 11 classes, including Caption, Footnote, Page-header, Picture, Table and Section-header. Source
- Dense Passage Retrieval used BM25 negatives: top-ranked passages that do not contain the answer but match most question tokens. Source
- RocketQA found about 70% of top-ranked unlabeled MS MARCO passages were actually positives or highly relevant, and introduced denoised hard negatives. Source
- NV-Retriever's positive-aware mining discards negatives whose teacher score exceeds a margin or percentage of the positive's score to reduce false negatives. Source
- The sentence-transformers mine_hard_negatives utility supports rank ranges, maximum score, absolute and relative margins, and optional cross-encoder rescoring. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.