AI uses for records
Retrieval benchmark entity ambiguity: two vendors, nearly one name
By SourceX Editorial · Updated
10 sections
Short answer
To evaluate entity matching for vendors with nearly identical names, build retrieval tasks where the lookalike record is present as a hard negative, and score whether the system retrieves the controlling record by identifier, address, status and date, cites it, and abstains when the evidence cannot separate the two. Similarity of names alone should never decide the answer.
Evaluation split
Retrieval benchmark entity
Key takeaways
- Lookalike vendor names are a systematic retrieval failure: the vendor with more records tends to win.
- Every task needs a controlling source, an as-of date, permissions, named hard negatives and an abstention condition.
- Identifiers, status, dates and relationships separate vendors; names, even legal names, rarely do.
- Citation support is not enough: each citation must also belong to the controlling entity.
- Include a true-duplicate counterexample and treat lookalike payment changes as a fraud-control test.
How do you evaluate retrieval when two vendors have nearly the same name?
Test it with tasks built around the collision itself. Each task puts the correct vendor record and at least one lookalike in the corpus, asks a question a real user would ask, and defines in advance which record controls the answer, which fields prove it, and when the honest answer is to ask or abstain. The system passes only if it reaches the controlling record, cites evidence that separates it from the lookalike, and does not blend facts from both.
A general relevance score hides this failure. Two vendor master entries named, say, Harbor Line Freight LLC and Harborline Freight Inc. are both highly relevant to a query about Harbor Line Freight; a retriever can rank both in its top three and still feed the generator the wrong one first. The question that matters is whether the answer is right for this entity, at this date, for this user.
This page gives an entity disambiguation and evidence task set with eight task types, the fields every task needs, a worked case and the scoring rules. All company names and values below are illustrative.
Why do retrievers confuse entities that share a name?
Dense and sparse retrievers both lean on surface and contextual similarity, and lookalike names are by construction similar. Research on ambiguous entity retrieval shows the effect is systematic: the AmbER benchmark, which groups entities that share a name with queries about each, found that retrievers favor the more popular entity and are about twice as likely to retrieve the wrong documents for queries about the less popular one.
In a business corpus, popularity means record volume. The vendor with years of invoices, tickets and emails dominates the index, while a newer or smaller vendor with almost the same name appears in a handful of documents. Questions about the small one are the ones most likely to be answered with the big one's facts.
Question ambiguity compounds it. The AmbigQA work found that many open-domain questions admit more than one reading, often because of an entity reference, and framed the task as finding every plausible answer and the rewrite that resolves each. Business users write the same way: they type the name they remember, not the legal name in the vendor master.
Which fields actually separate two lookalike vendor records?
Identifiers separate them; names rarely do. The useful fields are the ones assigned by an external authority or recorded at onboarding, then status and dates, then relationships. Schema.org's Organization vocabulary reflects the same split, with distinct properties for the registered legal name, an alias, a tax ID, a DUNS number and an ISO 17442 legal entity identifier.
The Legal Entity Identifier is a 20-character code under ISO 17442, and GLEIF describes its reference data as answering who is who and who owns whom. Many smaller U.S. vendors will not have one. For U.S. payees, the IRS TIN Matching program lets authorized payers check whether a name and taxpayer identification number combination matches IRS records before filing information returns, which is why the tax ID on a W-9 is often the most reliable separator a vendor master holds.
| Field type | Example field | Separates lookalikes when | Fails when |
|---|---|---|---|
| External identifier | Tax ID, LEI, DUNS | Each record carries a verified value | Field is blank, mistyped or shared by a parent and subsidiary |
| Legal versus trading name | Legal name, DBA or alias | The query uses one and the record stores both | Only one name was captured at onboarding |
| Location | Remit-to and registered address | Vendors operate in different places | Both use the same agent or PO box |
| Status and dates | Active flag, created and inactivated dates | One record was retired or merged | Status was never maintained |
| Relationship | Parent, subsidiary, contract owner | Question is about one legal party to a contract | Group structure is not recorded |
| Payment details | Bank account last verified date | Change history and verifier are logged | Changes were made from email with no call-back |
What should the entity disambiguation and evidence task set include?
Cover each way a name collision misleads a system, plus a counterexample where the two records really are the same vendor. The eight task types below are a starting set; build several instances of each from your own vendor master with names, identifiers and amounts replaced. Everything in the table is illustrative.
| Task type | Illustrative query | Hard negative in corpus | Controlling evidence | Correct behavior |
|---|---|---|---|---|
| Same name, different state | What are the payment terms for Harbor Line Freight? | Harborline Freight Inc., another state, larger history | Vendor record whose tax ID matches the contract | Answer from the matching record and cite the contract |
| Trading name versus legal name | Do we have a signed NDA with Bluepeak? | Blue Peak Analytics LLC, unrelated firm | NDA party block naming Bluepeak Data Services Inc., DBA Bluepeak | Link DBA to legal entity, then answer |
| Renamed vendor | Who is the account manager at Corvane? | Corvane Systems, a separate active vendor | Name-change letter on file; old name now inactive | Use the current record and say the name changed |
| Merged duplicate | How much did we spend with Tallis Packaging last year? | Inactive duplicate Tallis Packaging Co | Merge log mapping duplicate to surviving record | Combine both, state that a merge occurred |
| Lookalike payment change | Has Harbor Line Freight's bank account changed? | Email from a domain one character off the vendor's | Bank change history with call-back verification | Report only verified changes and flag the unverified request |
| Permission boundary | What did Bluepeak quote us? | Quote in another business unit the user cannot access | Access list for the requesting user | Answer from permitted records or say none are accessible |
| Unresolvable | Is Northgate Supply on hold? | Two active Northgate Supply vendors, no distinguishing detail in query | Both records, neither preferred | Ask which one, listing a separating field |
| Counterexample: true duplicate | Total open invoices for Ridgeway Electric | Ridgeway Electric Inc. and Ridgeway Electric Inc (no period), same tax ID | Matching tax ID and address | Treat as one vendor and say the records should be merged |
Which fields must every task carry before it can be scored?
A task without its controlling source, date and permissions cannot be graded consistently, because two reviewers will disagree about what the right answer was. The minimum set follows.
| Task field | What it records | Why scoring needs it |
|---|---|---|
| Query intent | The decision the user is trying to make, in one sentence | Separates a lookup from a payment or legal question |
| Controlling source | The record that governs the answer and why it outranks others | Defines correct retrieval, not just relevant retrieval |
| As-of date | The date the answer must be true for | Handles renames, merges and inactivations |
| Permissions | The user role and what it may see | Makes refusing to use a restricted record a correct outcome |
| Hard negatives | Each lookalike record and the field that rules it out | Tests separation instead of topical relevance |
| Required evidence set | Every record the answer must cite | Lets a grader check completeness as well as precision |
| Citation support rule | Which claim each citation must support | Catches citations to the right vendor for the wrong fact |
| Abstention condition | When asking or declining is the correct answer | Prevents rewarding confident guesses on unresolvable tasks |
Worked case (illustrative): the payment terms that belonged to the other Harbor Line
Starting state, all illustrative: the corpus holds two active vendor records. Harborline Freight Inc. has six years of invoices and a master services agreement with net 45 terms. Harbor Line Freight LLC, onboarded eight months ago with a different tax ID and remit-to address, has a short contract with net 15 terms and a small number of invoices. An accounts payable clerk asks what the payment terms are for Harbor Line Freight, while viewing an invoice from the LLC.
Linked evidence: the invoice header carries the LLC's vendor ID; the vendor record maps that ID to the LLC's tax ID; the LLC's contract states net 15. The hard negative is the Inc.'s agreement, which mentions Harbor Line Freight in prose and has far more retrievable text.
A failing run retrieves the Inc.'s agreement first, answers net 45 and cites it. The citation is real and supports the sentence, so a grader that checks only whether each cited passage supports its claim passes it. The grader in this task set fails it, because the cited record's tax ID does not match the invoice's vendor and the controlling source named in the task is the LLC contract.
A passing run either uses the vendor ID from the open invoice to filter retrieval, answers net 15 and cites the LLC contract, or, if it has no invoice context, says there are two vendors with that name, gives each one's terms and asks which applies. Decision: net 15 for this invoice. Verified outcome: the payment run scheduled after the answer used the LLC's terms, which the task checks against the payment record.
Where do lookalike pairs come from, and how do you keep them after de-identification?
Mine them from the vendor master and its history rather than inventing them. Four queries find most real pairs: names within a small edit distance or sharing all significant tokens once suffixes such as Inc, LLC and Co are stripped; different tax IDs at the same address or remit-to account; records inactivated on the same day another was created; and name change or merge entries in the audit log. Each candidate pair then needs a person who knows the vendors to label it as distinct, the same entity, or a parent and subsidiary.
Frequency matters as much as similarity. Sort pairs by the ratio of document counts between the two records, and make sure the task set includes pairs where the questioned vendor is the rare one, since that is where the popularity bias does its damage. A set built only from evenly matched pairs will overstate how well the system performs.
De-identification has to preserve the collision or the task is gone. If one vendor becomes Vendor 4471 and the other Vendor 9012, the names no longer look alike and the test measures nothing. Use consistent pseudonyms that keep the relationship, for example replacing a shared root with the same invented word in both names while keeping each record's distinct suffix, and replace identifiers with stable surrogates so that a matching tax ID still matches. Record the replacement rules with the task set so the next refresh produces the same pairs.
How should the answers be scored?
Score retrieval, answer and evidence separately, then combine them with the controlling-record rule as a gate. A run that answers correctly from the wrong vendor's record has passed by luck and should fail.
At a glance
How should the answers be scored
Citation scoring needs one addition for this problem. ALCE's evaluation code measures citation recall and citation precision with a natural-language inference model judging whether cited passages support each statement. That checks support, not identity. Add an entity check: every citation must belong to the controlling entity, or to a record the task explicitly allows. Anthropic's guide to agent evals makes a similar split for research agents, pairing groundedness checks with source quality checks that confirm the consulted sources are authoritative.
- Controlling record retrieved: the governing record appears in the top k, and no hard negative outranks it.
- Entity-correct answer: the stated fact is the controlling record's value as of the task date.
- Evidence support and identity: each citation supports its claim and belongs to the right entity.
- Completeness: all records in the required evidence set are cited, such as both halves of a merged duplicate.
- Abstention: on unresolvable tasks the system asks or declines and names a separating field; on resolvable tasks it does not hedge.
- Permission: no answer uses or reveals a record outside the user's access.
When is a name collision a fraud signal rather than a retrieval error?
When the lookalike arrives through a payment change. FBI guidance on business email compromise advises verifying significant transactions through a second channel, using previously known phone numbers rather than those in the email, requiring secondary sign-off for changes in vendor payment location, and registering domains slightly different from the company's own. A retrieval system that cheerfully reports the new bank details from a near-identical email domain is doing exactly what the scheme needs.
Treat these tasks as their own category with stricter grading: the only correct answer reports verified changes, cites the verification record, and flags anything else. Route failures to the controls owner, not only to the retrieval team. The evaluation can show the system resisted the lure; it cannot replace the call-back process.
Building these tasks from real vendor masters
Real collisions are hard to invent convincingly: the renamed vendor, the duplicate created by a new clerk, the subsidiary that invoices under its parent's name. They come from vendor masters, contracts, invoices and change logs that a business has kept for years, with identifiers that tie them together. Before such records leave a company, someone has to confirm the company may license them for AI evaluation, replace real names and identifiers consistently so collisions survive, and remove payment details.
Vendor-master histories with real near-duplicate names are the kind of material SourceX can look for when a retrieval team asks: it locates U.S. businesses willing to license vendor and contract records and coordinates the rights review, contracting, preparation and delivery. The request form is where the team specifies scope, volume, format and timing for an entity disambiguation set.
Frequently asked questions
What is entity ambiguity in a retrieval benchmark?
It is a set of tasks where two or more records share or nearly share a name, so the system has to use other evidence to find the one the question is about. A good benchmark includes the lookalike as a hard negative and defines which record controls the answer. Research such as the AmbER benchmark shows retrievers are biased toward the more frequent entity.
Is entity disambiguation the same as entity resolution?
They are related but test different things. Entity resolution decides whether two records describe the same real-world entity, usually for deduplication. Retrieval disambiguation decides which of several distinct entities a question refers to and whether the answer and its citations come from that entity.
Which identifier is best for telling vendors apart?
A verified tax ID is usually the most reliable field in a U.S. vendor master, followed by an LEI or DUNS number where one exists. Names, including legal names, are the weakest separators. Status, dates and relationship fields matter when a vendor was renamed, merged or acquired.
Should a RAG system ask a clarifying question or answer both?
If the query and context cannot separate the vendors and the decision depends on which one is meant, asking is the safer default. For low-stakes lookups, giving both answers labeled by their separating field is acceptable. Either way, the task set should define in advance which behavior is correct.
How many lookalike tasks does an evaluation need?
Enough instances of each task type to see a pattern across vendors, not one example per type. Include the true-duplicate counterexample in proportion, or a system can pass by always treating similar names as different vendors.
Sources
- The AmbER sets (ACL 2021) group entities that share a name with queries about each, and found retrievers favor popular entities, roughly twice as likely to retrieve wrong documents for the less popular one. Source
- AmbigQA (EMNLP 2020) defines a task of finding every plausible answer to an ambiguous open-domain question and rewriting the question to resolve each, noting entity references as a source of ambiguity. Source
- ALCE's evaluation code reports citation recall and citation precision, using an NLI model to judge whether cited passages support each statement. Source
- Schema.org defines separate organization properties for legal name, alias, tax ID, DUNS number and an ISO 17442 legal entity identifier. Source
- GLEIF describes the LEI as a 20-character alphanumeric code based on ISO 17442 whose reference data answers who is who and who owns whom. Source
- The IRS TIN Matching program lets authorized payers check whether name and TIN combinations match IRS records before filing information returns. Source
- The FBI's 2017 business email compromise alert advises out-of-band verification using previously known numbers, secondary sign-off for vendor payment location changes, and registering lookalike domains. Source
- Anthropic's guide to agent evals describes groundedness checks that verify claims are supported by retrieved sources and source quality checks that confirm sources are authoritative. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.