Skip to content

AI uses for records

Same invoice, new syntax: duplicate invoice semantic detection data

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Duplicate invoice semantic detection data should teach a model whether two documents record the same billable obligation, whatever their format. In the official EN 16931 examples, all nine UBL and CII pairs differ in raw text, yet seven agree on every shared business term once dates and amounts are normalized. The fields that separate recurring invoices from duplicates, such as the invoicing period, are optional, so labeled pairs need reviewer evidence.

Document stack

Same invoice, new syntax

This page covers same invoice, new syntax: duplicate invoice semantic detection data.Illustrative

Key takeaways

  • The unit of data for duplicate detection is a cluster of documents with a label and a payment outcome, never a single invoice.
  • Across nine EN 16931 example invoices published in both UBL and CII, no file pair is byte-identical, yet seven agree on every shared term after normalizing dates and amounts.
  • An invoice number is unique only within one seller's records; ten official UBL examples use just six numbers, and three groups share one.
  • In Peppol BIS Billing 3.0 only 9 of 21 candidate identity terms are always present, and the period, contract and order references that mark recurring charges are optional.
  • No EN 16931 or Peppol validation assertion counted here reads another document, so validation cannot catch a duplicate; labels need a qualified reviewer and evidence.

What should duplicate invoice semantic detection data teach a model?

It should teach one decision: do two documents record the same billable obligation, so that paying both would pay the seller twice? The file format has no bearing on that answer. A PDF emailed to the AP inbox, a UBL file from an e-invoicing network and a CII file uploaded to a supplier portal can carry one obligation, while two identical-looking monthly bills from the same vendor carry two.

A check that compares a handful of header fields fails in both directions. A re-sent invoice slips through because one field was keyed or formatted differently, and a legitimate recurring charge is held because it looks like last month's. Data for training or evaluating this task therefore has to come as labeled pairs or clusters, never as single invoices, and each label has to carry the evidence a qualified reviewer relied on.

The public EN 16931 e-invoicing artefacts make the problem concrete. The European standard's validation repository publishes the same example invoices in its two mandatory syntaxes, UBL 2.1 and UN/CEFACT Cross Industry Invoice (CII), with an optional EDIFACT form, and the examples reuse invoice numbers across different test scenarios. That is close to a ready-made set of easy positives and hard negatives. This article measures it, then turns it into a casebook and a reviewer rubric.

One invoice, three syntaxes: what changes and what survives

Almost every byte changes; the business terms survive. Of the nine example invoices that exist in both UBL and CII at commit b17b269 of the ConnectingEurope/eInvoicing-EN16931 repository, no pair of files is byte-identical and every pair differs in the literal text of at least one business term both files carry. After three normalization rules, seven of the nine pairs agree on every shared term.

Most of the difference is formatting. The issue date (BT-2) is written 2015-01-09 in UBL and 20150109 in CII; that formatting difference is the only change to the date in eight of nine pairs, and in the ninth the date itself differs. The invoice total with VAT (BT-112) differs only in formatting in four pairs, such as 4675.00 against 4675, and the invoicing period dates differ only in format in all five pairs where both files carry them. A detector that hashes files or compares raw strings would call every one of these different invoices.

The two pairs that still disagree are the instructive ones. In example 3 the UBL and CII files share the invoice number TOSL108 and the issue date, but the UBL version has two lines totaling 2005.00 and the CII version one line totaling 1125. In example 7 the number and the 3200.00 total match while the issue dates read 2013-03-11 and 2013-05-13. These are test files written to exercise validation rules, not real invoices, yet in an AP inbox they are exactly the pairs that need a person.

The EDIFACT versions agree with both XML files on invoice number, issue date and type code in eight of nine cases, example 7 again being the exception, and in example 9 the UBL file carries an invoicing period that the CII file omits. Per business term, across the nine UBL and CII pairs:

One invoice, three syntaxes: what changes and what survives
Business termPresent in both filesRaw text differs, value equalValue differs
Invoice number (BT-1)9 of 900
Issue date (BT-2)9 of 981
Seller VAT identifier (BT-31)8 of 900
Invoice total with VAT (BT-112)9 of 941
Amount due for payment (BT-115)9 of 941
Invoicing period dates (BT-73, BT-74)5 of 950
Number of invoice lines9 of 901
Two cards show EN 16931 example invoice 12115118 as UBL and as CII with differently written dates, and a normalized key below them in which number, date, total and line count all match, beside counts of 0 byte-identical pairs and 7 of 9 pairs agreeing after normalization.
EN 16931 example 1 in two syntaxes and the nine-pair result, from the ConnectingEurope repository at commit b17b269, 27 October 2026.

The invoice number is unique only inside one seller's books

An invoice number identifies an invoice only together with the seller who issued it. The Peppol BIS Billing 3.0 structure defines the invoice number (BT-1) as the sequential number required by Article 226(2) of the EU VAT directive, "to uniquely identify the Invoice within the business context, time-frame, operating systems and records of the Seller". Uniqueness is scoped to one seller's records, and even there to a time frame.

The example set shows what happens when that scope is ignored. The ten UBL examples use only six distinct invoice numbers, and seven of the ten share a number with at least one other example. Examples 2 and 3 share TOSL108 but come from differently named sellers with Norwegian and Danish VAT identifiers, so they are separate obligations. Examples 1 and 10 share 12115118, the seller, the issue date, the total and all 20 lines; example 10 differs only by additions: a tax currency, Swedish kronor, the VAT amount expressed in it and a repeated payment reference. That is a duplicate in different formatting created inside a single syntax.

Examples 4, 5 and 6 are the hard case. All three carry TOSL110, the seller name SellerCompany, the issue date 2013-04-10 and a 4675.00 total over three lines, yet each gives a different seller VAT identifier, and example 5 shows an amount due below its total. Adding issue date, total and line count to the invoice number separates only one of the three shared-number groups, examples 2 and 3. Whether 4, 5 and 6 describe one obligation is a question for vendor master data and the payment account, not for string matching.

The standard anticipates collisions in its own text. For a document that cites an earlier invoice, the Peppol structure says the preceding invoice issue date (BT-26) "Shall be provided in case the Preceding Invoice identifier is not unique".

Which fields can a duplicate key rely on?

Only on the fields every conforming invoice must carry, plus at least one seller identifier, and those are not enough to tell a recurring charge from a duplicate. Reading the Peppol BIS Billing 3.0 UBL structure at commit 261c458, and counting a term as always present only when it and every group above it have a minimum of one, 9 of 21 candidate identity terms appear on every conforming invoice.

Each seller identifier is optional on its own. Rule BR-CO-26 in the EN 16931 rule set requires at least one of the seller identifier, legal registration identifier and VAT identifier "In order for the buyer to automatically identify a supplier", so a key built on the VAT number alone will find nothing to compare on some valid invoices. The seller electronic address is always present in this Peppol specification; other EN 16931 implementations can treat it differently.

The terms that separate a legitimate recurring invoice from a duplicate all have a minimum of zero: the invoicing period dates, the contract reference and the purchase order reference. A monthly subscription invoice can arrive, and pass validation, with the same seller, amount and line text as last month's and no period at all. A dataset meant to teach a model to keep recurring charges therefore has to bring the contract schedule or ledger history from the buyer's own systems. The 21 candidate terms split into four groups:

  • Always present: invoice number (BT-1), issue date (BT-2), type code (BT-3), currency (BT-5), seller name (BT-27), seller electronic address (BT-34), total without VAT (BT-109), total with VAT (BT-112) and amount due (BT-115).
  • At least one of three, under BR-CO-26: seller identifier (BT-29), seller legal registration identifier (BT-30) and seller VAT identifier (BT-31).
  • Optional but decisive for recurring charges: invoicing period start and end (BT-73, BT-74), contract reference (BT-12) and purchase order reference (BT-13).
  • Optional but decisive for corrections and payment identity: preceding invoice number and date (BT-25, BT-26), remittance information (BT-83), payment account identifier (BT-84) and payment due date (BT-9).

Why no format validator will catch the duplicate

Because validation looks at one document at a time, and a duplicate exists only between documents. None of the 982 assertions in the EN 16931 UBL validation file at commit b17b269, none of the 810 in its CII counterpart and none of the 159 Peppol-specific UBL assertions at commit 261c458 calls an XPath function that opens another document.

That is the design, not a gap in the rules. A schematron rule set certifies that an invoice is complete and internally consistent, for instance that "An Invoice shall have an Invoice number (BT-1)" under rule BR-02. Whether the same obligation was already received, approved or paid sits in the buyer's AP subledger, payment runs and supplier correspondence. A model trained only on validated documents learns what a valid invoice looks like, never whether paying this one is right.

For a dataset buyer this fixes the unit of data. The record that teaches duplicate detection is a cluster: the candidate documents in every format they arrived in, the ledger state at the moment of the decision, the label, and the outcome that proves the label, such as a single payment, a blocked payment or a recovered overpayment.

Seven invoice pairs and the label each one needs

Six of the seven cases below come straight from the EN 16931 example files; the seventh is illustrative because the examples contain no recurring series. Between them they cover the outcomes a detector must keep apart: the same obligation in another format, the same obligation with extra fields, conflicting versions under one number, a seller identity conflict, a number collision between sellers and a recurring charge that has to be kept. Each row names the evidence a reviewer needs before the label is final:

Seven invoice pairs and the label each one needs
CaseSourceWhat matchesWhat differsLabel and evidence needed
A. Same invoice, two syntaxesUBL and CII example 1Number 12115118, issue date, seller VAT identifier, total, 20 linesDate written 2015-01-09 against 20150109; all markupSame obligation. Normalized key match and the normalization log
B. Same invoice, one extra blockUBL examples 1 and 10Number, seller, issue date, total with VAT, 20 linesExample 10 adds a tax currency, the VAT amount in it and a repeated payment referenceSame obligation. Only additive fields differ; amount due is equal
C. One number, two versionsUBL and CII example 3Number TOSL108 and issue dateTwo lines totaling 2005.00 against one line totaling 1125Conflicting versions. Hold; supplier states which version stands, plus any credit note
D. One number, two datesUBL and CII example 7Number, total 3200.00, invoicing periodIssue date 2013-03-11 against 2013-05-13Probable re-sent copy. Ledger shows one approved obligation, or supplier confirms
E. One seller name, three tax identifiersUBL examples 4, 5 and 6Number TOSL110, seller name, issue date, total 4675.00, three linesThree seller VAT identifiers; example 5 has a lower amount dueIdentity conflict. Vendor master check and callback to a known contact before payment
F. Shared number, different sellersUBL examples 2 and 3Number TOSL108Seller names, VAT identifiers, dates, totals and linesDistinct invoices. Differing seller identifiers and two purchase records
G. Same charge every month (Illustrative)Not in the example setSeller, buyer, amount and line textInvoice number and invoicing periodDistinct recurring invoices. Contract schedule and the prior period's payment

Six labels, the evidence behind each, and who signs off

A label is only as good as the evidence behind it, so the rubric fixes both. Six labels cover the casebook, each with a minimum evidence set. Pairs on which two reviewers disagree stay in the dataset with both labels and the adjudication, because disagreement marks exactly the ambiguity a model should learn to escalate rather than resolve.

Reviewer qualification is part of the label. An AP specialist can confirm that two files state the same terms; deciding that a re-dated invoice is not a second obligation, or that a credit note offsets it, is an accounting judgment that a qualified accountant should sign, with the sign-off stored in the pair record. The six labels and their minimum evidence:

  • Same obligation, different format: normalized seller identifier, invoice number, issue date, totals and line count all agree; evidence is both documents and the normalization log.
  • Same obligation, re-sent or re-dated: seller, number and amount agree while a date or channel differs; evidence is the ledger showing one approved obligation, or written supplier confirmation.
  • Conflicting versions under one number: same seller and number with different lines or totals; evidence is the supplier's statement of which version stands and any credit note, with payment held meanwhile.
  • Seller identity conflict: same number, date and amount with different tax identifiers or payment accounts; evidence is the vendor master record and a callback to a contact already on file, escalated before payment.
  • Distinct, number collision: different sellers happen to use the same number; evidence is the differing seller identifiers and a purchase record for each.
  • Distinct, recurring: same seller and amount over consecutive periods or contract schedule entries; evidence is the contract or subscription schedule and the prior period's payment.

Matching totals is not the same as a payment being right

Agreement on the arithmetic shows that two documents state the same numbers, not that either should be paid. The strongest alternative explanation for a model that scores well on duplicate matching is that it learned to compare amounts, and that skill breaks when a correct-looking invoice was never authorized, was already settled through another channel, or comes from a seller whose bank details just changed.

U.S. federal law draws the same line. The Payment Integrity Information Act definition at 31 U.S.C. 3351 treats as improper a payment that should not have been made or was made in an incorrect amount, and lists "any duplicate payment" among its examples, next to payments to ineligible recipients and payments for goods or services not received. The statute governs federal agency payments, not private AP departments, but it is a useful public template: a duplicate is one kind of improper payment, and a dataset that labels only duplicates leaves its neighbors unlabeled.

An evaluation therefore needs two scores. One asks whether the model found the same obligation; the other asks whether it routed the payment decision correctly, including holds for identity conflicts and version conflicts. A model that is right on the first and silent on the second is a matching utility, not an AP assistant.

Writing a data request for semantic duplicate detection

Ask for clusters with outcomes, not a pile of invoices. The specification below suits a pilot evaluation set; a training set scales the same fields.

At a glance

Writing a data request for semantic duplicate detection

SourceX sources datasets of this kind to a buyer's specification from businesses that hold the records, rather than from a stock it keeps: the request states scope, volume, language, format, timing and licensing needs, and the business approves the scope and terms and signs before anything is delivered. Whether a particular company's AP archive holds clusters with documented outcomes is established case by case; nothing here describes current holdings. A request for this task can be built from these lines:

  • Unit: a candidate cluster of two or more documents, with every arrival channel kept (PDF or image, UBL, CII, EDI, portal upload, manual ERP entry).
  • Fields: the EN 16931 business terms found in each document, the raw text as received, and buyer-side keys such as vendor master record, purchase order and goods receipt.
  • Ground truth: the rubric label, its evidence references and the dated payment outcome (paid once, blocked, reversed or recovered).
  • Look-alike negatives: recurring charges with consecutive periods, number collisions across sellers, and installment or partial invoices against one order.
  • Rights and privacy: authorization from the company that holds the records, removal of personal names and bank details, and a license that covers the intended training or evaluation use.
  • Acceptance test: agreement between two qualified reviewers on a held-out sample, plus a stated share of clusters whose outcome is documented.

What the example-file test can and cannot show

The counts come from two pinned repositories read as data. ConnectingEurope/eInvoicing-EN16931 at commit b17b2695e5e3e021945c38658326b78a76f80f84 (release notes at that commit list validation 1.3.17 of 9 October 2026) supplied ubl-tc434-example1 to 10, CII_example1 to 9 and EDIFACT_EXAMPLE1 to 9, paired by file number, and the preprocessed UBL and CII schematron files. OpenPEPPOL/peppol-bis-invoice-3 at commit 261c458474e27d58a25be629cccac28883171c92 supplied structure/syntax/ubl-invoice.xml with its included part files and the Peppol UBL schematron.

Normalization rules, applied as written: dates in YYYYMMDD form were rewritten as YYYY-MM-DD; amounts were compared as decimal numbers; identifiers were uppercased with every character other than A-Z and 0-9 removed. Raw text differs means the stripped element text differs. A term counts as always present when its element and every ancestor have a minimum occurrence of one, an element without a cardinality attribute counting as one; included part files are expanded in place, a reference naming two terms is split on the comma, and each term is credited to the first element that references it. An assertion reads another document when its test, after removing spaces before an opening parenthesis, calls doc(, document(, collection(, unparsed-text( or doc-available(; let values were checked the same way. The scripts invoice_identity.py, peppol_cardinality.py and schematron_scope.py print every count above.

The limits are real. These are test files built to exercise validation rules, so their shared numbers are artefacts of a test suite and say nothing about how often duplicates occur in any company. Scanned PDFs, OCR errors and email channels are not covered, and no buyer-side evidence exists for these examples. The next evidence needed is authorized, de-identified clusters from real AP archives, labeled under the rubric by qualified accountants, with reviewer agreement reported.

Frequently asked questions

Can matching on invoice number and amount find duplicate invoices?

It finds the easy ones and errs both ways on the rest. In the EN 16931 example set, matching on the number alone would merge two invoices from different sellers, while a raw string comparison would miss the same invoice written in two syntaxes. Seller identity, normalized dates and amounts, and line content all belong in the comparison, with a reviewer for conflicts.

How do you keep recurring invoices out of duplicate alerts?

Compare the invoicing period, contract reference and invoice number sequence, and check the contract or subscription schedule. In Peppol BIS Billing 3.0 the period, contract and purchase order references are all optional, so these fields often have to come from the buyer's ERP or contract records rather than from the invoice.

Does e-invoicing stop duplicate invoices?

It makes business terms comparable across formats but does not test for duplicates. EN 16931 and Peppol validation rules check one invoice at a time, and none of the assertions counted here reads another document. A paper copy and an e-invoice for the same obligation can both pass validation.

What labels should a duplicate invoice dataset use?

Use labels that separate kinds of sameness and difference: same obligation in another format, same obligation re-sent or re-dated, conflicting versions under one number, seller identity conflict, number collision between sellers, and recurring charge. Each label should cite the evidence used and the payment outcome.

Who should review labels for invoice duplicate detection data?

AP specialists can confirm that two documents state the same terms. Calls about whether a re-dated or revised invoice is a second obligation, or whether a credit note offsets it, need a qualified accountant, and the sign-off should be stored with the pair.

Sources

  • The UBL example invoices ubl-tc434-example1 to 10 used for the pair, collision and casebook counts. Source
  • The CII example invoices CII_example1 to 9 compared with their UBL counterparts. Source
  • UBL 2.1 and CII D16B are the EN 16931 mandatory syntaxes, EDIFACT is optional, and validation release 1.3.17 is dated 9 October 2026. Source
  • EN 16931 business rules BR-02 (invoice number required) and BR-CO-26 (at least one seller identifier). Source
  • The preprocessed EN 16931 UBL validation file holds 982 assertions, none calling a function that reads another document. Source
  • The preprocessed EN 16931 CII validation file holds 810 assertions, none calling a function that reads another document. Source
  • Peppol BIS Billing 3.0 definitions and cardinalities, including the BT-1 uniqueness scope and the BT-26 note on non-unique preceding invoice numbers. Source
  • The Peppol-specific UBL rule file holds 159 assertions, none calling a function that reads another document. Source
  • The federal definition of improper payment, which lists any duplicate payment among its examples. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies