Skip to content

Systems and records

Multimodal invoice reasoning dataset: one image, three linked systems

By SourceX Editorial · Updated

9 sections

Short answer

Build the task around the payables decision, not the page. A multimodal invoice reasoning dataset item pairs the invoice image with region-level anchors for every value used, the purchase order, receipt and vendor records as they stood on the decision date, the matching rule applied, the decision with its reason code, and later evidence such as a payment or credit memo that shows whether the decision held.

Document stack

Multimodal invoice reasoning

This page covers multimodal invoice reasoning dataset: one image, three linked systems.Illustrative

Key takeaways

  • Build each item around the payables decision, then collect every piece of evidence that decision needed.
  • Three-way matching compares the invoice with the purchase order and the receipt, so the image alone cannot answer it.
  • Anchor every value used to a file version, page and region, keeping the parser polygon and a percent rectangle.
  • Snapshot linked records as of the decision time and keep later changes in a separate dated layer.
  • Grade against outcome evidence such as payments and credit memos, not status fields or approval stamps.
  • Pair packets that share an image but differ in linked records, so models must use the records.

How do you build a document task when invoice evidence spans an image and several systems?

Start from the decision an accounts payable clerk actually made, approve, hold, pay in part, dispute or route for verification, and work backward to every piece of evidence that decision needed. The invoice image supplies what the vendor claims. The purchase order says what was agreed, the receiving record says what arrived, and the vendor master says who should be paid and where. A task that ships only the image asks a model to guess at three of those four.

Each value the answer relies on gets an anchor: the file, its version, the page, and the region on that page. Each linked record gets an as-of timestamp, because the receipt posted tomorrow changes today's correct answer. And each item records what happened afterward, so a grader can tell a decision that held from one that only produced a plausible message.

Why can the invoice image not answer the question by itself?

Because invoice matching is defined across documents. Microsoft's Dynamics 365 Finance documentation describes two-way matching as comparing the price on the invoice with the price on the purchase order, and three-way matching as adding a comparison of the invoiced quantity with the quantity on the product receipts selected for that invoice. Neither comparison can be made from the invoice alone.

Most public document datasets test the reading step, not the matching step. CORD, a receipt dataset from Clova AI, describes over 11,000 Indonesian receipts with box and text annotations and multi-level semantic labels for post-OCR parsing, and its public release is a 1,000-receipt sample. DocVQA poses about 50,000 questions on more than 12,000 document images, and its answers are extracted from the page. Both are valuable for layout and extraction; neither asks whether a quantity on the page was ever received.

Real payables files add difficulties those benchmarks leave out on purpose: scans with skewed pages and stamps over the totals, tables that continue across pages without repeating their headers, credit lines printed as negative quantities, and supplier-specific layouts that change when a vendor switches billing software. A cross-modal set should keep these as they occur, because they are where the extraction step hands the reasoning step a wrong number.

That gap is the reason for a separate dataset type. The model has to read a table on a scanned page, find the matching purchase order line, notice that a receipt covers fewer units, apply the tolerance in force, and choose an action a controller would sign off.

What should the cross-modal invoice task packet include?

One packet is one invoice decision. It bundles the image, the anchored extractions, the linked records with their as-of dates, the rule, the decision and the outcome. Values below are illustrative placeholders, and identifiers are synthetic.

What should the cross-modal invoice task packet include?
ComponentWhat it containsAnchor or formatCheck before release
Source fileInvoice image or PDF, content hash, received date, version (original, revised, duplicate)sha256 plus version labelRevised invoices link to the one they replace
Region anchorsFor each value used: page, bounding region, layout element (table cell, header, footnote, stamp)page index plus xywh or polygonRegion crops show the value a reviewer can read
Extraction recordRaw text, normalized value, OCR or parser confidence, extraction methodValue with confidence 0-1Low-confidence values are flagged, not silently fixed
Table contextColumn headers and units for each line item, including headers repeated across pagesHeader row anchor per lineQuantities are not read under the wrong column
Purchase order linesOrdered item, quantity, agreed unit price, terms, as of the decision datePO id, line, snapshot timeSnapshot predates the decision
Receipt linesReceived quantity per line, receipt date, partial or backorder statusReceipt id, line, posted timeReceipts posted after the decision are kept separate
Vendor master snapshotLegal name, remit-to details changed, change date, verification statusVendor id, version, change log entryBank and address fields are masked in the release copy
Rule appliedMatching policy (two-way or three-way), tolerance, approval threshold in forcePolicy id and effective dateRule matches the one configured at the time
DecisionAction taken, reason code, approver role, timeEnumerated action plus free-text noteReason is supported by at least one anchor
Outcome evidencePayment cleared, credit memo, vendor dispute, reversal, or none within the windowRecord ids with dates, window lengthOutcome window stated; status fields alone are not outcomes

How do you point to the exact region of the image an answer depends on?

Store a page index and a region for every value, using a format a viewer can render, and keep the region tied to the exact file version and page rendering it was drawn on. Without that, a reviewer cannot confirm the model read the right number, and a later re-scan silently breaks every label.

Two standard shapes are worth reusing. Google's Document AI data model attaches to each extracted entity its mention text, a confidence in the range 0 to 1, and a page anchor that refers to one or more page elements with a bounding polygon and an optional layout type such as a table, form field or token. W3C's Web Annotation Data Model offers a FragmentSelector that describes a segment of a resource with the fragment part of an IRI, and Media Fragments defines xywh as x, y, width and height in pixels by default or in percent, measured from the top-left corner.

In practice keep both: the polygon from the parser for machine use, and a normalized percent rectangle that survives re-rendering at a different resolution. Record the page size and rendering resolution next to them, so pixel anchors can be checked later.

  • Anchor every number used in the decision, not only the ones the parser happened to extract.
  • Anchor footnotes, stamps and handwritten notes as their own layout elements; they often carry the exception.
  • For multi-page tables, anchor the header row on the page where the line appears, not only on page one.
  • When a value is unreadable, anchor the region and label it unreadable rather than leaving it out.

Which linked record proves the decision, and which only looks like it does?

The proof is a record created after the decision by someone other than the clerk: the cleared payment, a vendor credit memo, a corrected invoice, or a dispute closed in the buyer's favor. A posted invoice status or an approval stamp shows that a decision was recorded, not that it was right.

That distinction matters for grading. A model that writes a convincing hold message has not solved the task if the eventual record shows the hold was unnecessary and the vendor was paid late. Set an outcome window, for example the next two payment cycles, and label items whose outcome did not arrive within it as unresolved rather than counting the decision as correct.

Outcome evidence also has to be linked as carefully as the inputs. Record the identifier of the payment, credit memo or dispute, the date it was created and the invoice line it refers to. A credit memo issued for a different shipment from the same vendor is not evidence about this invoice, and an automated join on vendor and amount alone will make that mistake often enough to poison the labels.

Snapshot every linked record as of the decision time and keep later changes in a separate, dated layer. The model sees the snapshot; the grader sees both. This is what lets the same packet test reasoning under the information that was actually available.

Worked task (illustrative): forty units billed, thirty-two received, a new remit-to address

This case is illustrative, built to show the packet, not taken from a real company. A distributor receives a two-page scanned invoice from a parts supplier. Page one's line table bills 40 units of one part at the agreed unit price. A footnote on page two adds a freight charge, and the remit-to block shows an address that differs from the one on file.

The purchase order line is for 40 units at the same unit price, so two-way matching on price passes. The receipt, posted the day before the invoice arrived, shows 32 units received and 8 on backorder, so three-way matching on quantity fails. The vendor master change log shows the remit-to details were changed six days earlier through an emailed request and are not yet verified.

Worked task (illustrative): forty units billed, thirty-two received, a new remit-to address
Question in the taskEvidence neededWhere it livesExpected answer
How many units are billed?Quantity cell under the Qty headerPage 1, line table, region anchor40, read with high confidence
How many units may be paid now?Receipt line quantity as of the decisionReceiving record snapshot32; the remaining 8 wait for receipt
Is the freight charge allowed?PO terms and the page 2 footnotePO header terms plus footnote anchorRoute for approval if the PO terms do not provide for freight
Can payment go to the new remit-to address?Vendor master change log and verification statusVendor record version and change entryNo; hold until the change is verified through a known contact
What happened afterward?Later payment and vendor recordsOutcome layer, within the stated windowPartial payment for received units; second invoice after the backorder ships

The counterexample: same image, different answer

Give the same image a second packet in which the receipt for all 40 units was posted an hour before the clerk reviewed the invoice. The pixels are identical; the correct quantity decision is now to pay in full. Pairs like this force a model to use the linked records rather than memorizing what invoices of a certain look tend to get.

Write the expected answer for both packets before anyone runs a model, and store them under the same pair id. When results come back, report accuracy on the pair as a unit as well as on each packet: a model that gets the full-receipt version right and the partial-receipt version wrong is reading the picture, not the records.

Other useful counterexamples: a revised invoice that supersedes an earlier one with the same number, a quantity printed in a unit of measure different from the purchase order, and a duplicate submission arriving through a second channel. Each should be its own packet with its own anchors, so the dataset teaches why the answer changed.

Quality checks before a packet counts as a training or evaluation item

A packet is ready when a second reviewer, given only the anchors and snapshots, reaches the same decision and reason code. If they need to open the original systems to understand it, a link or snapshot is missing.

At a glance

Quality checks before a packet counts as a training or evaluation item

  • Reproducibility: every reason code cites at least one region anchor or linked record id.
  • Time integrity: no linked record in the model-visible layer is dated after the decision.
  • Extraction honesty: low-confidence values keep their original confidence and are flagged, and corrections are recorded as corrections.
  • Balance: matched invoices, quantity exceptions, price exceptions, vendor changes and duplicates are each present in useful numbers, so the set does not reward approving everything.
  • Release copy: vendor names, bank details, addresses and personal names are masked or replaced consistently across the image and the linked records, so the links still resolve.
  • Versioning: the packet carries a dataset version and date, and any re-scan or re-parse produces a new version rather than editing anchors in place.

Getting real invoice packets with their linked records

Synthetic invoices can teach layout but rarely contain the receipts, vendor changes and later credit memos that make the reasoning hard. Those exist in a business's own payables history, where invoice images sit next to the ERP records and the outcome of each exception.

Payables histories like these can be licensed through SourceX, which sources each dataset from U.S. businesses once a buyer has described the scope, volume, format, timing and licensing needs. Nothing leaves the supplying company until it has approved the scope and terms and signed, and SourceX de-identifies the records through approved processing routes, including the image regions that show vendor and bank details.

Frequently asked questions

What is a multimodal invoice reasoning dataset?

It is a set of invoice decisions in which each item combines the invoice image or PDF with the purchase order, receipt and vendor records needed to make the decision, plus the decision itself and its later outcome. The model must read the page and reason across the linked records.

Why not just use an invoice OCR or document VQA dataset?

Those datasets test whether a model can read and extract values from a page. Payables decisions also depend on what was ordered, what was received and who should be paid, which never appear on the invoice. Extraction datasets are a useful first stage, not a substitute.

What is the difference between two-way and three-way matching?

Two-way matching compares invoice prices with the purchase order. Three-way matching also compares invoiced quantities with the quantities recorded as received. A task built on three-way matching needs the receiving record, not just the invoice and order.

How should an answer point to the image?

Record the file version, page index and a region for each value, ideally both the parser's polygon and a percent-based rectangle measured from the top-left corner. Add the layout element type, such as table cell or footnote, so a reviewer can see what was read.

What counts as the outcome of an invoice decision?

A later record that confirms or reverses the decision, such as a cleared payment, a credit memo, a corrected invoice or a closed dispute, within a stated window. An approval stamp or a posted status shows the decision was recorded, not that it was right.

Sources

  • Microsoft's Dynamics 365 Finance documentation defines two-way matching as comparing invoice and purchase order prices, and three-way matching as also comparing invoice quantities with product receipt quantities. Source
  • Google's Document AI data model gives each extracted entity mention text, a confidence from 0 to 1 and a page anchor with bounding polygons and an optional layout type. Source
  • CORD describes over 11,000 Indonesian receipts with box and text annotations and multi-level semantic labels for post-OCR parsing, and has released a 1,000-receipt sample. Source
  • DocVQA poses about 50,000 extractive questions over more than 12,000 document images. Source
  • The W3C Web Annotation Data Model defines a FragmentSelector that identifies a segment of a resource through the fragment component of an IRI, such as an xywh region. Source
  • Media Fragments URI 1.0 defines xywh as x, y, width and height in pixels by default or in percent, with the origin at the top-left corner. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies