AI uses for records
Multimodal evidence sufficiency: does the image support the business decision?
By SourceX Editorial · Updated
10 sections
Short answer
To test whether an image supports a business decision, split the decision into claims such as identity, condition, timing, coverage, integrity and outcome, and point each claim to a marked image region or linked record. Grade each claim supported, partly supported, unsupported or contradicted. The image supports the decision only when every material claim is supported, not when the photo merely looks relevant.
Document stack
Multimodal evidence
Key takeaways
- A photo attached to an approved decision is a pairing, not proof that the photo justified the approval.
- Grade each claim the decision depends on, such as identity, condition, timing, coverage, integrity and outcome, on its own.
- Every claim needs a pointer to a file version, a region and the linked record it also depends on.
- Keep contradicted as a separate label from unsupported; images that disprove a decision are the most valuable negatives.
- Capture time and edit history help but are often missing, so confirm them against linked business events.
How do you test whether an image supports the business decision attributed to it?
Split the decision into the specific claims it depends on, then check each claim against a marked region of the image or a linked record. A claim is supported only when that region is legible, shows the right subject at the right time, has a known edit history, and connects to the rule that turned the evidence into a decision. If any material claim rests on nothing but the caption, the status field or a reviewer's note, the image does not support the decision.
This matters for anyone building multimodal training or evaluation data from business records. A support ticket with a damage photo and an approved refund looks like a clean example of visual reasoning. Often the photo shows the damage but not which unit, not when, and not whether the policy covered it, and the approval was made on evidence that never reached the dataset.
The test below has two parts: an evidence sufficiency rubric that grades each claim, and a counterexample set of image-decision pairs that look supported but are not. Use the counterexamples both to train reviewers and as hard negatives in an evaluation.
Why a matching photo and an approval are not proof
A plausible pairing is the default state of business records, not evidence of support. Photos are attached because a form asked for one, decisions are recorded in a status field by someone who may have checked other systems, and nobody records which pixels mattered. A model trained on such pairs learns that a photo of a dented box predicts approval, which is a different skill from deciding whether the box justified it.
Well-built document benchmarks avoid this by recording where the answer comes from. The MMLongBench-Doc benchmark annotates each question with evidence pages and evidence source types such as text, table, chart and image. Its authors report that 33.0% of its questions need evidence from more than one page and that 22.5% are designed to be unanswerable, to cut shortcuts and detect hallucination. Business decision data needs the same two features: an evidence pointer for every claim, and cases where the honest answer is that the evidence is insufficient.
Which claims does the decision actually depend on?
Write the decision as a short list of propositions that would each have to be true for the decision to be correct. Each proposition gets its own evidence pointer and its own grade; a decision is never graded as a whole.
At a glance
Which claims does the decision actually depend on
For a refund approved for an item damaged in transit, the propositions usually look like this:
- Identity: the photographed item is the unit on this order, shown by a serial number, SKU label or packing slip in the frame.
- Condition: the damage is visible, and is the damage type the claim describes, such as a cracked housing rather than a scuffed carton.
- Timing: the damage existed at or before delivery, supported by a capture time after delivery and a carrier or receiving record.
- Coverage: the damage type and the claim date fall inside the written return or damage policy that applied on the order date.
- Integrity: the image has not been cropped, reused or edited in a way that removes context the reviewer needed.
- Outcome: the approval was carried out and not reversed, shown by a refund or replacement record and the absence of a later chargeback or reopened case.
Pointing to the exact region, page and file version
An evidence pointer must identify the file version and the region inside it, not just the attachment. Existing standards already provide the coordinates. The IIIF Image API defines a region as x,y,w,h in pixels from the top-left corner, or pct:x,y,w,h as percentages of the full image. The W3C Web Annotation Data Model lets an annotation target a fragment such as image.jpg#xywh=100,100,300,300 and supports an SVG selector for non-rectangular areas like a circled crack.
For scanned documents and photographed paperwork, the pointer also needs the layout role and the text the reviewer relied on. DocLayNet labels page regions with 11 layout classes across 80,863 pages, which is a useful reminder that a caption, a table and a footnote on the same page carry different weight. The hOCR format records x_wconf, an engine-specific confidence for recognized text, which it recommends expressing from 0 to 100 as an approximate probability.
| Pointer field | What it holds | Check before relying on it |
|---|---|---|
| source_file and file_hash | The attachment and a content hash of the exact version reviewed | The hash matches the delivered file; a re-uploaded or recompressed copy is a different version |
| page and region | Page index plus x,y,w,h or an SVG selector for the area that matters | The region still shows the evidence after any crop or rotation |
| region_role | Photo, table cell, table header, stamp, signature block, footnote or caption | Footnotes and headers can change what a cell or caption means |
| extracted_text and confidence | OCR or human transcription of text in the region, with engine confidence | Low-confidence digits in serials, dates or quantities are rechecked by a person |
| capture_time and edit_history | Camera capture time and any recorded edits | Capture time is plausible for the claim; edits are disclosed and did not remove context |
| linked_record | The order, work order, carrier scan or policy version the claim also depends on | The link is by identifier, not by matching names or dates by eye |
The evidence sufficiency rubric
Grade each proposition separately with one of four levels: supported, partly supported, unsupported or contradicted. The decision is supported only if every material proposition is supported, either by the image region or by a linked record named in the pointer. Partly supported means a person could reasonably accept it with one more check; record which check.
The rubric below defines the levels for the criteria that come up most often in photo-backed decisions. Adapt the wording to your decision type, but keep contradiction as its own level: an image that shows something incompatible with the decision is a different and more valuable label than one that simply shows too little.
| Criterion | Supported | Unsupported | Contradicted |
|---|---|---|---|
| Subject identity | Serial, SKU, asset tag or site marker readable in the region and matching the linked record | No identifier visible, or identifier too blurred to read | Identifier visible and belongs to a different unit, order or site |
| Visibility | The condition the decision depends on is visible at a resolution a reviewer can judge | Region too dark, distant, compressed or obstructed to judge | Region clearly shows the condition absent, such as an intact item |
| Scope match | The image shows the thing the decision is about | Image shows a related thing, such as outer packaging for an item-damage claim | Image shows a different defect type than the one claimed |
| Temporal fit | Capture time and linked records place the condition in the window the rule requires | No reliable capture time and no linked event to anchor it | Capture time or another record places the condition outside the window |
| Integrity | Original file, or edits recorded and harmless to the claim | Unknown history, such as a screenshot of a photo | Same image found on an earlier claim, or an edit removed relevant context |
| Rule linkage | The policy version and clause that turned the evidence into the decision are linked | Decision recorded with no rule or a rule that cannot be located | The linked rule excludes the case the image shows |
| Text in the image | Required text read with high confidence or confirmed by a person | Text present but low confidence and unconfirmed | Text reads differently from the value the decision used |
| Outcome | The decision was carried out and not reversed within the observation window | Outcome not yet mature or not linked | Decision reversed, charged back or reopened on the same evidence |
Where capture time and edit history can mislead
Capture time and edit history are both useful and both incomplete. The Exif 2.3 standard defines DateTimeOriginal as the date and time the original image data was generated, but many upload paths strip metadata, phone clocks can be wrong, and a screenshot of a photo carries the screenshot's time. Treat capture time as one input to temporal fit, confirmed by a linked event such as a delivery scan or a job check-in.
Content Credentials help when they are present. The C2PA specification records edits as actions in a signed manifest, with labels such as c2pa.cropped and c2pa.edited. Its explainer also states that provenance is not always complete: an asset cropped in a tool that does not support Content Credentials may not have that crop recorded, and adding provenance is optional. So the absence of a manifest is not evidence of tampering, and its presence does not prove that what is in the frame is the item in the claim.
Worked case (illustrative): a refund approved from one damage photo
This case is invented to show the rubric; no real customer or company is described. A customer reports a cracked blender jar and attaches one phone photo. A support agent approves a replacement the same day. The dataset row as exported contains the photo, the ticket text and status approved. The test asks whether the row supports that approval.
With every material proposition graded supported, or partly supported with a named check, the row qualifies as a supported example. Store each proposition grade in the label rather than just approved; training or evaluating on those grades teaches a model to reason about identity, timing and coverage instead of mapping any crack to approval.
Change one fact and the verdict flips. If the photo's capture time were three days before the delivery scan, temporal fit would be contradicted even though the image is identical. The approval may still have been right for reasons outside the row, such as a goodwill policy, but the image does not support it, and the row moves to the counterexample set with the reason recorded.
| Proposition | Evidence pointer | Grade | Why |
|---|---|---|---|
| Identity | Photo region around the base label; order record | Partly supported | Model number readable and matches the order, but no serial is visible; acceptable for a one-unit order |
| Condition | Region around the jar, SVG selector tracing the crack | Supported | Crack visible across the jar wall at full resolution |
| Timing | Capture time two hours after the carrier delivery scan; carrier record | Supported | Capture time and delivery event both fall inside the claim window |
| Coverage | Return policy version in force on the order date, damage clause | Supported | Clause covers breakage reported within the stated window |
| Integrity | File hash checked against earlier claims; no manifest present | Partly supported | No duplicate found; missing provenance noted but not penalized |
| Outcome | Replacement shipment record; no reopen or chargeback after an illustrative 60-day window | Supported | Decision carried out and not reversed |
The counterexample set: pairs that look supported but are not
Collect counterexamples deliberately; they are rare in approved records and are what separates a model that reads evidence from one that pattern-matches. Each item keeps the original image, the recorded decision, the failing criterion and a one-line reason a reviewer agreed with. Labels below are illustrative categories, not counts from any dataset.
| Counterexample | What the image shows | Failing criterion | Correct label |
|---|---|---|---|
| Wrong unit | Damaged unit whose serial differs from the order | Subject identity | Contradicted |
| Packaging only | Crushed carton with the item never shown | Scope match | Unsupported |
| Pre-existing damage | Clear damage, capture time before delivery | Temporal fit | Contradicted |
| Reused photo | Same file hash as a claim filed months earlier | Integrity | Contradicted |
| Tight crop | Crack visible, but crop removes the label and surroundings | Subject identity and integrity | Unsupported |
| Footnote override | Invoice scan where a footnote limits the credit line the decision used | Text in the image | Contradicted |
| Low-confidence digits | Scanned delivery note with a quantity read at low OCR confidence | Text in the image | Unsupported until confirmed |
| Reversed outcome | Convincing photo, approval later reversed after a return inspection | Outcome | Contradicted |
Which linked record proves the decision and its outcome?
The status field proves only that someone set a status. The decision is proven by the record that executed it, such as a refund transaction, replacement shipment, vendor credit memo or closed work order, and the outcome is proven by what happened next: no reopened case, no chargeback, a passed inspection or no callback within an agreed window.
When those records live in different systems, link them by identifier when the dataset is built and keep the link in the row. A case where the photo was sufficient but the approval was reversed is still useful, because it separates evidence quality from decision quality. Label both, and do not let a reversal silently relabel the image as bad evidence.
Sourcing photo-backed decisions with their linked records
Building this kind of set needs the whole chain from one business: attachments with original files, the tickets or work orders they belong to, the policy versions, and the transaction and follow-up records that show what happened. Public image sets rarely carry that chain.
A team building an evidence sufficiency set can describe that chain in a SourceX data request: the decision types, the linked records that must travel with each image, the outcome window, and the usual scope, volume, language, format, timing and licensing terms. SourceX holds no image inventory; it looks for U.S. businesses whose photo-backed decisions fit, qualifies them and runs rights review, contracting, preparation and delivery, with delivery waiting on the supplier's approval of scope and signature.
Frequently asked questions
What is multimodal evidence sufficiency?
It is the question of whether an image, table or scanned page, together with its linked records, contains enough to justify a specific claim or decision. It is graded claim by claim, so a photo can be sufficient for the condition of an item and insufficient for its identity or timing.
Is image-text alignment the same as evidence sufficiency?
No. Alignment asks whether a caption describes the image. Sufficiency asks whether the image and its linked records justify a decision, which also depends on identity, timing, the rule applied and what happened afterward. A perfectly aligned caption can still describe an image that does not support the decision.
Can EXIF data prove when a business photo was taken?
It can help but rarely proves it alone. The capture time can be missing, stripped on upload or wrong if the device clock was wrong. Confirm it against a linked event such as a delivery scan, a job check-in or a ticket timestamp.
Should insufficient-evidence examples be removed from a training set?
Usually not. Label them with the failing criterion and keep them, because they teach a model to say the evidence is insufficient instead of guessing. Remove only rows whose label you cannot verify at all.
How many counterexamples does an evaluation set need?
There is no fixed number. Cover every failing criterion in the rubric with several items, and check that a model cannot pass by always answering supported or always answering unsupported.
Sources
- MMLongBench-Doc records evidence pages and source types for each question; 33.0% of questions are cross-page and 22.5% are designed to be unanswerable. Source
- The IIIF Image API 3.0 specifies image regions as x,y,w,h in pixels or pct:x,y,w,h as percentages of the full image. Source
- The W3C Web Annotation Data Model allows targets such as image.jpg#xywh=... and an SVG selector for non-rectangular regions. Source
- DocLayNet provides layout bounding boxes for 11 class labels on 80,863 pages. Source
- The hOCR x_wconf property is an engine-specific confidence for recognized text, ideally expressed from 0 to 100 as an approximate probability. Source
- The CIPA Exif 2.3 standard (2012 English translation) defines DateTimeOriginal as the date and time when the original image data was generated. Source
- The C2PA specification records edits as actions assertions in signed manifests, with labels such as c2pa.cropped and c2pa.edited. Source
- The C2PA explainer says provenance is not always complete, for example after a crop in a tool without Content Credentials support, and that provenance is optional. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.