AI uses for records
Multimodal business chart interpretation without the source data
By SourceX Editorial · Updated
10 sections
Short answer
Test chart reasoning without the underlying data by asking only questions the image can settle, labeling each as printed, estimable, relational, needing another record or not determinable, and scoring answers on correctness, cited page region, precision and abstention. A model that returns an exact figure for an unlabeled mark fails, even if the guess is close. Keep a held-out split with tables withheld.
Document stack
Multimodal business chart
Key takeaways
- Classify every chart question by what the image can settle before writing a gold answer.
- Not determinable is a correct answer for unlabeled, occluded or scale-free values, and the test should reward it.
- Score the cited page region separately from the answer so lucky guesses do not pass as readings.
- Treat an exact figure for an unlabeled mark as a hard fail for invented precision.
- When the source table exists, use it for gold answers and reconciliation, but keep a split where it is withheld.
How do you test chart reasoning without access to the underlying data?
Write questions whose correct answer is fixed by what the image itself shows, label each one with how much the image can settle, and grade the model on whether its answer stays inside that limit. A printed data label can be read exactly. A bar height against gridlines can be estimated within a stated tolerance. A value hidden inside a stacked segment with no label and no scale cannot be known from the picture at all, and the right answer is to say so.
That last category is the one most chart tests leave out, and it is where invented numbers come from. When the source spreadsheet is missing, a model that returns a precise figure for an unlabeled segment is not reading the chart. It is producing a plausible guess and presenting it as a reading.
The method on this page has two parts: a chart grounded question set, where every item records its answerability class and the exact region of the page that holds the evidence, and an evidence rubric that scores the answer, the cited region and the decision to abstain separately. Both work on board packs, investor decks, monthly operations reports and screenshots pasted into tickets, where the table behind the chart was never kept.
Why do public chart benchmarks assume a table you rarely have?
Most public chart question answering sets ship the data with the picture. The ChartQA repository, for example, organizes each split into chart images, annotation files with bounding boxes for marks, axes and legends, and a tables folder holding the underlying data as CSV. That design makes grading easy: the gold answer comes from the table, not from a person squinting at pixels.
Some strong methods also lean on that table. DePlot, published at Findings of ACL 2023, translates a plot image into a linearized table and then hands that text to a large language model for reasoning. When the conversion is right, the reasoning step has clean numbers to work with. When the conversion has to guess a value the image never showed, the guess enters the table looking exactly like a measured value.
Business documents break the assumption in ordinary ways. A chart is copied into a slide as a picture, the workbook stays on someone's laptop, and a year later only the PDF survives in the shared drive or the CRM attachment. NIST's generative AI profile names the resulting failure confabulation: confidently stated but erroneous or false content. An image-only test is how you find out whether a model knows the difference between reading and reconstructing.
Which questions can a chart settle on its own?
Sort every candidate question into one of five answerability classes before you write a gold answer. The class decides what a correct response looks like, so a question cannot be scored until it has one.
| Class | What fixes the answer | Illustrative question | Correct model behavior |
|---|---|---|---|
| A. Printed | A data label, axis title, legend entry or note printed on the chart | What total does the chart print for the third quarter? | Return the printed value exactly and cite the label region |
| B. Estimable | Position of a mark against a labeled linear scale with gridlines | Roughly how many tickets did the phone channel handle in the first quarter? | Return an estimate with an explicit range, never a falsely precise figure |
| C. Relational | Visible ordering, direction or size difference between marks | Did chat exceed email in the second quarter? | Answer the comparison and cite both marks |
| D. Needs another record | A footnote, a definition elsewhere in the document or a linked file | Is the drop in the fourth quarter a real decline? | Name the record that is needed and answer only if it is supplied |
| E. Not determinable | Nothing in the image fixes it: no label, no scale, occluded or truncated | What exact count sits in the unlabeled middle segment? | Abstain and say what is missing |
What should the chart grounded question set include?
Each item is one question about one chart, stored with enough provenance that a second reviewer can find the same pixels and reach the same verdict without asking the author. The fields below are the minimum; a team can add fields, but dropping any of them makes the item impossible to audit.
At a glance
What should the chart grounded question set include
- item_id and set_version: a stable identifier and the dated version of the question set it belongs to.
- source_file and file_version: the PDF, slide deck or image the chart came from, with a hash or version so later edits are detectable.
- page and region: page number and the polygon or bounding box of the chart, recorded the way layout tools report figures. Microsoft's Document Intelligence layout model, for instance, returns each figure with bounding regions (page number plus polygon), spans into the document text, related element ids and a caption.
- chart_type and scale_type: bar, stacked bar, line, pie, combo or other, and whether the value axis is linear, logarithmic, broken or absent.
- question and answerability_class: the question text and its class A to E from the table above.
- gold_answer and tolerance: the exact value for class A and C, a range for class B, the required record for class D, and the literal answer not determinable for class E.
- evidence_regions: one or more boxes on the page that contain the label, mark or footnote the answer depends on.
- context_records: any other page, footnote, glossary entry or linked file a class D item needs, with its own location.
- label_ocr_confidence: the confidence your OCR step reported for the printed labels used as evidence, so low-confidence readings can be excluded or rechecked.
- author, reviewer and review_date: who wrote the item, who checked it, and when.
How should the evidence rubric score an answer?
Score four things separately and keep a hard fail for invented precision. A combined accuracy number hides the failure you most care about: a model that is right on printed values and wrong in a confident way on everything else can still post a respectable average.
CharXiv offers a useful precedent. Its descriptive questions tell the model to answer Not Applicable when, for example, a plot has no explicit axis label or no legend, and its grading instructions give credit only when the response expresses Not Applicable for those items. The rubric below generalizes that idea to business charts.
| Dimension | Full credit | Partial credit | No credit |
|---|---|---|---|
| Answer | Matches the gold value, or falls inside the stated range for class B | Right direction or rank with a wrong magnitude | Wrong value, or a value given for a class E item |
| Evidence citation | Points to the region that actually holds the label or mark | Points to the right chart but the wrong series or period | No citation, or a citation to a region that does not support the answer |
| Precision discipline | Reports a range or rounding that matches what the scale allows | Over-precise but inside tolerance | Exact figure for an unlabeled mark presented as read from the chart |
| Abstention | Declines class E items and names what is missing | Declines but gives no reason | Answers a class E item, or declines a class A item |
| Context use | Requests or uses the needed footnote or record for class D | Mentions uncertainty without naming the record | Ignores a footnote that changes the meaning |
How do the five classes apply to one stacked bar chart? (illustrative)
The following example is illustrative. Picture a stacked bar chart titled support volume by channel, with four quarterly bars split into phone, email and chat. Only the bar totals carry data labels. The value axis has gridlines every 2,000 tickets, and a small footnote reads that chat was counted from a new system starting in the third quarter.
Five questions cover the classes. The printed third-quarter total is class A. The phone segment in the first quarter, which sits between two gridlines, is class B with a tolerance of one gridline step either way. Whether chat exceeded email in the second quarter is class C, settled by segment heights. Whether the jump in chat from the second to the third quarter reflects real demand is class D, because the footnote about the new counting system decides it. The exact email count in the second quarter, a thin unlabeled middle segment, is class E.
A model that answers the class E item with a four-digit figure fails precision discipline and abstention, even if the number happens to land near the truth, because nothing on the page supports it. A model that calls the chat jump growth without mentioning the footnote earns no context credit. A strong response reads the printed total, gives a range for the phone segment, answers the comparison, flags the counting change and declines the exact email count while pointing out that the segment has no label.
Run the same five items against two or three charts of each type you care about before scaling up. Small pilots surface ambiguous wording quickly: if two reviewers disagree on an item's class, rewrite the question rather than averaging their answers.
Which charts belong in the set, and how many of each?
Sample charts the way your production documents will present them, not the way a benchmark does. If the model will read quarterly decks, most of the set should be slide exports with small fonts and partial labels; if it will read scanned board packs, include scans. A set built only from clean, fully labeled charts will mostly test class A reading and overstate how well the model copes with real files.
Stratify on three things you can record from the file alone: chart type, scale type and labeling density. Labeling density is the share of marks that carry a printed value, and it moves items between class A and class B or E more than anything else. A chart where every bar is labeled tests very little beyond OCR.
Within each stratum, aim for a spread of answerability classes rather than equal counts per chart. In practice, some charts support only classes A and C, while dense stacked charts are where class E items naturally arise. Track the class mix at the set level and top up thin classes with targeted questions on charts that support them.
Hold back a slice of the set from anyone tuning prompts or models. Chart sets are small enough that a team iterating against them can learn their quirks, and once that happens the score reflects familiarity with the questions rather than reading skill. Version the set, date each release, and record which models have been run against which version.
What changes when the source table does exist?
When the workbook behind a chart is available, use it to set gold answers, but keep it out of what the model sees and check that it matches the chart before trusting it. Tables drift: a deck may have been exported from an earlier version of the spreadsheet, a filter may have been applied for the slide, or values may have been rounded for display.
Record the table's version next to the chart's version and add a reconciliation check: for every class A item, the printed value must agree with the table. Disagreements are not noise; they tell you the chart and the table describe different states of the business, and the image is what a model will meet in production.
Keep a held-out split of charts whose tables are deliberately withheld, even when you have them. That split measures the behavior this page is about. A model that does well only when a clean table is available has been tested on extraction, not on reading.
Where does image-only testing stop and specialist review start?
Some charts cannot be fairly tested from the image alone, and forcing them into the set produces gold answers nobody can defend. Route these to a reviewer who knows the chart's domain, or leave them out and say so in the set's documentation.
- Logarithmic, broken or dual value axes, where class B estimates depend on which axis a series uses and readers often disagree.
- Three-dimensional bars and exploded pies, where perspective distorts heights and angles.
- Charts whose labels came back from OCR with low confidence, or that are scanned at a resolution where gridlines blur.
- Charts that summarize regulated or confidential figures, where even the question set needs a decision about who may see it.
- Charts with a published long description. The W3C tutorial on complex images says charts and graphs need a long description that conveys their essential information as text. Such a description is useful context, but treat it as a separate record with its own version, not as proof of the values.
How should a request for business charts be written?
If you are sourcing real charts for this kind of test, write the request around the answerability mix you need rather than a raw chart count: how many charts per type, what share must have tables for reconciliation, which scale types to include or exclude, and what context records must travel with each file.
Chart collections are among the business records SourceX takes through sourcing, rights review, contracting, preparation and delivery, and they come from U.S. businesses once a buyer asks rather than from a shelf. The request form's fields for scope, volume, language, format, timing and licensing are the natural home for an answerability mix and the reconciliation rules above.
Frequently asked questions
Can a vision model read exact values from a chart with no data labels?
Only within the precision the scale allows. With a labeled linear axis and gridlines, a careful reader can estimate a range; without labels or a scale, the exact value is not in the image. A test should reward ranges for estimable marks and abstention for values the picture does not show.
How is this different from chart-to-table extraction?
Extraction asks a model to rebuild the full data table, which forces it to output a value for every mark. An image-only reading test asks targeted questions and treats not determinable as a valid answer. The second catches invented precision that the first quietly rewards.
How many questions per chart are enough?
Enough to cover every answerability class the chart supports, which for a typical business chart is three to six questions. Coverage across chart types and scale types matters more than depth on any single chart. Start with a small pilot and expand once reviewers agree on classes.
Should the evidence region be required in the model's answer?
Yes, if you want to tell lucky guesses from readings. A right value with no citation, or a citation to the wrong series, earns only partial credit in the rubric. Grading the region also makes disagreements between reviewers easier to settle.
What if two reviewers disagree on whether a value is estimable?
Treat the disagreement as a signal that the item is ambiguous. Tighten the question, widen the tolerance, or move the item to class E. Averaging two conflicting gold answers produces an item nobody can defend later.
Sources
- The ChartQA repository describes its dataset as chart images, annotation files with bounding boxes for marks, axes and legends, and a tables folder with the underlying data as CSV. Source
- DePlot (Findings of ACL 2023) converts a plot or chart image into a linearized table that a large language model then reasons over. Source
- NIST AI 600-1 describes confabulation as the production of confidently stated but erroneous or false content. Source
- The Document Intelligence layout model reports each figure with bounding regions (page number and polygon), spans, related elements and a caption. Source
- CharXiv's descriptive questions instruct models to answer Not Applicable when a plot lacks, for example, an explicit axis label or a legend, and its grader credits only responses that express Not Applicable for those items. Source
- The W3C WAI complex images tutorial says graphs, charts and diagrams need a long description that gives their essential information in text. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.