Systems and records
Spreadsheet screenshot reasoning data: what a picture of a workbook leaves out
By SourceX Editorial · Updated
9 sections
Short answer
A spreadsheet screenshot keeps what one viewer saw: rendered text and layout. It loses formulas, the stored values behind formatted numbers, hidden rows, cells outside the window, other sheets, linked files and version. Screenshot-only data can teach table reading but not why a number is what it is. Pair each image with its native workbook, link answers to cells and image regions, and compare inputs on matched tasks.
Document stack
Spreadsheet screenshot
Key takeaways
- Number formats change how a value looks, not the value Excel calculates with, so the picture and the cell can disagree.
- Formulas, hidden rows, off-screen ranges, other sheets and external links are absent from a typical screenshot.
- Images are the right input when the deployed system will receive images, such as scanned reports or screen-based agents.
- A fair comparison runs identical questions on one frozen workbook state through image, workbook and paired inputs.
- Score answer correctness, evidence correctness and calibrated refusal per task family, not one overall number.
What is missing when a spreadsheet dataset contains only screenshots?
A screenshot keeps what one viewer saw at one zoom level: rendered text, layout and color. It drops the formulas behind each number, the stored values behind each formatted display, hidden rows and columns, every cell outside the visible window, the other sheets, linked files, comments and the version the image came from. Reasoning data built only from screenshots can teach a model to read a table, but it cannot teach or test why a number is what it is.
That gap matters for anyone buying spreadsheet reasoning data. If the question is "why did the forecast change", the answer lives in a formula chain and an input that changed, often on a sheet nobody photographed. A model trained on pixels alone learns to guess plausible explanations, and a grader working from the same pixels cannot tell a good guess from a correct one.
The fix is not to discard images. It is to pair each image with the native workbook it came from and link every answer to the exact cells and image region that support it, then compare how models do with each input on matched tasks.
Why the number in the picture is not the number in the cell
Excel number formats change how a value looks, not the value used in calculations. Microsoft's documentation says formats do not affect the cell value Excel calculates with, that a column too narrow for all decimal places shows a rounded number while calculations use the stored one, and that a custom format of three semicolons hides a value entirely while it remains in the cell.
So a screenshot that shows a margin of 4% may hide a stored value of 0.0449, and two cells that both display 4% may hold different numbers. Any task that asks a model to check a total, explain a variance or reconcile two sheets can fail for reasons that are invisible in the image. Formulas are hidden by default too: Excel shows results in cells and formulas in the formula bar unless someone switches on Show Formulas for the whole sheet.
Native files keep both layers. A library such as openpyxl can open the same workbook either with formulas or, using its data_only option, with the values Excel stored the last time it saved the sheet. A dataset that captures both, plus the rendered image, lets a grader check what was displayed, what was stored and how it was computed.
Which parts of a workbook never reach the image?
Before choosing a modality, list what each task actually depends on. The table below covers the workbook elements most often needed for reasoning questions and whether a typical screenshot preserves them.
| Workbook element | Where it lives | In a typical screenshot? | What reasoning loses without it |
|---|---|---|---|
| Formulas | Cell definitions, shown in the formula bar | No, unless Show Formulas was on | Why a value is what it is; dependency chains |
| Stored values | Cell value behind the number format | Only as rounded or formatted text | Exact checks, reconciliations, tie-outs |
| Hidden rows and columns | Still in the sheet, flagged hidden | No | Excluded items that still feed totals |
| Cells outside the window | Rest of the grid, up to 1,048,576 rows by 16,384 columns | No | Inputs, assumptions and lookup tables |
| Other sheets and named ranges | Same workbook | Only the active tab | Cross-sheet references and definitions |
| External links | References to other files | No | Which source file a figure came from |
| Comments and notes | Cell annotations | Only an indicator, if any | Analyst caveats and overrides |
| Version | File history or folder naming | No | Whether the image matches the file being graded |
When is an image the right input after all?
Images are the right input when the image is what the system will actually receive. The authors of MMTab, a multimodal table dataset released with the Table-LLaVA model, note that high-quality text versions of tables are hard to get from sources such as scanned documents and webpage screenshots, where images are far easier to obtain. Agents that work through a screen, reviewers reading pasted screenshots in tickets, and archives of scanned financial reports all fit that description.
Image tasks also have a mature evaluation pattern of their own. PubTabNet pairs more than 568,000 table images from scientific articles with the HTML structure of each table and, since version 2.0, the bounding box of each non-empty cell. That makes image-to-structure recognition gradeable. OCR engines can report a confidence value and bounding box for each word, which Tesseract's API documentation shows how to retrieve, so an image dataset can record where text extraction was shaky.
What images cannot do is stand in for the workbook on tasks whose answer depends on formulas, stored precision or content outside the frame. Match the input to the task, and when the task depends on both, provide both.
The modality tradeoff comparison with matched tasks
The comparison runs the same task, on the same workbook state, through three inputs: the image alone, the native workbook alone, and the pair with region links. Each task family below says which input should win if the data is built correctly and what would make the result misleading. Run every family through all three inputs and compare scores within a family, not across families.
- Freeze the workbook state, render the image from that exact state and store a hash of both.
- Write each question once and hand the identical wording to every input arm.
- Store the answer as a value and as the cell range that justifies it, plus the image bounding box where that range appears.
- Hold out whole workbooks, not individual questions, so the same file never appears on both sides of a split.
| Task family | Image only | Native workbook | Paired with region links |
|---|---|---|---|
| Read a displayed figure | Suited; tests what a viewer sees | Can over-answer with unrounded precision | Grade against the displayed text; keep stored value for audit |
| Verify a total or ratio | Cannot see hidden rows or stored precision | Suited; recompute from cells | Shows whether the visible total matches the cells |
| Explain a change between versions | Needs two images; misses formula edits | Suited if both versions are kept | Best: cell diff plus the region that changed on screen |
| Trace a figure to its inputs | Fails when inputs are off-screen | Suited; follow precedents across sheets | Answer cites cells; image anchors the starting point |
| Recover table structure | Suited; the PubTabNet pattern | Trivial; structure is explicit | Use the workbook as the structure answer key for the image |
| Answer from a scanned report with no file | Only option | Not available | Not available; record OCR confidence instead |
How to tie each answer to an image region and its supporting records
Every answer in a paired dataset should point to evidence a reviewer can open without asking anyone. The record below is the minimum that makes an answer checkable across both modalities, and it follows the evidence checklist a document AI engineer would apply.
At a glance
How to tie each answer to an image region and its supporting records
Two failure patterns justify the extra fields. The first is drift: the image was rendered from Monday's file, the answer was computed from Friday's, and nobody can tell which is wrong. The second is a silent dependency: a cited total on the visible sheet pulls from a lookup table in another workbook that was never collected, so the answer cannot be rechecked. Recording version on both sides and listing cross-file links catches both before a task enters the set.
Keep the record in a separate manifest rather than inside the image file name or a caption, so it survives renaming and can be validated by a script that opens the workbook, resolves each cited range and confirms the bounding box falls inside the rendered image.
Checklist · 8 items
- Source file: workbook name, content hash and the system it was exported from.
- Sheet, cell range and, for formula answers, the precedent ranges on other sheets.
- Image: render settings (zoom, visible range, active sheet) and the pixel bounding box for each cited range.
- Layout: header path for each cited cell, such as region, then quarter, then metric, including merged header cells.
- Footnotes and comments that change how a cited value should be read.
- Version: the file version the image was rendered from and the version the answer was computed on, which must match.
- OCR confidence for any text that was extracted from the image rather than read from the file.
- Cross-file links: the external workbook or system record a cited value was pulled from, if any.
0 of 8 ticked. Tick each item as you go.
Worked example: a margin that looks flat but moved
This case is illustrative and invented, and its figures are percentages chosen to show the mechanics. A regional operations workbook has a Summary sheet showing gross margin by region and a Lanes sheet listing shipping lanes with volumes and costs. The screenshot of Summary shows the Northeast margin at 4% in both March and April.
The question is: did Northeast margin change between March and April, and why? From the image alone, the honest answer is "no visible change". From the workbook, the stored values are 0.0449 in March and 0.0362 in April, both displayed as 4% because the format shows no decimals. The April drop traces through the Summary formula to the Lanes sheet, where two lanes, hidden after being marked inactive, still feed the regional sum and carried higher April costs.
The paired record cites the Summary cells and their bounding box, the precedent range on Lanes, the two hidden rows, and the format string that rounded the display. A model given only the image should answer "no visible change" and say it cannot see the underlying values; a model that confidently explains a cause from the image is hallucinating, and the grader can tell because the cited evidence is not in its input.
Exception: if the business question is "what did the April deck show the board", the image is the authoritative record and the displayed 4% is the correct answer. Keep that question type in the set so the comparison does not simply reward whichever input has more data.
How to score the comparison without rewarding the bigger input
Score each task family separately and for each input arm, with the paired arm as the reference. SpreadsheetBench, a NeurIPS 2024 benchmark built from 912 real questions posted on Excel forums, uses several test cases per instruction (2,729 in total) in an online-judge style, so a solution has to work on more than one version of the sheet. Borrow that idea: generate variant workbooks for each question by changing inputs, and require the answer to hold on all of them.
Report three things per arm: answer correctness, evidence correctness (did the cited cells or region actually support the answer) and calibrated refusal (did the model decline when its input could not support an answer). Without the third measure, image-only arms look worse than they should on unanswerable questions and native arms look better than they should on display questions.
State the limits with the results. Rendering choices such as zoom, fonts and gridlines change image difficulty, so publish render settings. Workbooks from one department or one template family will flatter whichever arm suits that template. No results are reported on this page; it describes the method only.
Sourcing native workbooks with their rendered views
Public spreadsheet sets are mostly forum examples or synthetic files. Business workbooks bring the parts that make this comparison informative: hidden rows with reasons, formats that round, cross-sheet chains and versions saved over months. They also carry customer names, pricing and personal data, so de-identification has to preserve formula semantics, and a reviewer has to confirm the business can license the files at all.
A buyer who wants native workbooks paired with rendered views can ask SourceX to find them; SourceX approaches U.S. businesses for each request and does not sell spreadsheet sets from stock. The request covers scope, volume, language, file format, timing and license. Files move only after the contributing business approves the scope and signs, and a collection too large to host can remain in that business's storage or be shipped on encrypted drives. It also helps to state the task families, whether rendered images are needed beside the files, and how many versions of each workbook you want.
Frequently asked questions
Can a vision model read formulas from a spreadsheet screenshot?
Only if the formulas were visible when the image was taken, which requires switching on Show Formulas for the whole sheet. In a normal view, cells show results and formulas appear only in the formula bar for the selected cell, so the image does not contain them.
Is a spreadsheet screenshot dataset useless for training?
No. It suits tasks where the image is what the model will receive, such as screen-based agents, scanned reports and table structure recognition. It is the wrong input for tasks that depend on formulas, stored precision or content outside the visible window.
What is the difference between a displayed value and a stored value in Excel?
The stored value is what Excel calculates with; the displayed value is that number after the cell's number format is applied. Formatting can round, add symbols or hide the value entirely without changing what is stored.
How do you link a screenshot region to the right cells?
Render the image from a frozen workbook state with recorded zoom and visible range, then map each cited cell range to its pixel bounding box. Store both, together with a hash of the workbook and the image, so either can be checked against the other.
Should OCR confidence be part of a spreadsheet reasoning dataset?
Yes, whenever any answer depends on text extracted from an image rather than read from the file. Recording word-level confidence lets graders separate reasoning errors from extraction errors.
Sources
- Microsoft says number formats change how a value appears without changing the value Excel calculates with, and a ;;; format hides a value that remains in the cell. Source
- Microsoft says cells show formula results by default, and the Show Formulas command switches the whole sheet to display formulas. Source
- Microsoft explains that hidden rows and columns remain in the worksheet and can be unhidden. Source
- Excel's specifications list a worksheet size of 1,048,576 rows by 16,384 columns. Source
- openpyxl's data_only option returns the value Excel stored at last save instead of the formula. Source
- The MMTab and Table-LLaVA authors note that text versions of tables are hard to obtain from scanned documents and webpage screenshots, where table images are more accessible. Source
- PubTabNet pairs 568k+ table images with HTML structure annotations and, since 2.0.0, cell bounding boxes. Source
- Tesseract's API documentation shows how to get a confidence value and bounding box for each word. Source
- SpreadsheetBench contains 912 real questions from Excel forums with 2,729 test cases and online-judge style evaluation. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.