AI uses for records
Agent evaluation artifact correctness: right answer, wrong workbook
By SourceX Editorial · Updated
10 sections
Short answer
A correct final answer delivered with an incorrect workbook or document should count as a failed task. Grade the reply and the artifact separately: recalculate the workbook, inspect its formulas, ranges and structure, and check every document that quotes it. Require both to pass for verified completion, and report answer accuracy, artifact correctness and the mismatch rate side by side, with severity set by the business owner.
Evaluation split
Agent evaluation artifact
Key takeaways
- A right answer with a wrong deliverable is a failed task; grade the reply and the file separately.
- Recalculate workbooks before grading, because files store formulas and last calculated values separately.
- Rerun the workbook with altered inputs to expose typed constants that match by coincidence.
- Check documents in their revision and comment state, and match every quoted figure to the source.
- Report the share of right answers with failed artifacts, and fill a coverage matrix of task families and mismatch types.
How do you evaluate a correct final answer that comes with an incorrect workbook?
Count it as a failed task. Grade the final answer and the artifact as two separate results, require both for verified completion, and record the artifact failure with a severity that reflects what the business would do with the file. Report answer accuracy, artifact correctness and verified completion side by side, so a model that talks well but leaves broken files cannot hide behind a single score.
The pattern is common in spreadsheet and document tasks. An agent states the right figure in its reply, but the workbook it saved holds a typed constant where a formula belonged, a range that stops short of the last row, or a memo that still shows last quarter's number. The reply looks finished; the file the business would actually reuse is wrong.
This page is for evaluation statisticians and business task owners designing agent evaluations for office work. It gives a checklist for spreadsheets and documents, a coverage matrix to find missing task families, and a severity rule for scoring mismatches between answer and artifact.
The file outlives the reply, which is why benchmarks grade deliverables
The workbook or document is what gets forwarded, recalculated next month and relied on by people who never saw the chat, so an evaluation of office work has to grade the file. Several public benchmarks have moved in that direction.
GDPval builds tasks from actual work product created by expert professionals, with deliverables such as spreadsheets, presentations and documents, and relies on blinded expert comparison of those deliverables. OSWorld pairs each task with an initial-state configuration and a custom execution-based evaluation script that inspects the result on a real operating system. SpreadsheetBench, built from real spreadsheet questions posted in Excel forums, checks each solution against several spreadsheets that share a structure but hold different data, and counts a solution as correct only if it works on all of them.
That last design matters for answer and artifact mismatches. A typed constant can match the expected value on one file by coincidence; it cannot match on three files with different data. Checking the artifact under more than one input is the cheapest way to separate a working workbook from a right number pasted into a cell.
What a grader actually reads when it opens a workbook
A saved workbook stores each formula and its last calculated value separately, so a grader that reads values without recalculating may be grading stale numbers. Microsoft's Open XML documentation describes the cell formula element and a separate cell value element that holds the cached result from the last time the formula was calculated; if the value is omitted, the reading application is responsible for computing it.
Common tooling inherits that split. The openpyxl library's data_only option returns either the formula, by default, or the value stored the last time Excel read the sheet. A file produced by a script may carry no cached values at all, and a file saved from Excel in manual calculation mode can carry values that do not reflect the latest edits, although Microsoft notes that choosing manual mode turns on recalculate before saving by default.
The practical rule: recalculate in a spreadsheet engine inside the grading environment, then read both formulas and values. Compare values with the expected outcome, and compare formulas with what the task required, such as a live reference to the source data rather than a typed result.
Documents hide errors in revisions, comments and stale figures
A document can read correctly on screen and still be the wrong deliverable. In WordprocessingML, tracked insertions and deletions are stored as revision elements with author and date information, so a file can contain unaccepted changes, deleted text still present in the markup, and comments the recipient was never meant to see.
Grade documents in the state the business would send them. Check whether the task asked for revisions to be accepted, rejected or left for review; confirm no comments or hidden text remain when the deliverable is external; and extract every figure, date and name in the text and compare it with the source workbook or record. Stale numbers copied from a previous version are the document equivalent of a cached spreadsheet value.
The artifact correctness checklist for spreadsheets and documents
Run every applicable check on every artifact the task names as a deliverable, after recalculation and in the accepted-revision state the task requires. A check that cannot run because the file does not open counts as failed.
| Check | Applies to | How to verify | Typical failure it catches |
|---|---|---|---|
| Opens with the business's tool and a strict parser | Both | Open in the target application or a strict library; log warnings and repairs | Corrupt file the reply described as saved |
| Recalculated values match the stated answer | Spreadsheet | Recalculate in an engine, then compare key cells with the answer and the gold outcome | Right reply, stale or wrong cached value |
| Live logic where the task requires it | Spreadsheet | Inspect formulas in target cells; rerun with altered inputs | Typed constant in place of a formula |
| Ranges and references cover the right rows and period | Spreadsheet | Trace precedents of each output cell to source rows; plant boundary rows | Range stops before the last row or spans the wrong month |
| Structure preserved | Spreadsheet | Diff sheet names, named ranges, hidden rows, filters and protected areas against the input | Deleted helper sheet or filter left on |
| Figures agree across deliverables | Both | Extract numbers and dates from memo, deck and workbook; match with unit and rounding rules | Memo still quotes the previous quarter |
| Revision and comment state as instructed | Document | Read revision and comment parts of the file, not the rendered page | Unaccepted changes or internal comments in an external file |
| Required sections and cross-references correct | Document | Check headings, tables and references against the template or spec | Missing section or table numbering that points nowhere |
| Out-of-scope content unchanged | Both | Diff everything outside the requested edit against the input file | Collateral edits to rows or paragraphs nobody asked to change |
| Name, location and format as requested | Both | Compare path, file name, format version and document properties with the spec | Saved to the wrong folder, wrong format, or author details left in properties |
Which task family or failure type is missing from your evaluation?
Lay the task families you care about against the mismatch types in a coverage matrix, and treat every empty cell as either a deliberate exclusion you write down or a gap you fill before reporting results. Most evaluation sets over-represent the easiest pairing: one workbook, one output cell, one number in the reply.
The rows below are illustrative families for office work; replace them with the families in your target workload. Fill each cell with the number of items you hold and the severity that drives it.
| Task family (illustrative) | Deliverables | Mismatch types to include | What drives severity |
|---|---|---|---|
| Month-end reconciliation | Workbook and exception list | Constant instead of formula, short range, wrong period | Whether the file is reused next month |
| KPI or variance reporting | Workbook and summary memo | Memo figure differs from workbook, stale cached value | Whether the memo goes to leadership or outside |
| Policy or procedure update | Document with tracked changes | Unaccepted revisions, missing section, wrong effective date | Whether staff act on the document |
| Data cleanup for import | Spreadsheet sent to another system | Wrong column order, text stored as number, deleted rows | Whether the import would fail or load bad data |
| Contract or report summary | Document drawn from source files | Figures or dates not matching source, leftover comments | Whether the document is external |
| Multi-file deliverable | Workbook, memo and slide | Figures disagree across files, one file not saved | Which file the reader trusts |
How much a wrong deliverable should cost the score
Score the artifact on a severity scale set by the business task owner before grading, and let the worst failure decide whether the task counts as verified complete. A cosmetic issue can pass with a note; a wrong figure in a file that leaves the team should fail the task even when the reply was perfect.
At a glance
How much a wrong deliverable should cost the score
- Critical: a deliverable figure, date or name is wrong in a file that will be sent externally or will drive a decision. Task fails.
- High: the logic is broken so the file will be wrong next time, such as constants in place of formulas or a range that will not extend. Task fails.
- Medium: the file is correct but needs rework before use, such as a missing section, revisions left open, or a filter left on. Task fails unless the spec says otherwise; record the rework.
- Low: naming, formatting or layout issues with no effect on content. Task passes with the issue logged.
- Report the share of right answers whose artifacts failed at critical or high severity separately. It measures how often a reader of the reply alone would be misled.
Worked example (illustrative): a right DSO figure in a broken KPI workbook
The case is invented; figures are illustrative counts and durations. An agent is asked: what was third-quarter days sales outstanding, and update the KPI workbook and the one-page summary memo. The verified answer is 47 days.
The agent replies with 47 days and a correct explanation. The answer grade passes. The artifact checks tell a different story. After recalculation, the workbook's third-quarter cell shows 47, but it holds a typed constant rather than the formula used in every other quarter, so the live-logic check fails. Rerunning with altered receivables data leaves the cell at 47, confirming it. The fourth-quarter column, which the agent also touched, now references a range that stops one month short, so the reference check fails. The memo quotes 74 days, a transposition, so the cross-deliverable check fails.
Severity: the memo error is critical because the memo goes to the leadership team; the constant and short range are high because next quarter's figure will be wrong. Verified completion: fail. Under a reply-only grader this run would have counted as a success, which is why the evaluation reports the mismatch rate as its own number.
When the artifact should not decide the grade
Do not grade scratch files as deliverables. If the task lets the agent build a working sheet and only names the final workbook as the output, the working sheet's state should not affect the score unless it leaked into the deliverable.
Grade meaning, not position, when several layouts are valid. If the business would accept the result in a new column or on a new sheet, check the semantics through named outputs or labels rather than fixed cell addresses, and add the alternative layouts as items that must pass.
Stop and narrow the scope when the task owner cannot say which properties of the file matter, when the gold deliverable itself has unresolved errors, or when the right figure depends on judgment experts disagree on. Those tasks need a business owner or domain reviewer to define the artifact standard before any automated check means anything.
Real deliverables with their verified final versions
Artifact checks need gold files: the workbook or document as the business actually finalized it, with the source data and the history of edits that produced it. Real deliverables carry the formulas, revisions and cross-file references that synthetic tasks tend to leave out.
Finalized workbooks and documents, together with their source files, can be specified in a request to SourceX. It sources such deliverables from U.S. businesses only when asked, with the buyer stating scope, volume, format, timing and licensing needs, and it manages rights review, contracting, preparation and delivery for the businesses licensing them.
Frequently asked questions
Should a correct answer with a wrong spreadsheet get partial credit?
Report it as correct on the answer and failed on the artifact, and do not count it as a verified completion. Partial credit merges two different facts into one number and hides how often the file a business would keep is wrong.
Why recalculate a workbook before grading it?
Spreadsheet files store a formula and its last calculated value separately, and libraries can read either. A file written by a script may have no stored values, and a file saved without recalculation can show stale ones, so the grader should recalculate in an engine before comparing values.
How do you catch a typed constant that happens to match the expected value?
Rerun the workbook with altered input data and check whether the output changes as it should. SpreadsheetBench applies the same idea by testing each solution on several spreadsheets with the same structure and different data.
What should a document check look at beyond the visible text?
The revision and comment parts of the file, the document properties, and every figure and date compared with the source data. Tracked changes and comments can remain in a file that looks clean on screen.
How many artifact checks does an evaluation item need?
Every check that applies to the deliverables the task names, usually four to eight for a workbook and memo task. Checks that cannot apply, such as revision state for a spreadsheet, are marked not applicable rather than passed.
Sources
- Open XML spreadsheets store each formula separately from a cached value holding the result of the last calculation. Source
- WordprocessingML records tracked changes as revision elements such as insertions, deletions and paragraph property changes, with author and date. Source
- openpyxl's data_only option returns either the formula or the value stored the last time Excel read the sheet. Source
- Excel's manual calculation mode updates formulas only on request and selects recalculate before saving by default. Source
- SpreadsheetBench tests each solution on several spreadsheets that share a structure but differ in data, and counts it correct only if all pass. Source
- OSWorld pairs each task with an initial-state configuration and a custom execution-based evaluation script. Source
- GDPval builds tasks from real expert work product, including spreadsheet and document deliverables, graded by blinded expert comparison. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.