AI uses for records
Audit evidence request dataset: training agents to find evidence, not invent it
By SourceX Editorial · Updated
9 sections
Short answer
Agents learn to find audit evidence from closed evidence requests linked to the exact documents, pages and fields that satisfied them, the approvals and corrections behind each transaction, and the final ledger state. The set must also include requests whose evidence is missing, superseded or contradictory, and scoring must fail any answer that cites a document not in the corpus. That makes not found a correct answer.
Checklist
Audit evidence request
Key takeaways
- The training unit is a closed request linked to the documents, locators, approvals, corrections and final ledger state that answered it.
- Requests with missing, superseded or contradicting evidence teach an agent that not found is sometimes the right answer.
- Demand locators a reviewer can re-inspect: document ID, page or field, and the value relied on.
- Any reference to a document that is not in the frozen corpus should fail the task outright.
- Company-side request histories, not the audit firm's workpapers, are the source records for this benchmark.
What training data helps agents locate audit evidence rather than invent it?
The data that teaches an agent to find audit evidence is a set of closed evidence requests, each linked to the exact documents that satisfied it, the location inside each document, the ledger entry it supports and the final state after corrections. It also needs requests where the evidence does not exist, so the agent learns that the right answer is sometimes not found. Without those negative cases, a model learns that every request has an answer and produces one.
Finance teams already hold most of this. Every year the audit produces a list of requests from the auditor, the company's responses, the attachments sent, follow-up questions and the adjustments that came out of them. Rebuilt as task packets, those histories become a benchmark in which the score depends on returning real document identifiers, and any invented reference fails the task.
This page gives the benchmark brief: the fields of a packet, the near-miss and missing-evidence items to include, the locator precision to demand, a worked example with an exception, and the scoring rules. It is written for the company side of an audit, the controller or finance operations lead who answers requests, not for the audit firm's own workpapers.
Why a fluent answer is the wrong target for audit support
Audit standards define evidence by its link to a claim and its source, not by how convincing it reads. PCAOB AS 1105 says appropriate evidence must be both relevant and reliable, that relevance is the connection to the assertion or control objective being tested, and that reliability depends on the nature and source of the evidence and the circumstances in which it was obtained. Evidence from knowledgeable sources outside the company is generally more reliable than evidence from company sources alone.
AS 1105 also says audit evidence includes information that contradicts management's assertions, not only information that supports them. An agent that searches until it finds something supportive and stops is optimizing for the wrong thing. The benchmark has to reward surfacing the credit memo that undercuts an invoice as much as finding the invoice.
Documentation rules point the same way. AS 1215 says documentation of procedures that inspect documents should identify the items inspected, which may be met by stating the source the items were selected from and the selection criteria. What an auditor needs from an assistant is therefore a list of identifiers and locators that someone else can re-inspect, not a paragraph. Train and score for that output.
The benchmark brief: what one evidence request packet must contain
One packet is one request as the auditor wrote it, plus everything needed to judge an answer without asking anyone. Fields below are required unless the note says otherwise.
| Field | Content | Where it usually comes from | Why the benchmark needs it |
|---|---|---|---|
| request_text | The request exactly as received, with its tracker ID and date | The request list or tracker the audit used | Agents must parse real wording, abbreviations and references to earlier requests |
| account_and_assertion | Account, period and the assertion or control in question, as best documented | Request list, trial balance mapping | Relevance is judged against the assertion, not the keywords |
| population_and_selection | Population definition and the selected item IDs, or the selection rule | Auditor's selection file sent to the company | The agent must locate evidence for these items, not similar ones |
| corpus_snapshot | The document store as it stood on the response date, with a manifest | ERP attachments, AP system, contract repository, shared drives | Prevents leakage from documents created after the fact |
| gold_evidence | For each item, document IDs and locators that the company actually provided | Response attachments and the tracker's closure notes | The answer key is what satisfied a human reviewer, not a guess |
| gold_status | found, partially found, not in corpus, or contradicting evidence found | Follow-up history and adjustments | Teaches abstention and escalation as correct outcomes |
| approval_chain | Who approved each transaction and where that approval is recorded | Workflow logs in the ERP or AP tool | Many requests ask for authorization, not the document itself |
| corrections | Reversals, credit memos, re-issued documents and their dates | General ledger and subledgers | Superseded documents are the most common wrong answer |
| final_state | The posted entry and reconciliation that the evidence supports | General ledger after close and audit adjustments | Lets the scorer check the answer reconciles to the books |
| follow_ups | Auditor questions after the first response and how they were resolved | Tracker comments, email threads | Shows which first answers were insufficient and why |
Which near misses and missing evidence should the benchmark include?
Build the hard items on purpose, because real request histories under-represent them. A useful mix is mostly ordinary requests with a deliberate share of near misses and genuinely missing evidence; record the proportions you chose in the brief so later versions stay comparable.
The precedent is reading comprehension research. The SQuAD 2.0 work by Rajpurkar, Jia and Liang added over 50,000 unanswerable questions written adversarially to look like answerable ones, and required systems to answer when the text supported an answer and abstain otherwise. An audit benchmark needs the same property: the missing-evidence items must look like normal requests.
| Item type | How to build it | Correct agent behavior |
|---|---|---|
| Evidence not in corpus | Remove the only supporting document from the snapshot for a selected item, or use a real case where none existed | Report not found, list where it searched and recommend escalation |
| Wrong period | Keep a matching vendor invoice dated one period outside the request | Reject it and say why; continue searching or report not found |
| Superseded document | Keep both the original invoice and its credit memo or reissue | Return the current document and flag the correction |
| Duplicate or near-duplicate | Two scans of one invoice with different file names | Return one, note the duplicate, do not count it twice |
| Approval outside the system | Approval exists only as an email or a signed scan | Locate it in the right store or report that the system record is missing |
| Contradicting evidence | A receiving record that shows fewer units than the invoice | Return both and flag the conflict rather than choosing one |
How precisely should the agent point at the evidence?
Require locators that a person can re-inspect in under a minute: document ID, page or field, and the exact text or value relied on. A folder path is not evidence; a document ID with no page forces the reviewer to redo the search, which is the work the agent was supposed to save.
FinanceBench, an open benchmark for financial question answering, shows a workable shape. Each evidence item records the extracted evidence text, the identifier of the document containing it, a zero-indexed page number and the full page text. The authors report that the full benchmark has 10,231 questions and that one tested model with a retrieval system answered incorrectly or refused on 81% of a 150-question sample. Their documents are public filings; an audit benchmark applies the same structure to invoices, contracts, approvals and bank records.
For structured sources, the locator is a record key and field name rather than a page. An approval in a workflow log is located by transaction ID, step and timestamp; a ledger entry by journal ID and line. Mixing both kinds in one packet is normal, so the locator schema should allow either.
Worked packet: a disbursement test with a missing approval (illustrative)
This packet is invented to show the structure. It does not describe a real company or audit, and every count is illustrative.
At a glance
Worked packet: a disbursement test with a missing approval (illustrative)
Request: support for 25 selected accounts payable disbursements in the third quarter, showing invoice, receipt of goods or services, approval and payment. The corpus snapshot holds the AP subledger, invoice images, receiving records, the approval workflow log and bank statements as of the response date.
- Gold evidence for 22 items: invoice ID and page, receiving record ID, approval step with approver and timestamp, and the matching bank statement line. Each set reconciles to the posted payment.
- One superseded item: the selected invoice was replaced by a reissued invoice after a pricing correction. Gold answer returns the reissue, links the credit memo and flags the change.
- One contradicting item: the receiving record shows a partial delivery. Gold status is contradicting evidence found, with both records returned.
- One missing approval: the payment posted but the workflow log has no approval step for it. Gold status is not in corpus for the approval, with the other three pieces located. An agent that returns an approval from a different transaction with the same vendor fails the item.
- Exception, outside the agent's reach: the auditor separately confirmed two vendor balances directly with the vendors. Under AS 2310 the auditor controls confirmation requests and responses, so that evidence never sits in the company's files. The brief marks those requests out of scope for a company-side agent instead of counting a not-found answer as a failure.
How to score an agent so that inventing a document always loses
Score the agent on what a reviewer can verify, and make fabrication the one error that cannot be offset by good performance elsewhere.
- Keep a frozen holdout of packets that never appears in training data, and record the corpus snapshot manifest with each scoring run.
- Report results by request type; an agent strong on invoices and weak on approvals should not hide behind an average.
| Metric | Definition | Pass rule |
|---|---|---|
| Fabricated reference rate | Share of returned document IDs or locators that do not exist in the corpus snapshot | Zero; any fabricated reference fails the packet |
| Evidence recall | Share of gold evidence items the agent located | Set per request type in the brief and keep it fixed across versions |
| Locator precision | Returned locators that point at the gold page, field or line | Reported separately from recall so vague answers do not score well |
| Abstention accuracy | Correct not-found or partial answers on missing-evidence items | Reported per item type; a confident wrong answer counts against it |
| Conflict surfacing | Contradicting items where both records were returned and flagged | Reported per item type |
| Reconciliation | Answers whose located evidence ties to the final ledger state | Checked automatically against final_state |
Where the source records live, and where this benchmark breaks
The company side holds what the benchmark needs: request lists and trackers, response attachments, the ERP and accounts payable records behind them, approval logs, bank statements, contracts and the adjustments that closed the audit. The audit firm's workpapers, risk assessments and review notes belong to the firm and are a different dataset with different owners.
Electronic records from outside the company need care. PCAOB amendments adopted in 2024 for fiscal years beginning on or after December 15, 2025 require auditors to evaluate the reliability of external information the company provides in electronic form, including understanding where it came from and how the company received and maintained it. A benchmark that includes bank exports or vendor portal downloads should record their source and retrieval path in the corpus manifest, because a reliable answer depends on them.
The benchmark breaks where the history is thin. A company with one audit cycle, a migration that dropped attachments, or approvals that lived in email nobody kept will produce packets with gaps that look like missing evidence but are really missing history. Mark those packets as incomplete rather than as not-in-corpus gold answers, or the agent will be trained to give up on evidence that a better search would have found.
Turning closed audit request histories into licensed training data
Closed audit requests with their attachments and ledger outcomes, kept over several years, are operational history of the kind SourceX helps companies license: it licenses business data directly from companies and runs rights review, contracting, preparation, delivery and payment. As a working threshold, suppliers bring two years or more of records kept the same way, tens of thousands of them in one system, while rarer work can qualify with fewer, and every case is assessed on its own.
Audit histories contain customer, vendor and employee data and are often covered by confidentiality terms, so the company approves the scope and terms and signs the agreement before any records are delivered.
Frequently asked questions
What is an audit evidence request dataset?
It is a set of real requests an auditor sent a company, each linked to the documents that satisfied it, their exact locations, the related ledger entries and the outcome. Used as training or evaluation data, it teaches an agent to return verifiable identifiers instead of plausible prose.
Why include requests with no supporting evidence?
Because an agent trained only on answerable requests learns that every request has an answer and fills gaps with something plausible. Missing-evidence items written to look ordinary teach it to report not found, say where it searched and escalate, which is what a reviewer needs.
Can a company use its auditor's workpapers to build this benchmark?
Usually not. Workpapers, risk assessments and review notes belong to the audit firm. The company side holds its own request lists, responses, attachments and the underlying ledger and approval records, which are enough to build the benchmark described here.
How do you stop an agent from inventing document references?
Make every returned reference checkable against a frozen corpus snapshot and fail any packet that cites a document or locator that does not exist. Report fabricated references separately from recall so a high hit rate cannot hide them.
Does an agent that finds evidence replace the auditor's judgment?
No. Locating evidence is retrieval; deciding whether it is sufficient and appropriate for an assertion is the auditor's judgment under the applicable standards. The benchmark measures whether the agent hands the right items to a person who makes that call.
Sources
- PCAOB AS 1105 defines appropriate audit evidence as relevant and reliable, ties relevance to the assertion tested, says reliability depends on nature, source and circumstances, and counts contradicting information as audit evidence. Source
- PCAOB AS 1215 says documentation of procedures that inspect documents should identify the items inspected, which may be met by stating the source and selection criteria. Source
- PCAOB amendments on technology-assisted analysis require auditors to evaluate the reliability of external electronic information provided by the company, effective for fiscal years beginning on or after December 15, 2025. Source
- PCAOB AS 2310 covers obtaining evidence from a knowledgeable external source by confirmation, with the auditor maintaining control over requests and responses. Source
- The FinanceBench repository describes 10,231 financial questions with evidence records holding the evidence text, document identifier, zero-indexed page number and full page, and reports an 81% incorrect-or-refused rate for one retrieval setup on a 150-case sample. Source
- The SQuAD 2.0 paper added over 50,000 adversarially written unanswerable questions and requires systems to abstain when the text supports no answer. Source
See if your company qualifies
A short company assessment. No data uploads are needed.