Skip to content

AI uses for records

Document bundle reasoning benchmark: testing agents on a 30-file case

By SourceX Editorial · Updated

10 sections

Short answer

Evaluate an agent on a thirty-file case by sampling whole case bundles rather than single documents, and by writing each question so its answer needs evidence from named regions in two or more files. Record the file, version, page and region behind every answer, rate each item on hops, modality, versions, scan quality and distractors, and score the answer and its evidence path separately.

Evaluation split

Document bundle reasoning

This page covers document bundle reasoning benchmark: testing agents on a 30-file case.Illustrative

Key takeaways

  • A case bundle tests things a single PDF cannot: choosing the right file, reconciling versions and linking records that share no obvious identifier.
  • The sampling unit should be the closed case, stratified by case type, bundle size and outcome, with whole cases held out.
  • Each item gets a difficulty profile across seven dimensions, so results can be reported by what made an item hard.
  • Every gold answer points to file, version, page, region and the other evidence it must be read with.
  • Grade the answer and the evidence path separately, and include questions the bundle cannot answer.
  • Confidence intervals should resample cases, not questions, because questions from one bundle are not independent.

How do you evaluate an agent on a case spanning thirty files?

Treat the case, not the document, as the unit of evaluation. Sample complete closed cases, write questions whose correct answer requires evidence from at least two files, record exactly where that evidence sits, and grade both the answer and whether the agent cited the right regions. A question that one file answers on its own belongs in a single-document test.

The reason is that real work arrives as bundles. A purchasing dispute, a claim, a loan file or a customer escalation is a folder of contracts, amendments, emails with attachments, scanned forms, spreadsheets and photos. An agent that reads any one of them well can still fail the case by trusting a superseded version, missing an attachment or joining two records that only look related.

The rest of this guide gives the framework: what to sample, how to rate difficulty, how to pin evidence, an illustrative case and a scoring plan.

What a single long PDF tests, and what a case file adds

Public document benchmarks have moved from single pages to long documents, but many still hand the model one file. MMLongBench-Doc, for example, contains 135 PDF documents averaging 47.5 pages, with 1,091 questions; its authors report that 33.0% of questions need evidence across pages and 22.5% are designed to be unanswerable. MP-DocVQA extended DocVQA to multi-page documents, and its baseline model also predicts which page holds the answer.

Multi-hop text benchmarks contribute the idea of labeled evidence. HotpotQA marks each question's supporting facts as pairs of paragraph title and sentence number, so a system can be checked on whether it used the right sentences, not only on its final answer.

A business case bundle adds four problems on top of length. Files differ in kind, from native spreadsheets to phone photos of signed pages. Several versions of the same document often coexist, and only one was in force. Records link through identifiers, names, dates or amounts that may be partial or inconsistent. And the bundle contains distractors that belong to the business but not to the question, such as a near-identical order from another month.

What should the document bundle sampling and difficulty framework include?

It needs two parts: a sampling plan that decides which cases enter the set, and a difficulty grid that rates each question. The grid below is the core asset. Rate every item on each dimension and store the ratings with the item, so you can later report accuracy separately for, say, three-file questions with a superseded version present.

What should the document bundle sampling and difficulty framework include?
DimensionLevel 1Level 2Level 3What to record
Hops across filesTwo filesThree or four filesFive or more filesList of file ids in the order a person would read them
Evidence modalityBody text in a born-digital fileTable cell, chart or form fieldScanned image, handwriting, stamp or photoElement type for each evidence region
Evidence locationMain bodyFootnote, header, appendix or table spanning a page breakAttachment inside an email or an embedded fileParent file and position for nested items
Version conflictOne version presentSuperseded draft present and clearly markedTwo signed or sent versions that disagreeVersion id, as-of date and which one governs
Scan qualityNative text layerClean scan with OCRDegraded, skewed or low-confidence OCROCR engine and confidence for each evidence region
Cross-file linkageShared explicit identifierName or reference with variationsImplicit link through dates, quantities or sequenceThe link field and how a person confirms it
DistractorsNone relevantSame document type, different periodNear-duplicate record from another case or customerDistractor file ids and why they look relevant

Which fields pin an answer to its evidence?

Every gold answer needs an evidence record precise enough that a second reviewer can open the bundle and find the same regions in under a minute. Without it, you cannot tell whether a wrong answer came from retrieval, reading or reasoning.

At a glance

Which fields pin an answer to its evidence

OCR confidence belongs in the record because it explains failures that are not reasoning failures. Tesseract's API returns a confidence value and bounding box for each word, and Amazon Textract returns line and word blocks with confidence on a 0 to 100 scale, noting that low-confidence detections may need human confirmation. If the gold annotator also struggled to read a region, mark it; items whose evidence is unreadable to people should be dropped or rated level 3 for scan quality.

  • Case id and the question id.
  • File id, original file name, file type and content hash, so later re-exports can be matched.
  • Version id and as-of date, plus whether this version governs the answer.
  • Page number and region coordinates, using the coordinate system of the rendered page.
  • Element type: paragraph, table cell, form field, footnote, figure, stamp, signature or handwriting.
  • For table cells, the full header path: column header, row header and any spanning header.
  • OCR engine and confidence for the region, or native text if no OCR was used.
  • Link to the other evidence records this one must be read with, and the reason, such as the same order line.
  • Role: primary evidence, corroborating evidence or a known distractor.

Pinning evidence inside an image, a table cell or an attachment

Name the smallest unit that contains the evidence and record its container. For a table, that is one cell and its header path; for an image, a crop of the page; for an attachment, the attachment file plus the message that carried it. A citation to a whole 40-page contract does not tell you whether the agent found the clause.

Layout tools can pre-annotate this work. Docling, an open-source converter, parses PDFs, Office files, email formats and images and recovers page layout, reading order and table structure. Its output gives annotators candidate regions and header paths, but a person should confirm every gold region, because the cases that matter most, such as merged headers and stamps over text, are where automated layout is least reliable.

Attachments need special care. An invoice sent as a PDF attached to an email and later saved again in a shared folder is two files with the same content; decide which one is the canonical evidence and mark the other as a duplicate, or the agent will be penalized for citing a correct copy.

How should you sample cases so the set does not flatter the agent?

Sample closed cases at random within strata, after defining the population, not by picking cases whose files look clean. Hand-picked bundles drift toward born-digital files and tidy version histories, which hides the failures you most need to see.

Hold out whole cases, never individual questions, so no file from a test case appears during development. Add unanswerable questions whose answer is absent from the bundle, so an agent that guesses scores worse than one that says it cannot find the evidence.

How should you sample cases so the set does not flatter the agent?
Sampling decisionRecommended choiceWhy it matters
PopulationClosed cases of one or more named case types over a stated periodOpen cases change while you annotate them and have no settled outcome
Sampling unitWhole case bundleQuestions from one bundle share files and are not independent
StrataCase type, bundle size band, outcome and periodLets results be reported for large and small bundles separately
ExclusionsCases with privileged material, active litigation or missing consent, recorded with countsKeeps the set clean and makes the exclusion bias visible
Unanswerable shareA fixed, stated share of items, for example one in five, chosen before annotation (illustrative)Measures abstention instead of rewarding guesses
HoldoutWhole cases, frozen with file hashes before any model is runPrevents leakage through repeated files

Worked example (illustrative): a 31-file purchasing dispute

Consider an illustrative closed case: a supplier disputes a short payment on one shipment. The bundle has 31 files: a master supply agreement, two price schedule amendments, four purchase order revisions, six receiving reports (four scanned), five invoices, a credit memo, eleven emails with attachments and a spreadsheet the buyer used to reconcile. The question is which unit price governed the March shipment, and whether the disputed invoice applied it.

The answer needs five files read in order. Its difficulty profile is level 3 for hops, level 2 for version conflict because a superseded purchase order revision is present, level 3 for location because the governing amendment arrived as an email attachment, and level 2 for linkage because the receiving report cites the order number with a typo. The table shows the evidence path.

Worked example (illustrative): a 31-file purchasing dispute
StepFileRegion and elementWhat it establishes
1Receiving report, scannedPage 1, header field and line tableShipment date and the order number, written with a transposed digit
2Purchase order revision 3Page 2, line item tableThe order the shipment belongs to; revision 2 is a distractor
3Email with price amendment attachedAttachment page 1, effective date clauseThe second amendment took effect before the shipment date
4Price schedule amendment 2Page 3, table cell under the product rowThe unit price in force for that product
5Disputed invoicePage 1, unit price cellThe invoice used the price from the first amendment, so it did not apply the governing price

Scoring the answer and the evidence path separately

Report answer correctness and evidence quality as separate numbers, then a joint score. HotpotQA's evaluation script follows this pattern: it scores supporting facts with their own precision and recall, and computes joint metrics by multiplying answer and supporting-fact precision and recall, so a lucky right answer with wrong evidence scores low.

For bundles, score evidence at the region level with a tolerance for overlapping boxes, and give partial credit for the right file with the wrong page. Score unanswerable items on abstention. Then break every metric down by the difficulty grid; a single average across all items hides whether the agent fails on scans, on versions or on links.

Estimate uncertainty by resampling whole cases, since items from one bundle share files and errors. Pre-register the scoring rules, tolerances and difficulty definitions before running any model, and keep annotator disagreement as data rather than forcing one gold answer when the records genuinely support two readings.

Where the framework reaches its limits

Some cases cannot be scored fairly however well they are annotated. Stop and narrow the scope when a bundle is missing a file the case depended on, when the business itself disputes which version governed, or when the decisive evidence is a phone call that was never written down. Those cases test the archive, not the agent.

Bring in a records expert when the bundle mixes retention regimes or contains privileged, personal or regulated material, and a domain expert when the correct answer depends on interpreting a contract rather than reading it. A benchmark built from real cases also inherits their skew: if the business mostly settles disputes one way, an agent can score well by predicting the usual outcome, so report outcome balance alongside accuracy.

Getting closed case bundles from real businesses

The hard part of this benchmark is not the method but the bundles: complete closed cases with every version, attachment and scan still attached. Buyers who need them can ask SourceX, which looks for case bundles among U.S. businesses on request rather than holding them, and the request spells out case types, bundle sizes, formats, timing and licensing needs.

A supplying company signs the agreement, after approving its scope and terms, before any bundle moves. Bundles too large to transfer stay in the seller's own storage or travel on encrypted drives, since SourceX does not host multi-terabyte datasets.

Frequently asked questions

What is a document bundle reasoning benchmark?

It is an evaluation in which each question is asked over a set of related files from one case, and the correct answer requires evidence from more than one of them. Items carry gold evidence regions and a difficulty profile. It differs from long-document benchmarks, which usually give the model one file per question.

How many files should a case bundle contain?

Use the sizes the business actually produces rather than a fixed number, and stratify by size band so results can be reported for small and large bundles. A thirty-file case is a useful upper band for many operational processes, but the right range depends on the case type.

Should the agent get the whole bundle or retrieve files itself?

Run both settings and report them separately. Giving the whole bundle tests reading and reasoning; making the agent search a folder also tests retrieval. Keep the same items and gold evidence so the two scores can be compared.

How do you handle duplicate files in the bundle?

Pick one canonical copy for each duplicated document, record the others as duplicates with their hashes, and accept a citation to any copy. Otherwise an agent is marked wrong for citing a correct duplicate.

Can synthetic bundles replace real case files?

They help for coverage and for testing specific failure patterns, but they rarely reproduce real version drift, scan damage and inconsistent identifiers. Use them alongside a held-out set of real closed cases rather than instead of it.

Sources

  • MMLongBench-Doc contains 135 PDFs averaging 47.5 pages and 1,091 questions, of which 33.0% are cross-page and 22.5% unanswerable. Source
  • MP-DocVQA extends DocVQA to multi-page documents, and its Hi-VT5 baseline predicts the page that holds the answer. Source
  • HotpotQA labels each question's supporting facts as paragraph title and sentence number pairs. Source
  • HotpotQA's evaluation scores supporting facts separately and computes joint metrics from answer and supporting-fact precision and recall. Source
  • Tesseract's result iterator returns a confidence value and bounding box for each recognized word. Source
  • Amazon Textract returns line and word blocks with confidence on a 0 to 100 scale and notes low-confidence detections may need human confirmation. Source
  • Docling parses PDF, Office, email and image formats and recovers page layout, reading order and table structure. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies