Systems and records
PDF embedded files in training data: where the real evidence hides
By SourceX Editorial · Updated
9 sections
Short answer
A document task should treat every PDF as a container, not a page stack. Before extracting text, list files in the embedded files name tree, scan each page for file attachment annotations, check for a collection and associated files, and recurse into what you find. Record each item in an inventory with its parent, hash, page and region, then require answers to cite that exact location.
Document stack
PDF embedded files training
Key takeaways
- A PDF can carry other files in four distinct ways, and common tools show only some of them.
- In hybrid formats such as Factur-X, the embedded XML holds the structured data the visible page only renders.
- Every embedded item needs its own inventory row: parent, mechanism, hash, page, region and extraction status.
- Answers should cite file, page and bounding box, plus OCR confidence when the text came from an image.
- Some attachments belong out of the dataset entirely; the inventory is how you prove they were found and excluded.
How should a document task discover evidence inside PDF attachments?
Discover it before extraction, by enumerating every file the PDF carries and treating each one as a record in its own right. A pipeline that renders pages and runs OCR sees what a person sees on screen. It does not see a spreadsheet stored in the file, an email pinned to page four or an XML invoice that the visible page was generated from.
The method has three parts. First, inventory the container: list every embedded item, where it is attached and what it claims to be. Second, extract each item with its provenance intact, recursing into files that carry files of their own. Third, link every answer the task produces to the exact file, page and region that supports it, so a grader can check the evidence rather than trust the output.
This matters for training and evaluation alike. If a task asks which credit was approved and the answer lives only in an attached workbook, a dataset built from page text teaches a model to guess from the cover letter. If the evaluation never records where the answer lived, nobody can tell whether the model found it or invented it.
Four places a file can hide inside a PDF
There are four mechanisms, and a careful inventory checks all of them because viewers and libraries disagree about which to show. The PDF Association notes that some older applications display only files attached through page annotations, others list only the document-wide embedded files list, and some show both.
PDF has supported embedded file streams since version 1.4, and PyMuPDF's documentation points out that an embedded file can itself contain embedded files. PDF/A-3, published in 2012, allows files of any format to be embedded as associated files, which is how archival and invoicing PDFs can carry payloads. PDF 2.0 formalized associated files with a relationship value such as Source, Data, Alternative or Supplement, and requires a MIME type on each embedded associated file.
| Mechanism | Where it is anchored | How it usually shows up | What a page-text pipeline misses |
|---|---|---|---|
| Embedded files name tree | Document catalog, a named list for the whole file | Attachments panel in a viewer; qpdf or pypdf attachment listing | The whole file, plus its name, description and dates |
| File attachment annotation | A specific page, usually drawn as a paperclip or pin icon | Icon on the page; may be absent from the document-wide list | The file and the page position that tells you what it explains |
| Collection (portfolio) | Document catalog, with a cover sheet and metadata fields per file | Portfolio view in some viewers; can look like a near-empty PDF | Every member file and the per-file properties the sender filled in |
| Associated file (PDF 2.0, PDF/A-3) | The document, a page or any object, with a relationship value | Often also listed in the name tree; the relationship is easy to drop | Why the file is there: source, data behind a chart, alternative or supplement |
When the attachment, not the page, is the record of truth
Decide per document family which copy is authoritative, and write that decision into the task. In some formats the visible page is a rendering and the embedded file is the data. Factur-X and ZUGFeRD invoices are described by their publishers as a hybrid of a PDF for people to read and an embedded XML file carrying structured invoice data for machines, built on PDF/A-3.
For a hybrid invoice, a task that asks for the tax total should treat the XML as the primary evidence and the page as a cross-check. A mismatch between them is itself a useful label, because it marks a document that a human would have flagged. For a scanned contract with an attached amendment, the reverse may hold: the page is the signed record and the attachment is a draft someone pinned for reference.
The relationship values in PDF 2.0 help when they are present. A file marked Source is original material behind the content; one marked Data holds information used to build a table or chart on the page. Many real files leave the relationship unset, so the inventory should record what the file declares and what a reviewer concluded, as separate fields.
What should the embedded artifact inventory include?
One row per item found, including the top-level PDF itself, with enough provenance that a second person could find the same bytes again. The values below are illustrative, drawn from the worked case later in this guide.
Keep the declared fields and the observed fields apart. A file can declare itself as a spreadsheet and contain something else, and a hash you computed is stronger evidence than a checksum the producer wrote into the PDF.
- Also record: table headers and units for any extracted table, so a number is never separated from its column name.
- Also record: footnotes and their markers, because qualifications often live there.
- Also record: cross-file links, such as the ticket, order or case number that ties the PDF to records outside it.
| Field | What to record | Illustrative value | Why it matters |
|---|---|---|---|
| item_id and parent_id | A stable id for this item and the id of the file that contained it | CM-0412.2, parent CM-0412 | Rebuilds the nesting so nothing is orphaned or counted twice |
| mechanism | Name tree, file attachment annotation, collection member or associated file | File attachment annotation | Explains how it was found and which tools would miss it |
| key and declared name | The internal key and the file name the producer gave, stored raw | Key credit_detail.xlsx, name credit_detail.xlsx | Names can repeat or differ from the key; never use them as ids |
| declared type and sniffed type | MIME subtype in the PDF and the type detected from the bytes | Spreadsheet declared, spreadsheet detected | Catches renamed or mislabeled payloads |
| relationship | AFRelationship value if present, else unset, plus reviewer conclusion | Unset; reviewer: Data | Separates what the file claims from what it does |
| anchor page and region | Page number and annotation rectangle for page-anchored items | Page 2, upper right margin | Tells you which paragraph or line the file explains |
| hashes | Your own SHA-256 of the bytes and the checksum stored in the PDF, if any | Computed hash recorded; stored checksum absent | Deduplicates repeats and proves what was extracted |
| dates and version | Creation and modification dates from the file specification, plus any version label | Modified two days after the memo date | Shows whether the attachment predates or postdates the decision |
| extraction status | Extracted, failed, encrypted, unsupported type or excluded, with a reason | Extracted | Makes gaps visible instead of silent |
| text source and confidence | Native text, text layer or OCR, with mean word confidence for OCR | Native cells, no OCR | Lets graders discount answers built on weak OCR |
The extraction checklist, in the order that avoids silent misses
Run the steps in order; stopping after step two is an easy way to miss files silently. The tools named are examples whose behavior is documented by their own projects, not requirements.
Checklist · 9 items
- 1. Open and note repairs. Record whether the library had to repair the file on open; PyMuPDF exposes this as is_repaired. Treat a repaired file as one to compare against the original export before trusting its attachment list.
- 2. List the embedded files name tree. qpdf's --list-attachments prints each key and stream number, and with --verbose adds the preferred file name, description and dates. The key is usually, but not always, the file name.
- 3. Walk every page for file attachment annotations. These can be absent from the document-wide list, so a name-tree listing alone is not enough.
- 4. Check for a collection and read its per-file properties. A portfolio can present a cover page with almost no text while every substantive document is a member file.
- 5. Look for associated files on the document, pages and objects, and copy each relationship value into the inventory.
- 6. Recurse. Extract each item and run steps 1 to 5 on any PDF or archive it turns out to be, keeping parent ids.
- 7. Hash, deduplicate and sanitize. pypdf warns that attachment names may repeat and can contain arbitrary characters, so write extracted files under generated ids, never under the declared name.
- 8. Extract content with structure. Keep table headers with their cells, footnotes with their markers, and native text separate from OCR output.
- 9. Link out. Record any identifiers that join the PDF to a ticket, order, claim or case, so a task can combine the document with the system record that explains it.
0 of 9 ticked. Tick each item as you go.
Tying each answer to a page, a region and a confidence
Require every answer in the dataset to point at an item_id, a page or sheet location and a bounding region, and to carry the OCR confidence when the supporting text came from an image. Without that pointer, a grader cannot tell a correct lookup from a lucky guess.
The data needed is already produced by common tools. Tesseract's TSV output, for example, reports left, top, width, height and a conf value for every recognized word, so a region and its confidence can be stored with the answer. For native PDF text, libraries return word boxes directly; for spreadsheets, a sheet name and cell range plays the same role.
Set a rule for weak evidence before labeling starts. A reasonable default is to flag any answer whose supporting words fall below a confidence floor your reviewers choose, and to send those items to a person rather than drop them, because hard scans are often where a model most needs examples.
Worked case (illustrative): a credit memo whose answer sits in the attached spreadsheet
This case is invented to show the method. A distributor's accounts receivable team closes a pricing dispute with a two-page PDF credit memo. The task asks: which invoice lines received the credit, and was it applied in full?
Page text answers only half the question. Page one says a credit was issued for incorrect pricing on an invoice; page two repeats the total. A page-text pipeline would label the answer as the invoice number and stop there.
The inventory finds three more items. A file attachment annotation on page two holds a spreadsheet listing the four affected lines. The name tree holds an email exported as a message file, approving the credit for three of the four lines. Inside that message is a second PDF, the original disputed invoice. In all, one container produced four inventory rows, one of them nested two levels deep.
The correct answer is three of four lines, credited in full, and it cites the spreadsheet rows and the approval sentence in the email. The modification dates add a useful check: the spreadsheet was edited after the approval, so a reviewer confirms the three lines still match before the item is accepted as ground truth.
Which attachments should stay out, and when to hand the file to a specialist?
Find everything, then exclude deliberately. The inventory is what lets you say an attachment was seen and kept out for a stated reason, rather than missed. Typical exclusions are vendor manuals and other material the company did not author, encrypted payloads you cannot open, executable or unknown types, and attachments that carry personal data the rest of the record does not need.
Exclusion has a cost. In the credit memo case, dropping the email would leave a dataset item whose correct answer can no longer be supported, so the whole item should go, not just the attachment. Record that dependency in the inventory so the decision is visible.
Bring in a document AI engineer when files are repaired on open, nested more than two levels or use collections with custom fields. Bring in a records manager when attachment dates conflict with the decision date, because which version governed is a records question, not an extraction one.
Asking for PDF archives with their attachments intact
If you are sourcing document data for training or evaluation, say in the request that embedded files must survive export and that you want the inventory alongside the PDFs. Some export paths flatten or print documents, which keeps the pages and drops everything inside them.
With SourceX, document datasets are gathered from U.S. businesses for a specific buyer request rather than pulled from inventory, and that request can set format, volume, timing and licensing needs, including whether embedded attachments are in scope. Large deliveries stay in the seller's own storage or ship on encrypted drives, which suits archives where nested files make the volume hard to estimate up front.
Frequently asked questions
Can a PDF contain other files?
Yes. A PDF can carry files in its document-wide embedded files list, in file attachment annotations on specific pages, as members of a collection (portfolio) and as associated files linked to the document or to objects inside it. Embedded files can themselves contain further embedded files.
Why does my PDF tool show no attachments when the file has a paperclip icon?
Viewers and libraries differ in which mechanism they read. Some list only the document-wide embedded files list, while a paperclip icon is usually a file attachment annotation on the page. Walk every page's annotations as well as the name tree before concluding a file has no attachments.
Is the XML inside a Factur-X or ZUGFeRD invoice better training data than the page?
For structured fields, usually yes, because the XML is the machine-readable data and the page is a rendering for people. The useful practice is to keep both, treat the XML as primary for field values and record any disagreement between them as a label.
How should OCR confidence be stored with an answer?
Store the region of the supporting words and their confidence values with the answer, not just the extracted text. Tesseract's TSV output already gives a box and a confidence per word, so the dataset can flag answers that rest on weak recognition for human review.
Should every attachment be included in a document dataset?
No. Include the attachments a task needs and that the supplier can license and clean, and exclude the rest with a recorded reason. If excluding an attachment removes the evidence an answer depends on, drop the whole item rather than keep an unsupported answer.
Sources
- The PDF Association explains that file attachment annotations sit on pages and need not appear in the embedded files name tree, that collections are a separate feature, and that applications differ in which they display. Source
- PDF 2.0 associated files link a file to a PDF object with an AFRelationship value such as Source, Data, Alternative or Supplement, and embedded associated files need a MIME type. Source
- PDF/A-3, published in 2012, allows files of any format to be embedded as associated files. Source
- Factur-X/ZUGFeRD is a hybrid invoice: a PDF for people and an embedded XML file with structured data for machines, based on PDF/A-3. Source
- PDF has supported embedded files since version 1.4, an embedded file can contain embedded files, and PDF 1.7 added collections. Source
- pypdf documents that attachment names may not be unique and can contain arbitrary characters that must be sanitized before writing files. Source
- qpdf --list-attachments shows each embedded file's key and stream number, with names, descriptions and dates under --verbose; the key is not always the file name. Source
- PyMuPDF lists embedded file names, reports a stored checksum that may not be MD5, and flags PDFs repaired on open with is_repaired. Source
- Tesseract TSV output gives each recognized word a bounding box (left, top, width, height) and a confidence value. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.