AI uses for records
Agent hidden attachment failure: did the agent open the file?
By SourceX Editorial · Updated
10 sections
Short answer
Test it with a task whose correct answer exists only inside an attachment, while the message body offers a plausible but outdated answer. Grade the trajectory as well as the reply: the agent passes only if it fetched the attachment, read the relevant passage, answered from it and cited it. A right answer without the fetch is a lucky pass, and a wrong answer from the body is an evidence retrieval failure.
Agent run
Agent hidden attachment
Key takeaways
- An agent that answers from a message body can sound complete while missing the one document that decides the question.
- Mail and tool APIs often return attachment references, not content, so reading the file is a separate step the agent must choose to take.
- Build the task so the body and the attachment disagree; otherwise the test cannot tell reading from guessing.
- Score evidence retrieval and answer correctness separately, and treat a correct answer with no fetch as unsupported.
- Not opening a file is correct when permissions or policy forbid it, as long as the agent says what it could not check.
How do you test whether an agent opens the attachment needed to answer a question?
Give the agent a question whose answer lives only in an attachment, plant a different, plausible answer in the visible message text, and grade the run on what the agent retrieved as well as what it said. A passing run shows a fetch of the attachment content, an answer that matches the attachment, and a citation to it. A run that answers from the body fails, however fluent it is.
The design matters because many business answers are in files: an amended contract, a revised quote, a signed change order, an inspection report, a screenshot in a support ticket. An agent working an inbox or a ticket queue sees the thread first and the files only if it asks for them. The failure is quiet: the reply cites real text from a real message, just not the message part that decides the question.
Why do agents miss attachments in the first place?
Because the systems they read hand over pointers to attachments, not the attachments themselves. Opening the file is an extra call, sometimes two, and nothing in the first response forces the agent to make it.
Email shows the pattern clearly. In the Gmail API, a message part's body can carry an attachmentId, and when it does the content must be retrieved with a separate messages.attachments.get request; the filename field is present only on parts that represent attachments, and container parts such as multipart types hold their content in nested child parts. In Microsoft Graph, attachments are fetched through a separate List attachments request, and the message's hasAttachments flag does not count inline attachments, so a message whose only attachments are inline reports false. MIME itself, in RFC 2183, separates inline body parts meant to display automatically from attachment parts kept apart from the main body.
Agent tool protocols add one more hop. Under the Model Context Protocol, a tool may return a resource link, a URI the client can fetch or subscribe to, instead of the content, and the specification notes that such links may not appear in a resources/list request. An agent that treats the link's name and description as the content has read the label on the folder, not the file.
| Where the content sits | What the agent sees first | What it must do to read it | What a grader checks |
|---|---|---|---|
| Gmail attachment part | A part with a filename and an attachmentId, no data | Call messages.attachments.get with that ID | Attachment fetch for the right message and ID in the log |
| Gmail nested multipart | A container part with child parts | Walk the child parts, including nested ones | Every child part holding the answer was visited |
| Graph message attachments | Message fields, with hasAttachments | Call List attachments, then read the file content | List call made even when the flag is false |
| Graph inline attachment | hasAttachments false; a cid reference in the body | Parse the body for inline references and fetch them | Inline item fetched when it holds the answer |
| MCP resource link | A URI with a name, description and MIME type | Fetch the resource the link points to | Resource read call after the tool result |
| Scanned attachment | A file of image type | Extract the text with an OCR or vision step | Extraction output contains the deciding passage |
Starting state: a contract question answered by an amendment
Everything in this case is illustrative: the companies, tools, IDs, durations and messages are invented to show the structure of a task. An operations manager asks an inbox agent: how much notice do we need to give the logistics vendor to end the agreement?
The mailbox holds a thread with the vendor. An older message quotes the master agreement: either party may terminate with 30 days' written notice. The newest message says only: terms otherwise unchanged, signed amendment attached. The attached PDF changes the notice period to 60 days for terminations after the first year. The agent has three tools: search_messages, get_message and get_attachment.
The operating rule given to the agent says it must answer contract questions from the most recent signed document in the thread, open every attachment on a message newer than the clause it relies on, and cite the document and section it used. That rule turns a judgment call into a required condition a grader can check.
Build at least one mirror case where the body and the attachment agree. If an agent passes the disagreement case but fails the mirror case, something other than retrieval is driving its answers, and the grader needs to know before reading too much into either result.
The evidence retrieval completeness task packet
Package the case so any grader can reproduce the verdict from linked evidence, not from the transcript alone. The fields below are the reusable template; the values describe the illustrative amendment case.
At a glance
The evidence retrieval completeness task packet
Keep the evidence map honest. If two items in the thread could each support a correct answer, list both, and grade retrieval as complete when either was read. Overly strict maps punish agents for finding a valid alternative route, which is the opposite of what the task is meant to reward.
| Packet field | Illustrative content | Evidence reference | Why the grader needs it |
|---|---|---|---|
| Case label | Illustrative; not a real company, contract or mailbox | Packet header | Stops invented cases being cited as incidents |
| Question | How much notice is needed to end the vendor agreement? | Task prompt | Defines what a complete answer must contain |
| Starting state | Thread of four messages; newest has one PDF attachment | Mailbox snapshot with message and attachment IDs | Fixes what was available to the agent |
| Evidence map and distractor | Answer in the attachment, section 2; an older message quotes the superseded 30-day clause | Annotated attachment and distractor message ID | Shows which item decides the question and makes body-only answers wrong |
| Applicable rule | Answer from the most recent signed document; open newer attachments; cite | Policy text given to the agent | The required condition being measured |
| Required retrieval steps | Find the thread, read the newest message, fetch and read the attachment | Reference trajectory | Lets the grader score completeness step by step |
| Failure step | Agent read the newest message but never called get_attachment | Tool-call log at step 3 | Pinpoints the missing retrieval |
| Final reply | You need to give 30 days' written notice, per the master agreement | Final message text | The claim being graded |
| Resulting state and severity | High: the manager would send notice on a timeline the amendment does not allow | Business impact note and severity rubric | Explains why the error matters and sets its priority |
| Recovery check | Reply gives 60 days after year one, cites the amendment, notes the clause it supersedes | Reference response | Defines what a passing run looks like |
Which required condition failed despite the agent's plausible reply?
The agent never fetched the attachment, so it could not apply the rule to answer from the most recent signed document. Its reply quoted real contract language from a real message in the thread, which is why a grader reading only the transcript is likely to pass it.
Trace each required condition and mark it met, not met or not applicable. Recording the trace for every case, rather than a one-line verdict, is what lets a team count which retrieval steps its agents skip most often.
- Locate the vendor thread: met.
- Read the newest message in the thread: met.
- Notice that the newest message references an attachment newer than the quoted clause: not met.
- Fetch the attachment content with get_attachment: not met.
- Answer from the most recent signed document: not met.
- Cite the document and section relied on: not met; the reply cited the superseded master agreement.
Grading retrieval and the answer as separate scores
Score two things. Retrieval completeness asks whether the trajectory contains a successful read of every item in the evidence map. Answer correctness asks whether the reply matches the deciding item. Anthropic's guidance on building agents says agents need ground truth from the environment at each step, such as tool call results, to assess their progress; retrieval completeness checks whether the agent actually collected that ground truth before answering.
Retrieval should be scored from the tool-call log, not from the reply. Check that the fetch targeted the right message and attachment ID, that it returned content rather than an error, and that later reasoning refers to that content. A citation in the reply to a document the log shows was never fetched is a fabricated citation and should be scored as such.
Crossing the two scores gives four outcomes, and each one means something different for training and acceptance.
Run each case several times. An agent that fetches the attachment in some runs and skips it in others has an unreliable retrieval habit, and a single passing run hides it. Report the share of runs with complete retrieval alongside the share with correct answers, so buyers can see whether accuracy rests on consistent evidence gathering or on chance.
| Attachment fetched and read | Reply matches the attachment | Verdict | Training use |
|---|---|---|---|
| Yes | Yes, with citation | Pass | Positive example of retrieve, read, then answer |
| No | Yes | Lucky pass; answer unsupported | Negative for retrieval even though the answer is right |
| No | No, answered from the body | Hidden attachment failure | High-severity negative; pair with the reference recovery |
| Yes | No | Reading or reasoning failure | Separate category: retrieval worked, comprehension did not |
Variants that make the task harder
Once agents pass the basic case, vary where the deciding content sits. Each variant targets a different shortcut, and a library of cases should cover several of them so a pass on one does not hide a gap in another. Microsoft Graph, for example, also has a reference attachment type that is only a link to a file on OneDrive for work or school or other supported storage, which adds a third hop.
At a glance
Variants that make the task harder
Introduce variants one at a time and keep the question and distractor fixed, so a change in pass rate can be traced to the variant rather than to a new question. Tag each case with its variant in the packet header; a library summary by variant then shows exactly which hiding places a given agent handles and which it does not.
- Inline only: the deciding image is inline, so a mail API flag reports no attachments.
- Reference attachment: the message carries a link to a file in cloud storage rather than the file.
- Forwarded message: the deciding document is inside an attached email, nested one level down.
- Earlier message: the attachment is on a message two replies back, not the newest one.
- Tool resource link: an MCP tool returns a link and description, and the content must be fetched separately.
- Scanned file: the attachment is an image of a signed page and needs text extraction before it can be read.
A counterexample: when not opening the file is the right call
Some runs should pass without a fetch. If the agent's permissions do not allow reading content, it cannot open the file, and the correct behavior is to say so. In the Gmail API, for instance, the full and raw message formats cannot be used with the gmail.metadata scope, and the metadata format returns only the message ID, labels and headers. An agent with that scope that reports a signed amendment is attached, explains that its access does not let it read attachments, and warns that the 30-day clause may be out of date has behaved well.
The same applies when policy forbids opening a file, for example an attachment from an unverified sender flagged by security tooling. Grade these runs on honesty: did the agent name the unread evidence and decline to give a confident answer it could not support? Record in the packet which runs are permission-limited, so graders do not penalize correct refusals.
A second kind of counterexample is the irrelevant attachment. A newsletter PDF, a logo image or a calendar invite on the same thread does not bear on the notice question, and an agent that skips it is not failing anything. The evidence map is what separates the two: only items that could change the answer count toward retrieval completeness, so graders should mark irrelevant attachments explicitly rather than leave them unlisted.
Limits of the task and what it does not measure
A retrieval completeness task tests whether the agent collected evidence, not whether the evidence is true or whether the agent was entitled to act on it. It also reflects the tools and permissions of one environment; a change to a tool's response shape, such as a new attachment type, can reopen a gap that earlier runs closed, so re-run the cases whenever tools change.
Attachments in real mailboxes and tickets often contain personal data, signatures and third-party documents. Building cases from them needs the same privacy and rights review as any other business record, and in sensitive domains the severity rubric should be set with the business operator who owns the process, not by the evaluation team alone.
The task also does not measure cost or latency. An agent that opens every attachment on every message will pass retrieval completeness while wasting time and reading files it had no reason to touch. If efficiency matters, add a separate count of unnecessary fetches rather than folding it into the retrieval score.
Sourcing real attachment-dependent cases
Synthetic mailboxes can reproduce the mechanics, but real business records show where deciding information actually hides: amendments sent as attachments, revised specifications in ticket uploads, signed change orders that never made it into the system of record. A thread, its attachments and the action taken afterward together form a gradable case, because the later action shows which document the business relied on. Cases drawn this way also carry the messy details that synthetic threads leave out: wrong file names, duplicate uploads and replies that quote the wrong version.
Those records belong to the businesses that hold them, so ownership, privacy permission and a license covering AI use have to be settled first; removing names from an email archive makes it safer to share but does not create a right to license it. SourceX keeps no stock of mailbox or ticket datasets; it sources them from U.S. businesses when an AI buyer asks, and the buyer's request spells out the scope, volume, language, format, timing and licensing terms the cases need. Every business that supplies such threads approves the scope and terms and signs before a single message or attachment is delivered.
Frequently asked questions
What is an agent hidden attachment failure?
It is a run in which an agent answers a question from the visible message text when the deciding evidence is in an attachment it never retrieved. The reply often looks well supported because it quotes real text, just not the document that controls the answer.
Why don't agents read attachments automatically?
Most mail and tool APIs return references to attachments, such as an attachment ID, a list endpoint or a resource link, rather than the content. Reading the file is an extra call the agent must decide to make, and nothing forces it to unless the task or policy requires it.
How do I stop body-only answers from passing my evaluation?
Make the body and the attachment disagree, so a body-only answer is wrong, and grade retrieval from the tool-call log separately from the answer. A correct answer with no fetch of the deciding item should be scored as unsupported.
Should an agent open every attachment it sees?
No. It should open attachments that could change the answer, such as files newer than the evidence it is relying on, within its permissions and security policy. When it cannot open one, it should say which evidence it could not check.
Can these cases be built from real business email?
Yes, if the business has the right to license the records and privacy review is done first. Real threads show where deciding documents actually hide and what action followed, which synthetic mailboxes rarely capture.
Sources
- Gmail attachment content is retrieved by a separate messages.attachments.get call using attachmentId, and full or raw formats are unavailable with the gmail.metadata scope. Source
- Microsoft Graph hasAttachments excludes inline attachments, so a message with only inline attachments reports false. Source
- Microsoft Graph retrieves a message's attachments through a separate List attachments request. Source
- A Microsoft Graph reference attachment is a link to a file on OneDrive for work or school or other supported storage. Source
- MCP tools may return resource links the client can fetch, which may not appear in resources/list. Source
- RFC 2183 separates inline body parts displayed automatically from attachment body parts kept apart from the main body. Source
- Agents need ground truth from the environment at each step, such as tool call results, to assess their progress. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.