AI uses for records
Missing attachment AI performance study: a paired three-arm protocol
By SourceX Editorial · Updated
9 sections
Short answer
To measure what a missing attachment costs an AI system, run the same tasks in paired arms: file intact, file removed with its reference kept, and file swapped under a misleading label. Use only tasks where experts confirm the file is decisive, score accuracy alongside unsupported certainty, clarification and wrong actions, and weigh the difference against file recovery hours. This page publishes the missing attachment AI performance study as a protocol; no results yet.
Value drivers
Missing attachment AI
Key takeaways
- Only tasks where the attachment is shown to be decisive belong in the study; a text-only pass screens out the rest.
- Three paired arms isolate the file: intact, missing with its reference kept, and swapped under a misleading label.
- Accuracy alone hides the costly failure, a confident answer or an irreversible action taken without the evidence.
- Paired analysis on the same tasks, with errors clustered by source case, needs fewer tasks than comparing separate samples.
- Recovering files pays only where the measured difference across affected tasks outweighs recovery and clearance hours.
How do you measure what a missing attachment costs an AI system?
Run matched versions of the same tasks that differ only in the attachment, and score what the system does, not only whether its final answer is right. The difference between arms on identical tasks is the measured cost of the missing file.
The question matters commercially because files carry much of the context business tasks depend on. Anthropic's September 2025 Economic Index report counts files and datasets sent with a request as part of the input context and finds that complex tasks draw on disproportionately more of it. Public benchmarks have started to treat files as part of the task too: SWE-bench Multimodal asks whether AI systems generalize to visual software domains, and its v2 release in September 2026 made 480 tasks available for local evaluation.
This page publishes the study protocol; results are not yet published. SourceX has not run it, and a study would preregister the design, use only attachments it has the rights to use, preserve task and model versions and publish null and negative results. It is narrower than the existing context-ablation protocol, which removes whole categories of context, and different from the missing-attachment audit, which grades whether a delivered ticket still makes sense without its file. This design tests how a system behaves when a decisive file is absent or wrong.
Which tasks qualify: proving the file is decisive
A task qualifies only when its correct resolution demonstrably depends on the attachment: two domain experts reach the right answer with the file, cannot reach it without the file, and agree on both verdicts. Every other task is a helpful-file or incidental-file task and stays in the study as a control.
At a glance
Which tasks qualify: proving the file is decisive
Do not trust the reference alone. Multimodal benchmark builders have documented the problem: the MMStar authors report that visual content is unnecessary for many samples in the vision-language benchmarks they examined, and the first construction step of MMMU-Pro filters out questions that text-only models can answer. Run a text-only pass with the file removed before the study begins; any task a model or an expert answers correctly without the file leaves the decisive set.
The agreement bar comes from Anthropic's January 2026 guidance on agent evaluations, which counts a task as good only when two domain experts, each grading on their own, would land on the same verdict of pass or fail. Record each expert's verdict with and without the file, and hold disagreements out of the decisive set until they are resolved.
Rights come before design. Attachments often contain third-party documents and personal data, so the study uses only files the data owner has licensed for evaluation and reviewed for personal data, with the same redaction applied in every arm. Grade each task into one of three groups:
- Decisive: the correct resolution is reachable only with the file, and both experts agree.
- Helpful: the resolution is reachable without the file, but more slowly or with less certainty.
- Incidental: the file adds nothing to the resolution; these tasks test whether a system asks when it should not.
Three arms that change the file and nothing else
Build each task three times, changing only the attachment, so any difference in behavior traces to the file. The task text, the reference to the file, the tools, the token budget, the model version and the sampling settings stay identical across arms.
The misleading-metadata arm goes beyond simple removal and mirrors the archive defects a missing-attachment audit looks for: a file labeled as the signed contract that holds the unsigned draft, a placeholder left behind by redaction, or an attachment linked to the wrong record. It tests whether the system checks content against its label or simply trusts the label.
An optional fourth arm makes the file retrievable through a tool instead of attached. OpenAI's Model Spec tells the assistant to fill information gaps from context or trusted sources, such as file searches, before asking the user; this arm measures whether a system goes and gets the file when it can.
| Arm | What changes | What stays fixed | What it tests |
|---|---|---|---|
| Intact | Nothing: the decisive file is attached | Task text, reference, tools, budget, model and settings | The ceiling: performance with full evidence |
| Missing | The file is removed; the reference to it stays | Everything else, including the reference text | Whether the system notices, asks or guesses |
| Misleading metadata | The file is swapped for a wrong version, a placeholder or another record's file, label unchanged | Everything else, including filename and description | Whether the system checks content against the label |
| Retrievable (optional) | The file is removed but can be fetched with a search tool | Everything else; the tool is offered in every arm | Whether the system retrieves before answering or asking |
What to score besides accuracy
Score four behaviors on every trial besides accuracy: whether the system asserted content it never saw, whether it asked for or flagged the missing evidence, whether it noticed a mismatch, and whether it took an action that changed state wrongly. Two systems with the same accuracy can differ sharply on these, and the one that guesses confidently costs more to deploy.
Write the correct behavior into each task, because it depends on stakes. The Model Spec says the assistant should err on the side of asking for confirmation or clarification before potentially costly actions, and otherwise should weigh the cost of a wrong assumption against the inconvenience of asking, often proceeding while stating its assumption. A missing delivery receipt on a disputed freight claim calls for a question; a missing screenshot on a password-reset ticket may not.
Published benchmarks suggest these behaviors deserve separate measurement. AbstentionBench, which spans 20 datasets across six abstention scenarios including underspecified context, reports that abstention remains a problem for frontier models, that scale has almost no effect and that reasoning fine-tuning hurts abstention. QuestBench reports accuracy of only 40 to 50 percent at picking the right clarifying question on its logic and planning tasks, even for models that solve the fully specified versions, and finds that on the planning tasks models tend not to hedge even when offered a not-sure option.
Grade outcomes rather than narration. Anthropic's guidance defines the outcome as the final state in the environment, separate from what the agent says it did, and recommends giving model graders an Unknown option when they lack the information to judge. Keep the incidental-file controls in every run, so a system that asks for every attachment is penalized for wasted questions; the same guidance recommends testing cases where a behavior should not occur as well as where it should.
| Metric | Counts as | Grader | Arm where it matters most |
|---|---|---|---|
| Answer accuracy | Final answer or end state matches the expert resolution | Code check against the expected state, or a calibrated model grader | All arms; intact sets the ceiling |
| Unsupported certainty | States specific content of a file the system never saw, without a caveat | Model grader with a rubric and an Unknown option, plus human spot checks | Missing and misleading |
| Clarification or retrieval | Asks for the file, flags it as missing, or fetches it with a tool | Transcript check for the request, flag or tool call | Missing and retrievable |
| Conflict detection | Notes that the file's content contradicts its label or the record | Rubric item scored by a model grader and a human sample | Misleading metadata |
| Incorrect action | A state-changing tool call the expert resolution would not make | Code check on the environment's final state | All arms, weighted by reversibility |
Illustrative walkthrough: a freight damage claim without its delivery receipt
Illustrative case: a shipper's claim says a pallet arrived crushed and points to the attached delivery receipt, which the consignee signed with a damage notation. The expert resolution approves the claim because the notation exists. The case is invented to show the protocol's logic.
In the intact arm, a good system answers from the file: it reads the notation and approves, and the only failure to score is misreading evidence that is there. In the missing arm the reference remains but the receipt is gone, and because approving or denying a claim is a costly action, the correct behavior is to ask for the receipt or route the claim for review. The errors to watch for are a confident approval that cites a notation the system never saw and a denial on the grounds that no evidence appears.
In the misleading arm the attachment is still labeled as the signed receipt but holds a clean, unsigned copy. The correct behavior is to flag the mismatch, not to deny the claim on a document that cannot be the one the claim describes. A system that scores well here is checking content; a system that denies is trusting the label.
Scored on all five metrics, two systems with identical accuracy across the three arms could still be told apart: one asks in the missing arm and flags in the misleading arm, the other guesses in both. That difference is what the study exists to measure.
Analysis: paired differences, clustered errors and enough tasks
Analyze each task's arms as pairs, cluster uncertainty by source case, and size the study with a power calculation before running it. Because every task appears in every arm, the comparison removes the variance that comes from differences in task difficulty.
Anthropic's Adding Error Bars to Evals (November 2024) recommends a paired-differences test whenever the same questions are scored under two conditions, and reports that frontier models' question scores on popular evals correlate at 0.3 to 0.7; the higher the correlation, the smaller the standard error of the difference. The same paper recommends clustering standard errors when questions are related, which applies whenever several tasks are built from one source case, resampling answers several times when they vary, and power analysis to choose the number of tasks.
Report consistency, not just averages, wherever actions are scored. Anthropic's guidance notes that an agent with a 75 percent per-trial success rate passes all three of three trials only about 42 percent of the time, so a system that occasionally takes the wrong state-changing action needs pass-all-trials reporting in the arms where actions matter.
Preregister the decisive-task rule, the arms, the metrics and rubric, the model, harness and grader versions, and the analysis plan. Then publish every arm's results, including differences that turn out to be zero.
Recovery-cost worksheet: is the missing file worth getting back?
Recover attachments when the measured difference between the intact and missing arms, multiplied by the number of affected tasks, is worth more than the hours needed to recover, check and clear the files. The worksheet keeps every input in counts, points and hours so each buyer can apply its own rates.
Recovery is rarely one step. Files may have to be re-requested from the source system while the account is still active, matched back to their parent records, checked for personal data and third-party content, redacted in images and PDFs, and logged in a manifest. Each step has a measurable effort per hundred tasks, and some files will not come back at all.
The decision rule follows from the lines: recover when utility difference and review savings across affected tasks exceed recovery and clearance hours at the buyer's rates. Otherwise exclude the decisive-missing tasks, or relabel them as missing-evidence tasks, which remain useful for testing whether a system asks.
| Worksheet line | How to measure it | Unit |
|---|---|---|
| Affected tasks | Tasks in the candidate dataset whose decisive file is missing or mislabeled | Count, by cohort |
| Utility difference | Intact-arm minus missing-arm result on accuracy and incorrect actions | Percentage points, with a confidence interval |
| Review savings | Expert reviews avoided when intact cases are handled without escalation | Reviews per 100 tasks |
| Recovery effort | Retrieval, re-linking and validation of the files | Hours per 100 tasks |
| Clearance effort | Rights check, personal-data review and redaction of attachments | Hours per 100 tasks |
| Residual loss | Files that cannot be recovered or cleared | Count, stated in the delivery notes |
A buyer checklist for attachment completeness
Before accepting a dataset with attachments, ask for evidence on each item below, measured on the delivered data rather than promised in a proposal.
Buyers can make these items part of the requirement from the start. Because datasets are sourced from U.S. businesses for each request instead of pulled from stock, the request-data form, where buyers give scope, volume, language, format, timing and licensing needs, is the place to state that decisive attachments must arrive intact and labeled. On the supply side, the order is fixed: the owning company has to approve the scope and terms, and sign, before a single file is delivered, so attachments outside that approved scope never enter the dataset.
Checklist · 8 items
- Reference detection: how tasks that mention a file were found, and the share found through metadata versus text.
- Availability states: for every expected file, whether it is present, missing, expired, redacted or attached to the wrong record, with counts.
- Integrity: a hash for every delivered file and its link to the parent record and comment.
- Decisiveness grades: which tasks have decisive, helpful or incidental files, and the rule that assigned them.
- Label checks: whether file content was compared with its filename and metadata, and how many mismatches turned up.
- Attachment redaction: how images, PDFs and embedded objects were reviewed, separately from text.
- Text-only baseline: the share of decisive-labeled tasks that a text-only pass still answered correctly.
- Rights: the permitted use for third-party documents inside the attachments.
0 of 8 ticked. Tick each item as you go.
What has not been measured, and how results would be reported
Nothing on this page is a measured result. SourceX has not run this study, and no figure here describes the cost of a missing attachment for any model, dataset or company.
A report from the study would state the tasks screened and admitted, with the decisive rate and expert agreement; results per arm and metric with paired differences, clustered intervals and trial counts; model, harness and grader versions; worksheet inputs measured on the actual dataset; and every deviation from the preregistration. A finding that attachments matter less than expected for a task family is as useful to a buyer as the opposite, so null results are published too.
Methods note: the arms, metrics, walkthrough, worksheet and checklist are editorial frameworks built on 2026-10-26. They draw on Anthropic's agent-evaluation guidance (January 2026), statistical recommendations (November 2024) and Economic Index report (September 2025), OpenAI's Model Spec, and the MMStar, MMMU, SWE-bench, AbstentionBench and QuestBench repositories. Not measured: accuracy or behavior differences between arms for any system, the share of business tasks with decisive attachments, and recovery hours for any real dataset.
Frequently asked questions
Has anyone measured how much a missing attachment hurts AI performance on business tasks?
This page does not report a measurement. Multimodal benchmark builders have found that many questions can be answered without their images, which is why the protocol screens tasks for decisiveness first; the cost for business tasks depends on the task family and has to be measured on it.
What is a misleading-metadata variant?
It keeps the file's label and reference unchanged but swaps the content for a wrong version, a redaction placeholder or another record's file. It tests whether a system checks what a file contains or trusts what it is called, a common failure mode in real archives.
Should an AI agent ask for a missing attachment or proceed?
It depends on the cost of being wrong. Before a costly or irreversible action, asking or routing for review is the safer behavior; for low-stakes tasks, proceeding while stating the assumption is often better. The study writes the expected behavior into each task so both errors can be scored.
How many tasks does a missing attachment study need?
Fewer than separate samples would, because each task is compared with itself across arms. Use a power calculation for the smallest difference that matters, cluster uncertainty by source case when several tasks come from one record, and run several trials per task when outputs vary.
Is a redacted attachment treated as missing?
It is never treated as intact. A redaction placeholder that keeps the original label belongs in the misleading-metadata arm; a reference with no file behind it belongs in the missing arm.
Sources
- Anthropic's January 2026 agent-evaluation guidance says a good task is one where two domain experts would reach the same verdict, defines the outcome as the final state in the environment, recommends giving model graders an Unknown option, recommends testing where a behavior should and should not occur, and shows that a 75 percent per-trial success rate passes all three of three trials about 42 percent of the time. Source
- Anthropic's Adding Error Bars to Evals (November 2024) recommends paired-differences analysis, reports question-score correlations of 0.3 to 0.7 between frontier models on popular evals, and recommends clustered standard errors, resampling and power analysis. Source
- Anthropic's September 2025 Economic Index report counts files and datasets sent with a request as input context and finds that complex tasks require disproportionately more context. Source
- The MMStar authors report that visual content is unnecessary for many samples in earlier vision-language benchmarks and built MMStar from 1,500 human-selected, vision-indispensable samples. Source
- MMMU-Pro's construction begins by filtering out questions that text-only models can answer, to ensure questions require multimodal understanding. Source
- SWE-bench Multimodal studies whether AI systems generalize to visual software domains, and its v2 release in September 2026 made 480 tasks available for local evaluation. Source
- AbstentionBench spans 20 datasets over six abstention scenarios, including underspecified context, and reports that abstention remains a problem for frontier models, scale has almost no effect and reasoning fine-tuning hurts abstention. Source
- QuestBench reports that state-of-the-art models reach only 40 to 50 percent accuracy at selecting the right clarification question on Logic-Q and Planning-Q, even when they can solve the fully specified problems, and that on Planning-Q models tend not to hedge even when offered a not-sure option. Source
- OpenAI's Model Spec says the assistant should err on the side of asking for confirmation or clarification before potentially costly actions, weigh the cost of wrong assumptions against the inconvenience of asking, and fill information gaps from context or trusted sources such as file searches before asking the user. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.