AI uses for records
Benchmark answer leakage in attachments: the file name gives it away
By SourceX Editorial · Updated
9 sections
Short answer
Detect benchmark answer leakage through attachments by inventorying every attachment field a model can see, including names, types, sizes, dates and embedded properties; searching them for label terms and timing after the decision; and scoring a metadata-only baseline. If that baseline beats chance clearly, remove or neutralize the leaking fields and split by company, case family and time.
Evaluation split
Benchmark answer leakage
Key takeaways
- File names written by the person who made a decision, or after it, are labels in disguise.
- Leaks also travel through content type, size, modification time, inline flags and properties embedded in documents.
- A metadata-only baseline is the fastest proof: if it beats chance clearly, the set leaks.
- Timing checks against the decision timestamp remove the largest leaks, because outcome documents did not exist at the prediction point.
- Split by account, case family, handler and time so shared templates and naming habits stay on one side.
How do attachment names leak evaluation answers?
Attachment names leak answers when the people who handled a case named files after the outcome, such as a file called denied_final.pdf on a claim the model must classify, or refund_approved_v2.xlsx on a dispute it must decide. A model that sees the name, or metadata such as size, type or modification date, can score well by reading the label instead of solving the task, and the benchmark result means little.
Detect it in three moves. Inventory every attachment field the model can see, including names, types, sizes, dates and embedded document properties. Search those fields for answer terms and for patterns that correlate with the label. Then train or prompt a metadata-only baseline; if it beats chance by a clear margin, the set leaks.
Scikit-learn's documentation gives the general definition that applies here: data leakage occurs when information that would not be available at prediction time is used when building the model, which produces overly optimistic performance estimates. In an evaluation built from business records, an attachment named after the decision is information the system would not have at the moment the decision was made.
Where answer-bearing metadata hides in an attachment
The file name is the obvious channel, but it is one of several. Email and help desk systems expose a set of attachment fields alongside the content, and office documents carry their own properties inside the file.
Microsoft Graph's fileAttachment resource, for example, exposes name, contentType, size, isInline and lastModifiedDateTime, and documents that the name is the text displayed below the attachment's icon and need not be the actual file name. Zendesk's ticket attachment object carries file_name, content_type, size and an inline flag. Office's Document Inspector exists because documents hold hidden data such as document properties, comments, revisions and author information that travel with the file.
| Field | Illustrative leaking value | What it can reveal | First check |
|---|---|---|---|
| File name | claim_4471_DENIED_final.pdf | The label itself | Search names for label terms and their abbreviations |
| Display name versus stored name | Display: settlement letter; stored: approved_offer.docx | Outcome hidden in the second name | Inspect every name field the export carries |
| Content type | Only escalated cases have a .msg forwarded email | Process path that predicts the label | Compare type distribution across labels |
| Size | Denial letters are a fixed template size | Template identity, hence outcome | Bin sizes and test against labels |
| Last modified time | Edited after the decision date | Information from after the prediction point | Compare to the case's decision timestamp |
| Embedded document properties | Title or subject field reads Final denial | Outcome or reviewer identity | Read core and custom properties |
| Comments and revisions inside the file | Tracked change: change to approved | Decision history | Run a hidden-content inspection |
| Inline flag and count | Approved cases include an inline signature image | Workflow step reached | Compare counts and inline flags across labels |
Why a model reads the label instead of the case
Models take the easiest reliable signal, and a label in a file name is about as easy as signals get. Lapuschkin and colleagues showed the pattern in image classification: a classifier trained on PASCAL VOC 2007 relied on a source tag present in about one fifth of horse images, failed to recognize a horse when the tag was removed, and called a car a horse when the tag was added. Its accuracy looked fine until someone checked what it was using.
Language models given an attachment list are no different. A file name is short, high-signal text in a prominent place. If one in five denied claims carries a file whose name says so, a model can raise its score on exactly those items without reading the claim narrative or the policy.
This is not a rare research curiosity. Kapoor and Narayanan's survey of leakage in machine-learning-based science found errors across many research fields and proposed a taxonomy of eight leakage types, from textbook mistakes to open research problems. Attachment metadata adds one more route for the textbook case, where the input carries information about the target that would not exist when the real decision is made.
The practical lesson is that a strong score is a claim to verify, not a result to report. When a model does unexpectedly well on a set built from business files, the first question is what it could have read, and attachment metadata belongs near the top of that list because nobody looks at it while building the set.
The attachment metadata leakage checklist
Run the checklist on every evaluation set built from records that carry files, before any model is scored on it. Each item has a pass condition, so two reviewers running it should reach the same answer. Keep the completed checklist with the dataset version.
At a glance
The attachment metadata leakage checklist
Order matters. Fix timing first, because removing files that postdate the decision usually removes the largest leaks and changes what the later checks see. Then run label-term and distribution checks, then embedded properties, and finish with the metadata-only baseline as the summary test of everything before it.
Checklist · 12 items
- Inventory: list every attachment field visible to the model at inference, per source system, including stored and display names.
- Prediction point: write down when the decision being predicted was made, and the timestamp field that marks it.
- Timing: flag any attachment created or modified after the prediction point; remove it or move the item out of the set.
- Label terms: search names, titles and subject properties for label words, synonyms, abbreviations and codes used by the business, such as DEN, APPR or WO.
- Distribution: compare file type, size bins, attachment count and inline flags across label classes, and investigate any field that separates them.
- Embedded properties: extract document core and custom properties, comments and tracked changes, and apply the label-term search to them.
- Metadata-only baseline: score a model or simple classifier that sees only metadata; record its result next to the full model's.
- Entity groups: confirm that no company, customer, vendor or case family appears in both the development and the evaluation split.
- Time boundary: confirm that every evaluation case is later than every development case when the task predicts future decisions.
- Near duplicates: check for the same attachment, by hash or near-hash, across splits.
- Access history: record who could see evaluation items and their files, and whether any were sent to external services.
- Scan limits: note which file types could not be opened or inspected, and how many items that affects.
0 of 12 ticked. Tick each item as you go.
Worked example: a claims triage set with a leaking file name
The example is illustrative and uses invented counts. A team builds an evaluation set of 1,000 insurance claim cases from a claims system, each with its narrative, policy summary and attachments, and asks a model to predict pay, deny or refer. Reviewers find that adjusters often saved the outcome letter with the decision in the file name, and that the letter was attached to the case after the decision.
A metadata-only baseline that sees just file names and dates gets most decisions right on the cases where an outcome letter is present. The full model's score drops sharply once letters created after the decision timestamp are removed, which shows how much of the original score came from the names. Applying the timing check alone fixes the main leak, because the outcome letter never existed at the prediction point.
Two smaller leaks remain. Referred cases more often include a forwarded email file, so file type separates one class; the team keeps the files but replaces types with a neutral placeholder in the model's view. A handful of adjusters wrote the decision into the document title property of their working notes, which the label-term search on embedded properties catches. The team records each fix, reruns the metadata-only baseline and documents that it is now close to the class balance.
The exception in this case is instructive. Some claim files legitimately include a document the claimant sent before the decision, such as a repair estimate, whose name the claimant chose. Those files stay, names included, because a human adjuster sees them at the same point in the process and they carry no information from after the decision.
Splits that keep related files out of the test set
Split by the units that share files and habits: company or customer, case family and time. A random split by item puts sibling cases, with the same templates, naming conventions and sometimes the same attachments, on both sides, so the evaluation partly measures memory of a group rather than skill on new cases.
Scikit-learn describes the same principle for grouped data: when samples are dependent within groups, the validation fold should contain only groups not represented in the paired training fold, which GroupKFold enforces. For time-ordered data it notes that classical cross-validation assumes independent samples and recommends evaluating on future observations, which TimeSeriesSplit does.
For business records, the group key is usually the customer, vendor or employer account, and sometimes the handler, because one adjuster's file naming habits form a group of their own. The time boundary is the decision date. When in doubt, use the stricter split; a lower but honest score is more useful than a high one built on shared files.
| Split unit | What it prevents | Group or boundary key | Watch out for |
|---|---|---|---|
| Company or account | Same templates and names on both sides | Customer, vendor or employer ID | Parent and subsidiary accounts with shared files |
| Task or case family | Same case pattern memorized | Claim type, product line or queue | Families defined so narrowly that one class disappears |
| Time | Later outcomes informing earlier predictions | Decision timestamp | Attachments modified after the cut-off |
| Handler | Personal naming habits that predict outcomes | Adjuster, agent or approver ID | Small teams where this removes most data |
When an attachment name is legitimately part of the task
Sometimes the name is evidence the model should use. A document classification task may legitimately read the name a sender gave a file, because a person doing the job sees it too. An email triage task may treat attachment names in incoming messages as part of the input, since the sender chose them before any decision was made.
The test is who wrote the name and when. A name written by the sender before the prediction point is input. A name written by the person who made the decision, or after it, is a label in disguise. RFC 2183 describes the filename parameter as a suggestion for the name to store the body part under, so names travel with messages from sender to recipient and are a natural part of incoming mail; renames inside the receiving organization are a different story.
When names are kept as input, document it, keep the metadata-only baseline in the results, and report scores with and without names. Readers can then see how much of the performance depends on them.
What a metadata scan misses, and when to hand it to specialists
A scan finds what it can read. Encrypted, corrupted or proprietary files, images of documents and archives within archives may hide properties the scanner never sees, so count them and report the number. Office's own documentation notes limits too: the Document Inspector does not detect some objects, such as shapes, charts and pictures, that may sit in hidden Excel columns.
Label terms are local. A team may use internal codes, color words or reviewer initials that no generic search would catch. Ask someone who did the work to read a sample of names and properties for each class; they will spot conventions an outsider misses.
Bring in a dataset engineer and the evaluation lead together when the metadata-only baseline remains well above chance after fixes, when attachments cannot be inspected at scale, or when removing leaking items would change the class balance enough to alter what the set measures. Attachments in sensitive records also raise privacy questions that need their own review before the files are used at all.
Requesting evaluation records with attachments handled up front
When requesting evaluation data built from business records, ask for the attachment field inventory, decision timestamps, group keys for account, case family and handler, and a statement of what was renamed, removed or neutralized and why. Ask for the metadata-only baseline the supplier ran, if any, and the count of files that could not be inspected.
Evaluation data from SourceX never comes from stock: datasets are sourced from U.S. businesses after a data request that sets scope, volume, language, format, timing and licensing needs, so attachment handling rules can be part of the request from day one. The supplying company, in turn, gets sourcing, rights review, contracting, preparation and delivery handled by SourceX.
Frequently asked questions
What is answer leakage through attachments?
It is when an evaluation item's attachments reveal the expected answer through their names, types, sizes, dates or embedded properties. A model can then score by reading the metadata rather than solving the task, which inflates results.
How do I test whether attachment metadata leaks labels?
Train or prompt a baseline that sees only attachment metadata and compare it with chance and with the full model. If the metadata-only baseline performs well, inspect which fields drive it and fix those items before using the set.
Should I remove all attachments from an evaluation set?
Usually not, because the content of attachments is often what the task needs. Remove or neutralize the fields that carry the answer, drop files created after the prediction point, and keep the rest.
Is file metadata the same as benchmark contamination?
No. Contamination usually means test items appearing in a model's training data. Attachment leakage is a flaw inside the evaluation set itself, where the input carries the label, and it can exist even for a model that has never seen the data.
Which split prevents attachment leakage between development and test data?
Group splits by company or account, case family and handler, combined with a time boundary at the decision date, keep shared templates, naming habits and duplicate files on one side. Random item-level splits do not.
Sources
- Scikit-learn defines data leakage as using information that would not be available at prediction time, which produces overly optimistic performance estimates. Source
- Scikit-learn's GroupKFold keeps each group out of the paired training fold, and TimeSeriesSplit evaluates time-ordered data on later observations. Source
- Microsoft Graph file attachments expose name, contentType, size, isInline and lastModifiedDateTime, and the name need not be the actual file name. Source
- Zendesk ticket attachments carry file_name, content_type, size and an inline flag. Source
- RFC 2183 says the filename parameter can suggest a filename for storing the body part. Source
- Office's Document Inspector finds hidden data such as document properties and personal information, and does not detect some objects in hidden Excel columns. Source
- A classifier trained on PASCAL VOC 2007 relied on a source tag in about one fifth of horse images; removing or adding the tag flipped its predictions. Source
- Kapoor and Narayanan found leakage errors across many fields using machine learning and proposed a taxonomy of eight leakage types. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.