Skip to content

AI uses for records

Benchmark answer leakage in attachments: the file name gives it away

By SourceX Editorial · Updated

9 sections

Short answer

Detect benchmark answer leakage through attachments by inventorying every attachment field a model can see, including names, types, sizes, dates and embedded properties; searching them for label terms and timing after the decision; and scoring a metadata-only baseline. If that baseline beats chance clearly, remove or neutralize the leaking fields and split by company, case family and time.

Evaluation split

Benchmark answer leakage

This page covers benchmark answer leakage in attachments: the file name gives it away.Illustrative

Key takeaways

  • File names written by the person who made a decision, or after it, are labels in disguise.
  • Leaks also travel through content type, size, modification time, inline flags and properties embedded in documents.
  • A metadata-only baseline is the fastest proof: if it beats chance clearly, the set leaks.
  • Timing checks against the decision timestamp remove the largest leaks, because outcome documents did not exist at the prediction point.
  • Split by account, case family, handler and time so shared templates and naming habits stay on one side.

How do attachment names leak evaluation answers?

Attachment names leak answers when the people who handled a case named files after the outcome, such as a file called denied_final.pdf on a claim the model must classify, or refund_approved_v2.xlsx on a dispute it must decide. A model that sees the name, or metadata such as size, type or modification date, can score well by reading the label instead of solving the task, and the benchmark result means little.

Detect it in three moves. Inventory every attachment field the model can see, including names, types, sizes, dates and embedded document properties. Search those fields for answer terms and for patterns that correlate with the label. Then train or prompt a metadata-only baseline; if it beats chance by a clear margin, the set leaks.

Scikit-learn's documentation gives the general definition that applies here: data leakage occurs when information that would not be available at prediction time is used when building the model, which produces overly optimistic performance estimates. In an evaluation built from business records, an attachment named after the decision is information the system would not have at the moment the decision was made.

Where answer-bearing metadata hides in an attachment

The file name is the obvious channel, but it is one of several. Email and help desk systems expose a set of attachment fields alongside the content, and office documents carry their own properties inside the file.

Microsoft Graph's fileAttachment resource, for example, exposes name, contentType, size, isInline and lastModifiedDateTime, and documents that the name is the text displayed below the attachment's icon and need not be the actual file name. Zendesk's ticket attachment object carries file_name, content_type, size and an inline flag. Office's Document Inspector exists because documents hold hidden data such as document properties, comments, revisions and author information that travel with the file.

Where answer-bearing metadata hides in an attachment
FieldIllustrative leaking valueWhat it can revealFirst check
File nameclaim_4471_DENIED_final.pdfThe label itselfSearch names for label terms and their abbreviations
Display name versus stored nameDisplay: settlement letter; stored: approved_offer.docxOutcome hidden in the second nameInspect every name field the export carries
Content typeOnly escalated cases have a .msg forwarded emailProcess path that predicts the labelCompare type distribution across labels
SizeDenial letters are a fixed template sizeTemplate identity, hence outcomeBin sizes and test against labels
Last modified timeEdited after the decision dateInformation from after the prediction pointCompare to the case's decision timestamp
Embedded document propertiesTitle or subject field reads Final denialOutcome or reviewer identityRead core and custom properties
Comments and revisions inside the fileTracked change: change to approvedDecision historyRun a hidden-content inspection
Inline flag and countApproved cases include an inline signature imageWorkflow step reachedCompare counts and inline flags across labels

Why a model reads the label instead of the case

Models take the easiest reliable signal, and a label in a file name is about as easy as signals get. Lapuschkin and colleagues showed the pattern in image classification: a classifier trained on PASCAL VOC 2007 relied on a source tag present in about one fifth of horse images, failed to recognize a horse when the tag was removed, and called a car a horse when the tag was added. Its accuracy looked fine until someone checked what it was using.

Language models given an attachment list are no different. A file name is short, high-signal text in a prominent place. If one in five denied claims carries a file whose name says so, a model can raise its score on exactly those items without reading the claim narrative or the policy.

This is not a rare research curiosity. Kapoor and Narayanan's survey of leakage in machine-learning-based science found errors across many research fields and proposed a taxonomy of eight leakage types, from textbook mistakes to open research problems. Attachment metadata adds one more route for the textbook case, where the input carries information about the target that would not exist when the real decision is made.

The practical lesson is that a strong score is a claim to verify, not a result to report. When a model does unexpectedly well on a set built from business files, the first question is what it could have read, and attachment metadata belongs near the top of that list because nobody looks at it while building the set.

The attachment metadata leakage checklist

Run the checklist on every evaluation set built from records that carry files, before any model is scored on it. Each item has a pass condition, so two reviewers running it should reach the same answer. Keep the completed checklist with the dataset version.

At a glance

The attachment metadata leakage checklist

Order matters. Fix timing first, because removing files that postdate the decision usually removes the largest leaks and changes what the later checks see. Then run label-term and distribution checks, then embedded properties, and finish with the metadata-only baseline as the summary test of everything before it.

Checklist · 12 items

  1. Inventory: list every attachment field visible to the model at inference, per source system, including stored and display names.
  2. Prediction point: write down when the decision being predicted was made, and the timestamp field that marks it.
  3. Timing: flag any attachment created or modified after the prediction point; remove it or move the item out of the set.
  4. Label terms: search names, titles and subject properties for label words, synonyms, abbreviations and codes used by the business, such as DEN, APPR or WO.
  5. Distribution: compare file type, size bins, attachment count and inline flags across label classes, and investigate any field that separates them.
  6. Embedded properties: extract document core and custom properties, comments and tracked changes, and apply the label-term search to them.
  7. Metadata-only baseline: score a model or simple classifier that sees only metadata; record its result next to the full model's.
  8. Entity groups: confirm that no company, customer, vendor or case family appears in both the development and the evaluation split.
  9. Time boundary: confirm that every evaluation case is later than every development case when the task predicts future decisions.
  10. Near duplicates: check for the same attachment, by hash or near-hash, across splits.
  11. Access history: record who could see evaluation items and their files, and whether any were sent to external services.
  12. Scan limits: note which file types could not be opened or inspected, and how many items that affects.

0 of 12 ticked. Tick each item as you go.

Worked example: a claims triage set with a leaking file name

The example is illustrative and uses invented counts. A team builds an evaluation set of 1,000 insurance claim cases from a claims system, each with its narrative, policy summary and attachments, and asks a model to predict pay, deny or refer. Reviewers find that adjusters often saved the outcome letter with the decision in the file name, and that the letter was attached to the case after the decision.

A metadata-only baseline that sees just file names and dates gets most decisions right on the cases where an outcome letter is present. The full model's score drops sharply once letters created after the decision timestamp are removed, which shows how much of the original score came from the names. Applying the timing check alone fixes the main leak, because the outcome letter never existed at the prediction point.

Two smaller leaks remain. Referred cases more often include a forwarded email file, so file type separates one class; the team keeps the files but replaces types with a neutral placeholder in the model's view. A handful of adjusters wrote the decision into the document title property of their working notes, which the label-term search on embedded properties catches. The team records each fix, reruns the metadata-only baseline and documents that it is now close to the class balance.

The exception in this case is instructive. Some claim files legitimately include a document the claimant sent before the decision, such as a repair estimate, whose name the claimant chose. Those files stay, names included, because a human adjuster sees them at the same point in the process and they carry no information from after the decision.

Split by the units that share files and habits: company or customer, case family and time. A random split by item puts sibling cases, with the same templates, naming conventions and sometimes the same attachments, on both sides, so the evaluation partly measures memory of a group rather than skill on new cases.

Scikit-learn describes the same principle for grouped data: when samples are dependent within groups, the validation fold should contain only groups not represented in the paired training fold, which GroupKFold enforces. For time-ordered data it notes that classical cross-validation assumes independent samples and recommends evaluating on future observations, which TimeSeriesSplit does.

For business records, the group key is usually the customer, vendor or employer account, and sometimes the handler, because one adjuster's file naming habits form a group of their own. The time boundary is the decision date. When in doubt, use the stricter split; a lower but honest score is more useful than a high one built on shared files.

Splits that keep related files out of the test set
Split unitWhat it preventsGroup or boundary keyWatch out for
Company or accountSame templates and names on both sidesCustomer, vendor or employer IDParent and subsidiary accounts with shared files
Task or case familySame case pattern memorizedClaim type, product line or queueFamilies defined so narrowly that one class disappears
TimeLater outcomes informing earlier predictionsDecision timestampAttachments modified after the cut-off
HandlerPersonal naming habits that predict outcomesAdjuster, agent or approver IDSmall teams where this removes most data

When an attachment name is legitimately part of the task

Sometimes the name is evidence the model should use. A document classification task may legitimately read the name a sender gave a file, because a person doing the job sees it too. An email triage task may treat attachment names in incoming messages as part of the input, since the sender chose them before any decision was made.

The test is who wrote the name and when. A name written by the sender before the prediction point is input. A name written by the person who made the decision, or after it, is a label in disguise. RFC 2183 describes the filename parameter as a suggestion for the name to store the body part under, so names travel with messages from sender to recipient and are a natural part of incoming mail; renames inside the receiving organization are a different story.

When names are kept as input, document it, keep the metadata-only baseline in the results, and report scores with and without names. Readers can then see how much of the performance depends on them.

What a metadata scan misses, and when to hand it to specialists

A scan finds what it can read. Encrypted, corrupted or proprietary files, images of documents and archives within archives may hide properties the scanner never sees, so count them and report the number. Office's own documentation notes limits too: the Document Inspector does not detect some objects, such as shapes, charts and pictures, that may sit in hidden Excel columns.

Label terms are local. A team may use internal codes, color words or reviewer initials that no generic search would catch. Ask someone who did the work to read a sample of names and properties for each class; they will spot conventions an outsider misses.

Bring in a dataset engineer and the evaluation lead together when the metadata-only baseline remains well above chance after fixes, when attachments cannot be inspected at scale, or when removing leaking items would change the class balance enough to alter what the set measures. Attachments in sensitive records also raise privacy questions that need their own review before the files are used at all.

Requesting evaluation records with attachments handled up front

When requesting evaluation data built from business records, ask for the attachment field inventory, decision timestamps, group keys for account, case family and handler, and a statement of what was renamed, removed or neutralized and why. Ask for the metadata-only baseline the supplier ran, if any, and the count of files that could not be inspected.

Evaluation data from SourceX never comes from stock: datasets are sourced from U.S. businesses after a data request that sets scope, volume, language, format, timing and licensing needs, so attachment handling rules can be part of the request from day one. The supplying company, in turn, gets sourcing, rights review, contracting, preparation and delivery handled by SourceX.

Frequently asked questions

What is answer leakage through attachments?

It is when an evaluation item's attachments reveal the expected answer through their names, types, sizes, dates or embedded properties. A model can then score by reading the metadata rather than solving the task, which inflates results.

How do I test whether attachment metadata leaks labels?

Train or prompt a baseline that sees only attachment metadata and compare it with chance and with the full model. If the metadata-only baseline performs well, inspect which fields drive it and fix those items before using the set.

Should I remove all attachments from an evaluation set?

Usually not, because the content of attachments is often what the task needs. Remove or neutralize the fields that carry the answer, drop files created after the prediction point, and keep the rest.

Is file metadata the same as benchmark contamination?

No. Contamination usually means test items appearing in a model's training data. Attachment leakage is a flaw inside the evaluation set itself, where the input carries the label, and it can exist even for a model that has never seen the data.

Which split prevents attachment leakage between development and test data?

Group splits by company or account, case family and handler, combined with a time boundary at the decision date, keep shared templates, naming habits and duplicate files on one side. Random item-level splits do not.

Sources

  • Scikit-learn defines data leakage as using information that would not be available at prediction time, which produces overly optimistic performance estimates. Source
  • Scikit-learn's GroupKFold keeps each group out of the paired training fold, and TimeSeriesSplit evaluates time-ordered data on later observations. Source
  • Microsoft Graph file attachments expose name, contentType, size, isInline and lastModifiedDateTime, and the name need not be the actual file name. Source
  • Zendesk ticket attachments carry file_name, content_type, size and an inline flag. Source
  • RFC 2183 says the filename parameter can suggest a filename for storing the body part. Source
  • Office's Document Inspector finds hidden data such as document properties and personal information, and does not detect some objects in hidden Excel columns. Source
  • A classifier trained on PASCAL VOC 2007 relied on a source tag in about one fifth of horse images; removing or adding the tag flipped its predictions. Source
  • Kapoor and Narayanan found leakage errors across many fields using machine learning and proposed a taxonomy of eight leakage types. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies