Privacy and preparation
Enterprise archive AI contamination study: a measurement protocol
By SourceX Editorial · Updated
10 sections
Short answer
This is a study protocol; it reports no results. To estimate AI-generated text in enterprise archives, date each record against the archive's own tool rollout and interaction logs, use pre-tool text as a negative control to measure the detector's false positives, measure its hit rate on records known to be AI-drafted, correct the raw flag rate for both errors and report shares per record type and period with intervals.
Evaluation split
Enterprise archive AI
Key takeaways
- Treat detector output as one weak evidence stream; dated rollout records, interaction logs and in-record markers decide a unit wherever they exist.
- Use each archive's pre-tool period as a negative control to measure false positives on that company's own writing.
- Correct flag rates for measured sensitivity and specificity, carry their uncertainty into the interval, and skip strata where the detector barely beats chance.
- Report shares by record type and period with a stated denominator, never one pooled figure.
- This page publishes the method only; no measured shares or findings are reported or implied.
How could a study estimate AI-generated archive content while reporting detection uncertainty?
This page is a protocol, and it reports no results. A defensible study would not count records that a detector flags. It would date each record against the archive's own evidence of when AI writing tools became available, use the period before any tool existed as a negative control to measure the detector's false positive rate on that archive's text, measure its hit rate on records known from interaction logs to be AI-drafted, correct the raw flag rate for both errors and report the share per record type and period with an interval.
The design rests on one admission: detection is imperfect, and the imperfection differs by archive, writer and text length. A study that hides this behind a single percentage measures its detector, not the archive. The method below treats detector output as one evidence stream among several and lets the dated records do most of the work.
Everything that follows is the method an archive owner, researcher or journalist could pre-register and run. No measured shares, archive counts or findings are stated or implied.
What counts as contamination, and what is the denominator?
Contamination is a loaded word, so the study defines it narrowly before collecting anything. The unit is a single authored text field: an email body without quoted replies and signatures, a ticket reply, a CRM note, a document section. Each unit is assigned to one of four classes.
The headline estimate is the share of units in a stratum that are AI-drafted or AI-revised, with the denominator stated as the number of eligible authored units in that stratum, after boilerplate and quoted text are removed. Character-weighted shares can be reported as a secondary measure; they answer how much text, not how many records.
| Class | Definition | Typical evidence | Counted in the headline share |
|---|---|---|---|
| AI-drafted | Text produced by a generative tool and sent or saved with light edits | Interaction log linked to the saved text, tool footer, generated-reply flag | Yes |
| AI-revised | Human-written text substantially rewritten by a tool beyond spelling and grammar | Revision history showing a tool action, interaction log near the save time | Yes |
| Human | Text written by a person, including spell-check and fixed templates | Pre-tool period, revision history with no tool action | No |
| Indeterminate | Evidence is missing or conflicts | Detector score alone, or streams that disagree | Reported separately, never folded into either side |
Population, sampling frame and the two periods every archive contributes
The population is the authored text in operational systems of participating companies: email, help desk, CRM, project tools and shared documents. Because companies differ far more from each other than records within one company do, sampling happens in two stages: archives first, then units within each archive, stratified by system and record type.
Each archive contributes two periods. The control period ends before any generative writing tool was available to staff, set conservatively from public availability and the company's own rollout records. The exposure period starts at the earliest documented rollout of any such tool in that archive. A buffer between them is excluded, because pilots and personal accounts rarely leave clean dates.
Sample sizes are set per stratum from the interval width the study wants to report, inflated for clustering within archives. The protocol fixes the number of archives, the strata and the per-stratum targets before any detector is run, and it records how archives were recruited, because volunteer companies may be the ones most or least likely to use AI tools.
The comparison that gives the estimate meaning is within each archive, not across companies. The control period is the baseline: it shows what the evidence streams and the detector produce on text that predates the tools. The exposure period is compared with it stratum by stratum, so a difference reflects a change in that archive rather than differences in industry, writing style or language mix between companies. Where a company rolled tools out to some teams before others, the not-yet-enabled teams in the same months provide a second, concurrent baseline, which guards against changes in writing that have nothing to do with AI, such as a new template or a reorganized support desk.
The mixed-evidence dating method
Dating is the core of the method. Each unit receives a dated exposure status from records that exist independently of the text, and only then is any text-based detector applied. The streams are ranked from strongest to weakest, and the strongest stream available for a unit decides its class.
Interaction logs can be strong evidence where they exist. Microsoft Purview, for example, documents audit records generated automatically when users interact with Copilot and other AI applications, showing who interacted, when and where. Retention limits mean such logs usually cover only recent periods, which the study records per archive.
| Stream | What it establishes | Strength | Main failure mode |
|---|---|---|---|
| Rollout records | Date a tool became available to a team: licenses, admin enablement, feature flags | Bounds exposure; cannot classify a single unit | Unofficial personal use before rollout |
| Interaction logs | A specific user invoked a tool at a specific time and place | Strong when linked to the saved text by user, object and time | Short retention; no link from prompt to final text |
| In-record markers | Generated-reply flags, tool footers, revision events naming a tool | Strong for the marked unit | Markers stripped on copy, export or edit |
| Watermark detection | Text came from a model using a known watermark configuration | Strong positive for that configuration only | Absent for every other tool; silent on human text |
| Statistical detectors | Text resembles model output | Weak per unit; usable only after calibration | False positives that vary by writer and length |
| Expert review with author confirmation | Ground truth for a small validation subsample | Strongest, and most expensive | Recall bias; authors may not remember |
Why detector output is calibrated on the archive's own text
Published detector accuracy does not transfer to a given archive. OpenAI withdrew its own AI text classifier on July 20, 2023, citing its low rate of accuracy; at launch it reported that the classifier identified 26 percent of AI-written text in its tests while labeling 9 percent of human-written text as AI-written, and that it was very unreliable on short texts. Business replies and notes are often short.
Error also varies by writer. Liang and colleagues found that several widely used GPT detectors misclassified many essays by non-native English writers as AI-generated while classifying native writers' essays accurately. An archive with many second-language writers can produce a high flag rate in its control period for reasons that have nothing to do with AI.
Watermarks do not close the gap. Google DeepMind's SynthID Text reference implementation describes detectors that score whether text was generated with a given watermarking configuration, and its Bayesian detector must be trained for each watermarking key. That is useful evidence for text from a cooperating tool and no evidence at all about text from any other.
The protocol therefore estimates specificity from control-period units, which are presumed human, and sensitivity from exposure-period units linked to interaction logs or markers, which are known AI-drafted. Both estimates are made per record type, because a detector that behaves well on long documents may fail on two-line ticket replies.
Variables recorded for every sampled unit
- Archive id, system, record type and team, with the archive's recruitment route.
- Creation and last-modified timestamps with time zone, and the period assignment (control, buffer, exposure).
- Unit length in characters after removal of quoted text, signatures and templates, with what was removed.
- Author attributes the archive can disclose without identifying anyone: role family and whether the author writes mainly in a second language, where the company records it.
- Each evidence stream's result for the unit, including missing, with the record ids that support it.
- Detector name, version, threshold and raw score, frozen before scoring begins.
- Final class from the strongest available stream, and whether the unit is in the human validation subsample.
From flag rate to an estimate with honest uncertainty
For strata where classification depends on detector output, the raw flag rate is corrected for measured error. The standard correction, from Rogan and Gladen's work on estimating prevalence from an imperfect screening test, adjusts the observed positive rate using sensitivity and specificity; the corrected share equals the flag rate plus specificity minus one, divided by sensitivity plus specificity minus one, truncated at zero.
The correction is unstable when sensitivity plus specificity is close to one, which is exactly the situation the OpenAI figures describe. The protocol states in advance that a stratum is not reported when the lower bound of that sum, from the validation data, falls below a pre-set level. Reporting nothing beats reporting a corrected number that the data cannot support.
Intervals come from resampling whole archives, then resampling validation units within each, and recomputing sensitivity, specificity and the corrected share each time. This carries the uncertainty of the calibration into the final interval rather than treating the detector's error rates as known.
Where per-unit classification is unreliable, a corpus-level method is a recognized alternative. Liang and colleagues estimated the fraction of AI-modified text in a large set of peer reviews with a maximum likelihood approach over the whole corpus rather than classifying and counting individual documents. The protocol allows that estimator as a secondary analysis, with the same control-period check.
What the design would not support
- Any statement that a particular email, ticket or document was written by AI, unless an interaction log, marker or author confirmation says so; corrected shares are population estimates.
- Any claim about archives, industries or countries that were not sampled, or about companies that declined to take part.
- Any conclusion about quality, accuracy or legal status of AI-written text; the study measures presence, not effect.
- A single pooled percentage across record types, since detector error differs by text length and writer.
- Comparisons of individual employees or teams for performance purposes; the variables are collected for stratification, not evaluation.
- Treating the control period as perfectly clean: template text, autocomplete and grammar tools existed earlier, which is why the human class includes them and why control-period flags measure specificity rather than prove zero.
Pre-registration checklist for an archive AI content study
At a glance
Pre-registration checklist for an archive AI content study
Checklist · 10 items
- Research question, unit definition and the four classes, written before data access.
- Recruitment route, number of archives, strata and per-stratum targets, with the clustering assumption used to size them.
- Control, buffer and exposure boundaries per archive, and the documents used to set them.
- Evidence streams used, their ranking and the rule that picks a unit's class.
- Detector names, versions, thresholds and preprocessing, frozen and hashed before scoring.
- Size of the validation subsample, how it is drawn and how author confirmation is collected and recorded.
- Correction method, minimum sensitivity plus specificity for reporting, resampling scheme and number of replicates.
- Secondary analyses allowed, including any corpus-level estimator, labeled as secondary in advance.
- Privacy handling: aggregate outputs only, no unit text leaving the archive, and who can see per-archive figures.
- A statement of the conclusions the design would not support, to be repeated in any report.
0 of 10 ticked. Tick each item as you go.
Running the method on one company's archive before a licensing review
A single company can run a reduced version on its own records: set the control and exposure boundaries from its rollout history, pull interaction logs while they are still retained, mark the units that carry tool markers and leave detector scores as a weak, calibrated stream. The output is a dated map of where AI-assisted text is likely, by system and period, with its uncertainty stated.
That map is what an AI data buyer needs to scope a dataset to the periods and record types that fit the use. Licensing those periods through SourceX means rights review, preparation and delivery are handled for the company, which approves the scope and signs before anything is delivered; a short company assessment, with no records uploaded, comes first.
Frequently asked questions
Does this page say how much enterprise data is AI-generated?
No. It sets out a study design and reports no measured shares. Any figure would require the sampling, dating, calibration and uncertainty steps described here, carried out and disclosed.
Why not run an AI detector over the whole archive and report the result?
Detector error varies with text length and writer, and published accuracy does not transfer to a given archive. Without measuring false positives on the archive's own pre-tool text and hits on known AI-drafted text, the result reflects the detector more than the archive.
What makes the period before AI tools useful?
Text from before any generative writing tool was available can be presumed human, so the detector's flags there estimate its false positive rate on that company's writing. That calibration is what allows the exposure-period flag rate to be corrected.
Can a watermark detector find all AI-written text?
No. A watermark detector checks for one tool's watermark configuration and says nothing about text from tools that do not use it. It is strong positive evidence for some units and silent on the rest.
What population and conclusions does the design cover?
It covers authored text in the operational systems of the archives actually sampled, split by record type and period. It does not support claims about individual records, unsampled companies or the quality of AI-written text.
Sources
- Liang and colleagues estimated the fraction of AI-modified text in a large corpus of peer reviews with a corpus-level maximum likelihood method instead of classifying individual documents. Source
- OpenAI withdrew its AI text classifier on July 20, 2023 for low accuracy; at launch it identified 26 percent of AI-written text, flagged 9 percent of human text and was unreliable on short texts. Source
- Liang and colleagues found widely used GPT detectors misclassified many essays by non-native English writers as AI-generated. Source
- SynthID Text detectors score whether text was generated with a given watermarking configuration, and the Bayesian detector must be trained for each watermarking key. Source
- Rogan and Gladen showed that the frequency of positive results from an imperfect test is a poor prevalence estimate and gave the correction using sensitivity and specificity. Source
- Microsoft Purview automatically generates audit logs when users interact with Copilot and AI applications, recording who interacted, when and where. Source
See if your company qualifies
A short company assessment. No data uploads are needed.