Manufacturing
Why factories threw away the data AI needs, and how to stop
By SourceX Editorial · Updated
Short answer
Factories discarded production data because historians, SCADA logs and MES databases were configured to save storage and keep systems fast, not to preserve history. AI developers now want exactly that history: alarms, failures, operator actions and outcomes. The fix is to find every setting that deletes or compresses data, change it deliberately, and record metadata and rights.
Key takeaways
- Most production data loss comes from defaults: historian compression, rolling alarm buffers, MES purge jobs and camera overwrite cycles.
- Alarm, event and operator-action logs are cheap to keep and carry the failure and recovery signal AI developers look for.
- Data without a tag dictionary, units and asset hierarchy is hard for anyone, including your own engineers, to use later.
- System migrations are the riskiest moment, so snapshot the old system before it is switched off.
- Keeping data longer is a decision for the retention policy owner and counsel, not only for IT.
Why did factories discard so much production data?#
Factories discarded production data because, for most of the history of plant automation, it was treated as exhaust. Plant servers had limited storage, historians and MES databases slowed down as they grew, and nobody outside the control room asked for old sensor readings or alarm logs.
So the defaults did the deciding. Historians stored a value only when it changed by more than a set amount. HMI alarm buffers overwrote themselves. MES vendors shipped purge jobs that cleared old transactions, and integrators left them switched on. Machine PCs saved logs to local drives that were later unplugged, boxed and forgotten.
None of this was careless at the time. Each choice protected uptime or saved money. The problem is that the choices were made one system at a time, without anyone deciding what the plant's history as a whole was worth.
Where production data usually disappears#
Production data usually disappears at the control and machine layers rather than in the ERP. Financial and order records tend to survive because accounting and tax rules push companies to keep them. Machine-level data has no such pressure, so it goes first and goes quietly.
| Source | How data gets lost | What is lost for AI use |
|---|---|---|
| Process historian | Compression and deadband settings drop small changes; archive files rotate off disk | Fine detail just before faults and slow drift |
| SCADA and HMI | Alarm and event buffers overwrite; logs stay on a panel PC | Alarm sequences and operator acknowledgments |
| MES | Purge jobs clear old work orders; archive tables are skipped in upgrades | Links between lots, machines, operators and outcomes |
| Inspection and CMM | Only a pass or fail summary or a PDF report is kept | Raw measurements that show drift before failure |
| Vision systems | Images of passed parts are discarded; storage fills and rolls over | Examples of good parts and borderline cases |
| CMMS | Free-text notes are truncated or dropped in migration | Technician reasoning and repair outcomes |
| Robot and CNC controllers | Logs overwrite; only programs get backed up | Faults, retries and recoveries |
What changed: AI developers want the messy history#
AI developers now want plant history because systems that work in physical settings need examples of real operations, including the parts that went wrong. Developers of industrial and physical AI look for alarm sequences, failure modes, operator interventions and the outcome that followed, since these are hard to simulate and rare in any single plant.
The useful unit is an episode with context: what the process looked like, what happened, what a person did and what the result was. A historian trend alone is a curve. Joined to an MES work order, a maintenance work order and an NCR, it becomes a documented case. That linkage is exactly what the discarded data broke.
This changes the value of old choices. Under the SourceX Enterprise Data Value Framework, scale, recency, human-generated signal and data cleanliness all raise value. Compression, missing metadata and broken links lower cleanliness and add preparation cost, which reduces net value even when the raw volume looks large.
Is compressed or partial data still worth keeping?#
Compressed historian data is still worth keeping in most cases, as long as the compression settings are documented. Swinging-door and deadband compression keep the shape of a trend while dropping small movements, so long-term patterns, shift effects and major events usually survive.
What compression cannot do is restore the detail just before a fault. Do not fill missing values by interpolation and present them as recorded data. Label each dataset with its compression settings and the dates they changed, so any user, internal or external, knows what it is looking at.
Partial archives follow the same rule. A gap between an old MES and a new one is a fact to document, not a defect to hide. Engineers and outside users alike can work with known gaps; they cannot work with silent ones.
A stop-the-loss checklist for plants#
A stop-the-loss review walks through every system that deletes, compresses or overwrites data automatically and decides each setting on purpose. OT and IT should run it together, with the owner of the retention policy in the room.
- Retention settings: list historian archive rotation, compression and deadband values, HMI alarm buffer sizes, MES purge jobs, camera recorder overwrite cycles and controller log limits.
- Change what you can afford: extend or disable purges for alarms, events and work orders first, since they are small and dense with signal.
- Cold storage: move aging historian archives, MES exports and image sets to lower-cost storage instead of deleting them.
- Migration snapshots: before any ERP, MES or historian upgrade, take a full export of the old system and keep a way to read it.
- Old drives and panel PCs: collect, label and image disconnected drives before they are wiped or recycled.
- Metadata: save the tag dictionary, engineering units, asset hierarchy, time zone, shift calendar and the compression settings for each period.
- Join keys: keep work order numbers, lot and serial numbers and machine IDs consistent across systems.
- Rights: note which data involves customer-owned processes, export-controlled work, vendor-hosted systems or identifiable operators.
- Policy and holds: confirm every change with the retention policy owner and counsel before keeping or discarding anything.
Keep, sample or let go#
Plant data does not all have to be kept at full resolution: keep event and decision records whole, and sample routine high-volume streams. A simple decision table helps OT teams set defaults that balance storage cost against future use, within whatever the retention policy and any legal hold require.
| Data type | Default | Reason |
|---|---|---|
| Alarm and event logs | Keep all | Small files with dense failure and recovery signal |
| Operator actions and acknowledgments | Keep all; de-identify before outside use | Shows human decisions during abnormal conditions |
| High-frequency sensor data | Keep full detail around events; downsample routine periods | Detail matters most just before and after faults |
| Inspection raw measurements | Keep with part, lot and gauge IDs | Shows drift that pass or fail summaries hide |
| Vision images | Keep rejects and borderline cases; sample passes | Balanced examples beat rejects alone |
| Routine camera footage | Follow policy; keep clips linked to incidents | Large volume and mostly repetitive |
Illustrative: an injection molder finds a box of drives#
Illustrative: a fictional injection molding company is moving from an on-premises MES to a cloud MES. While clearing a server room, its IT/OT director finds a box of drives pulled from old press-side PCs, plus historian archives that were about to roll off disk.
Instead of wiping them, the team images the drives, exports the historian archives with their tag dictionary and compression settings, and takes a full snapshot of the old MES, including mold, lot and downtime-reason tables. They turn off the purge job on alarm history in the new MES and add a cold-storage tier for historian archives.
The outcome is an indexed archive in which press alarms, mold changes, scrap codes and lot numbers can be joined across the old and new systems, with gaps and settings documented. Whether or not the company ever licenses any of it, process engineers are already using it to study recurring short shots.
How SourceX handles recovered plant archives#
SourceX starts with a fit check that collects metadata only: which systems hold the archive, the years covered, the record types and any known restrictions. No files move at that stage, and the company decides whether to go further.
If a package proceeds through the SourceX five-step transaction, large archives stay in the company's own storage or ship on encrypted drives; SourceX never hosts multi-TB datasets. The SourceX Evidence Packet records provenance, including compression settings and known gaps, alongside licensing rights, permitted use, the privacy record and release authorization.
Frequently asked questions
Does keeping more plant data increase legal risk?
It can. Data you keep can be requested in litigation, and some logs identify individual operators. That is why extended retention should be a written decision in the retention policy, reviewed with counsel, rather than an IT default. Keeping alarm logs and process data deliberately, for a stated purpose, is different from keeping everything by accident.
Can data be recovered from old historian archives and backups?
Often, if you can still read the format. Keep a working copy of the old historian or MES software, or export archives to open formats such as CSV or Parquet while the software still runs. Image old drives before anyone wipes them, and record which machine and period each drive came from.
Who should own retention settings in a plant, IT or OT?
Both, with clear roles. OT engineers know which tags, alarms and logs matter and how compression affects them. IT manages storage, backup and security. The retention policy owner, often in finance or legal, decides how long each record class is kept. Write the settings into the retention schedule so they survive staff changes.
Do operators need to be told that their actions are logged and kept?
Operator logins, acknowledgments and manual actions can identify individuals, so employee notices and workplace monitoring rules may apply depending on your state and any union agreement. Review notices with counsel before extending retention, and plan to replace operator IDs with pseudonyms before any use outside the company.
Related resources
See if your company qualifies
A short company assessment. No data uploads are needed.