Short answer
Manufacturing execution system (MES) records log which unit was at which station, when work started and ended, and the result. Aligned to video through shared timestamps and join keys, they provide ground-truth timing and outcome labels for footage. They are structured events, not visual data, so on their own they cannot train perception or control.
Why MES records matter for footage
A station camera shows what happened but not what it meant. Was this cycle good or reworked? Which unit was it? When did the work begin by the system's clock? The MES already knows. It logs unit IDs, operations, start and end events, operator logins, parameter results and pass/fail outcomes, because traceability requires it.
Joined to video, those events turn hours of unlabeled footage into indexed, outcome-labeled clips. That is the same idea as pick confirmations used as labels in the warehouse, applied to a production line. It also makes it possible to search for what is rare, such as rework cycles or stoppages, without watching every hour.
What an MES export usually contains
Systems differ, but exports commonly hold the following. Field names below are generic.
| Field group | Example | Value for labeling |
|---|---|---|
| Unit identity | Serial, lot, work order, variant | Join key to video, images, test data |
| Station and operation | Station ID, operation code, step | Locates the clip in the process |
| Time | Start, end, event time | Cuts clips to the cycle |
| Operator | Badge or login ID | Personal data; pseudonymize |
| Parameters and results | Torque, test values, pass/fail | Outcome labels |
| Exceptions | Defect code, rework route, scrap | Rare event labels |
| Downtime | Stop reason, duration | Context; see andon and downtime logs |
Complementary records include error-proofing records, such as poka-yoke triggers, and quality results described under manufacturing quality records.
Alignment: where projects fail
The value of MES data depends on joining it to video correctly. Four things go wrong.
Clock mismatch. MES servers, PLCs and cameras may run on different clocks. A drift of a few seconds can mislabel a short cycle. The Precision Time Protocol defined in IEEE 1588-2019 synchronizes clocks in networked systems, with sub-microsecond accuracy possible with minimal resources (IEEE, 2019). Ask whether the plant used PTP, NTP or nothing, and whether offsets were measured. Lacking synchronization, ask whether an offset was measured at least daily; that is a practical fallback.
Weak join keys. The video has no serial number. The link is "station 12 at 10:42" and the MES says unit 88317 was at station 12 from 10:41:52 to 10:42:30. If two units overlap or the station is a buffer, the join is ambiguous. Ask for scanner events or cycle triggers that the video also records.
Granularity. MES events record operation start and end, not the steps within. Fine step labels still need annotation, using MES events as anchors.
Event semantics. A "complete" event can be a worker's button press rather than a sensor reading. Ask how each event is generated.
Formats and containers
Record exports are often CSV or database dumps, but when events from controllers, cameras and sensors must live together, a container for time-synchronized multimodal logs is useful. MCAP is an open-source container for multimodal log data that holds multiple channels of timestamped, pre-serialized messages, is serialization-agnostic (ROS 1, ROS 2, Protobuf, JSON and others) and is the default ROS 2 bag storage since the Iron release (Foxglove, MCAP). Whether you need it depends on your pipeline; for MES tables, a documented CSV or Parquet export with a time-base note may be enough.
What the data can support, and what it cannot
MES records aligned to video can support:
- Clip segmentation by unit and cycle.
- Outcome labels such as pass, rework or scrap for each clip.
- Weak step labels, from operation codes and parameter events.
- Cycle-time analysis and anomaly retrieval.
- Splits by lot, shift or line for evaluation.
They cannot support:
- Visual or manipulation learning without footage. Events do not contain images, poses, or motions.
- Robot actions. No joint, gripper or force data unless the system includes controller logs from a robot.
- Cause. A reject code says a unit failed, not why. Video and annotation are needed for cause.
- Generalization. One plant's operation codes and quality definitions are idiosyncratic.
Illustrative aligned package
# Illustrative only - synthetic values, not an available dataset
package: mes_video_aligned_v0
video: station_S07 overhead + side, 30 fps, files per hour
mes_export: parquet, one row per operation event
join:
key: unit_serial_hash + station_id
video_offset_s: measured per day (e.g. +1.8 on day_03), method documented
clock_source: ptp | ntp | unsynchronized (state which)
labels: cycle_start, cycle_end, result, defect_code, rework_flag
operator_id: pseudonymized per package
checks: 100 random cycles verified manually, match rate reported
excluded: customer fields, pricing, supplier names
Permissions and privacy
MES data identifies operators, lots, and sometimes customers. Operator IDs can reveal individual productivity, which raises worker-privacy concerns separate from the video itself. Customer and part identifiers may be commercially sensitive; see manufacturing data confidentiality and worker consent and identifiable people. Pseudonymization reduces exposure but does not by itself settle whether a supplier may license the data.
Limitations and additional preparation
Expect preparation: extract the relevant fields, remove or hash identifiers, measure clock offsets, build join tables, and verify a sample by eye. Many plants keep MES data for a limited time, so footage and events may cover different periods. Records for processes with manual steps may be sparse. The process monitoring camera footage page covers the video side.
Choosing what to request
Request the smallest set of fields that gives you the labels you need. For most footage-labeling uses that is unit key, station, operation code, start and end time, result and defect code. Operator IDs, customer fields and pricing are rarely needed and increase risk. Ask the supplier for a data dictionary that defines each event and code, because a code such as "R2" means nothing without it. Ask for a sample covering a few days so you can test the join before agreeing to a larger export.
A worked use of event records on the plant floor is line-side kitting and replenishment.
Questions to ask suppliers
- Which MES events and fields can be exported, and in what format?
- How are unit IDs linked to the video: scanner event, cycle trigger or time only?
- Are clocks synchronized, and has the offset between systems been measured?
- How are "start" and "complete" events generated, by sensor or by operator action?
- How are rework and scrap recorded, and can they be tied to video?
- How long were records and footage retained, and do the periods overlap?
- Which fields identify operators, customers or suppliers, and how will they be handled?
- Can you share a sample join with a manually checked match rate?
- Do you hold controller or tool logs that sit alongside the MES events?
Discuss your requirements
If you need footage with outcome and timing labels, describe the stations, record types and permitted uses to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. A request does not guarantee a match; SourceX follows up about potential sources. For the wider map, start at the humanoid training data guide or the manufacturing and assembly hub. The site's entry on sensor and IoT data describes related records.
Discuss your humanoid training-data requirements with SourceX
Sources
- MCAP — open-source container for timestamped multimodal logs.
- IEEE 1588-2019 - IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems — Precision Time Protocol for clock synchronization.