Short answer
Food assembly line recordings show repetitive hand tasks such as placing, folding, spreading, portioning and tray loading, performed under hygiene rules and at line pace. They can support step recognition, error detection and evaluation. They do not supply robot actions, grip forces or food-contact behavior, and gloves, hairnets and fast cycles limit what annotators and hand trackers can see.
What makes line work different
Assembly line food work is short-cycle and repeated hundreds of times per shift. A worker builds a sandwich, fills a meal-kit tray or places components into compartments in the same order with small variations. Items are soft, sticky or slippery, and the line may move while the worker places.
This is not the same as commercial kitchen food prep, which involves cutting, heating, mixing and variable recipes. Line work trades variety for rhythm, consistency checks and hygiene routines. The question for a buyer is whether the repeated pattern is the thing a model needs to learn, or whether the long tail of variation is.
Task anatomy
| Element | Examples | Why it matters to data |
|---|---|---|
| Station layout | Ingredient bins, tray conveyor, scale, label printer | Defines where hands travel and what objects appear |
| Component handling | Pick slices, scoop sauce, drop portions, place garnish | Deformable, sticky, variable shape |
| Placement rules | Position in tray compartment, stack order, orientation | The structured part a model can learn |
| Quality checks | Weight, count, appearance, label match | Needed for success labels |
| Hygiene routine | Glove change, hand sanitizing, contamination stop | Constrains capture and shapes the sequence |
| Line pace | Units per minute, takt time | Affects motion blur and annotation effort |
Failure modes: wrong component count, misaligned tray, torn or crushed product, sauce on the rim, glove contamination, and a missed pick that propagates down the line.
What the data can support
- Step and sequence understanding. Recipe and procedure datasets show how step-level labels are built. YouCook2 annotated 2,000 videos covering 89 recipes (176 hours) with temporal boundaries and descriptions for each procedure segment (Zhou et al., 2017). Line-work video can be annotated the same way, though line steps are much shorter.
- Error recognition. CaptainCook4D recorded 384 egocentric recordings (94.5 hours) of recipes in real kitchens, with both normal and intentionally erroneous executions, 5.3K step annotations and 10K fine-grained action annotations, and used them to benchmark error recognition and procedure learning (Peddi et al., 2023). A food-line dataset that includes deliberate mistakes serves the same purpose for quality detection.
- Rhythm and cycle time. Fixed-camera video yields per-unit durations, useful for working-pace analysis.
- Component recognition and placement checking. Images of finished trays support visual QA models that decide whether an assembly is complete.
- Evaluation sets. Held-out lines or products test whether a model's step labels generalize.
What it cannot support
- Robot actions. A hand sprinkling cheese or sliding a slice from a stack carries timing and finger motion that no robot uses directly. Retargeting needs 3D hand pose, calibration and robot data; see the embodiment gap.
- Grip force on soft items. Squeezing a bun, peeling a tortilla or lifting a thin slice depends on touch. Video does not show it.
- Food variability. One day's tomatoes differ from the next. A single product run captures one distribution.
- Hygiene compliance as a robot rule. Observing glove changes shows human behavior, not a validated robot hygiene design.
- Hand tracking under gloves. Gloved hands in white-on-white scenes confuse trackers; see gloved hands in workplace video.
Capture and labeling choices
At line pace, 30 frames per second may blur fast placements. Specify shutter speed and frame rate, and consider a second view looking along the tray. Segment at the level your model will use: unit-level (one sandwich), component-level (add ham) or placement-level (place slice, release). The finer the level, the more annotation per hour. Step segmentation annotation covers granularity and agreement.
Hygiene rules often limit what can be worn on the head or hands. A head-mounted camera may need approval from the food-safety manager, and a fixed overhead camera may be the only practical choice. Plan for glass, steam and washdown cleaning in camera mounts. Procedures such as sanitation steps may also exist as written records; see food safety and sanitation procedures.
Illustrative package for a meal-kit tray line
# Illustrative only; synthetic values
package: meal_kit_line_v0
sites: site_017, site_021 (2 production lines)
operators: operator_A01..A14 (pseudonymous), mixed tenure
capture:
fixed_overhead: 60 fps, shutter fixed, global view of station
head_camera: where permitted by food-safety lead
scale_log: per-tray weight timestamps
labels:
unit_segments: tray_start, tray_end
placement_events: component_type, compartment_id, timestamp
errors: missing_item, wrong_item, misplaced, contamination_stop
outcome: pass / rework / reject (from line QC record)
excluded: robot actions, force; recipe formulas and supplier names removed
splits: by line and shift
Acceptance criteria
- Unit boundaries reconcile with the line's own count or scale log within a tolerance you set.
- Error classes appear with enough examples to evaluate; if none are labeled, assume none were kept.
- Motion blur does not hide hand-object contact in a random sample of placement events.
- Product mix, shifts and operators are listed so you can split evaluation sets.
- Recipe or formula data is redacted or licensed, not accidentally embedded in labels.
Permissions and provenance
Production lines hold trade secrets in recipes, supplier labels and line design, and often customer-branded packaging. Ask whether the supplier can license footage that shows a retailer's private-label product. Workers on a line are identifiable by build, hands and badge; see worker consent and identifiable people. Guidance from the EDPB on video devices is one reference on notice and proportionality (EDPB, 2020). This is not legal advice.
Questions to ask suppliers
- Which products, line speeds and shifts were recorded?
- Is the footage from production or from a staged session?
- Are errors, rework and line stops preserved?
- Does a QC or weight log exist that can be linked to the video?
- What approvals did the food-safety lead give for cameras, and can the setup be repeated?
- Do retailer or brand customers have rights in what is visible?
- How many distinct operators appear, and were they told how the video would be used?
- Does a written assembly specification accompany the recordings, and can it be licensed too?
Discuss your requirements
For line-work data, describe the products, camera views, error classes and permitted uses to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. If no existing recordings fit, SourceX can scope a commissioned collection. A request does not guarantee a match; SourceX follows up about potential sources.
Start at the humanoid training data guide or the workplace task demonstration hub. Related catalog entries include first-person video of skilled manual work and robotics and embodied AI use cases. For recovery behavior, see failure and recovery demonstrations.
Describe the workplace demonstration data your robotics project needs
Sources
- CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities — Egocentric recipe recordings with normal and erroneous executions and step annotations
- Towards Automatic Learning of Procedures from Web Instructional Videos — Procedure segmentation with temporal boundaries on cooking videos
- Guidelines 3/2019 on processing of personal data through video devices (Version 2.0) — Guidance on personal data processed through video devices