Pick Confirmation via Scans and Weights as Automatic Labels

The buyer's problem

A data lead with thousands of hours of pick footage faces an annotation bill. Many of the labels they want, such as whether the pick succeeded, which unit was handled and when the step ended, are already written by the warehouse system for its own purposes. Confirmation scans, checkweigher results and order-complete events all record outcomes.

The temptation is to treat these as ground truth. They are useful, but each has a failure mode: a scan proves a code was read, not that the right object was in the hand. This page specifies what to request so system-generated labels are usable, how to test them and where to stop trusting them. For the pick-error side, see pick errors and mispicks; for the scan task itself, barcode scanning workflows.

What each system signal can label

The GS1 Serial Shipping Container Code is an 18-digit key that uniquely identifies a logistic unit such as a case, pallet or parcel, and it keys ASN or despatch advice data (GS1). Where a site uses it, scans can link a video interval to shipment-level records. Order picking is a heavy cost area, estimated at as much as 55% of total warehouse operating expense (de Koster et al., 2007); that figure concerns picking cost generally and says nothing specific about confirmation steps.

What this data can support, and what it cannot

Can support. Weak outcome labels at scale for pick-and-place video; step segmentation at scan timestamps; success/failure classifiers for reward or progress estimation, trained with spot-checked labels; filtering for clean vs problem picks; and stratified sampling for annotation, so human effort goes to the ambiguous cases. Record-based labels also support evaluation bookkeeping, such as the share of picks confirmed at first scan.

Cannot support. The labels contain no information about how the pick was done. They provide no robot actions, joint states or forces, and they are blind to anything between scans. A confirmed pick may have involved a drop and recovery; a flagged pick may have been fine. They also reflect the process rules of one site. Treat them as a way to cut annotation cost, with human verification, and never as a substitute for demonstration or teleoperation data. For what video alone can do, see what human demonstration video can and cannot do.

Illustrative specification

# Illustrative only - synthetic spec a buyer would adapt
spec: system_generated_pick_labels_v0
inputs:
  video: per-station files, constant frame rate, clock source stated
  events: [scan, weight_check, order_complete, override, exception]
event_schema:
  fields: [event_id, t_utc, station_id, event_type, unit_token, location_token, result, tolerance_class]
  time_precision: milliseconds preferred; stated if coarser
join:
  keys: [station_id, unit_token, task_id]
  max_offset_s: set by buyer after offset test
derived_labels:
  step_end: from scan timestamp
  outcome: confirmed | mismatch | out_of_tolerance | overridden
  confidence: high | medium | low   # by rule, see below
rules:
  high: scan matched and weight within tolerance and no override
  medium: only one of scan or weight available
  low: override or exception present
human_audit: stratified sample per site, share set by buyer, disagreement rate reported
excluded: [operator_name, customer_data, client_sku_names]
holdout: by site, station and time period

Quality and acceptance tests

1. Offset test. Sample events across shifts, mark the visible moment in video and report the offset distribution. The tolerance is the buyer's; the measurement must be the supplier's evidence.
2. Audit test. Humans review a stratified sample of labels. Report agreement by confidence tier and by site.
3. Coverage report. Share of video minutes with at least one event, and share of events with video.
4. Tolerance transparency. Checkweigher tolerance bands are stated per item class; items with similar mass cannot be separated by weight.
5. Override rate. Report how often scans were keyed manually or overridden; overridden picks get the low tier.
6. Join integrity. Duplicate keys, reused tokens and retries are listed.
7. Split discipline. Hold out sites, stations and periods; never let derived labels leak across the split.

Supplier types

Fulfillment centers with WMS scan and weight stations produce the events; their WMS vendors store them. Video is separate, held by site security or operations. These sources must be joined, and the supplier must be able to supply both or coordinate with whoever holds each. Client contracts often govern order and SKU fields; see client-owned inventory rights and 3PL providers as warehouse data suppliers. Related footage and record pairings are in pack station camera recordings, and records measuring size and mass are in dimensioning and weighing station records.

Limitations and additional preparation

System labels need alignment, join keys, tokenization of customer and employee identifiers and a documented rule for each derived label. They encode the site's process: a site with double-scan verification labels differently from one that scans once. Timestamps may be written at message arrival, not at the scan. Weight checks can fail for packaging variability rather than a wrong item. Document the final package with a datasheet; Datasheets for Datasets proposes covering motivation, composition, collection process and recommended uses (Gebru et al., 2018). Policy training still needs robot or teleoperated demonstrations.

When automatic labels mislead

Three situations deserve explicit tests. In the first, an operator scans an item and then drops it, picks it up and places it; the label says confirmed, the video shows a recovery. In the second, two items with near-identical mass are swapped; the weight check passes. In the third, a pack-verification scan occurs well after the pick, so using its timestamp as the end of the pick misplaces the label by minutes.

Handle these by assigning confidence tiers (as in the specification), sampling for human review in each tier, and reporting audit results with the dataset. State the labeling rules in the delivery notes, so a later user does not mistake derived labels for human judgment. If labels will feed an evaluation, keep a separate, human-labeled subset for that purpose and do not tune on it.

Questions to ask suppliers

1. Which system events are logged per pick, and at what time precision?
2. How are event clocks and camera clocks synchronized, and can you show measured offsets?
3. Which identifiers link events to video and to each other?
4. How are weight tolerances set, and do they vary by item class?
5. What share of events are manual entries or overrides?
6. Can you share an audit of event-to-video agreement from a sample?
7. Are shipment-level keys, such as logistic unit codes, retained?
8. Do records contain operator, customer or recipient identifiers, and how are they handled?
9. Who owns the order and item data, and who can authorize licensing?

Discuss your requirements

If you can state the signals, join keys and audit level you need, describe the requirement to SourceX. SourceX reviews it, follows up about potential sources among US businesses, and checks the data and the supplier's licensing permissions before coordinating the license and delivery. A request does not guarantee a match. See the humanoid training data guide and the warehouse and fulfillment hub; catalog context is on the sensor and IoT data page.

Discuss your humanoid training-data requirements with SourceX

Sources

• Serial Shipping Container Code (SSCC) — logistic unit key linking scans to shipping data.
• Design and control of warehouse order picking: A literature review — order-picking cost share estimate.
• Datasheets for Datasets — documenting dataset composition and collection.