Short answer
Retail product recognition data pairs reference images of each SKU with in-store images labeled at GTIN level, so a robot can find and identify items on shelves, in backroom cases and in totes. Grocery is the hard case: look-alike variants, constant SKU churn, promotional packaging, glass-door freezers and unpackaged produce. Grocers, brands and merchandising services hold pieces of it; expect gaps for case-level and robot-height views.
Why stores are a distinct recognition problem
A humanoid stocking shelves, picking online orders or auditing gaps has to name the exact product in front of it, not just its category. Generic detection datasets do not prepare a model for that. In a store the hard parts are specific:
- Look-alikes. Flavors, sizes and multipacks often share nearly identical artwork. The difference may be a color band or a small number.
- Churn. New items, seasonal packs, promotional "20% more" packs and redesigns arrive constantly, so a fixed label set ages fast; see packaging variants and SKU appearance changes.
- Three forms of the same item. Eaches on the shelf, shipping cases in the backroom and shelf-ready trays in between look different and carry different identifiers.
- Shelf tags. Price and item tags on the shelf edge are how staff confirm a slot. Linking a tag to the product above it is its own task, close to the work on labels, barcodes and text on packages.
- Unpackaged items. Produce, bakery and deli items vary unit to unit and have no packshot.
- Optics. Glass freezer doors add reflections and condensation; glossy film adds glare.
- Robot viewpoints. Bottom shelves seen from knee height and top shelves from below are rare in photos people take.
How many SKUs your robot must tell apart, and whether new ones must be added without retraining, decides the label level; instance vs category recognition covers that choice.
Two halves of a retail recognition dataset
Most retail recognition data has two parts that must share identifiers.
A reference gallery holds clean images of each SKU, ideally from several sides. It is the identity side: when a new item arrives, it is added to the gallery. Catalog and brand imagery fills most of it; product catalog data for robot perception covers that supply.
Query images are what the robot sees: shelves, cases and totes, with boxes or masks labeled to the same GTINs. These are the scarce half.
Public datasets mirror this split. RPC separates single-product images taken under controlled conditions from multi-product checkout images, with several levels of annotation (Wei et al., 2019). Products-10K mixes in-shop and customer photos of the same SKUs under a hierarchical label graph (Bai et al., 2020). ABO defines a multi-view retrieval benchmark on catalog imagery (Collins et al., 2021), a formulation suited to catalogs that change often.
Density needs its own attention. SKU-110K targets tightly packed, often identical shelf items, and its authors added a Soft-IoU layer and an EM-Merger unit to the detector to handle them (Goldman et al., 2019). Ask whether a supplier's labels separate adjacent identical facings or box whole rows.
For coarse levels of the hierarchy, a standard classification avoids inventing one. GS1's Global Product Classification has four tiers, Segment, Family, Class and Brick, and assigns exactly one brick per GTIN (GS1).
Label fields that matter in stores
| Field | Why it matters | Common problem in supplier data |
|---|---|---|
| Box or mask per facing | Counting and picking need instances, not rows | Rows boxed as one object |
| GTIN | Joins to catalog, planogram and inventory | Store-internal codes only |
| Variant group | Lets you evaluate look-alike confusion | Not recorded anywhere |
| Item form | Each, case or shelf-ready tray | Cases labeled with the each's GTIN |
| Shelf tag link | Connects the tag to the product slot | Tags unlabeled |
| Visibility | Fully visible, partly occluded, behind glass | Missing; occluded items dropped |
| Packaging version | Separates old and new artwork | Inferred from date only |
| Capture context | Store format, fixture, camera height, date | Not kept with images |
Supplier types
| Supplier | What they typically hold | Strength | Watch for |
|---|---|---|---|
| Grocers and retail chains | Store audit photos, gap-scan images, item-setup images, planograms | Real shelves in real stores | Shoppers and staff in frame; images supplied by brands |
| Consumer-goods brands | Packaging artwork with version history, packshots | Authoritative variants and change dates | Only their own products |
| Retail execution and merchandising firms | Shelf photos taken by field representatives | Many stores, regular cadence | Brand clients may control the photos |
| E-grocery and micro-fulfillment operators | Pick and substitution photos in totes | Robot-like viewpoints | Mostly eaches; tote clutter |
| Shelf-scanning technology operators | Fixed-camera or scanning-robot images | Consistent geometry and frequency | Technology vendor may own the data |
Shelf photos labeled against a target layout are a separate asset with their own checks; see planogram compliance images. For tote picking in dark stores, see micro-fulfillment grocery picking.
Illustrative package contents
# Illustrative only — synthetic stores, SKUs and counts, not an available dataset
package: retail_recognition_supplier_G04_v0
assortment: 9,000 GTINs across ambient grocery, frozen and household
gallery:
per_gtin: 4-8 views (brand packshots + retailer item-setup photos)
artwork_versions: retained where the brand supplied dated files
query_images:
sources: [store_audit_photos, overnight_gap_scans, backroom_case_photos]
stores: 18 (12 train, 3 validation, 3 held out for test)
labels: {box_per_facing, gtin, variant_group, item_form, shelf_tag_link, visibility}
camera_height_bands_cm: [30-80, 80-140, 140-200]
fresh_items: produce and bakery labeled at category level only
excluded: images with identifiable shoppers; price values on tags (masked)
rights: brand packshots licensed per brand; audit photos owned by retailer
Holding out whole stores, not random images, is what makes the test meaningful; the same aisle photographed twice in a week is nearly a duplicate.
What this data cannot support
- Handling. Recognition data says nothing about pulling a can from a tight row, threading a peg hook or catching a tipping bottle. Those skills need demonstrations and robot data; retail shelf restocking demonstrations covers the task side.
- Physical behavior. Bread squashes, bags shift and multipacks flex; images do not record it.
- In-hand appearance. Fingers and gloves hide labels during handling, a view shelf photos rarely show.
- Prices. Prices change constantly and should not be learned as part of identity.
- The future assortment. Items launched after the data cutoff are absent unless the gallery is refreshed.
Quality and acceptance criteria
- Identifier validity. GTIN check digits pass, and case and each identifiers are not mixed.
- Gallery coverage. The share of your target assortment with gallery images meets your threshold.
- Label audit. An independent re-label of a sample agrees with supplier labels at the GTIN level above a rate you set, reported separately for look-alike groups.
- Freshness. Capture dates are recent relative to packaging changes in your categories.
- Splits. Test stores and dates are held out entirely.
- Viewpoint coverage. Counts per camera-height band are reported.
- Privacy handling. The process for excluding or masking shoppers and staff is documented.
Questions to ask suppliers
- Who took the shelf photos, and under which contract: store staff, a merchandising firm or a technology vendor?
- Are labels at GTIN level, and are adjacent identical facings boxed separately?
- Do you keep dated packaging artwork or item-setup images for old and new versions?
- Are backroom cases and shelf-ready trays photographed and labeled?
- At what heights and angles were images taken?
- How are shoppers and staff kept out of images or masked?
- Which gallery images came from brands, and on what terms?
- How often could the gallery and query sets be refreshed?
Permissions
In-store images can include shoppers and staff, and brand artwork appears in every frame. Masking people changes what is identifiable but does not by itself establish permission to use the images; ask how notice was given and under what terms. Photos taken by merchandising firms may belong to their brand clients. See rights in product images.
Discuss your requirements
Describe the assortment, store formats, viewpoints and label level you need. Send the requirement to SourceX: it looks for US businesses that hold suitable images, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. SourceX reviews each request and follows up about potential sources; a request does not guarantee a match. If no existing asset fits, SourceX can scope a commissioned collection of new recordings. See also the product and object data hub, the humanoid training data guide and the page on licensing product catalogs and descriptions.
Discuss your humanoid training-data requirements with SourceX
Sources
- Precise Detection in Densely Packed Scenes — dense shelf detection and the methods it required.
- Products-10K: A Large-scale Product Recognition Dataset — SKU-level labels across shop and customer images.
- RPC: A Large-Scale Retail Product Checkout Dataset — controlled single-product images alongside cluttered scenes.
- ABO: Dataset and Benchmarks for Real-World 3D Object Understanding — multi-view retrieval on catalog imagery.
- GS1 Global Product Classification (GPC) — standard product hierarchy.