Short answer
Grocery micro-fulfillment picking builds mixed-temperature orders into totes from a huge, fast-changing range of fragile, deformable and rigid items. Grocers and micro-fulfillment operators may hold tote-building video and order records, though client and brand rights apply. The data supports item recognition and tote-packing logic; handling skill needs robot or teleoperation data, not footage or catalog data alone.
Why grocery picking is its own problem
General each-picking draws from a catalog of mostly rigid, stable products. Grocery adds items that bruise, leak, crush or slump: produce, bakery, eggs, bagged goods, glass bottles, yogurt cups, meat trays. It adds temperature classes, since an order may span ambient, chilled and frozen goods that must be kept apart or packed in a defined order. And it adds substitutions, because the item on the shelf may not be the item ordered.
A micro-fulfillment center, or MFC, is a compact picking facility close to demand, either automated or semi-automated, often attached to or inside a store. Operators describe themselves in different ways, so confirm what the supplier means: a store aisle picked by staff, a back-of-store automated cell, or a dedicated site. The each-picking demonstrations page covers picking in general; this page covers what grocery changes.
Task anatomy
Confirm with the supplier which of these happen where:
- Receive the order and its tote or bag plan, including temperature classes.
- Locate items by aisle, shelf, bin or goods-to-person port; see goods-to-person station tasks.
- Pick the correct item, size and variant.
- Check freshness, damage or code date where the operation requires it.
- Place into a tote by a pack plan: heavy and rigid items first, fragile on top, temperature classes separate.
- Handle substitutions and shorts where the item is out of stock.
- Close and stage totes for pickup or delivery.
The tote is the task's unit of success, not the item. A correct pick placed on top of a fragile item or put into the wrong temperature tote is still a failure. For container handling beyond grocery, see tote and container handling.
What the data can support
| Data | Supports | Does not provide |
|---|---|---|
| Product images and catalog attributes | Item recognition, variant discrimination | How items behave when handled |
| Video of tote building | Pack-order logic, fragile-item placement, step segmentation | Actions or forces |
| Order and substitution records | Ground truth for correct picks and exception types | Spatial detail |
| Pick-confirmation scans | Time-stamped weak labels | Cause of errors |
| Damage and complaint records | Examples of consequential errors | Sparse and aggregated |
Retail recognition data can help with the item-identity half. RPC contains single-product images under controlled conditions and multi-product checkout images across 200 product categories grouped into 17 meta categories, up to 83k images (Wei et al., 2019). It was built for automatic checkout counters, not for grocery fulfillment, so it is useful only as a reference for how product images and category hierarchies can be labeled. See product catalog data for robot perception and deformable product properties for the product-side data that recognition alone misses.
Suppliers and rights
Grocers, delivery fulfillers, MFC operators, store-based pickers and distribution centers serving grocery may hold order histories, pick-confirmation logs and some video. Stores may have loss-prevention camera coverage that includes customers, which is a significant privacy issue; staff-only back-of-house areas are likely to be more suitable. Order histories contain customer identities, addresses and purchase patterns, which a buyer will usually want removed entirely.
Rights are layered: the grocer may own the order data, brands own packaging imagery, and customers appear in store footage. This is not legal advice. Ask which party can license each layer, and see worker consent and identifiable people. Keep robot-perception evaluation separate from any productivity measurement of pickers.
Commissioned collection
Commissioned capture lets a buyer choose the order mix. A plan might cover ambient, chilled and frozen items, a graded range of fragile products, bagged and boxed items, and substitution events. Useful views include a fixed camera over the tote, a picker's-eye view and the shelf face. Item identity, tote plan and pack order should be logged alongside the video. Capture in chilled or frozen areas inherits the constraints on cold-chain and freezer handling.
Expect limits from live operations: customers share the space in some store formats, fresh stock changes daily, and recording should avoid interfering with order cut-off times.
Illustrative order record
# Illustrative only - synthetic values, not an available dataset
order_id: ord_000931
site_id: site_021
format: store_based_picking # or automated_cell
temperature_classes: [ambient, chilled]
totes:
- {tote_id: t1, class: ambient, pack_plan: [heavy_rigid, bagged, fragile_top]}
- {tote_id: t2, class: chilled}
picks:
- {line: 1, item_class: class_3, qty: 1, result: correct}
- {line: 2, item_class: class_9, qty: 2, result: substituted, reason: out_of_stock}
video:
views: [tote_overhead, picker_chest]
sync: shared_clock
customer_data: removed
excluded: robot actions, bruising forces, produce ripeness grading
A sizing sketch could count three temperature classes, four fragility grades and two container types, which makes twenty-four cells. Those inputs are arbitrary and not minimums; the number of examples needed depends on the architecture, task and embodiment.
What this data cannot support
- Robot handling. Picking a ripe peach, a bag of chips and a glass jar without damage needs robot or teleoperation data, ideally with tactile information. Video, product images and order records alone do not contain the actions, contact forces or compliance a controller must learn.
- Freshness and quality judgment. Grading produce is a separate perception task with its own labels.
- Item-range coverage. Grocery assortments change frequently; a captured set samples one period and one store type.
- Food-safety conclusions. Nothing here supports judgments about temperature control or food handling practice, and no compliance conclusion should be drawn.
- Customer-facing behavior. Interactions with shoppers are a different problem.
Evaluate on held-out items, sites and temperature classes, and never mix evaluation orders back into training.
Questions to ask suppliers
- Which formats exist: store-based picking, an automated cell, or a dedicated MFC?
- What temperature classes are handled, and how are they separated in totes?
- Do cameras cover tote building, and are customers ever in frame?
- Are order, substitution and short records exportable, and with what identifiers?
- How are fragile and damaged items logged?
- Which items are produce, bakery or unpackaged, where appearance varies?
- What customer data is in the records, and can it be fully removed?
- Which brands or suppliers restrict use of packaging imagery?
- Were store pickers told their footage might be licensed, and who can license footage that also shows shoppers?
Discuss your requirements
If you are scoping grocery picking data, describe the formats, temperature classes and permitted uses to SourceX. SourceX reviews the requirement, looks for US businesses that may hold suitable material, checks the data and the supplier's licensing permissions, and follows up about potential sources. If no existing asset fits, SourceX can scope a commissioned collection of new recordings. A request does not guarantee a match. For orientation, see the humanoid training data guide and the warehouse data hub; the catalog page on product catalogs and descriptions is related.
Describe the workplace demonstration data your robotics project needs
Sources
- RPC: A Large-Scale Retail Product Checkout Dataset — product images and category hierarchy from checkout settings; a recognition-label reference only, not grocery fulfillment.