Privacy and preparation
Retail returns fraud ambiguity: labeling suspicious returns in a dataset
By SourceX Editorial · Reviewed by Noah Loul ·
8 sections
Short answer
Label a suspicious return by what the evidence established, not by what staff suspected. Keep three separate fields: the signal that raised suspicion, the investigation state, and the confirmed finding with its evidence type and decision date. Returns nobody reviewed stay unlabeled, neither cleared nor fraud. Escalate when evidence conflicts, the policy version is unclear, or the label could drive a decision about a specific customer.
Redaction
Retail returns fraud
Key takeaways
- A fraud flag is a prediction made by an earlier rule or person; only a reviewed finding with named evidence is a label.
- Unreviewed flagged returns belong in an unlabeled pool, not among the legitimate examples or the fraud positives.
- Replacement shipments, stock receipts, promotions, approvals and policy versions can each turn an apparent mismatch into an ordinary event.
- A policy breach such as a late or worn return is a separate class from deception confirmed by evidence.
- Keep each reviewer's label and the agreement rate; disagreement marks the cases where the evidence is thin.
How do you label an ambiguous return without treating suspicion as proof?
Write down what the evidence established, and keep it apart from what someone suspected. A suspicious return gets a signal field (why it was flagged), an investigation state (whether anyone looked) and a finding (what was confirmed, by which evidence, on which date). Only the finding is a fraud label. A flag on its own is someone else's prediction, and training on it teaches a model to copy yesterday's rules.
Commerce systems already separate these layers, even if exports flatten them. Shopify's Admin API, for example, gives a return one of five statuses: requested, open, declined, closed or canceled. None of them says fraud. A declined return records a merchant decision, not a finding about the customer, so a dataset that maps declined to fraudulent has invented a label.
Payment systems draw the same line. Stripe's early fraud warning object is a card-issuer signal, and Stripe marks it actionable only while the charge has no dispute and has not been fully refunded. The warning, the dispute and the refund are three different records with three different meanings, and a returns dataset should keep the same discipline.
The rest of this page turns that principle into a working rubric: the fields to keep, the linked facts that change a reading, the uncertainty levels, how to handle reviewer disagreement, an illustrative case and the cases that should never reach a label queue at all.
Which fields keep a flag, a review and a finding apart?
Keep at least nine fields per flagged return, each with one job. The table below is a field specification a returns or loss-prevention team can adopt as is; the example values are illustrative.
The most common failure is a single fraud_flag column that mixes rule hits, reviewer opinions and confirmed findings. Once those are merged, nobody can recover which rows were proven, and every model trained on the column inherits the merge.
| Field | What it records | Example values | Mistake it prevents |
|---|---|---|---|
| suspicion_signal | Why the return was flagged | serial_mismatch, weight_short, high_return_frequency, receipt_not_found | Treating the reason for a look as the result of the look |
| signal_source | Who or what raised the flag | rules_engine_v12, vendor_score, store_associate, warehouse_inspector | Training a model to imitate an old rule set |
| investigation_state | Whether anyone reviewed the case | not_reviewed, in_review, review_closed | Counting unreviewed flags as negatives or positives |
| evidence_type | What the reviewer actually examined | inspection_photo, serial_scan, carrier_weight, fulfillment_record, cctv_note | Findings with no traceable basis |
| finding | The reviewed conclusion | confirmed_mismatch, policy_breach, cleared_explained, inconclusive | Collapsing abuse, error and fraud into one class |
| finding_date | When the conclusion was recorded | 2026-03-14 | Leaking later outcomes into earlier training windows |
| decided_by_role | Role of the person who decided | returns_lead, loss_prevention_analyst | Mixing associate guesses with specialist findings |
| policy_version | Return policy in force on the purchase date | returns_policy_2025_11 | Judging a return against a rule that did not apply |
| customer_outcome | What happened to the customer | refunded, store_credit, return_declined, escalated | Confusing the business response with the finding |
Which order, stock or promotion fact can flip the reading of a return?
A return is rarely ambiguous on its own; it becomes ambiguous when the linked records are missing. Before anyone writes a finding, pull the order line, the fulfillment history, the stock movements and the promotion that applied at purchase. Each of these can turn an apparent mismatch into an ordinary event.
The table lists the linked facts that most often change the valid reading, with the record that settles each one. A labeling guide should require a reviewer to check every row that applies before choosing any finding above inconclusive.
| Linked fact | How it can change the reading | Record to pull |
|---|---|---|
| Replacement or split shipment | A different serial may be the one the customer actually received | Fulfillment and replacement order history |
| Inventory receipt timing | An item returned before the matching stock was received cannot be the unit sold | Warehouse receipt log with timestamps |
| Bundle or free-gift promotion | Returning the paid item but keeping the gift may be allowed or may require a price adjustment | Promotion rules version on the purchase date |
| Manager approval or override | An out-of-window return may have been approved in advance | Approval record with approver role and time |
| Carrier scan and weight | A short weight can come from carrier damage or a packing error | Carrier tracking events and dock weight |
| Account merge or address change | Apparent serial returning may be one household with merged accounts | Customer master change history |
| Policy change between purchase and return | A return valid under the old window may look late under the new one | Policy version table keyed to dates |
Which uncertainty level and escalation route does each flagged return get?
Assign every flagged return one level, using the minimum evidence in the third column; when the evidence for a higher level is missing, the case stays at the lower one. The levels describe how much is known, not how bad the case looks.
The rubric deliberately separates a policy breach from deception. A worn item returned after the window breaks a rule, but the record rarely shows intent, so it belongs in its own class rather than in the fraud positives. Treat this rubric as a draft for your own retail operations and rights reviewers to adapt, not as a finished standard.
- Escalate to a second reviewer when the signal and the linked records point in different directions.
- Escalate to the policy owner when the policy version for the purchase date cannot be found.
- Escalate to loss prevention when a mismatch is confirmed but the explanation table has an unchecked row.
- Escalate to counsel when a case involves a dispute, a claim against the customer or an outside agency.
| Level | Meaning | Minimum evidence | Label to write | Next step |
|---|---|---|---|---|
| L0 | No signal raised | Routine return record | not_flagged (not proof of honesty) | None |
| L1 | Signal only, nobody reviewed it | A rule hit, score or associate note | flagged_unreviewed, excluded from fraud positives and negatives | Sample for review if the class is needed |
| L2 | Reviewed, evidence conflicts or is missing | Reviewer notes naming what was checked | inconclusive with a reason code | Second reviewer or more evidence |
| L3 | Reviewed, an ordinary explanation found | A linked record that explains the signal | cleared_explained with evidence_type | Close |
| L4 | Policy breach, intent not shown | Policy version plus the breached condition | policy_breach, kept separate from fraud | Policy owner decides on exceptions |
| L5 | Physical or system mismatch confirmed | Inspection photo, serial scan or weight record that rules out the listed explanations | confirmed_mismatch with evidence | Loss prevention review before any person-level conclusion |
| L6 | Formal finding recorded | Closed loss-prevention case with documented finding by an authorized role | confirmed_finding with case reference | Restricted handling and counsel review before reuse |
What should happen when two reviewers read the same return differently?
Record both readings and the agreement rate, then route the case to a third reviewer instead of averaging it away. In ambiguous returns, disagreement is information: it marks the cases where the evidence really is thin, and those are the cases a model most needs to handle with care.
Measure agreement on a sample of double-reviewed cases with Cohen's kappa. The scikit-learn documentation describes kappa as intended to compare labelings by different human annotators and notes that scores above 0.8 are generally considered good agreement. A low score on L2 versus L3 usually means the reason codes are unclear, not that reviewers are careless.
Keep the per-reviewer labels in the delivered file. Research on subjective annotation found that modeling each annotator's judgments, rather than a majority vote, matched or beat aggregated labels and gave uncertainty estimates that tracked disagreement better. A buyer can always collapse the labels later; nobody can rebuild them once they are merged.
Disagreement also matters downstream. If a model will output a probability that a return is fraudulent, that probability is only useful when it is calibrated: scikit-learn's guide explains that among cases scored near 0.8, roughly 80 percent should actually be positive. An evaluation set that keeps inconclusive and cleared cases lets a buyer check that property; one that drops them cannot.
Illustrative worked example: a serial mismatch with two explanations
This example is invented to show the method; it describes no real retailer, customer or result. A consumer electronics store receives a returned headset whose serial number does not match the serial recorded on the order line. The rules engine flags it as serial_mismatch.
The reviewer pulls the linked records before choosing a level. The fulfillment history shows that the first headset was reported damaged on delivery and a replacement unit shipped two days later. The replacement's serial matches the returned unit. The order line still carries the original serial because the replacement was created as a zero-value order. The case is labeled L3, cleared_explained, with evidence_type fulfillment_record.
Now the exception. A second flagged headset in the same batch matches neither the original nor any replacement serial, and the warehouse receipt log shows that serial was never stocked by the store. The reviewer records L5, confirmed_mismatch, with the serial scan as evidence. That is still not a finding that the customer committed fraud: a carrier or warehouse swap remains possible until loss prevention closes a case. The dataset row says exactly that much and no more.
The difference between the two rows is one linked record. A dataset that kept only the flag would have labeled both as fraud, and a model trained on it would learn that replacement orders look suspicious.
Which return cases belong with loss prevention or counsel instead of the label queue?
Pull a case out of the general label queue when the label could be used to make a decision about a specific person. Return-review records can feed customer-level decisions, and the U.S. Consumer Financial Protection Bureau's list of consumer reporting companies includes a retail entry for a company that monitors and reports retail return and exchange fraud and abuse to merchants. Whether a given dataset or model raises consumer-reporting obligations is a question for counsel on the specific facts.
The cases below should go to a specialist route rather than an annotator:
- Open disputes, chargebacks or claims against the customer, until they close and counsel approves reuse.
- Cases referred to an outside agency or involving employee conduct.
- Records where the only evidence is an associate's description of a person rather than of the item or transaction.
- Fields that reveal or proxy for protected characteristics, which should be removed rather than labeled.
- Reviewer and associate names, which should be replaced with role codes before any export.
What makes return-review records worth licensing?
If you are considering licensing returns records, the rubric above is also what makes them worth reviewing: separated signal, review and finding fields, linked fulfillment and policy records, and labels that admit uncertainty. Retailers that want to license such records can do it through SourceX, which qualifies the company and then manages rights review, contracting, preparation and delivery.
On size, the usual buyer expectation is two-plus years of consistent history and tens of thousands of records or more from one system; rarer work can get by with less, and each set is weighed deal by deal. Customer details are de-identified by SourceX through approved processing routes, nothing moves before the retailer approves the scope and terms in a signed agreement, and eligibility is checked through a short questionnaire that needs no return records.
Frequently asked questions
Is a declined return the same as a fraud label?
No. A declined return records the business response, which can follow from a late return, a missing receipt or a policy rule. Fraud requires a reviewed finding with evidence, so keep the decline in customer_outcome and the conclusion in a separate finding field.
Should flagged returns that nobody reviewed be used as legitimate examples?
No. An unreviewed flag is neither cleared nor confirmed, so it belongs in an unlabeled pool. A survey of positive-unlabeled learning notes that methods treating unlabeled examples as negatives are working with label noise, and a returns dataset can avoid that by marking these rows flagged_unreviewed.
What is the difference between return abuse and return fraud in a dataset?
Abuse usually means a policy breach, such as wearing an item and returning it late, where intent is not shown in the record. Fraud implies deception confirmed by evidence, such as a swapped or empty package. Label them as separate classes so a model does not learn to treat every late return as dishonest.
How many reviewers does an ambiguous return need?
One reviewer is enough for clear L0, L3 and L4 cases. Inconclusive cases and any case heading to L5 or L6 should get a second reviewer, and a sample of double-reviewed cases should be scored for agreement so unclear reason codes can be fixed.
Can labels built this way be used to refuse a customer's return?
That depends on how the labels and any model are used and on the law that applies, which counsel should review. The rubric is designed for training and evaluating models on well-documented cases; using outputs to make decisions about individual customers raises separate legal and fairness questions.
Sources
- Shopify's Admin API gives a return one of five statuses: requested, open, declined, closed or canceled. Source
- Stripe treats an early fraud warning as actionable only if it has not received a dispute and has not been fully refunded. Source
- Positive-unlabeled learning methods that treat unlabeled examples as negatives are handling label noise. Source
- Modeling individual annotators matched or beat majority-vote labels and gave uncertainty estimates that correlate better with disagreement. Source
- Cohen's kappa compares labelings by different human annotators, and scores above 0.8 are generally considered good agreement. Source
- For a well-calibrated classifier, about 80 percent of samples given a probability near 0.8 belong to the positive class. Source
- The CFPB's consumer reporting companies list includes a retail company that monitors and reports retail return and exchange fraud and abuse to merchants. Source
See if your company qualifies
A short company assessment. No data uploads are needed.