Skip to content

Privacy and preparation

Retail returns fraud ambiguity: labeling suspicious returns in a dataset

By SourceX Editorial · Reviewed by Noah Loul ·

8 sections

Short answer

Label a suspicious return by what the evidence established, not by what staff suspected. Keep three separate fields: the signal that raised suspicion, the investigation state, and the confirmed finding with its evidence type and decision date. Returns nobody reviewed stay unlabeled, neither cleared nor fraud. Escalate when evidence conflicts, the policy version is unclear, or the label could drive a decision about a specific customer.

Redaction

Retail returns fraud

This page covers retail returns fraud ambiguity: labeling suspicious returns in a dataset.Illustrative

Key takeaways

  • A fraud flag is a prediction made by an earlier rule or person; only a reviewed finding with named evidence is a label.
  • Unreviewed flagged returns belong in an unlabeled pool, not among the legitimate examples or the fraud positives.
  • Replacement shipments, stock receipts, promotions, approvals and policy versions can each turn an apparent mismatch into an ordinary event.
  • A policy breach such as a late or worn return is a separate class from deception confirmed by evidence.
  • Keep each reviewer's label and the agreement rate; disagreement marks the cases where the evidence is thin.

How do you label an ambiguous return without treating suspicion as proof?

Write down what the evidence established, and keep it apart from what someone suspected. A suspicious return gets a signal field (why it was flagged), an investigation state (whether anyone looked) and a finding (what was confirmed, by which evidence, on which date). Only the finding is a fraud label. A flag on its own is someone else's prediction, and training on it teaches a model to copy yesterday's rules.

Commerce systems already separate these layers, even if exports flatten them. Shopify's Admin API, for example, gives a return one of five statuses: requested, open, declined, closed or canceled. None of them says fraud. A declined return records a merchant decision, not a finding about the customer, so a dataset that maps declined to fraudulent has invented a label.

Payment systems draw the same line. Stripe's early fraud warning object is a card-issuer signal, and Stripe marks it actionable only while the charge has no dispute and has not been fully refunded. The warning, the dispute and the refund are three different records with three different meanings, and a returns dataset should keep the same discipline.

The rest of this page turns that principle into a working rubric: the fields to keep, the linked facts that change a reading, the uncertainty levels, how to handle reviewer disagreement, an illustrative case and the cases that should never reach a label queue at all.

Which fields keep a flag, a review and a finding apart?

Keep at least nine fields per flagged return, each with one job. The table below is a field specification a returns or loss-prevention team can adopt as is; the example values are illustrative.

The most common failure is a single fraud_flag column that mixes rule hits, reviewer opinions and confirmed findings. Once those are merged, nobody can recover which rows were proven, and every model trained on the column inherits the merge.

Which fields keep a flag, a review and a finding apart?
FieldWhat it recordsExample valuesMistake it prevents
suspicion_signalWhy the return was flaggedserial_mismatch, weight_short, high_return_frequency, receipt_not_foundTreating the reason for a look as the result of the look
signal_sourceWho or what raised the flagrules_engine_v12, vendor_score, store_associate, warehouse_inspectorTraining a model to imitate an old rule set
investigation_stateWhether anyone reviewed the casenot_reviewed, in_review, review_closedCounting unreviewed flags as negatives or positives
evidence_typeWhat the reviewer actually examinedinspection_photo, serial_scan, carrier_weight, fulfillment_record, cctv_noteFindings with no traceable basis
findingThe reviewed conclusionconfirmed_mismatch, policy_breach, cleared_explained, inconclusiveCollapsing abuse, error and fraud into one class
finding_dateWhen the conclusion was recorded2026-03-14Leaking later outcomes into earlier training windows
decided_by_roleRole of the person who decidedreturns_lead, loss_prevention_analystMixing associate guesses with specialist findings
policy_versionReturn policy in force on the purchase datereturns_policy_2025_11Judging a return against a rule that did not apply
customer_outcomeWhat happened to the customerrefunded, store_credit, return_declined, escalatedConfusing the business response with the finding

Which order, stock or promotion fact can flip the reading of a return?

A return is rarely ambiguous on its own; it becomes ambiguous when the linked records are missing. Before anyone writes a finding, pull the order line, the fulfillment history, the stock movements and the promotion that applied at purchase. Each of these can turn an apparent mismatch into an ordinary event.

The table lists the linked facts that most often change the valid reading, with the record that settles each one. A labeling guide should require a reviewer to check every row that applies before choosing any finding above inconclusive.

Which order, stock or promotion fact can flip the reading of a return?
Linked factHow it can change the readingRecord to pull
Replacement or split shipmentA different serial may be the one the customer actually receivedFulfillment and replacement order history
Inventory receipt timingAn item returned before the matching stock was received cannot be the unit soldWarehouse receipt log with timestamps
Bundle or free-gift promotionReturning the paid item but keeping the gift may be allowed or may require a price adjustmentPromotion rules version on the purchase date
Manager approval or overrideAn out-of-window return may have been approved in advanceApproval record with approver role and time
Carrier scan and weightA short weight can come from carrier damage or a packing errorCarrier tracking events and dock weight
Account merge or address changeApparent serial returning may be one household with merged accountsCustomer master change history
Policy change between purchase and returnA return valid under the old window may look late under the new onePolicy version table keyed to dates

Which uncertainty level and escalation route does each flagged return get?

Assign every flagged return one level, using the minimum evidence in the third column; when the evidence for a higher level is missing, the case stays at the lower one. The levels describe how much is known, not how bad the case looks.

The rubric deliberately separates a policy breach from deception. A worn item returned after the window breaks a rule, but the record rarely shows intent, so it belongs in its own class rather than in the fraud positives. Treat this rubric as a draft for your own retail operations and rights reviewers to adapt, not as a finished standard.

  • Escalate to a second reviewer when the signal and the linked records point in different directions.
  • Escalate to the policy owner when the policy version for the purchase date cannot be found.
  • Escalate to loss prevention when a mismatch is confirmed but the explanation table has an unchecked row.
  • Escalate to counsel when a case involves a dispute, a claim against the customer or an outside agency.
Which uncertainty level and escalation route does each flagged return get?
LevelMeaningMinimum evidenceLabel to writeNext step
L0No signal raisedRoutine return recordnot_flagged (not proof of honesty)None
L1Signal only, nobody reviewed itA rule hit, score or associate noteflagged_unreviewed, excluded from fraud positives and negativesSample for review if the class is needed
L2Reviewed, evidence conflicts or is missingReviewer notes naming what was checkedinconclusive with a reason codeSecond reviewer or more evidence
L3Reviewed, an ordinary explanation foundA linked record that explains the signalcleared_explained with evidence_typeClose
L4Policy breach, intent not shownPolicy version plus the breached conditionpolicy_breach, kept separate from fraudPolicy owner decides on exceptions
L5Physical or system mismatch confirmedInspection photo, serial scan or weight record that rules out the listed explanationsconfirmed_mismatch with evidenceLoss prevention review before any person-level conclusion
L6Formal finding recordedClosed loss-prevention case with documented finding by an authorized roleconfirmed_finding with case referenceRestricted handling and counsel review before reuse

What should happen when two reviewers read the same return differently?

Record both readings and the agreement rate, then route the case to a third reviewer instead of averaging it away. In ambiguous returns, disagreement is information: it marks the cases where the evidence really is thin, and those are the cases a model most needs to handle with care.

Measure agreement on a sample of double-reviewed cases with Cohen's kappa. The scikit-learn documentation describes kappa as intended to compare labelings by different human annotators and notes that scores above 0.8 are generally considered good agreement. A low score on L2 versus L3 usually means the reason codes are unclear, not that reviewers are careless.

Keep the per-reviewer labels in the delivered file. Research on subjective annotation found that modeling each annotator's judgments, rather than a majority vote, matched or beat aggregated labels and gave uncertainty estimates that tracked disagreement better. A buyer can always collapse the labels later; nobody can rebuild them once they are merged.

Disagreement also matters downstream. If a model will output a probability that a return is fraudulent, that probability is only useful when it is calibrated: scikit-learn's guide explains that among cases scored near 0.8, roughly 80 percent should actually be positive. An evaluation set that keeps inconclusive and cleared cases lets a buyer check that property; one that drops them cannot.

Illustrative worked example: a serial mismatch with two explanations

This example is invented to show the method; it describes no real retailer, customer or result. A consumer electronics store receives a returned headset whose serial number does not match the serial recorded on the order line. The rules engine flags it as serial_mismatch.

The reviewer pulls the linked records before choosing a level. The fulfillment history shows that the first headset was reported damaged on delivery and a replacement unit shipped two days later. The replacement's serial matches the returned unit. The order line still carries the original serial because the replacement was created as a zero-value order. The case is labeled L3, cleared_explained, with evidence_type fulfillment_record.

Now the exception. A second flagged headset in the same batch matches neither the original nor any replacement serial, and the warehouse receipt log shows that serial was never stocked by the store. The reviewer records L5, confirmed_mismatch, with the serial scan as evidence. That is still not a finding that the customer committed fraud: a carrier or warehouse swap remains possible until loss prevention closes a case. The dataset row says exactly that much and no more.

The difference between the two rows is one linked record. A dataset that kept only the flag would have labeled both as fraud, and a model trained on it would learn that replacement orders look suspicious.

Which return cases belong with loss prevention or counsel instead of the label queue?

Pull a case out of the general label queue when the label could be used to make a decision about a specific person. Return-review records can feed customer-level decisions, and the U.S. Consumer Financial Protection Bureau's list of consumer reporting companies includes a retail entry for a company that monitors and reports retail return and exchange fraud and abuse to merchants. Whether a given dataset or model raises consumer-reporting obligations is a question for counsel on the specific facts.

The cases below should go to a specialist route rather than an annotator:

  • Open disputes, chargebacks or claims against the customer, until they close and counsel approves reuse.
  • Cases referred to an outside agency or involving employee conduct.
  • Records where the only evidence is an associate's description of a person rather than of the item or transaction.
  • Fields that reveal or proxy for protected characteristics, which should be removed rather than labeled.
  • Reviewer and associate names, which should be replaced with role codes before any export.

What makes return-review records worth licensing?

If you are considering licensing returns records, the rubric above is also what makes them worth reviewing: separated signal, review and finding fields, linked fulfillment and policy records, and labels that admit uncertainty. Retailers that want to license such records can do it through SourceX, which qualifies the company and then manages rights review, contracting, preparation and delivery.

On size, the usual buyer expectation is two-plus years of consistent history and tens of thousands of records or more from one system; rarer work can get by with less, and each set is weighed deal by deal. Customer details are de-identified by SourceX through approved processing routes, nothing moves before the retailer approves the scope and terms in a signed agreement, and eligibility is checked through a short questionnaire that needs no return records.

Frequently asked questions

Is a declined return the same as a fraud label?

No. A declined return records the business response, which can follow from a late return, a missing receipt or a policy rule. Fraud requires a reviewed finding with evidence, so keep the decline in customer_outcome and the conclusion in a separate finding field.

Should flagged returns that nobody reviewed be used as legitimate examples?

No. An unreviewed flag is neither cleared nor confirmed, so it belongs in an unlabeled pool. A survey of positive-unlabeled learning notes that methods treating unlabeled examples as negatives are working with label noise, and a returns dataset can avoid that by marking these rows flagged_unreviewed.

What is the difference between return abuse and return fraud in a dataset?

Abuse usually means a policy breach, such as wearing an item and returning it late, where intent is not shown in the record. Fraud implies deception confirmed by evidence, such as a swapped or empty package. Label them as separate classes so a model does not learn to treat every late return as dishonest.

How many reviewers does an ambiguous return need?

One reviewer is enough for clear L0, L3 and L4 cases. Inconclusive cases and any case heading to L5 or L6 should get a second reviewer, and a sample of double-reviewed cases should be scored for agreement so unclear reason codes can be fixed.

Can labels built this way be used to refuse a customer's return?

That depends on how the labels and any model are used and on the law that applies, which counsel should review. The rubric is designed for training and evaluating models on well-documented cases; using outputs to make decisions about individual customers raises separate legal and fairness questions.

Sources

  • Shopify's Admin API gives a return one of five statuses: requested, open, declined, closed or canceled. Source
  • Stripe treats an early fraud warning as actionable only if it has not received a dispute and has not been fully refunded. Source
  • Positive-unlabeled learning methods that treat unlabeled examples as negatives are handling label noise. Source
  • Modeling individual annotators matched or beat majority-vote labels and gave uncertainty estimates that correlate better with disagreement. Source
  • Cohen's kappa compares labelings by different human annotators, and scores above 0.8 are generally considered good agreement. Source
  • For a well-calibrated classifier, about 80 percent of samples given a probability near 0.8 belong to the positive class. Source
  • The CFPB's consumer reporting companies list includes a retail company that monitors and reports retail return and exchange fraud and abuse to merchants. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies