Short answer
Success labels record whether a demonstrated task met a defined goal, and ideally why it did not. They feed success detectors, reward models and replanning loops. A usable specification defines the success condition per task in observable terms, separates outcome from process quality, labels partial success and failure causes, and states who judged each clip and from which camera view.
The buyer's problem
A video of a person restocking a shelf does not say whether the shelf ended up correct. Somebody has to decide that, using a rule. Without a rule, one annotator marks "success" when the item is on the shelf, another when it is front-facing and aligned, and a third when the manager did not complain. The resulting labels are noisy in a way no model can repair.
Outcome labels matter in three places. Success detectors tell a robot whether to continue or retry; published planners use success detection as one source of closed-loop feedback (Huang et al., 2022). Reward and progress models need a terminal signal to anchor them. Evaluation sets need unambiguous pass/fail ground truth. This page is about the labels, not the failures themselves; for collecting failed attempts see failure and recovery demonstrations.
What outcome labels can and cannot support
They support training and testing classifiers that judge "done / not done" from video, generating reward targets from sparse outcomes, filtering demonstrations to keep only successful ones, and measuring evaluation results.
They do not give the robot an action to take. They cannot establish success for things that are invisible in the footage, such as torque applied to a fastener or a sealed container's internal pressure. They also reflect the camera view they were judged from: a label made from an overhead camera may differ from what a wrist camera on a robot would show. Footage of experienced workers is mostly successful, so a detector trained on it alone lacks negative examples; the counterexamples are discussed under failures.
Published work on failure explanation
Failure-aware work shows what outcome and cause labels are used for. AHA is an open-source vision-language model that detects and explains manipulation failures in natural language; its FailGen module perturbs successful simulated demonstrations to generate failure trajectories, and the authors report generalization to real-world failure datasets (Duan et al., 2024). REFLECT builds hierarchical summaries of multisensory robot data (RGB-D, audio, robot states) for an LLM to explain failures and plan corrections, and introduces the RoboFail dataset of failed executions in simulation and the real world (Liu et al., 2023). Both rely on cause descriptions, not bare binary flags, and both use robot data; labels on human video are an adjacent use of the same idea.
Illustrative outcome schema
# Illustrative only - fields and values are examples a buyer would set
clip_id: clip_000412
task_id: restock_shelf_v0
rubric_version: 1.2
success_condition: >
All 6 target items on the shelf, front-facing, within 20 mm of the shelf
edge, no items on the floor, observed in final 2 s of the best camera view
outcome:
binary: success | failure | undecidable
graded: 0 | 0.25 | 0.5 | 0.75 | 1.0 # fraction of sub-goals met
sub_goals:
- {id: sg1, name: items_on_shelf, met: true}
- {id: sg2, name: front_facing, met: false}
failure_cause: [misplaced_item, dropped_item, wrong_item, incomplete, timeout]
recovered: true
time_to_outcome_s: 41.5
judgement:
judged_from: cam_overhead
annotator_id: ann_07
second_annotator_agree: true
outcome_source: visual | system_log | supervisor_report
Fields to adapt: the graded scale, the cause list and the physical tolerances. Pick tolerances your sensors can actually measure.
Specification guidance
Define success per task, in observables. Write the condition so two people can apply it from the video alone. "Tidy" fails this test; "no items on the floor and every label facing the aisle" passes.
Separate outcome from quality. A task can succeed slowly, clumsily or unsafely. Keep outcome, efficiency and safety_flag as separate fields so they do not contaminate one another.
Keep an "undecidable" option. When the end state is out of frame, forcing a label adds noise. Track the undecidable rate by camera and site; a high rate points to a capture problem.
Label partial success. A graded score or sub-goal list gives reward models more than a binary flag, and it supports step-level analysis alongside step segmentation.
Record failure causes with a closed list. Free text is useful for review but not for training; use both.
Use system records as a cross-check. Where a business system confirmed completion (a scan, a work-order close), it can serve as independent evidence; see aligning demonstration video with workplace system logs. Treat it as a check, since a record can be wrong or late.
Acceptance criteria
- Inter-annotator agreement. Double-label a sample and report agreement on the binary and graded labels per task; set the pass threshold yourself.
- Gold set. Include clips with known outcomes decided in advance and report accuracy against them.
- Class balance. Report the success rate per task and site; a set that is 98% success cannot train a failure detector.
- Undecidable rate. Report it per camera view.
- Rubric traceability. Every label links to a rubric version, and relabeling after a rubric change is tracked.
- Evaluation separation. Labels for evaluation clips come from a different annotation pass than training labels, and the clips are held out by site or operator.
Process checks that apply to all temporal labels are in measuring annotation quality in task video. Outcome labels also feed human video for reward models.
Using outcome labels downstream
Outcome labels are consumed in different ways, and the way affects the specification. A success classifier uses the terminal frames and needs the end state visible from the camera the robot will have. A reward model uses graded progress across the clip and benefits from sub-goal timestamps. A filter for imitation learning only needs a binary flag but needs it to be reliable, since mislabeled failures enter the training set as demonstrations. An evaluation set needs the strictest version: labels fixed before models are run, no revision after seeing results. State which of these you are buying for. If you plan more than one use, ask for the richest label set and derive the simpler ones, rather than the other way round.
Questions to ask suppliers
- Who defines success: the buyer, the business that recorded the work or the annotator?
- Is the outcome judged from video only, or confirmed by a system record or supervisor?
- How are partial successes, retries and recoveries counted?
- What fraction of clips is double-labeled, and what was the disagreement?
- Which tasks have a meaningful number of failures, and how were they identified?
- How do you handle clips where the end state is not visible?
- Can the rubric be versioned and the labels regenerated if it changes?
- Does the footage include failures that the business removed or edited out before supply?
- Were workers told their outcomes would be labeled, and could that have changed how they worked?
Discuss your requirements
If you need outcome labels for evaluation or training, describe the tasks, success definitions and intended use to SourceX. SourceX reviews the requirement and follows up about potential sources; annotation can be arranged case by case, and a request does not guarantee a match. Start from the humanoid training data guide or the workplace task demonstration hub. For robot-side failure data, see robot manipulation failure datasets.
Tell SourceX what your evaluation set needs to cover
Sources
- Inner Monologue: Embodied Reasoning through Planning with Language Models — success detection as closed-loop planner feedback.
- AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation — failure detection and explanation; generated failure trajectories.
- REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction — failure explanation from multisensory data; RoboFail.