Short answer
In-collection quality control finds unusable recordings while re-recording is still cheap: a camera check before each session, automated tests on every upload, a human review of a sample within a day, and written re-record criteria. Defects are mostly capture faults such as obstructed lenses, drifted clocks and missing metadata, so the checks target them directly and results go into a quality log.
The buyer's problem
Discovering a problem after delivery is expensive. If a head-mounted camera was tilted too low for two weeks, the hands are cropped out of most clips, and the demonstrators, site access and kit have moved on. Re-recording then means a new schedule, a new notice to the workforce and a new bill. In-flight checks move detection from months to hours.
This page is about quality control during capture. Acceptance after delivery is a separate step covered by demonstration video acceptance criteria; the two should use compatible thresholds so nothing that passes in the field fails at acceptance. Multi-location logistics are in running collection across multiple sites.
What goes wrong, and when you can see it
| Defect | Cause | Visible in | Best check |
|---|---|---|---|
| Hands out of frame | Camera tilt, mount slipped | First minute of a clip | Pre-session framing check; automated hand-in-frame rate |
| Lens obstructed or dirty | Smudge, dust, hair, strap | Any frame | Automated blur/occlusion score |
| Motion blur, low light | Exposure, lighting change | Frame statistics | Automated sharpness and brightness |
| Dropped frames, variable rate | Storage or CPU limits | File metadata | Frame-count vs duration test |
| Clock drift or wrong time | Device clock unset | Timestamps | Sync event at start and end |
| Wrong task or protocol deviation | Misunderstood script | Content | Human sample review |
| Missing audio/consent flags | Setup error | Metadata | Metadata validator |
| Corrupt or partial file | Battery, card, transfer | File | Decode test, checksum |
| Calibration missing | Skipped step | Metadata | Required-field check |
| Bystanders or screens in frame | Layout | Content | Automated flag plus human review |
Most items are capture faults that checks can catch within a day. Content-level problems, such as task variants nobody recorded, show up only in coverage reports; see demonstration coverage plan.
Published collection practices
Large programs store the information needed to judge each episode along with it. DROID stores camera calibration, depth and a language instruction with each episode, and replicated one rig at 13 institutions with 50 collectors (Khazatsky et al., 2024). That per-episode metadata is what makes automated QC possible: you can check calibration age or instruction presence without watching the clip. Ego4D reports consenting participants, de-identification procedures and more than 250,000 hours of annotator effort for 3,670 hours of video (Grauman et al., 2021); effort of that size suggests why catching problems before annotation can save money.
Three layers of control
Before each session (on-site). A short checklist the collector completes and records:
- Lens clean, battery above a set level, storage free.
- Clock check against a reference, or sync event recorded.
- Calibration run if the camera moved.
- A 10-second test clip reviewed on the spot: hands visible, work area framed, exposure acceptable.
- Consent and notice confirmed for people in the area.
After each upload (automated). Run on every file within hours:
- Decode test, frame count against duration, constant frame rate.
- Resolution, codec and container as specified.
- Sharpness, brightness and occlusion scores against thresholds.
- Hand-in-frame or person-in-frame rate using a detector, where relevant.
- Required metadata present and valid.
- Duplicate and zero-motion detection.
Daily and weekly (human). A reviewer samples clips per collector and per site, scoring against the protocol. Weekly, the coordinator reviews rates and trends and decides whether to retrain a collector or adjust the kit.
What QC can and cannot do
QC reduces the share of unusable clips and keeps sites comparable. It cannot establish that the footage is valuable for a particular model, nor that coverage is adequate, and automated scores are proxies. A threshold on blur may reject clips of fast legitimate work; a hand detector may fail on gloves. See gloved hands in workplace video. Calibrate every automated check against human judgment on a sample before using it to reject data.
Illustrative re-record criteria
# Illustrative only - thresholds are examples a buyer would set and tune on a pilot
re_record_if:
- hands_visible_fraction < 0.70 over the clip # head-mounted tasks
- frame_drop_rate > 0.5%
- variable_frame_rate == true
- clock_offset_s > 2.0 at session end
- blur_score_below_threshold_frames > 10%
- lens_occlusion_detected
- task_not_matching_protocol (human review)
- missing_required_metadata
- bystander_in_frame_without_permission
repair_instead_of_rerecord_if:
- metadata_missing_but_recoverable
- single_clip_corrupt_but_session_has_duplicate_camera
review_if:
- rare_event_in_clip # do not auto-reject unusual footage
quality_log_fields: [clip_id, site_id, collector_id, check_name, value, pass, reviewer, action_taken, date]
Never auto-reject on unusual content. Failures, interruptions and odd objects are what a dataset often lacks; route them to review, not deletion. See rare events in demonstrations.
Quality and acceptance criteria for the QC process itself
- A written QC plan with checks, thresholds and owners exists before recording starts.
- A quality log is kept and delivered, including rejected-clip counts and reasons per site and per collector.
- A pilot calibrates thresholds; see pilot collection scoping.
- The feedback loop for flagged issues is defined, so a collector hears about a problem before the next session, not after the program ends.
- Re-recorded takes link to the originals.
- Automated check accuracy is verified against human review on a sample.
Tuning thresholds on a pilot
Set thresholds from evidence. Record a pilot, have people label which clips they would accept, then compute each automated score on the accepted and rejected sets and place thresholds where they separate them. Revisit after the first week of production, when real conditions appear: seasonal lighting, a new shift, a busier floor. Track the rejection rate per collector and per site. A sudden rise usually indicates an equipment or procedure change; a rate near zero may mean a check is not working. Keep the rejected clips for a short time before deletion, in case a threshold proves too strict, and make sure retention is consistent with the permissions granted.
Questions to ask suppliers
- Which checks run before, during and after each session, and which are automated?
- Who reviews samples, and how soon after capture?
- What are the re-record criteria, and who decides?
- How are rejected clips handled: deleted, kept or logged?
- Can I see a sample quality log from a previous program?
- How do you check collectors' cameras across sites?
- How do you avoid rejecting rare but valuable events?
- How does QC change when recording on a live site with restricted access?
Discuss your requirements
If you are scoping a new recording program, describe the tasks, quality thresholds and intended use to SourceX. The site describes a separate physical-world path for new recordings of hands-on work coordinated with businesses, where each opportunity has its own requirements. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match, and annotation or specialist capture partners can be arranged case by case. See the humanoid training data guide and the workplace task demonstration hub for the surrounding decisions, and recording protocol on live work sites for on-site rules. Label checks after capture are in annotation quality for task video.
Describe the workplace demonstration data your robotics project needs
Sources
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — per-episode calibration and metadata in a distributed collection.
- Ego4D: Around the World in 3,000 Hours of Egocentric Video — consent, de-identification and annotation effort at scale.