Short answer
Audio in workplace demonstrations can carry contact sounds, machine state cues and spoken narration that video misses, and some robot datasets record it alongside vision and force. It also captures bystander speech, which raises separate consent questions. Decide per use whether you need contact sound, machine cues or narration, and record only those with a stated plan and permission.
The buyer's problem
Much workplace video is delivered with audio stripped or was never recorded with sound. That is often sensible, since microphones pick up conversations nobody agreed to record. But some of what happens in manual work is audible before it is visible: a snap-fit seating, a drill changing pitch under load, a mixer slowing as dough thickens, a cart wheel catching. A buyer designing a perception stack for contact-rich tasks needs to decide whether sound is a signal worth the added risk.
This page sorts audio into three uses with different value and different consent profiles, so you can specify one without taking the others.
Three kinds of workplace audio
| Kind | Examples | Possible use | Privacy exposure |
|---|---|---|---|
| Contact and process sounds | Click of a connector, scrape, impact, liquid pouring | Event detection, success cues, anomaly spotting | Low if separated from speech |
| Machine and alarm cues | Tool pitch, conveyor start, beeps | State recognition, timing labels | Low to moderate |
| Speech | Demonstrator narration, coworker talk, instructions | Language grounding, step labels | High; includes bystanders |
Contact sound and machine cues are properties of the task. Speech is a property of the people present, and it needs its own plan.
What published work shows
RH20T is a robot dataset of more than 110,000 contact-rich real-world manipulation sequences, with visual, force, audio and action data per sequence, and each robot sequence paired with a human demonstration video and a language description (Fang et al., 2023). It shows audio recorded alongside force and action in a robot setup, so the audio channel is tied to the physical contact events that produced it.
Narration appears in the other direction too. EPIC-KITCHENS-100 used "pause-and-talk" narration as its first annotation stage, followed by transcription, parsing and temporal segmentation (Damen et al., 2020). There, speech was an annotation process rather than a training input. The distinction matters when you choose whether speech stays in the delivered product. See narrated demonstrations for how that choice affects the data.
Neither study establishes how much audio helps a particular model. Treat audio as a hypothesis to test in a pilot.
What audio cannot support
- Robot actions or joint states. Sound is an observation, not a command.
- Reliable force estimates. Sound correlates with some contact events but is not a force sensor; instrumented rigs measure force directly.
- Clean signals on a live floor. Plants and kitchens are loud. Ambient noise, hearing protection and enclosed machines can bury cues.
- Cross-site comparability. Room acoustics and microphone placement vary widely.
- A shortcut for labels. Spoken instructions are often partial and informal; see language annotations for demonstrations.
Capture choices
The recording design should follow from which kind of audio you want.
- Microphone type and placement. A head-mounted microphone hears the wearer's speech strongly and contact sounds weakly; a close-contact or tool-mounted microphone does the reverse. Wrist-worn and tabletop microphones fall between.
- Separate tracks. Record narration on a separate channel from environmental sound where possible, so that speech can be dropped without losing contact sound.
- Sample rate and dynamic range. Set according to the sound band you care about, and avoid automatic gain that flattens impacts.
- Sync. Align audio to video with a shared clock or a clap marker at session start.
- Protocol. Agree on a quiet-zone rule for breaks and personal conversation, and a stop signal.
Head-mounted capture specifications can include an audio block, and recording protocols on live sites cover how to run quiet zones on a working floor.
Consent and law
Audio is regulated differently from silent video in many places. Some US states require the consent of all parties to record a confidential communication; California's Penal Code section 632(a), for example, makes intentionally recording a confidential communication without the consent of all parties an offense (California Legislature). That is one state's rule, and others differ.
Diligence questions for you and the supplier: Who will be present within microphone range? How will each person agree, including visitors, customers and coworkers who are not demonstrators? Is speech retained, transcribed, or removed before delivery? Does the permission cover model training and redistribution? Removing audio later does not erase the earlier recording event. This is not legal advice; ask counsel for each recording location. See also worker consent and identifiable people.
Illustrative audio channel specification
# Illustrative only - synthetic values, set by the buyer
audio:
purpose: [contact_sounds, machine_cues] # speech excluded unless consented
tracks:
- {id: aud_env, mic: "wrist-worn", sample_rate_khz: "buyer sets", gain: "fixed, no AGC"}
- {id: aud_voice, mic: "head-mounted", status: "disabled"}
sync: "shared clock with video; clap marker at session start"
quiet_zone_rule: "mic off in breaks and designated areas"
delivered: [wav_tracks, sync_log, mic_placement_photos]
speech_policy: "none recorded; if recorded, separate track and separate consent"
qa: "reject clips with clipping or speech from non-participants"
metadata: [site_id, ambient_noise_note, ppe_note, session_id]
Where audio is most and least useful
Audio tends to help most where sound is tied to a physical event: seating a connector, sealing a container, pouring, cutting, or a tool changing pitch under load. It helps least in open, noisy spaces where the target sound is masked, and in tasks that are silent by nature. Kitchens illustrate both cases. Sizzle and timer beeps carry state information, but extraction fans and crowd noise bury quieter cues; see commercial kitchen food prep for the setting.
A cheap test is to ask annotators to label the same clips with and without sound. Where labels change, audio carries information. Where they do not, the channel adds privacy exposure and storage for little gain.
Quality checks
Sample twenty clips and listen. Confirm that the target sound is audible against noise, that clipping and gain pumping are absent, that track labels match the spec, and that no speech appears on a track where it should not. Compare audio events with video events by scrubbing to a few known impacts. Disagreement of more than a frame or two signals a sync problem.
Questions to ask suppliers
- Which sound types will be recorded, on which microphones, in which positions?
- Is speech excluded by design, recorded on a separate track, or removed before delivery?
- How will people within microphone range, including non-demonstrators, be informed and give permission?
- How do state recording laws affect each recording location, and who reviewed them? This is not legal advice.
- What noise levels are typical, and what is the plan when hearing protection or machinery masks the target sound?
- How are audio and video clocks synchronized, and how is offset reported?
- Is audio normalized, filtered or compressed, and are raw files retained?
- Can the supplier delete audio on request, and what happens to derived transcripts?
- Can I receive a short pilot sample to test whether audio improves my model before ordering volume?
Describe your needs
If you want to test whether sound adds value for your tasks, describe the tasks, sound types and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources, and can scope a commissioned collection of new recordings if no existing asset fits; a request does not guarantee a match. For wider navigation, see the workplace task demonstration hub and the humanoid training data guide. The catalog entry on voice and speech data covers a separate speech use case.
Describe the workplace demonstration data your robotics project needs
Sources
- RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot — robot sequences with audio, force, vision and paired human video.
- California Penal Code Section 632 (eavesdropping on or recording confidential communications) — example of an all-party-consent recording rule.
- Rescaling Egocentric Vision — pause-and-talk narration in EPIC-KITCHENS-100 annotation.