Short answer
Remote-assistance session recordings capture a field technician's camera view together with a remote expert's speech, on-screen markup and chat during a live repair. That pairs first-person video with step-by-step instruction, which suits guidance-following, step understanding and error-detection research. Framing is uneven, the technician's hands may be out of view, and the recordings involve customer premises and two-party conversations, so rights review comes first.
What a session recording contains
Remote-support tools stream a technician's camera, from a phone, smart glasses or a body-worn camera, to an expert who talks, points, draws on the shared view, sends documents and chats. What a given platform records, and for how long, varies, so treat the table as a checklist of questions, not a promise.
| Component | Typical content | What it supports | Common problem |
|---|---|---|---|
| Technician video | First-person or handheld view of the equipment | Step and object recognition in context | Framing follows the conversation, not the task |
| Two-way audio | Technician and expert speech | Instruction language, questions, corrections | Noise, PPE, customers audible |
| Expert markup | Arrows or circles drawn on the live view | Linking words like "that valve" to a region | Often stored as overlays, or not stored at all |
| Chat and shared files | Text, photos, manual pages sent | Procedures and parts cited | May live in a separate system |
| Session metadata | Case ID, asset, timestamps, participants, outcome | Joins to work orders; filtering | Outcome often free text |
Why this is its own asset
Much demonstration footage has at most one voice. A remote-assistance session has two, and the second is reacting to what it sees: asking for a different angle, correcting a wrong terminal, confirming a step. That makes the recording an interactive guidance dialogue grounded in a live scene. Instruction-paired demonstration datasets and narrated demonstrations cover instruction attached to a performer's own work; here the instruction comes from a different person in real time. Technician body-camera footage has the view but not the expert, and repair video demonstrations cover the repair as a whole.
What public work shows
Two public references show the shape of the data. HoloAssist is titled as an egocentric human interaction dataset for interactive AI assistants in the real world (HoloAssist, 2023); read the paper for its capture setup and scale before using it as a benchmark for what a business archive should contain.
TEACh is a closer model of the dialogue structure. It contains more than 3,000 human-human dialogues in which a Commander guides a Follower through household tasks in simulation, the Follower can ask questions, and three benchmarks cover dialogue understanding, language grounding and task execution (Padmakumar et al., 2021). It is simulated and domestic; real equipment, real faults and real expert speech are what business recordings add.
The robot-side counterpart is deciding when to ask. KnowNo aligns a language-model planner's uncertainty so that a robot asks for help when needed (Ren et al., 2023). Session recordings show how human technicians ask for and receive help, the human half of that problem.
What the data can support
- Grounding language in equipment scenes. Utterances plus markup link instructions to image regions.
- Step recognition with expert-supplied names. Experts say what step comes next, which can reduce annotation effort.
- Correction and error cues. Expert interventions mark moments where a correction was needed; see human procedural mistake videos.
- Diagnostic dialogue. How an expert narrows a fault through questions, the spoken form of diagnostic reasoning traces.
- Assistant evaluation. Held-out sessions test whether a model's guidance matches what experts gave.
What it cannot support
- Robot actions, joint states or forces. The technician's hands generate none of these.
- Reliable hand views. A technician holding a phone has one hand free, and the camera tends to point where the expert asked to look, not at the hands. Glasses and body cameras help but follow the head.
- Complete tasks. Sessions usually end when the problem is solved or escalated, so repairs can continue off-recording, and many sessions are diagnosis only.
- Authoritative procedure. Expert guidance is unscripted and sometimes wrong; it is not an SOP, and it may differ from the written method, as discussed in procedural drift.
- A humanoid's viewpoint. The camera is a handheld or head-worn device; mapping it to a robot's sensors is a separate step, covered in matching camera viewpoint to humanoid sensors.
- Unambiguous language. "That one on the left" has no meaning without the frame and markup.
Illustrative session package
# Illustrative only: synthetic ids and events, not an available dataset
session_id: ra_000731
captured: [technician_video, two_way_audio, expert_markup_overlay, chat_log]
participants: {technician: tech_B19, expert: expert_E07} # pseudonymous
equipment: {class: commercial_refrigeration_unit, asset_id: AST_0042, site_id: site_041}
work_order_ref: WO-000-118734 # joins to symptom, parts, fix
timeline:
- {t: "00:00:12", event: technician_describes_symptom}
- {t: "00:01:40", event: expert_requests_view, target: condenser_fan_motor}
- {t: "00:01:58", event: expert_markup, shape: circle, frame_ref: f_2958}
- {t: "00:04:20", event: expert_correction, note: "wrong terminal"}
outcome: {resolved_on_session: false, escalated: true, revisit_within_30d: true}
annotations: diarized_transcript, step_segments, referent_links, intervention_type
excluded: customer names and speech, addresses, screens showing customer data
rights: training and internal evaluation; customer-contract status recorded per session
Buyers typically need annotation layers beyond what the platform stores: a speaker-labeled transcript, step segments, links from utterances to image regions, and an intervention type for each expert turn (instruction, question, clarification, correction or confirmation). SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case.
Who holds these recordings
Field service organizations that adopted remote-support tools are the primary route, followed by dealer and manufacturer service desks, and plant maintenance teams that call vendors for help. Each has a different rights position. Equipment makers and platform vendors may hold or control stored sessions, and a service firm may not be able to license what the platform stores on its behalf. Ask who actually controls the files before discussing volume. Related record types include work order histories, which can supply the outcome for a session.
Quality and acceptance criteria
- Audio. Intelligibility is sampled and transcript accuracy is reported on a double-checked sample.
- Work-area visibility. The share of session time in which hands, tool and part are in frame is reported.
- Joins. The share of sessions linked to a work order and a known outcome is stated.
- Coverage. Counts of distinct technicians, experts, sites, equipment classes and fault types; one expert's habits should not dominate.
- Markup. Overlays are stored with timestamps, or the supplier states they were not kept.
- Synchronization. Audio, video and markup share a time base, with the offset measured on a sample.
- Redaction effects. Any bleeped audio or blurred regions are logged, with their effect on instructions noted.
- Unresolved sessions. Escalations and abandoned sessions stay in the set.
- Splits. Held-out sessions are separated by technician, expert, site and equipment model, following training vs evaluation datasets.
Questions to ask suppliers
- Which platform produced the recordings, and who controls the stored sessions: you or the vendor?
- What is recorded by default (video, audio, markup, chat, files), and how long is each kept?
- What notice did technicians, experts and customers receive, and was every party on a call told it was recorded?
- Do customer contracts permit using recordings, or data derived from them, for AI training?
- Can sessions be joined to work orders and outcomes, and at what rate?
- How many distinct technicians, experts and sites appear, and in which equipment classes?
- How often are the technician's hands and the part in view?
- Are escalated and unresolved sessions included?
- Are experts your employees, contractors or equipment-maker staff, and does that affect who can license their speech?
Permissions and provenance
A session is a conversation. Some US states require every party's consent to record a confidential communication; California's Penal Code 632(a), for example, makes intentionally recording one without the consent of all parties an offense (California Legislature). State rules differ, so ask what notice was given on each call and how it was documented. This is not legal advice.
The picture also shows customers' premises, equipment serial numbers, addresses and sometimes on-screen records. Customer contracts may limit reuse, and expert speech may belong to a manufacturer's employee. Worker consent and identifiable people and confidential business information in footage set out the questions. Blurring faces or bleeping audio reduces exposure but does not settle whether a recording may be licensed.
Discuss your requirements
If you need guidance-paired repair video, describe the equipment types, session elements and permitted uses to SourceX. SourceX investigates whether US businesses hold suitable recordings, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. It reviews the requirement and follows up about potential sources; a request does not guarantee a match, and nothing is contracted until a supplier agrees.
See the maintenance, repair and tool-use hub and the humanoid training data guide for neighboring assets, and the catalog pages on video recordings and field service and maintenance work orders for related record types.
Discuss your humanoid training-data requirements with SourceX
Sources
- HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World — existence and general nature of a public assistant-interaction dataset.
- TEACh: Task-driven Embodied Agents that Chat — human-human guidance dialogues in simulated tasks.
- Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners — robots asking for help when uncertain.
- California Penal Code Section 632 — consent of all parties for recording confidential communications.