Skip to content

Privacy and preparation

Robotics task boundary ambiguity: where does one manual task end?

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

To find where one manual task ends and the next begins in workplace video, tie each task to a change in a work object: start at the first action committed to that change and end when the new state is visible and checked. Write rules for interruptions, rework, hand-offs and off-camera endings, then have two trade-aware annotators mark the same footage blind and measure boundary offsets, segment overlap and label agreement.

Video and sensors

Robotics task boundary

This page covers robotics task boundary ambiguity: where does one manual task end.Illustrative

Key takeaways

  • Define a manual task by the work object and the state change that ends it, not by the worker's sense of being done.
  • Work order closures and spoken narration only locate a task in a recording; they rarely mark its true edges.
  • Write a rule for each common ambiguity, including interruptions, interleaved jobs, rework, hand-offs, off-camera ends and abandoned work, before annotation starts.
  • Score boundary offsets, segment overlap and label agreement separately, by task type, and report inferred ends on their own.
  • Repeated disagreement after one revision usually means the task dictionary needs a new task type or a narrower scope.

How do you find where one manual task ends and the next begins in workplace video?

Tie every task to a change in the state of a work object. A task starts at the first action committed to that change, such as the hand reaching for the panel that must come off, and ends at the first frame where the new state is visible and checked, such as the restored unit running with the panel closed. Everything between those two frames belongs to the task, including waits and mistakes.

That rule sounds simple, and it fails on real jobs in predictable places: a technician fetches a tool halfway through, two jobs share one trip to the van, a step is redone after a failed test, or the decisive moment happens off camera. The answer is to write a rule for each of those cases before anyone annotates, then have two people mark the same footage independently and measure how far apart their boundaries land.

This guide is for a business that records hands-on work, such as field service, assembly, maintenance or warehouse handling, and wants the footage to be usable as task-level training data. It gives the boundary rules, an agreement protocol a small team can run on a pilot batch, and the evidence each task package needs before anyone outside the company sees it.

Why the work order, the narration and the footage give three different end times

The business system almost never marks the true boundary. A work order is closed when someone taps complete in the field app, which may be minutes after the last bolt is tightened or the next morning. Treat system timestamps as anchors that place a task within a recording, not as boundaries.

Spoken commentary has the same problem. The EPIC-KITCHENS-100 annotation files keep a narration_timestamp, recorded when the participant described the action, separately from the start_timestamp and stop_timestamp of the action segment itself. A spoken note lands near the action, but annotators still have to find the segment edges in the video.

Workers also disagree with each other about what counts as one task. One technician treats testing as part of the repair; another treats it as a separate inspection. Neither is wrong. The boundary has to come from a written definition tied to the object state, not from the individual's sense of being done.

Which kinds of ambiguity show up on real jobs, and what rule settles each?

Most disagreement on manual-work footage falls into a handful of patterns. Decide the rule for each before the pilot, and record which rule was applied on every boundary that needed one.

Which kinds of ambiguity show up on real jobs, and what rule settles each?
AmbiguityWhat it looks like on videoBoundary ruleRecord on the segment
PreparationLaying out tools, reading the job sheet, putting on glovesBelongs to the task only when it is specific to this job; generic setup is its own segmentprep_scope: job-specific or generic
InterruptionWorker stops to answer a call or help a colleague, then resumesThe task stays open; the interruption becomes a nested pause segmentpause start, pause end, reason if visible
InterleavingTwo units serviced in alternation on one visitOne task per work object, even when segments alternate in timeobject_id on every segment
ReworkA test fails and a step is repeatedRework stays inside the original task; mark the failed check and the repeatattempt number and the failing check
Hand-offA second worker finishes what the first startedOne task, two actors; the boundary does not move at the hand-offactor change time and role
Off-camera endFinal check happens outside the field of viewEnd at the last visible evidence and flag the end as inferredend_evidence: visible or inferred
Abandoned taskJob stopped for missing parts and left openEnd at the stop, with outcome set to abandoned, never merged into a later visitoutcome and stop reason

What should the inter-annotator task boundary agreement protocol include?

The protocol below is written for a pilot of a few hours of footage and two or three annotators who know the trade, plus one adjudicator. It produces three things: boundaries you can defend, a measured level of agreement, and a list of rules that need rewriting before the full batch.

What should the inter-annotator task boundary agreement protocol include?
StepWhat happensWhoOutputPass condition
1. Define the task unitWrite one sentence per task type naming the work object and the state change that ends itTrade lead with the annotation leadTask dictionary, version 1Every task type has a start event and an end state
2. Write ambiguity rulesAdopt or adapt the rules for preparation, interruption, interleaving, rework, hand-off, off-camera and abandoned workAnnotation leadBoundary rulebook with one example clip per ruleNo rule depends on what the worker intended
3. Set toleranceAgree how many seconds two boundaries may differ and still count as the same, per task typeTrade lead and data userTolerance tableTolerance is shorter than the shortest real step
4. Select the pilot samplePick recordings across sites, workers, shifts and job difficulty, including at least some failed or reworked jobsAnnotation leadSample list with reasonsNo single worker or site dominates the sample
5. Annotate blindEach annotator marks start, end, task type, outcome and rule used, without seeing the othersTwo or three annotatorsIndependent segment filesEvery boundary has a rule code or none
6. Score agreementCompare boundary offsets, segment overlap and labelsAnalystAgreement report by task type and ambiguity typeThresholds set in step 3 are met or the miss is explained
7. AdjudicateReview each disagreement outside tolerance and record the final boundary and the reasonAdjudicatorAdjudication logEach decision cites a rule or creates a new one
8. Revise and re-testRewrite the rules that caused most disagreements and re-annotate a fresh sliceAnnotation leadRulebook version 2 and a second reportAgreement holds or improves on new footage

How do you score boundary agreement without hiding the hard cases?

Report three numbers separately: how far apart the boundaries are, how well the segments overlap, and whether the annotators agreed on the label and outcome. One blended score hides the case that matters most, where both annotators call it the same task but disagree about when it ended.

For boundary distance, list the offset in seconds between matched start points and between matched end points, and report the median and the worst tenth by task type. Starts and ends behave differently on manual work: starts are usually crisp, while ends drift because checks and tidying blur into the next job.

For segment overlap, the evaluation script in the MS-TCN action segmentation repository is a useful reference. It computes an F1 score at overlap thresholds of 0.1, 0.25 and 0.5, where a predicted segment counts only if its intersection over union with a same-label reference segment reaches the threshold, and an edit score from the Levenshtein distance between the two ordered segment label sequences. Run it with one annotator as reference and the other as prediction, then swap. Text segmentation offers window-based alternatives: NLTK implements WindowDiff, which counts sliding windows where two segmentations disagree on the number of boundaries.

For labels and outcomes, use a chance-corrected coefficient. Krippendorff's alpha works with any number of annotators and tolerates missing ratings, which matters when one annotator marks a segment the other skipped. Report it per task type, because a high pooled value can come from easy tasks drowning out the hard ones.

Worked example (illustrative): one rooftop unit, two annotators, three disagreements

This example is invented to show how the protocol plays out; the durations and counts are illustrative. A body-camera recording of an HVAC technician lasts 52 minutes and covers a filter replacement and a belt inspection on one rooftop unit, with a phone call in between.

Annotator A marks two tasks. Annotator B marks three, splitting the belt inspection into inspection and adjustment. Both agree the filter task starts when the technician unlatches the access panel. They differ by 40 seconds on its end: A ends it when the new filter is seated, B when the panel is closed and latched. The rulebook says the end state is the closed unit, so the adjudicator moves A's end and records rule 'end at restored state'.

The phone call produces the second disagreement. A ends the filter task at the call; B keeps it open. Under the interruption rule the task stays open and the call becomes a pause segment. The third disagreement is real ambiguity: the belt adjustment. The trade lead decides adjustment is a separate task type when it involves loosening the motor mount, because that changes the work object state, and adds the case to rulebook version 2.

The exception worth noting: the final belt tension check happens with the technician's head turned away from the unit, so the end is inferred from audio and the latching sound. The end is flagged as inferred, and pairs that rely on inferred ends are reported separately so they do not inflate agreement.

What operating variation and recovery behavior is missing from the footage?

Agreement on clean jobs proves little. Boundaries are hardest, and footage is most useful, when work goes wrong, so a task package should show what varied and what the worker did about it. Large research collections make the same point: the Ego4D paper describes video from 931 camera wearers in 74 locations across 9 countries, gathered with consenting participants and de-identification where relevant, and Ego-Exo4D's keystep annotations carry start and end times plus a flag for whether a step is essential or optional. A single site's footage will never match that spread, so it has to document what it covers and what it does not.

Before you call a pilot representative, check each task package against the evidence list below.

What operating variation and recovery behavior is missing from the footage?
EvidenceWhy it matters for boundariesMinimum to recordGap that should stop the batch
Consent and rightsFootage of people and client sites cannot be used without the right permissionsConsent record per worker and site permission per locationAny worker or bystander without a recorded basis
Camera metadataFrame rate and clock drift move boundariesDevice, frame rate, resolution, mount positionUnknown frame rate or variable frame rate without a time map
SynchronizationBoundaries from a second camera or a work order must line upOffset between each camera and the job system clockOffset unknown by more than the agreed tolerance
Worker visibilityHands and the work object must be in frame at start and endShare of boundaries with visible versus inferred evidenceMost end boundaries inferred
ConditionsLighting, weather and site layout change what can be seenSite type, indoor or outdoor, lighting noteAll footage from one site and one condition
InterventionHelp from a supervisor or a second worker changes the taskActor changes and reasonInterventions present but unmarked
OutcomeA task without a verified result cannot be judgedCompleted, failed, reworked or abandoned, plus the check usedOutcome missing on more than a handful of tasks

When disagreement means the task definition, not the annotators, is wrong

Low agreement after one round of revision usually points at the task dictionary. If annotators keep splitting the same activity in different places, the activity is probably two task types, or one task with an optional sub-step that needs its own label.

Narrow the scope when a task type depends on information the camera cannot see, such as a torque value, a pressure reading or a customer's sign-off. Either bring the reading into the record from the work order or instrument, with a known clock offset, or drop that task type from the pilot. Do not ask annotators to guess.

Stop and get specialist review in three cases: the footage shows people who have not agreed to be recorded, the work involves regulated or safety-critical steps where a mislabeled end could teach the wrong practice, or the client whose site appears has terms that may restrict recording or reuse. Counsel and a workplace privacy reviewer should look at the specific facts before more footage is annotated.

Getting boundary-annotated job footage ready for a licensing conversation

Footage with written boundary rules, a measured agreement report and complete outcome evidence is far easier for an AI team to evaluate than raw recordings. It also makes it clear what the business is offering: specific task types, from specific sites, with known gaps.

For field service, assembly or maintenance companies, SourceX turns selected footage and job records into licensed datasets for AI, covering sourcing, rights review, contracting, data preparation and delivery. The first step is a short company assessment with nothing uploaded, and no footage leaves the site until the company has approved the scope and terms and signed the agreement.

Frequently asked questions

What is the difference between a task boundary and an action boundary?

An action boundary marks a short movement such as grasping a tool or turning a screw. A task boundary marks a goal-level unit of work, such as replacing a filter, that contains many actions. Task boundaries depend on the state of the work object, so they need rules for waits, rework and hand-offs that action labels rarely cover.

How many annotators does a task boundary pilot need?

Two independent annotators who know the trade are the minimum, with a third person to adjudicate. A third annotator helps when the task dictionary is new, because it shows whether a disagreement is one person's habit or a real ambiguity in the rules.

Can we use work order timestamps as task boundaries?

Use them to locate a task within a recording, not to mark its edges. Field apps record when someone tapped complete, which can be minutes or hours away from the moment the work ended. Record the offset between the job system clock and each camera so the anchor is reliable.

What tolerance should we use for boundary agreement?

Set it per task type and keep it shorter than the shortest step that matters inside the task. A few seconds may suit a quick replacement, while a long inspection can tolerate more. Write the tolerance down before scoring, so it is not tuned to make the numbers look good.

Should failed and reworked jobs be included in the pilot?

Yes. Failures, rework and interventions are where boundary rules break and where the footage is most informative for a model learning recovery behavior. A pilot built only from clean jobs will overstate agreement and miss the rules the full batch needs.

Sources

  • The EPIC-KITCHENS-100 annotation files record a narration_timestamp for when the participant narrated the action, separate from the start_timestamp and stop_timestamp of the action segment. Source
  • The MS-TCN evaluation script scores segmentation with F1 at overlap thresholds of 0.1, 0.25 and 0.5 between same-label segments and an edit score based on the Levenshtein distance between segment label sequences. Source
  • NLTK implements the WindowDiff segmentation metric of Pevzner and Hearst, which compares the number of boundaries two segmentations place in each sliding window. Source
  • Krippendorff's alpha can be computed for any number of observers and for incomplete or missing data. Source
  • Ego4D reports 3,670 hours of daily-life video from 931 camera wearers in 74 locations across 9 countries, collected with consenting participants and de-identification procedures where relevant. Source
  • Ego-Exo4D keystep annotations include start and end timestamps, a keystep label and a flag for whether the step is essential or optional. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies