Short answer
A troubleshooting evaluation set turns closed service cases into test items: symptom and context in, diagnosis or next repair step out, with the technician's confirmed outcome as the answer. Service organizations hold the source cases; annotators can add structure. The set must be held out by site, asset model or time, and it measures reasoning over text, not physical repair ability.
The buyer's problem
A team building a maintenance assistant, or a planner that decides what a robot does next, needs to know whether the model reasons well about faults. Public benchmarks cover technical support in software; few cover equipment repair. Building your own from service records is feasible, but only if the cases are closed, the outcomes are confirmed and the items are kept away from anything the model trained on.
Training data and evaluation data are different assets. The records that help a model learn repair language, such as work order histories, can also supply test items, but each case may serve only one purpose. This page is about the evaluation side: what to source, how to shape items, and where the approach stops. See the maintenance hub and the humanoid data guide for context.
A published design to learn from
TechQA is a technical-support question answering dataset built for domain adaptation. It has 600 training, 310 development and 490 evaluation question-answer pairs, with questions drawn from IBM developer forums, accepted answers linked to IBM Technotes, and a companion corpus of 801,998 Technotes as of 4 April 2019 (Castelli et al., 2019). Three design lessons carry over to repair.
First, the evaluation split is small compared with the corpus, and quality of the answer key matters more than count. Second, answers are tied to a document, which lets a grader check grounding. Third, the questions come from real users in their own words, with all the noise that implies. A repair set should keep that realism while resolving ambiguity in the key.
TechQA is software support, not equipment repair, and says nothing about physical tasks. Use it as a structure, not as evidence about maintenance performance.
Source material: where cases come from
Closed work orders with a complaint, diagnosis, action and parts. The strongest items come from cases with a confirmed fix and no callback; see first-time fix and callback labels.
Technician notes and logbooks. MaintNet gathers logbook data from aviation, automotive and facilities domains with abbreviation lists, termbanks and Python tools for pre-processing and clustering (Akhbardeh et al., 2020). That kind of normalization comes before a case becomes an item.
Fault trees and guides. Documented decision paths supply expected diagnostic order; see troubleshooting guides and fault trees.
Error codes and screens. Cases that start from a fault code allow items with a clear anchor; see error codes and HMI screens.
Remote support transcripts and video. These provide dialogue and, where recorded, visual evidence for text-plus-video items.
Hard cases. Multi-visit repairs and misdiagnoses are the most informative and the hardest to find; see difficult and failed repairs.
Item types
| Item type | Input | Expected output | Grading |
|---|---|---|---|
| Next-step prediction | Case so far (symptom, readings, steps done) | Next action | Match against confirmed action, with accepted alternatives |
| Diagnosis | Symptom, context, evidence | Root cause or fault class | Exact class or ranked list |
| Ordering | Shuffled steps | Correct sequence | Rank correlation or exact match |
| Evidence selection | Case plus candidate readings | Which reading or test to run | Pick from options |
| Parts prediction | Case | Parts list | Set overlap |
| Safety check | Plan | Missing isolation or verification step | Flag/no flag |
| Text-plus-video | Short clip and question | Step label or issue | Match to annotation |
Next-step items need care: several actions may be reasonable at a point in the case, and the technician's action is only one of them. Provide an accepted-alternatives field from expert review, or grade by rubric.
What these sets can measure, and what they cannot
They can measure whether a model reads symptoms and readings correctly, orders diagnostic steps plausibly, uses terminology correctly, names the right parts and notices missing safety steps.
They cannot measure physical ability. A model that predicts "replace the contactor" has not shown that a robot can do so. Text evaluation is not a robot evaluation, and an evaluation set built from human cases is not an action benchmark. They also cannot tell whether the technician's chosen path was optimal; records show what was done, and the outcome shows only that it worked. Cases from one fleet or one model family may not predict performance elsewhere.
Holdout and contamination
Evaluation material must be held out by site, operator, object or time as the claim requires, and never mixed back into training.
- By site if you claim generalization to new facilities.
- By asset model or family if you claim transfer to unseen equipment.
- By time if the assistant will meet future faults.
- By technician if you want to avoid learning a person's style.
- Against public text. Manuals and forum answers may be in a model's pretraining; check overlap between items and public documents.
Keep the answer key and the case text out of any fine-tuning set, and record who has seen it. Rotate items when a set is used repeatedly.
Illustrative item
{
"_note": "Illustrative only - synthetic content",
"item_id": "ts_000231",
"type": "next_step",
"asset_class": "rooftop_air_handler_type_Q",
"site": "site_017",
"case_so_far": [
"Complaint: no cooling on zone 3",
"Checked: thermostat calling, contactor energized",
"Reading: compressor amp draw 0.0, fan running"
],
"expected_action": "Test capacitor and compressor windings before replacing the contactor",
"accepted_alternatives": ["Check high-pressure switch continuity"],
"outcome": {"fix": "start capacitor replaced", "callback_within_30d": false},
"source_refs": ["wo_88123", "fault_tree_node_FT-12"],
"review": {"expert": "reviewer_03", "agreement": "2 of 2"},
"split": "test_site_heldout"
}
Quality and acceptance criteria
- Every item has a confirmed outcome and a callback status.
- Two experts agree on the key, with disagreements stored not discarded.
- Normalization documented, original text preserved.
- Customer names, serial numbers and addresses removed or controlled as the agreement requires.
- Coverage table by asset class, fault family and site, reported with counts.
- Difficulty tiers defined, including cases that needed escalation.
- A separate development set exists, so tuning does not touch the test set.
Permissions and provenance
Service cases carry customer data and sometimes proprietary diagnostic knowledge from OEMs. Diligence questions: who authored the records, whether customers consented to secondary use, whether OEM-derived text is reproduced, and whether the supplier can license derived test items. Removing names does not by itself settle who may license the content. Because evaluation sets are small and reused, a leak of the key is costly, so agree on handling and access terms.
Questions to ask suppliers
- How many closed cases are available, by asset class and fault family?
- Which cases have a confirmed fix and callback status?
- Is there free-text reasoning from the technician or only action codes?
- Can cases be split by site, model and date?
- How were abbreviations and codes defined, and who normalized them?
- Do any cases link to video, images, error codes or fault trees?
- Are escalated and multi-visit repairs included?
- What customer data is present, and under what terms may derived items be used?
Discuss your requirements
If you are assembling an evaluation set, tell SourceX the asset classes, item types and holdout rules you need. SourceX reviews the requirement and follows up about potential sources among US businesses. Annotation and expert review are scoped case by case, and a request does not guarantee a match. The use-case entry for evaluation sets and the catalog entry for field service maintenance records describe related material, and diagnostic reasoning traces cover the training-side counterpart.
Tell SourceX what your evaluation set needs to cover
Sources
- The TechQA Dataset — technical-support QA split sizes and companion corpus.
- MaintNet: A Collaborative Open-Source Library for Predictive Maintenance Language Resources — maintenance logbook data and normalization tools.