Privacy and preparation
Expert task reconstruction interviews: replay the case before the task
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Rebuild the case timeline from the records first, then walk the expert through it in order, revealing only what was known at each step and stopping before the outcome. At each decision point, ask what they noticed, what they considered and what evidence would have changed the call. Log every gap between what the expert used and what the records hold, and author the task only from evidence that survives.
Redaction
Expert task reconstruction
Key takeaways
- Build the case timeline from system history before the session, with the time each fact became visible to the person deciding.
- Replay forward and hide the outcome until the end; outcome knowledge reshapes how experts recall and judge earlier decisions.
- Probe specific moments for cues, expectations, options and the one fact that would have changed the call.
- Trace every cue to a record; log each gap by type and decide to retrieve, restate, flag or drop before writing the task.
- Have a second qualified reviewer check a sample of tasks blind, and record reasonable alternative answers.
How do you replay a business case with an expert before authoring a training task?
Replay it forward from the records, not backward from the expert's memory. An expert task reconstruction interview starts with a timeline assembled from system history before the session, walks the expert through that timeline step by step with only the information that existed at each moment, and stops short of the outcome. At every point where a decision was made, the interviewer asks what the expert noticed, what they expected, which options they weighed, and what would have made them choose differently.
The second half of the method is the evidence gap guide. Every cue the expert names is traced back to a field, document or message in the records. When a cue cannot be found, the gap is logged with its type and a decision about the task: retrieve the missing evidence, restate the task so it does not depend on it, keep the case with a flag, or drop it. Only then is the training or evaluation task written, using the evidence that survived.
This order matters for anyone preparing business records for AI work. A task written from an expert's summary tends to give the model information nobody had at the time, or to ask it to infer things that were never in the record. A task written after a replay matches what a person in that seat could actually see, which is what makes it a fair test and a useful training example.
Why the expert's memory of a closed case is not yet a training task
Because memory of a finished case is shaped by how it ended. Decades of research on judgment show that outcome knowledge changes how people recall and evaluate earlier decisions, and experts are not exempt.
Baruch Fischhoff's work on hindsight bias, which he reviewed in a 2025 retrospective, shows that knowing an outcome makes it seem more predictable than it was in foresight. A related effect, outcome bias, means the same decision is judged better when it turned out well than when it turned out badly; a pre-registered replication of Baron and Hershey's original experiment found the effect again, including among participants who said outcomes should not affect their judgment. An expert asked to explain a case that went well will tend to tell a cleaner, more inevitable story than the one they lived through.
The interview techniques that work around this come from cognitive task analysis. The critical decision method, as its original IEEE paper describes it, is a variant of Flanagan's critical incident technique extended with probes for the basis of perceptual and conceptual discriminations, typicality judgments and critical cues, and it has been used with fireground commanders, engineers, paramedics and programmers. A systematic review of cognitive task analysis studies found the critical decision method and related interviews among the most frequently used methods.
Replaying real material also helps. Studies of verbal reporting distinguish concurrent think-aloud from retrospective reports, and a 2024 study of hardware reverse engineers notes that cued retrospective reporting, where participants watch a recording of their own session, has been found to elicit more action, how and metacognitive information than uncued retrospection. Business records are the closest thing most companies have to that recording.
Rebuilding the case timeline before the expert sees it
Assemble a timestamped timeline from system history before the session, so the expert reacts to what the record shows rather than narrating from memory. The timeline is the backbone of the interview and later of the task.
Most business systems keep the history you need, within limits worth checking early. Salesforce field history tracking records changes on up to twenty selected fields per object, and its overview says standard history is retained for about eighteen months in the interface and about twenty-four months through the API unless Field Audit Trail is used. Jira Cloud's REST API returns an issue's changelog as a paginated list sorted from the oldest change. Untracked fields, retention windows and overwritten values are the first sources of gaps, so record which fields were tracked during the case period.
Mark every timeline entry with when it became visible to the person who decided. A note typed into a ticket at the end of the day may describe a phone call from the morning; for the replay, that information arrives at the time of the call only if the expert confirms it, and otherwise at the time it was written. Choose cases deliberately: a few routine ones to establish the baseline, several hard ones where the decision was not obvious, and at least one where colleagues disagreed.
| Timeline field | What it holds | Why the replay needs it |
|---|---|---|
| event_time | When the event happened in the business | Orders the replay |
| visible_time | When the decider could first see it | Prevents leaking later information into earlier steps |
| source_ref | System, record ID and field or document | Lets each cue be traced to evidence |
| actor_role | Role of the person or system that produced the event | Separates the expert's actions from others' decisions |
| tracked | Whether the field was under history tracking at the time | Flags values that may have been overwritten |
| redaction_note | Personal or customer data removed before the session | Keeps the replay usable without exposing people |
The reconstruction interview guide, pass by pass
Run the interview in six passes, each with a different job, and do not skip ahead to the outcome. Sessions work best with one interviewer who drives the timeline and one note-taker who maintains the gap log in real time.
Prepare the expert as well as the timeline. Tell them in advance that the session is about one case, that the goal is to understand how the decision looked at the time rather than to judge it, and that saying they do not remember is a useful answer. Experts who think they are being audited give the policy answer; experts who understand that gaps are expected point to them. Keep the free-recall notes separate from the replay notes, because the differences between the two are often where the most useful cues hide.
The task check in the last pass is where the interview pays off. Put the draft task in front of the expert exactly as a model would see it, with the inputs, the question and the expected answer, and ask them to decide from it cold. If they reach for something that is not there, that is a gap the earlier passes missed.
| Pass | What the interviewer does | Example question | What gets recorded |
|---|---|---|---|
| 1. Free recall | Ask for the case in two minutes, before showing any record | What do you remember about this case and why was it hard? | The expert's own framing, to compare with the replay later |
| 2. Timeline replay | Reveal timeline entries in visible_time order, stopping before the outcome | This is what you had on Tuesday morning. What did you make of it? | Expert reading of each step, with timestamps |
| 3. Decision probes | At each decision point, ask about cues, expectations, options and goals | What told you this one was different from the usual? | Cues, expectations, options considered, goal at that moment |
| 4. Alternative paths | Change one fact and ask whether the decision changes | If the customer had paid the last invoice on time, would you have held it? | Decision boundaries and acceptable alternative decisions |
| 5. Evidence trace | For each cue, find where it lives in the records | Where would I find that the account was in dispute? | Source reference or a gap entry |
| 6. Task check | Show the draft task inputs and ask what is missing or too much | Could someone in your seat decide from just this? | Changes to inputs, expected answer and grading notes |
Probes that find cues, options and what would have changed the call
Ask about specific moments, not about policy in general, and ask for the evidence behind each answer. The probes below adapt the critical decision method's focus on cues and discriminations to business casework; adjust the wording to the domain.
- What was the first thing you looked at when this landed, and what were you checking for?
- Which detail made this case different from the ones you handle without thinking?
- What did you expect to happen next, and did it?
- Which options did you consider, even briefly, and why did you set each one aside?
- Who else would you normally ask in a case like this, and did you?
- If a newer colleague had handled this, where would they most likely have gone wrong?
- What single piece of information, if it had been different, would have changed your decision?
- Which part of your decision would another experienced person reasonably have made differently?
- How sure were you at the time, and what made you more or less sure as the case went on?
- Where is that recorded, and if it is not, how did you know?
The evidence gap guide: when the record cannot carry the decision
Every gap gets a type, a consequence for the task and a resolution before anything is authored. Gaps are not failures of the interview; they are the main output that keeps a training task honest.
Count the gaps by type across cases as well as within them. If the same unrecorded cue appears in most interviews, such as a phone call that never gets logged, the problem is the records, not the case selection, and the honest response is to scope the task set around what the systems do capture or to change how that evidence is recorded going forward.
| Gap type | How it shows up | Consequence for the task | Resolution |
|---|---|---|---|
| Unrecorded cue | Expert cites a phone call or hallway conversation | Model would need information it cannot see | Retrieve a later note if one exists; otherwise restate the task or flag it |
| Overwritten value | Field was not tracked and now shows the final value | Task inputs leak the outcome | Use only tracked history; drop the field from inputs |
| Outcome contamination | Expert's reasoning cites what happened later | Expected answer reflects hindsight | Re-ask with the timeline cut at the decision point; flag if it persists |
| Off-system evidence | Decisive data lived in a personal spreadsheet or inbox | Records alone cannot support the decision | Collect the artifact with permission or drop the case |
| Policy drift | The rule in force then differs from today's | Grader would apply the wrong rule | Attach the dated policy version to the task |
| Tacit pattern | Expert relies on experience across many cases | Single-case inputs may not justify the call | Test the pattern against history; keep as a labeled heuristic if it holds |
| Someone else's decision | A key step was another person's call | Expert's rationale for it is a guess | Interview that person or mark the step as given |
Worked example (illustrative): a credit hold released ahead of payment
This example is invented to show the method; the company, people, durations and counts are illustrative and do not describe any real records. A distributor's credit manager released an order that a credit hold had stopped, before the customer's overdue invoice was paid. The team wants an evaluation task asking an agent whether to hold, release or escalate.
Free recall was short: the customer is reliable and always catches up. The replay told a richer story. On the first morning the expert saw the hold, the aging report showing one invoice overdue by about three weeks, and the order's requested ship date two days out. They expected a payment because the customer's payables contact had promised one, a promise recorded only in a call note typed late that afternoon. The decision point came the next morning, when a second, smaller order arrived.
Probes surfaced three cues: the overdue invoice was the only one past terms, the customer had paid late in the same season in each of the previous two years, and a missed ship date would cost the customer a scheduled installation. The alternative-path pass showed the boundary: with two invoices overdue, or with no payment promise, the expert would have escalated to the controller. The expert also said a colleague might reasonably have released only the first order, which became an acceptable alternative answer.
The gap log had three entries. The payment promise was an unrecorded cue until the call note, so the task's visible inputs at the decision point include that note. The seasonal pattern was tacit; a check of payment history in the records confirmed late payments in the two prior years, so it stayed as evidence. The ship-date consequence lived in a sales rep's email the expert had been copied on; it was collected with the rep's agreement. The authored task gives the agent the hold, aging, order details, payment history, call note and email; accepts release or partial release; and records escalate as wrong only under the facts as they stood.
Cases to drop, cases to keep with a flag, and the limits of one expert
Drop a case when the decisive evidence cannot be recovered or when the decision depended on facts no record holds. A task whose correct answer needs information the inputs do not contain teaches a model to guess, and an evaluation built on it measures luck.
Keep a case with a flag when the expert's knowledge is the point. Tacit patterns that hold up against history, such as the seasonal payment behavior in the example, are valuable precisely because they are not written down; label them as heuristics with the supporting evidence, so a task can test whether an agent finds the same pattern from the records.
Remember what one interview cannot do. A single expert's reconstruction is one defensible answer, not the standard. Have a second qualified person review a sample of authored tasks without seeing the first expert's answer, and treat disagreement as information: either the task needs an accepted alternative or the case is genuinely contested and belongs in a separate set. Recall also fades, so prefer recent cases, and stop scheduling interviews when new sessions stop adding gap types or decision boundaries.
Turning reconstructed cases into records you can license
Interview material contains more personal data than the case records themselves: names of customers and colleagues, opinions about them, and recordings of the expert. Agree with the expert how the session will be used, remove personal and customer details from the timeline before the session where you can, and keep raw recordings out of anything shared externally. Whether a company may license reconstructed tasks depends on its rights in the underlying records and on its agreements with customers and staff, which counsel should review; being able to read the records does not by itself give that right.
Reconstructed cases are one way businesses turn operational history into material AI developers can use. When a company wants to license the case records behind such tasks, SourceX qualifies the records and runs the rights review, contracting, preparation and delivery. The interview timelines and gap logs never have to leave the building to find out whether the company fits, since the opening step is a short assessment without uploads, and nothing is delivered until the company has approved scope and terms and signed.
Frequently asked questions
How long should an expert task reconstruction interview take?
Plan for one case per session for hard cases and two or three for routine ones, with the timeline prepared beforehand. Sessions that try to cover many cases slide into general policy talk, which loses the specific cues the replay is meant to find.
Should the expert see the outcome during the replay?
Not until the decision probes and alternative paths are done. Showing the outcome first invites hindsight, and the expert's account starts to explain the result instead of the decision. Reveal the outcome at the end and ask whether anything about it surprised them.
What if the expert disagrees with what the records show?
Log it as a gap and find out which is wrong. Records can be incomplete or entered late, and memory can be reshaped by the outcome. Where neither can be confirmed, keep the case out of evaluation sets or flag the step as uncertain.
Is a reconstruction interview the same as a knowledge-capture session before someone retires?
It overlaps but has a narrower goal. Knowledge capture preserves what an expert knows across many situations; a reconstruction interview turns one recorded case into a task whose inputs and correct answers are traceable to evidence.
Can an AI note-taker run the gap log?
It can transcribe and draft entries, but a person should confirm each gap type and resolution, because classifying a gap is a judgment about the records. Check the company's policy on recording staff and customer information before using any recording tool.
Sources
- The critical decision method is a variant of Flanagan's critical incident technique extended with probes for perceptual and conceptual discriminations, typicality judgments and critical cues, used with fireground commanders, engineers, paramedics and programmers. Source
- A systematic review of cognitive task analysis studies found the critical decision method and CTA interviews were the most frequently used methods. Source
- Fischhoff's retrospective on fifty years of hindsight bias research describes how outcome knowledge makes events seem more predictable than they were in foresight. Source
- A pre-registered replication of Baron and Hershey's outcome bias experiment found that outcomes still affected evaluations of decision quality, including among participants who said they should not. Source
- A study of hardware reverse engineers describes cued retrospective reporting, where participants view recordings of their own session, as eliciting more action, how and metacognitive information than uncued retrospective think-aloud. Source
- Salesforce field history tracking covers up to 20 fields per object and, without Field Audit Trail, keeps history about 18 months in the interface and about 24 months through the API. Source
- Jira Cloud's REST API returns an issue's changelog as a paginated list sorted by date, starting from the oldest change. Source
See if your company qualifies
A short company assessment. No data uploads are needed.