Skip to content

AI uses for records

Agent trajectory outcome delay: labeling results that arrive next week

By SourceX Editorial · Updated

10 sections

Short answer

Label delayed business outcomes by keeping the agent's trajectory unchanged and attaching each outcome as a separate, timestamped record linked by case ID. Fix an observation window per outcome type before labeling. When a window closes, record an observed success or failure. If the window is still open at the freeze date, or the outcome can no longer be tied to the case, label it censored and keep it out of success rates.

Agent run

Agent trajectory outcome

This page covers agent trajectory outcome delay: labeling results that arrive next week.Illustrative

Key takeaways

  • Outcome delay is the gap between an agent's last step and the business learning whether the work held.
  • Labels frozen at task end overstate success, because reopens, disputes and cancellations arrive late.
  • Set an observation window per outcome type from the source system's lifecycle before labeling.
  • Use explicit censored and competing-event states instead of forcing unresolved cases into success or failure.
  • Store outcomes as linked records with event and observed timestamps so labels can be recomputed and versioned.

How do you label a business outcome that arrives after the agent's task ends?

Leave the trajectory untouched and attach the outcome as a separate, timestamped event linked by the case ID. Before labeling anything, fix an observation window for each outcome type. When the window closes, each case gets a success or failure label from what the source system did or did not record. A case whose window is still open at the freeze date, or whose outcome can no longer be tied to it, is labeled censored with the reason, and is never counted as a success or a failure by default.

Agent trajectory outcome delay is the gap between the last step an agent took and the moment the business learns whether the work held. A support reply is sent on Monday and the customer reopens the ticket on Thursday. A refund is approved today and a card dispute arrives weeks later. A renewal call goes well and the account cancels at term end. If labels are frozen at the end of the task, every one of these cases looks like a success it was not.

Why does closing the label at task end give the wrong answer?

Labeling too early treats not yet observed as did not happen. Recommender-system research has studied this as delayed feedback: Ktena and colleagues describe positive labels that only arrive after a possibly long, random delay, and note that counting an example as negative until a positive arrives tends to underestimate the rate being predicted. Agent outcomes have the same structure with the sign flipped: complaints, reopens and reversals are the late events, so early labels overstate success.

Survival analysis names the missing part. The lifelines documentation calls subjects whose event has not yet occurred right-censored and warns that dropping them is a common mistake that biases estimates. The NIST engineering statistics handbook describes the fixed-time version, where a test runs for a set time and units that have not failed by then are recorded as censored. Both apply directly: a trajectory whose window is still open, or that was cut off for an unrelated reason, is censored, not successful.

Business systems make the delay concrete. Zendesk's default automation closes a solved ticket four days after it is solved, configurable up to 28 days; a reply before closing reopens the ticket, and a reply after closing creates a follow-up ticket that references the original. Stripe's documentation says card networks typically allow cardholders to dispute a payment within 120 days, and sometimes longer. Any outcome label set before those points can still change.

Which delayed outcomes need a window and a censoring rule?

This is the asset: one row per delayed outcome type, filled in before any labeling. Window lengths are set by the business process owner from the source system's own lifecycle; the values shown are illustrative or quoted from vendor documentation where noted.

Which delayed outcomes need a window and a censoring rule?
Delayed outcomeSource record and link keyObservation window ruleLabel at window closeCensoring rule
Ticket reopenedHelp-desk ticket status history; ticket IDUntil the ticket auto-closes (Zendesk default four days after solved, configurable)Failure if reopened in window; success if closed without reopenCensored if the ticket is merged or deleted before closing
Follow-up ticket on the same issueFollow-up ticket referencing the closed ticket; parent ticket IDFixed window after closing, set by the owner (illustrative: 14 days)Failure if a linked follow-up names the same issueCensored if the requester's account is closed during the window
Card dispute or chargebackPayment dispute record; payment or charge IDNetwork dispute window (Stripe: typically 120 days from payment)Failure if a dispute is opened and lost; flag if wonCensored if the window is still open at dataset freeze
Refund reversalBilling or ERP credit memo; order IDUntil the next billing cycle closesFailure if the refund is reversed or reissuedCensored if the order is transferred to another account
Renewal or cancellationCRM subscription record; account IDUntil the term end date recorded at task timeSuccess if renewed; failure if cancelled for a reason tied to the taskCompeting event if cancelled for an unrelated reason such as acquisition
Invoice paid after collection callAccounts receivable ledger; invoice IDUntil the promised payment date plus a grace period set by the ownerSuccess if paid in full; partial if part paidCensored if the invoice is written off for a reason unrelated to the call
Field-service callbackWork order history; site or asset IDOwner-set warranty window (illustrative: 30 days)Failure if a callback cites the same faultCensored if the asset is replaced or the site leaves service
Escalation or formal complaintComplaint log or escalation queue; case IDOwner-set window after resolutionFailure if escalated about the task's handlingCensored if the link to the case is ambiguous

Which label states keep censored cases out of the success column?

Use more than two label values. A binary success field forces every unresolved case into one bucket or the other, and that choice is where the bias enters.

At a glance

Which label states keep censored cases out of the success column

  • observed_success: the window closed and no failure event occurred, or a positive event was recorded.
  • observed_failure: a failure event linked to the task was recorded inside the window.
  • censored_window_open: the dataset was frozen before the window closed; relabel later or exclude from success rates.
  • censored_lost_link: the outcome exists but cannot be tied to this case reliably, such as a merged ticket or a changed account ID.
  • competing_event: something unrelated ended the observation, such as an account acquired or a contract ended by the customer for other reasons.
  • not_applicable: the outcome type does not apply to this task, such as a dispute check on a task with no payment.
  • Each label also carries label_version and label_computed_at, so a later recomputation never silently replaces an earlier one.

Store the outcome as its own record that points back to the trajectory, not as a field written into the trajectory's last step. The OpenTelemetry specification uses the same idea for traces: a span can link to causally related spans inside its own trace or in a different trace, which lets work that happens later and elsewhere declare what it relates to without rewriting the earlier record.

A minimum outcome record holds the case ID, trajectory ID, the step ID judged responsible if known, the outcome type from the linkage table, the source system and its record ID, the time the outcome occurred, the time it was observed by the data pipeline, the window end date, the label state and the label version. Help desks already provide some of this: Zendesk's tickets API lets a follow-up ticket reference the closed ticket it continues, which is a ready-made link key.

Keep two timestamps per outcome. The event time says when the customer reopened the ticket; the observed time says when your extract saw it. Mixing them makes a dataset frozen on one date look as if it knew about events it had not yet received.

Replay check: could another team rebuild the same labels from what ships?

Yes, if the dataset ships with an as-of rule and the inputs to every label. Give the second team the trajectory, the outcome records, the linkage table version and the freeze date, and they should compute identical label states. If they cannot, the labels depend on something undocumented.

Two leaks break replay most often. The first is future information in the observation: a ticket's final status or tags copied into the state the agent saw at step one. The second is a moving freeze date, where a later extract quietly turns censored cases into failures without a new label version. Check for both by recomputing labels from scratch on a sample and diffing against the shipped labels.

How do labels change for a refund that is disputed weeks later?

This example is illustrative and uses counts and durations only. An agent handles 500 refund requests from one help desk in a single week. Each trajectory ends when the agent sends the decision and the ticket is solved.

At the end of that week, labeling at task end would show nearly every case as a success. Applying the linkage table instead: the ticket-reopen window closes four days after each ticket is solved, so by the following week, say, 20 tickets have reopened and are observed failures. The dispute window runs much longer, so at a freeze date 30 days later every refund-related payment is still censored_window_open for disputes, and the dataset reports them that way. Five accounts are merged during the period and become censored_lost_link.

The reported success rate for reopen outcomes is computed over cases whose reopen window closed; the dispute outcome is reported as not yet observable at this freeze, with a planned relabel date. Nothing in the trajectories changes; only the outcome records and their label version do.

When should a late outcome not be credited to the agent?

Some late outcomes are real but not caused by the trajectory. A customer may cancel because their company was acquired, or a callback may be for a different fault on the same asset. Crediting those to the agent teaches the wrong lesson. Treat them as competing events, and require the linkage rule to name what evidence ties an outcome to the task, such as the same issue code or a reference to the original ticket.

Human handoffs need the same care. If a person took over after the agent and the outcome followed the person's action, record the outcome against the combined case but mark which actor took the last material step. Teams that cannot make that call from the records should narrow the dataset to outcomes with an unambiguous link rather than guess, and ask the task expert who owns the process to review the attribution rules.

How should censored labels be used in training and evaluation?

For evaluation, compute success rates only over cases whose window has closed, report the censored share next to every rate, and state the freeze date. For training, either exclude censored cases from outcome-based rewards, weight them by how much of their window has elapsed, or model time to the outcome directly with survival methods; pick one and document it.

Plan relabeling. A dataset of agent trajectories with delayed outcomes is not finished on delivery day; it has a schedule on which open windows close. Publishing that schedule with the dataset lets the buyer know which labels are final and when the rest will be.

Report the dataset in a way that makes the timing visible. For each outcome type, give the number of cases with a closed window, the observed failure count among them, the censored count by reason and the freeze date. A reader who sees a high success rate next to a large censored share knows to wait for the relabel before trusting it, which is exactly the judgment a single success column hides.

Where do records with already-observed outcomes come from?

Historical business records have a quiet advantage here: for cases old enough, the windows have already closed, so outcomes are observed rather than censored. That only helps if the export keeps the downstream records, such as status histories, follow-ups and billing events, with stable keys to the original case.

Teams without such records can have SourceX search among U.S. businesses on their behalf, with no off-the-shelf catalog involved; the request records what they need in scope, volume, format, timing and licensing. Outcome sources and the freeze date belong in the written scope, which the supplying company approves, together with the terms, by signing ahead of delivery.

Frequently asked questions

What is outcome delay in agent trajectory data?

It is the time between an agent's last action on a task and the moment the business records whether the result held, such as a reopen, refund reversal or cancellation. During that gap the true label is unknown, so it should be treated as censored rather than as a success.

What does censored mean for a trajectory label?

A censored label means the outcome was not observed, either because its window is still open or because something unrelated ended observation. It is a known lack of information, and treating it as success or failure biases any rate computed from the data.

How long should the observation window be?

Set it per outcome type from the source system's lifecycle and the process owner's judgment, before labeling. For example, a help desk's auto-close delay bounds reopen outcomes, while card disputes typically allow a much longer window.

Should delayed outcomes be written into the trajectory itself?

No. Keep the trajectory as it happened and store outcomes as separate linked records with event and observed timestamps. That way labels can be recomputed and versioned without rewriting the episode.

Can delayed outcomes be used as rewards for training agents?

Yes, once their windows have closed or with a documented method for censored cases, such as exclusion or survival modeling. Rewards built from open windows overstate success, because failure events such as reopens and disputes are the ones that arrive late.

Sources

  • The lifelines documentation defines right-censored subjects as those whose event has not yet been observed and warns that ignoring them is a common mistake that biases estimates. Source
  • The NIST/SEMATECH handbook describes Type I censoring, where a test runs for a fixed time and units without a failure by then are recorded as censored, also called right-censored data. Source
  • Ktena and colleagues describe positive labels that arrive only after a possibly long, random delay and note that counting examples as negative until a positive arrives tends to underestimate the rate. Source
  • The OpenTelemetry specification lets a span link to zero or more causally related spans in the same trace or in different traces. Source
  • Zendesk's default automation closes tickets four days after they are solved, adjustable up to 28 days; a reply to a solved ticket reopens it, and a reply to a closed ticket creates a follow-up ticket. Source
  • Zendesk's tickets API creates a follow-up for a closed ticket through the via_followup_source_id parameter that points to the closed ticket. Source
  • Stripe says card networks typically allow cardholders to initiate a dispute within 120 days of the payment, with longer windows in some situations. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies