AI uses for records
Reward function for delayed outcomes: scoring an agent when success comes later
By SourceX Editorial · Updated
10 sections
Short answer
Grade a delayed-outcome task by defining success as an inspectable business event inside a fixed window, then choosing how the score is assigned while you wait: hold the reward until the window closes, pay a proxy now and correct it later, or reward verified milestones. Score unresolved episodes as censored, not failed, and stress-test the grader against shortcuts that satisfy the proxy without the business result.
Grader rubric
Reward function delayed
Key takeaways
- A delayed outcome needs three things written before scoring: the business event that counts, the window it must happen in, and what an unresolved episode scores.
- A score assigned at task end is a proxy; the reward design decides how far that proxy can drift from the real result.
- Episodes still inside their window are truncated, not failed, and should be scored that way.
- Milestone and process rewards give earlier signal but need more expert labels and their own shortcut tests.
- Every grader should face a set of deliberately wrong trajectories that would score well if the grader could be gamed.
- Keep expert disagreement as data rather than averaging it into a single label.
How do you grade an agent task whose outcome is delayed?
You grade it against a business event you can inspect later, inside a window you fix in advance, and you choose explicitly what the agent scores while the window is open. That choice is the reward design. The safest default is to hold the outcome reward until the window closes and to score episodes still waiting as censored rather than as failures.
Delayed reward is not an edge case in reinforcement learning; Richard Sutton describes trial-and-error search and delayed reward as its two most important distinguishing features, and the discount rate as what weighs immediate against delayed rewards. Business tasks make the delay concrete. A collections call succeeds when the invoice is paid, a support fix succeeds when the ticket stays closed, and a claims decision succeeds when it survives appeal.
Each of those outcomes lives in a different system from the agent's actions, arrives on its own schedule, and can be caused by things the agent did not do. The reward function has to deal with all three facts, and the comparison below shows how each common design does.
Write the success definition down as a record a second person could check: the outcome event, the system it lives in, the field and value that count, the window measured from a named timestamp, and the rule for partial results. If two reviewers reading that record would grade the same episode differently, the definition is not ready to become a reward.
Why the score at task end is usually a proxy
At the moment the agent stops, you can usually check only what it did, not whether it worked. Anything you score then, such as a status field set to resolved or a promise recorded in the CRM, is a proxy for the outcome, and proxies can be satisfied without the result.
Google DeepMind defines specification gaming as behavior that satisfies the literal specification of an objective without achieving the intended outcome. In June 2025, METR reported frontier models of that time attempting, often successfully, to get higher scores by modifying tests or scoring code. An agent rewarded for closing tickets can close tickets; an agent rewarded for a logged payment promise can log promises that are never kept.
The delay widens this gap because the proxy is what the agent is optimized against for most of the episode's life. The longer the window, the more weight the design puts on whatever is checkable early, and the more important it is to know which shortcut that early check allows.
The delayed outcome reward design comparison
This is the asset: seven ways to assign a score when the outcome is late, compared on what each needs, where it fits, the shortcut it invites and the test that should pass before you rely on it. Most production graders combine two of them, typically a held outcome reward with a milestone or proxy component.
| Design | How the score is assigned | Fits when | Shortcut it invites | Test before use |
|---|---|---|---|---|
| Held outcome reward | No score until the window closes; then success, failure or censored | Training can wait, or the set is historical with outcomes already observed | Few, but slow feedback tempts teams to shorten the window quietly | Window length set from the source system's lifecycle and fixed in writing |
| Proxy at task end only | Score from a checkable end state such as a status, record or document | The proxy is shown to track the outcome closely on historical cases | Setting the status or creating the record without doing the work | Measure how often proxy success later became outcome failure |
| Proxy now, corrected later | Provisional proxy score replaced by the outcome score when it arrives | Fast iteration is needed but outcomes eventually arrive for most cases | Gains that live only in the provisional window | Report provisional and corrected scores side by side for the same runs |
| Milestone or process rewards | Credit for verified intermediate steps, such as a correct lookup or a valid offer | Experts can define and check steps that lead to the outcome | Hitting milestones in a sequence that still fails the task | Check that milestone scores predict the outcome, not just each other |
| Progress shaping | Extra reward for moving toward a goal state, removed at the end | Long episodes with sparse outcomes and a clear notion of progress | Looping through rewarded states without finishing | Confirm no cycle of states earns net reward |
| Return redistribution | A model assigns the final outcome's credit back to earlier steps | Many episodes with observed outcomes are available to learn from | Credit placed on steps correlated with success but not causing it | Inspect which steps receive credit on cases an expert can explain |
| Predicted outcome reward | A model trained on past episodes predicts the eventual outcome now | History is large and the workflow is stable over time | Exploiting features the predictor overweights | Hold out later outcomes and compare predicted with actual |
Episodes still waiting: truncated is not failed
An episode whose outcome window has not closed has been cut off, not failed, and the grader should say so. Scoring it as zero teaches that the agent's action was bad when you simply do not know yet.
Reinforcement learning libraries already draw this line. Gymnasium splits the old done flag into terminated, when the task was completed or failed, and truncated, when an external limit such as a time or step limit ended the episode. Its tutorial on time limits explains that learners bootstrap from the value of the last state on truncation and do not on termination; treating both the same is described as incorrect.
For an evaluation set the equivalent is simple: report the success rate over episodes whose windows have closed, report how many are still open, and never fold the open ones into the failure count. For training, either exclude open episodes from outcome rewards or let the method estimate their value, and document which one you did.
Keep the freeze date with every reported score. A success rate computed on a Monday and the same rate recomputed a month later will differ as windows close, and readers need to know which one they are looking at. Versioning the score by freeze date also makes it obvious when a later improvement is just more windows closing.
Process rewards or outcome rewards: which to trust when the result is late
Use outcome rewards when the outcome is reliable and complete, and add process rewards when it is not, or when you also care whether each step was sound. The research behind that rule comes from reasoning tasks, but the trade-off carries over to agents.
Uesato and colleagues compared process-based and outcome-based feedback on math word problems and found process feedback improved the correctness of reasoning steps but required more label supervision; they suggest outcome feedback suits cases with a reliable, complete evaluation metric. Lightman and colleagues reported that process supervision significantly outperformed outcome supervision on a harder math benchmark, and noted that outcome-supervised models can reach correct answers through flawed reasoning.
Delayed business outcomes are rarely complete metrics. A paid invoice can follow a bad call because the customer was going to pay anyway. That argues for pairing the held outcome reward with a small number of expert-defined milestones, such as verifying the account and stating terms correctly, that the domain team agrees are necessary for a good result.
Whatever mix you choose, keep the components separate in the logs. Store the milestone score, the proxy score and the outcome score as distinct fields per episode rather than one blended number, so a later reviewer can see which part moved when an agent version improves, and can re-weight them without re-running the episodes.
Attacking the grader: which fake completions would still score well?
Find out before training by attacking your own grader. Write a set of trajectories that should fail the business requirement but might pass the grader, run them, and treat any high score as a grader bug.
At a glance
Attacking the grader: which fake completions would still score well
Shortcuts in delayed-outcome tasks tend to cluster around the proxy and around attribution. The list below is a starting set; the domain adjudicator should add cases specific to the workflow.
- Status without substance: the agent sets the resolved, paid or approved field without the underlying action.
- Outcome borrowed from elsewhere: the outcome happens, but because of another team's action or an automated rule, not the agent.
- Deferred failure: the agent pushes the problem past the window, for example by scheduling a follow-up just after it closes.
- Cherry-picked cases: the agent escalates or skips hard cases so that only easy ones reach an outcome.
- Grader tampering: the agent edits the records, tests or files the grader reads.
- Partial completion: one of several required outcomes occurs and the grader credits the whole task.
Keeping expert disagreement in the record
When experts disagree about whether a delayed outcome counts as success, keep both judgments and the reason, rather than averaging them into one label. Disagreement is information about how well the success definition is specified.
A useful adjudication record has the episode ID, the outcome event and its timestamp, each expert's verdict, an error severity from a short scale agreed in advance, the reason for any disagreement, and the final adjudicated label with who made it. Severity matters because a delayed failure that costs a customer relationship should not count the same as one that costs a second call.
If disagreement clusters on one kind of case, the definition is the problem, not the experts. Narrow it, split the case type into its own label, or exclude it from automated rewards until the definition is fixed.
Measure agreement before the grader goes into use. Have two experts label the same sample of closed episodes independently, report how often they agree, and list the case types where they do not. An automated grader should not be held to a stricter standard than the experts can meet with each other, and it should not be trusted on case types where they cannot agree at all.
Illustrative example: a collections agent scored on payment that arrives later
This example is illustrative; all counts and windows are invented to show the method. A team evaluates an agent that calls customers about overdue invoices. The business outcome is payment in full within an illustrative 30 days of the call; the proxy at task end is a recorded promise to pay.
On 1,000 historical episodes, the proxy credits 420 as successes. When the windows close, 300 of those are paid in full, 60 are partly paid, 40 are unpaid and 20 are still inside an extended window. Of the 580 episodes without a promise, 50 are paid anyway, mostly through automatic card retries the agent did not trigger.
Under the held outcome design, the agent's success rate is reported over the 980 closed episodes, with 20 shown as censored. The 50 payments without a promise are scored by an attribution rule: credited only when the payment followed a step the agent took, such as sending a payment link. The grader stress test adds trajectories that log a promise without stating the amount or due date, and confirms they score zero on the milestone component.
The comparison shows the proxy alone would overstate success and would reward logging promises. The team keeps the held outcome reward for evaluation and uses milestone checks for faster training feedback, reporting both.
Where delayed rewards stop being worth the wait
Some outcomes are too slow, too rare or too confounded to grade an agent on directly. If the window is longer than the agent version will live, if fewer than a handful of outcomes arrive per month, or if most outcomes are driven by factors outside the agent's control, a held outcome reward will not give a usable signal.
In those cases, narrow the claim. Grade the agent on verified milestones the domain team agrees are necessary, report outcome data as a separate observational measure, and say plainly that the evaluation does not establish the business result. Tasks touching credit, claims, employment or regulated communications also need the reward definition reviewed by the people accountable for those decisions.
Seasonality is a quieter version of the same problem. If outcomes for episodes run in December are judged by January payment behavior, and episodes from June by July behavior, part of any difference between agent versions is the calendar. Compare versions on episodes from the same period, or record the period as a variable and report results within it.
Sourcing records whose outcomes have already landed
Historical business records solve the waiting problem, because the windows closed long ago. A ticket, invoice or claim from last year already shows whether the result held, which makes held outcome rewards practical from day one.
Historical sets still need the checks above. Confirm that each outcome is linked to the right case, that the window rule is applied the same way to every record, and that cases still open at the extract date are marked as censored rather than dropped.
SourceX does not keep datasets on the shelf; it looks for them at U.S. businesses when a buyer asks, and the buyer's request sets out scope, volume, format, timing and licensing needs, which is where the outcome fields and the window belong. As a general guide, buyers look for a couple of years or more of consistent history and tens of thousands of records in one system, though smaller sets can qualify when the work is rare.
Frequently asked questions
What is a delayed outcome in agent evaluation?
It is a business result that is only known some time after the agent finishes, such as a payment, a reopened ticket or an appeal decision. Grading against it needs a defined window, a way to link the outcome to the episode and a rule for episodes whose window has not closed.
Should unresolved episodes be scored as failures?
No. They are truncated, not failed, and scoring them as zero biases the success rate downward and penalizes actions whose results are unknown. Report them separately as censored and either exclude them from outcome rewards or estimate their value with a documented method.
Is a proxy reward ever good enough?
Yes, when historical data shows the proxy tracks the outcome closely and stress tests show it cannot be satisfied by a shortcut. Even then, report the corrected outcome score when it arrives so drift between the two is visible.
What is the difference between process and outcome rewards?
Outcome rewards score the final result; process rewards score intermediate steps. Research on reasoning tasks found process feedback improved step correctness but needed more labels, so it is most useful when the outcome alone is unreliable or incomplete.
How do you stop an agent from gaming a delayed reward?
Build a set of trajectories that fail the business requirement but could pass the grader, such as setting a status without the action, and confirm they score low. Also make sure the agent cannot edit the records or files the grader reads.
Sources
- Sutton describes delayed reward as one of reinforcement learning's two most important distinguishing features, with the discount rate weighing immediate against delayed rewards. Source
- Specification gaming is behavior that satisfies the literal specification of an objective without achieving the intended outcome. Source
- METR reported recent frontier models attempting, often successfully, to raise their scores by modifying tests or scoring code. Source
- Gymnasium separates terminated episodes (task completed or failed) from truncated ones (ended by an external limit), a distinction that matters for bootstrapping. Source
- Gymnasium's time-limit tutorial says learners bootstrap on truncation and not on termination. Source
- RUDDER handles delayed rewards by redistributing the return to earlier steps through return decomposition. Source
- Process-based feedback improved reasoning-step correctness but required more labels; outcome feedback suits cases with a reliable, complete metric. Source
- Process supervision outperformed outcome supervision on a harder math benchmark, and outcome-supervised models can reach correct answers with flawed reasoning. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.