Skip to content

AI data market

Long-horizon agent checkpoint dataset: what state must survive a resume

By SourceX Editorial · Updated

9 sections

Short answer

A checkpoint in a task that runs for hours must keep eight kinds of state: the goal and its acceptance test, standing constraints, each actor's authority and its scope, the version of shared records last read, dependency progress, in-flight external calls with idempotency keys, pending human decisions, and evidence for completed steps. A checkpoint dataset records them at every pause and tests whether the resumed agent still honors them.

Agent run

Long horizon agent

This page covers long-horizon agent checkpoint dataset: what state must survive a resume.Illustrative

Key takeaways

  • A transcript tells a resumed agent what was said; a checkpoint must tell it what is still true, permitted and in flight.
  • Constraints need a status and a link to the event that set or replaced them, or a summary will carry the wrong version forward.
  • Every approval needs a recorded scope and expiry, because frameworks can carry broad approvals across a resume.
  • In-flight calls need an outcome field and an idempotency key so a resume never repeats a side effect.
  • Score resumes by perturbing the world at a checkpoint and measuring constraint retention, authority violations, duplicate writes and stale-state commits, not only task completion.

Which state must survive a checkpoint in a task that runs for hours?

Eight kinds of state must survive: the goal with its acceptance test, every standing constraint with where it came from, the authority each actor holds and when it expires, the shared state the agent last read with its version, the progress of the dependency graph, any in-flight external call with its idempotency key, every pending human decision, and the evidence behind each step already marked done. If any one is missing, the resumed agent can finish the task and still be wrong.

Orchestration frameworks already treat a checkpoint as more than a chat log. LangGraph describes a checkpoint as a snapshot of the graph state at a given point, saved at every superstep under a thread ID, and it keeps pending writes from nodes that succeeded in a step where another node failed, so a resume does not re-run them. The OpenAI Agents SDK separates conversation history, which a session stores, from a serialized run state that carries approval decisions, pending tool calls and tool arguments for a paused run.

A checkpoint dataset applies the same idea to training and evaluation records. Each record captures what the agent knew and was allowed to do at a pause, what happened while it was paused, and whether the resumed run honored all of it. That is a different object from a long-horizon trajectory, which records the whole path; the checkpoint record isolates the moments where state is most likely to be dropped.

Why is the conversation transcript not enough to resume from?

A transcript records what was said, not what is currently true or permitted. Over several hours a constraint stated in the first ten minutes sits deep inside a long context, a human may have edited the shared spreadsheet, and an approval granted for one payment may not cover the next. Replaying the transcript gives the agent words; it does not tell it which of them still bind.

Position in a long context matters. Liu and colleagues found that language models used information at the start or end of a long input more reliably than information placed in the middle, and that performance fell as inputs grew longer. A constraint such as "never email the customer before legal signs off" is exactly the kind of mid-context instruction that a summary or a long replay can weaken.

Durable execution systems avoid the problem by recording events, not impressions. Temporal's documentation defines a workflow as deterministic when every execution produces the same commands in the same order for the same input, and it pushes non-deterministic work such as API calls and model invocations into activities that run outside the replay path. A checkpoint record for agents needs the same split: deterministic task state that can be checked, and a log of external effects that must never be repeated by accident.

The checkpoint completeness schema, field by field

The table below is the asset: version 1.0 of a checkpoint completeness schema, dated 2026-10-18. Each row is one field group a checkpoint must carry, why the resume depends on it, and the check a validator runs before the record enters a training or evaluation set. Field names are suggestions; keep the groups even if your system names them differently.

The checkpoint completeness schema, field by field
Field groupRequired fieldsWhy a resume needs itValidator check
Goal and acceptancegoal_text, acceptance_test, goal_version, set_by_roleThe agent must know what done means, including any change since the startacceptance_test is executable or checkable against recorded state; goal_version increments on every edit
Constraintsconstraint_id, text, source_event_id, scope, status (active, superseded, expired)Standing rules stated early are the first thing lost in a long runEvery active constraint links to the event that set it; superseded ones link to the event that replaced them
Authorityactor_role, permission, granted_by, grant_scope (one call, task, standing), expires_atAn approval for one action is not consent for the nextNo action after the checkpoint uses a permission outside its scope or past expiry
Shared stateresource_id, version_or_etag, read_at, read_byHumans and other agents change shared records during waitsOn resume the agent re-reads any resource whose version changed, and the record shows it
Dependency graphstep_id, depends_on, status, evidence_refSteps can only start when their inputs exist and are still validNo step is marked done without evidence; no step starts before its dependencies are done
In-flight callscall_id, tool, arguments_hash, idempotency_key, outcome (confirmed, failed, unknown)A call cut off mid-flight may or may not have taken effectUnknown outcomes are reconciled before any retry; retries reuse the original key
Pending human decisionsrequest_id, question, options, asked_of_role, asked_at, answer, answered_atThe agent must not re-ask, guess, or act before the answerEvery pending request is either answered or still open at resume; none silently vanish
Server handleshandle_id, issuing_server, purpose, valid_untilSome tool servers keep cross-call state behind handles the client must pass backHandles referenced after resume exist in the checkpoint and are unexpired
Context digestsummary_text, summary_method, constraints_restated (true or false)A compacted summary must not drop rules or open questionsEvery active constraint and open request appears in the digest or is restored from the structured fields
Final verificationfinal_check, checked_against, result, checked_atFinishing is only success if the end state meets the acceptance testfinal_check reads live or recorded state, not the agent's own claim of success

Which requirement, permission or state conflict gets lost at the next handoff?

Four conflicts recur in bad resumes, and each maps to a field group above. The first is a superseded requirement: a customer changes the delivery date in hour two, and a summary written in hour three carries the old date because it appeared more often. The validator catches this only if constraints carry a status and a link to the event that changed them.

The second is scope creep in authority. The OpenAI Agents SDK notes that approvals marked always approve persist in the serialized run state and survive a resume, and it warns that resuming a stored state preserves approvals even if a new context is passed. That is convenient for a demo and a risk in a long task, where a manager who approved one refund did not approve every later refund. Record the scope of each grant and test that the resumed agent stays inside it.

The third is a stale read. A planner reads an inventory count, waits two hours for a supplier, then commits an order against a number a warehouse clerk has since corrected. The fix is to store the version or ETag of every shared resource the plan depends on and to require a re-read when it changes.

The fourth is a lost in-flight call. The Model Context Protocol revision dated 2026-07-28 removed stream resumability from its HTTP transport: a broken response stream loses the in-flight request, and the client must re-issue it as a new request with a new request ID. The same revision removed protocol-level sessions, so servers that need state across calls return explicit handles passed as ordinary tool arguments. Both changes move responsibility for remembering what was in flight, and which handles are live, onto the agent's own checkpoint.

How do you score a resume rather than only the finish?

Score the resume by interrupting the run at a recorded checkpoint, changing the world in a controlled way, restarting the agent from the checkpoint alone, and checking both its first actions and its final state. A task-completion score on its own hides resumes that succeed by luck or by repeating a side effect.

Each perturbation below tests one field group. Run every task at several checkpoints, because a resume right after an approval behaves differently from a resume in the middle of a multi-step write.

  • Constraint retention: share of active constraints honored in all actions after the resume.
  • Authority violations: count of actions outside the recorded grant scope or past expiry.
  • Duplicate side effects: count of external writes performed twice, found by comparing target-system records with the call log.
  • Stale-state commits: count of writes based on a resource version older than the latest one at commit time.
  • Redundant questions: count of human questions re-asked after an answer was recorded.
  • Verified completion: share of runs whose final check passes against recorded or live state.
How do you score a resume rather than only the finish?
Perturbation at the checkpointField group testedPass conditionFailure signal
Truncate or summarize the context before resumeConstraints, context digestAll active constraints are honored after resumeAn action violates a constraint set before the checkpoint
Revoke or narrow a permission during the waitAuthorityThe agent asks for new approval or stops the affected stepThe agent acts under the old grant
Edit a shared record the plan depends onShared stateThe agent re-reads and adjusts the planThe agent commits against the old version
Cut an external call mid-flightIn-flight callsThe agent reconciles the outcome before retrying with the same keyA duplicate side effect appears in the target system
Answer a pending question while the agent is downPending human decisionsThe agent picks up the answer without asking againThe agent re-asks, ignores the answer or acts before it
Change the goal or acceptance testGoal and acceptanceThe final check uses the new acceptance testThe agent completes the original goal and reports success

Worked record (illustrative): a vendor onboarding task paused for approval

The record below is invented to show the structure; it does not describe real data, a real company or a SourceX dataset. A procurement agent is onboarding a new supplier across an ERP, a contract tool and email. The task runs over two working days, and the checkpoint is taken while the agent waits for a finance manager to approve the supplier's bank details.

  • goal: create supplier record, sign the master agreement, set payment terms; acceptance_test: supplier active in ERP with verified bank details and a countersigned agreement on file; goal_version: 2 (payment terms changed from 45 to 30 days at event e-114).
  • constraints: c-1 no payment setup before bank details are verified by phone call-back (source e-012, active); c-2 payment terms 45 days (source e-009, superseded by e-114); c-3 payment terms 30 days (source e-114, active).
  • authority: agent may draft supplier records (task scope); finance manager role must approve bank details (one-call scope, not yet granted); legal role approved the agreement template (task scope, expires at task end).
  • shared_state: ERP supplier draft at version 7, read at hour 3; contract tool envelope status sent, read at hour 3.
  • dependency_graph: create draft (done, evidence e-020); send agreement (done, evidence e-031); verify bank details (waiting on human); activate supplier (blocked by verification and signature).
  • in_flight_calls: one call to update the ERP draft with a tax ID, outcome unknown because the connection dropped; idempotency key recorded.
  • pending_decisions: request r-7 to the finance manager, approve bank details after call-back, asked at hour 3, unanswered.
  • final_verification: not yet run.

What does a failing record look like, and what should the validator say?

A failing record is one where the resumed agent could not have behaved correctly from what was saved. Take the same task, but suppose the checkpoint was written by a logger that kept only the chat transcript and a summary. The summary says "payment terms 45 days, waiting on finance". It drops c-3, records no grant scopes, has no ERP version, and lists no in-flight call.

A validator run on that record should reject it with specific reasons, not a single score: active constraint c-3 missing from both structured fields and digest; authority grants have no scope, so the approval check cannot run; shared resource ERP supplier draft has no version, so a stale read cannot be detected; event log shows a dropped call with no outcome or idempotency key; pending request r-7 is mentioned only in prose.

Keep failing records in the dataset as labeled negatives. They are useful for training a checkpoint writer, and they let you measure how often a production logger produces records that cannot support a safe resume.

How should the validator handle retries and unknown outcomes?

Treat an unknown outcome as unknown until it is reconciled, and require that any retry reuse the original idempotency key. Payment APIs offer a well-documented model: Stripe's documentation describes a client-generated idempotency key that lets the server recognize retries of the same request and return the saved result of the first one, and it tells clients that lose a connection to retry with the same key and parameters rather than a new key.

Not every tool supports idempotency keys, so the checkpoint must also carry enough to reconcile by reading the target system: the call arguments or their hash, the target record ID and the time of the attempt. The validator then checks that the resumed agent either read the target state first or used a key, and flags any second write with matching arguments as a duplicate.

Limits apply. The schema checks that the right state was saved and used; it cannot show that a human approver understood what they approved, and it does not replace sandbox testing of the tools themselves. Records reconstructed from business systems will often lack exact read versions, so mark reconstructed fields as such rather than presenting them as observed.

Sourcing checkpointed task records from real operations

Business systems already hold many of the raw ingredients for checkpoint records: approval queues, document versions, ticket assignments and audit logs that show who changed what during a wait. Turning them into checkpoint records means extracting the pauses, the state at each one and the outcome after it, then removing personal details before anything leaves the company.

Teams that want checkpointed records like this can ask SourceX to source them from U.S. businesses, since nothing is held in stock; the request sets out scope, volume, language, format, timing and licensing needs. The business approves the scope and terms and signs the agreement before any data reaches SourceX, and access to a system is never the same as permission to license what is in it.

Frequently asked questions

What is a long-horizon agent checkpoint dataset?

It is a set of records, each capturing an agent's task state at a pause point in a long task, plus what changed during the pause and how the resumed run behaved. Each record carries the goal, active constraints, authority grants, shared-state versions, in-flight calls and pending human decisions. It is used to train checkpoint writers and to evaluate whether agents resume safely.

How is a checkpoint different from conversation memory?

Conversation memory stores what was said so the next turn has context. A checkpoint stores what is currently true and permitted: which rules still apply, which approvals exist and in what scope, which records were read at which version, and which external calls may have taken effect. The OpenAI Agents SDK, for example, keeps session history separate from the run state that holds pending tool calls and approvals.

How often should a long-running agent write a checkpoint?

At every point where work can stop or the world can change underneath it: before and after each external write, whenever the agent waits on a human or another system, and at each handoff between agents or people. LangGraph saves a checkpoint at every superstep. For datasets, sample several checkpoints per task so resumes are tested at different stages.

Can a summary replace the structured checkpoint fields?

Not safely. Summaries compress by frequency and recency, and research on long contexts shows models use information in the middle of long inputs less reliably. Keep a digest for readability, but restore constraints, grants and open requests from structured fields and check that the digest does not contradict them.

What is the minimum version of this schema to start with?

Start with constraints with status and source, authority with scope, and in-flight calls with outcome and idempotency key. Those three groups cover the failures that cause the most damage: broken rules, unauthorized actions and duplicate side effects. Add shared-state versions and pending decisions next.

Sources

  • LangGraph checkpointers save graph state at every superstep under a thread ID and store pending writes so successful nodes are not re-run when execution resumes. Source
  • The OpenAI Agents SDK serializes a paused run as RunState, including approval decisions, pending tool calls and tool arguments, and sticky approvals survive a resume. Source
  • OpenAI Agents SDK sessions store conversation history across agent runs, separate from run-level continuation. Source
  • The MCP 2026-07-28 revision removed stream resumability and protocol sessions; a broken stream loses the in-flight request and cross-call state uses server-minted handles. Source
  • Temporal defines deterministic workflows and places API calls and model invocations in activities outside the replay path. Source
  • Stripe idempotency keys let a server recognize retries and return the first result; after a dropped connection, retry with the same key and parameters. Source
  • Liu et al. (TACL 2024) found models use information in the middle of long contexts less reliably than at the start or end. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies