Skip to content

AI uses for records

Enterprise RL environment state design: what belongs in the state

By SourceX Editorial · Updated

9 sections

Short answer

An enterprise RL environment's state should hold everything that decides the next transition and the grader's verdict: the business records in play, each case's lifecycle status, pending messages, a pinned clock, policy and permission versions, approvals, hidden facts the agent must discover, and an episode log. Derive the agent's partial view from that state, and validate every initial state before use.

Grader rubric

Enterprise RL environment

This page covers enterprise RL environment state design: what belongs in the state.Illustrative

Key takeaways

  • If a field changes what an action does or how the grader judges the episode, it belongs in the state.
  • Store the full hidden record and derive the agent's partial view from it, then check that nothing hidden leaks.
  • Eight field groups cover most back-office tasks: records, case status, messages, clock, versions, permissions, hidden facts and bookkeeping.
  • Reuse a published lifecycle, such as Peppol's invoice response statuses, to get transition rules a validator can enforce.
  • Validate every initial state and ship at least one failing example that names the rule it breaks.

What belongs in the state of an enterprise RL environment?

The state should hold everything that determines what happens next and how the episode is judged: the business records the task touches, the process status of each case, pending messages, the clock, the policy and permission versions in force, any approvals, the facts the agent must discover but cannot see, and a log of what has happened in the episode. If a field changes the result of an action or the grader's verdict, it belongs in the state.

A useful test is to ask whether two environments with identical state would always respond identically to the same action. If the answer is no, something that matters lives outside the state, often a live clock, an unpinned policy document, a counterparty simulator's memory or a value computed on the fly from a service the environment does not control. Those are the fields that make results drift.

This page builds the answer as a schema and validator for one concrete task, a supplier invoice under dispute, so each field can be checked against a real business process rather than an abstract definition. It sits next to separate questions about acceptance tests and reset reproducibility, which take a defined state as given.

Why keep hidden state, agent view and grader view apart?

Design the state as the complete hidden record, then derive what the agent sees from it, rather than starting with the screens and working backward. Gymnasium's core environment interface notes that an environment can be partially or fully observed, that reset returns an initial observation, and that each step returns an observation drawn from the declared observation space. In enterprise tasks the observation is almost always partial: the agent sees what a clerk would see through the system's tools, not the database.

Separate three views. The full state is what the simulator stores and the transition logic reads. The agent view is the subset the agent can reach through its tools and messages, filtered by its permissions. The grader view is what the scoring logic reads, which often includes hidden facts such as the quantity actually received. Keeping these apart is what makes it possible to check later that nothing hidden leaked into what the agent saw.

Why keep hidden state, agent view and grader view apart?
ViewContentsWho reads itTypical mistake
Full stateEvery record, status, message, clock value, version and hidden factTransition logic and resetFields stored in a live service instead of the snapshot
Agent viewTool results, inbox messages and documents within the agent's permission scopeThe agentReturning whole records when the tool should return selected fields
Grader viewTarget end conditions, hidden facts and the episode logScoring logicGrader reading a field the schema never documents
Episode logOrdered actions, tool results, messages and status changesGrader, replay and auditLogging only final state, so a correct outcome by a forbidden path passes

Which eight field groups does every enterprise state need?

Group fields by the role they play, not by the source system's tables. The eight groups below cover enterprise back-office tasks well; a team can add groups, but dropping one usually means a dependency has moved outside the state.

Open benchmarks show different ways of pinning these groups down. OSWorld describes each task with a snapshot identifier for the starting environment and a config of setup steps that produce the initial state, plus an evaluator. AgentDojo stores a suite's initial environment as a data file that must parse into the environment's typed model. tau2-bench describes an initial state made of initialization data and initialization actions replayed before the episode. Any of these works if every group below ends up in the pinned definition.

Which eight field groups does every enterprise state need?
Field groupWhat it holdsWhy the environment needs itValidator check
RecordsRows the task reads or writes: invoices, orders, receipts, vendorsTransitions change these; the grader compares themSchema valid; every foreign key resolves
Case statusLifecycle state of each case with reason and requested actionDetermines which actions are legal nextStatus in the allowed set; order rules respected
MessagesInbox, outbox and threads with counterparties, with sender rolesThe agent learns facts and must communicate outcomesEvery message references a known case and party
ClockCurrent business date and time, working-day calendarDeadlines, aging and response windows depend on itFrozen or explicitly offset; never read from the host
Policy and versionsPolicy text version, tool schema version, tolerance settingsSame action can be allowed under one policy and not anotherVersion identifiers present and resolvable
Permissions and approvalsAgent role, granted scopes, pending and granted approvalsLimits what the agent may do and whenNo approval recorded before its request
Hidden factsGround truth the agent must discover, such as the received quantityMakes the task a search for evidence rather than a lookupAbsent from every agent-facing tool result
Episode bookkeepingStep count, action budget, log pointer, termination flagsEnds episodes consistently and supports replayCounters start at zero; log empty at reset

Worked state schema: a disputed supplier invoice (illustrative)

The example below is illustrative, built for this article, and not drawn from any company's records. A buyer's accounts payable team has received an invoice for 120 units against a purchase order for 120 units, but the goods receipt shows 100 units. The buyer has put the invoice under query with the supplier. The agent's task is to resolve the dispute within policy: confirm the received quantity, request the right correction from the supplier and move the invoice to the correct status.

Each row gives a field, its value at reset, and whether the agent can see it. Values are counts, codes and dates only.

Worked state schema: a disputed supplier invoice (illustrative)
FieldValue at resetAgent can see itPurpose
invoice.id, linesINV-1047; 1 line, item A, quantity 120Yes, via invoice lookupThe disputed record
po.linesPO-5521; item A, quantity 120Yes, via order lookupReference for what was ordered
receipt.linesGR-3308; item A, quantity 100Yes, via receipt lookupEvidence that conflicts with the invoice
case.statusUQ (under query), first response sent on business day 2YesConstrains next legal statuses
case.reason, case.actionQTY (quantity incorrect); action not yet requestedYesThe agent must choose the correct requested action
thread.messages2 messages: buyer query, supplier reply asking which quantity was receivedYes, via inboxThe agent must answer with evidence
clockBusiness day 4, calendar version 2026-cal-1Yes, as current date onlyResponse windows and aging
policy.version, toleranceap-policy-v3; quantity tolerance 0 unitsPolicy text yes, version id noDefines whether 20 units over is acceptable
permissionsRole ap-clerk: read all, write case status and messages, no payment releaseIndirectly, via tool errorsBounds the action space
hidden.truthReceived quantity 100; supplier will issue a partial credit if asked with receipt evidenceNoGrader target and simulator behavior

Why borrow the dispute lifecycle from a published invoice response standard?

Do not invent the dispute lifecycle; reuse one that a standards body has already defined. The Peppol invoice response specification, published by OpenPEPPOL, sets a fixed list of statuses a buyer reports to a seller: message acknowledged, in process, under query, conditionally accepted, rejected, accepted, and paid, with partial payment marked by a clarification code. Statuses must advance in a defined order, though any may come first and any may be skipped, and under query may repeat before the case moves on.

The specification also defines which statuses need a clarification, and its code lists separate reasons from requested actions. Reason codes include quantity incorrect, prices incorrect and references incorrect; action codes include provide information, issue new invoice, credit fully and credit partially. Rejected and fully paid are final. That gives the environment a ready-made transition table and a set of invariants that a validator can check mechanically.

It also supplies a clock rule worth modeling. The specification says the seller should receive a first response within 3 working days, which is why the worked state records a business calendar rather than wall-clock time. Using a published lifecycle has a second benefit: an agent trained on it learns a process that real buyers and sellers recognize, not one invented for the benchmark.

Which checks must a state pass before an episode starts?

Run the validator on every initial state and after every step in reference runs. Each check is deterministic and produces a named failure, so a broken state is rejected before any agent sees it, rather than discovered later as an unexplained low score. The checks below follow from the field groups and the lifecycle rules.

At a glance

Which checks must a state pass before an episode starts

  • Schema: every record parses against the declared types, and required fields are present.
  • Referential integrity: invoice lines reference an existing order line and item; every receipt references an order.
  • Lifecycle order: the case status history never moves backward in the specification's sequence, except that under query may repeat.
  • Clarification rule: a status that requires a clarification carries at least one reason code or reason text.
  • Finality: no status change follows a final status such as rejected or fully paid.
  • Hidden-field isolation: no hidden fact appears in any tool result, document or message the agent can reach at reset.
  • Solvability: a reference sequence of actions from this state reaches a state that satisfies the grader; tau2-bench derives its target database state by replaying reference actions on a fresh environment, and the same idea works here.
  • Determinism: two resets from the same specification produce byte-identical full states and the same first observation.

What does a failing state look like, and what does the validator report?

This failing state is also illustrative. It differs from the valid one in four fields, and each difference is a common authoring error. The validator rejects it before an episode starts.

What does a failing state look like, and what does the validator report?
FieldFaulty valueValidator resultFix
case.status historyUQ, then AP, then UQLifecycle order: status moved backward from accepted to under queryRemove the later query or model a new case
case.reasonStatus UQ with no reason code or textClarification rule: under query requires a clarificationAdd QTY with a short reason text
receipt lookup resultTool returns the receipt with a note quoting the supplier's planned creditHidden-field isolation: simulator behavior leaked into an agent-facing resultStrip the note; keep the planned credit only in hidden state
po.linesLine references item B while invoice and receipt reference item AReferential integrity: invoice line has no matching order lineCorrect the order line or change the task to an item mismatch

Which environment assumptions change what the grader concludes?

Four assumptions change outcomes most often, and each must be recorded in the state rather than left to the harness. The first is the clock: if the business date moves with the host, a run on a Monday and a run on a Friday see different response windows. The second is the policy version: changing the quantity tolerance from zero to a small allowance turns a correct dispute into an unnecessary one.

The third is the counterparty simulator: if the supplier's behavior is a model, its version, prompt and sampling settings belong in the state, along with any memory it keeps between messages. The fourth is the permission profile: an agent allowed to release payments faces a different task from one that is not, even with identical records.

Treat a change to any of these as a new environment version. Report results against that version, and when two versions disagree, compare their state definitions field by field before blaming the agent.

Where do believable dispute histories come from?

Synthetic records satisfy the schema but rarely capture how disputes actually unfold: partial receipts, replies that answer a different question, credit notes that arrive after the invoice was accepted, and policies that changed mid-quarter. De-identified dispute histories from real accounts payable and receivable systems give a state generator the distributions that make episodes believable, and the business must have the right to license them before they are used.

Such histories are found per request, not pulled from a shelf: SourceX locates them at U.S. businesses once an environment builder has set out the scope, volume, language, format, timing and licensing terms on the request-data form, and it then takes the holding business through qualification, rights review, contracting, preparation and delivery.

Frequently asked questions

What is the difference between state and observation in an RL environment?

The state is the complete record the environment uses to decide what happens next and how an episode is judged. The observation is what the agent receives after reset or each step. Enterprise environments are usually partially observed, so the agent sees tool results and messages, not the full database.

Should hidden ground truth be part of the state?

Yes. Facts the agent must discover, such as the quantity actually received, belong in the state so transitions and the grader can use them. They must never appear in any tool result or message the agent can reach at reset, and a validator should check that.

Why record the clock and policy version in the state?

Deadlines, aging and response windows depend on the business date, and the same action can be allowed or forbidden depending on policy. If either is read from outside the state, two runs from the same start can diverge, and a grader cannot tell an agent error from an environment change.

What valid and invalid examples should a state schema include?

At least one valid initial state that passes every validator check and one or more failing states that each break a specific rule, such as a backward status change, a missing clarification, a leaked hidden field or a broken reference. Each failing example should name the check it fails and the fix.

Can a published business standard define the state machine?

Often, yes. The Peppol invoice response specification defines invoice statuses, their order, which ones need clarification and which are final, with separate code lists for reasons and requested actions. Reusing it gives an environment tested transition rules and makes the trained behavior recognizable to real businesses.

Sources

  • Gymnasium's environment interface says an environment can be partially or fully observed, that reset returns an initial observation and that each step returns an observation from the declared observation space. Source
  • OSWorld describes each task with a snapshot identifier, a config of setup steps that produce the initial state, and an evaluator. Source
  • AgentDojo defines a suite's initial environment state in a data file that must parse as the environment's typed model. Source
  • tau2-bench defines a task's initial state as initialization data and actions, and derives the target database end state by replaying reference actions on a fresh environment. Source
  • The Peppol invoice response specification fixes seven status codes, their order and which are final, lets under query repeat, and expects a first response within 3 working days. Source
  • Peppol's status reason code list includes quantity incorrect, prices incorrect and references incorrect. Source
  • Peppol's status action code list includes provide information, issue new invoice, credit fully and credit partially. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies