AI uses for records
Enterprise RL environment state design: what belongs in the state
By SourceX Editorial · Updated
9 sections
Short answer
An enterprise RL environment's state should hold everything that decides the next transition and the grader's verdict: the business records in play, each case's lifecycle status, pending messages, a pinned clock, policy and permission versions, approvals, hidden facts the agent must discover, and an episode log. Derive the agent's partial view from that state, and validate every initial state before use.
Grader rubric
Enterprise RL environment
Key takeaways
- If a field changes what an action does or how the grader judges the episode, it belongs in the state.
- Store the full hidden record and derive the agent's partial view from it, then check that nothing hidden leaks.
- Eight field groups cover most back-office tasks: records, case status, messages, clock, versions, permissions, hidden facts and bookkeeping.
- Reuse a published lifecycle, such as Peppol's invoice response statuses, to get transition rules a validator can enforce.
- Validate every initial state and ship at least one failing example that names the rule it breaks.
What belongs in the state of an enterprise RL environment?
The state should hold everything that determines what happens next and how the episode is judged: the business records the task touches, the process status of each case, pending messages, the clock, the policy and permission versions in force, any approvals, the facts the agent must discover but cannot see, and a log of what has happened in the episode. If a field changes the result of an action or the grader's verdict, it belongs in the state.
A useful test is to ask whether two environments with identical state would always respond identically to the same action. If the answer is no, something that matters lives outside the state, often a live clock, an unpinned policy document, a counterparty simulator's memory or a value computed on the fly from a service the environment does not control. Those are the fields that make results drift.
This page builds the answer as a schema and validator for one concrete task, a supplier invoice under dispute, so each field can be checked against a real business process rather than an abstract definition. It sits next to separate questions about acceptance tests and reset reproducibility, which take a defined state as given.
Why keep hidden state, agent view and grader view apart?
Design the state as the complete hidden record, then derive what the agent sees from it, rather than starting with the screens and working backward. Gymnasium's core environment interface notes that an environment can be partially or fully observed, that reset returns an initial observation, and that each step returns an observation drawn from the declared observation space. In enterprise tasks the observation is almost always partial: the agent sees what a clerk would see through the system's tools, not the database.
Separate three views. The full state is what the simulator stores and the transition logic reads. The agent view is the subset the agent can reach through its tools and messages, filtered by its permissions. The grader view is what the scoring logic reads, which often includes hidden facts such as the quantity actually received. Keeping these apart is what makes it possible to check later that nothing hidden leaked into what the agent saw.
| View | Contents | Who reads it | Typical mistake |
|---|---|---|---|
| Full state | Every record, status, message, clock value, version and hidden fact | Transition logic and reset | Fields stored in a live service instead of the snapshot |
| Agent view | Tool results, inbox messages and documents within the agent's permission scope | The agent | Returning whole records when the tool should return selected fields |
| Grader view | Target end conditions, hidden facts and the episode log | Scoring logic | Grader reading a field the schema never documents |
| Episode log | Ordered actions, tool results, messages and status changes | Grader, replay and audit | Logging only final state, so a correct outcome by a forbidden path passes |
Which eight field groups does every enterprise state need?
Group fields by the role they play, not by the source system's tables. The eight groups below cover enterprise back-office tasks well; a team can add groups, but dropping one usually means a dependency has moved outside the state.
Open benchmarks show different ways of pinning these groups down. OSWorld describes each task with a snapshot identifier for the starting environment and a config of setup steps that produce the initial state, plus an evaluator. AgentDojo stores a suite's initial environment as a data file that must parse into the environment's typed model. tau2-bench describes an initial state made of initialization data and initialization actions replayed before the episode. Any of these works if every group below ends up in the pinned definition.
| Field group | What it holds | Why the environment needs it | Validator check |
|---|---|---|---|
| Records | Rows the task reads or writes: invoices, orders, receipts, vendors | Transitions change these; the grader compares them | Schema valid; every foreign key resolves |
| Case status | Lifecycle state of each case with reason and requested action | Determines which actions are legal next | Status in the allowed set; order rules respected |
| Messages | Inbox, outbox and threads with counterparties, with sender roles | The agent learns facts and must communicate outcomes | Every message references a known case and party |
| Clock | Current business date and time, working-day calendar | Deadlines, aging and response windows depend on it | Frozen or explicitly offset; never read from the host |
| Policy and versions | Policy text version, tool schema version, tolerance settings | Same action can be allowed under one policy and not another | Version identifiers present and resolvable |
| Permissions and approvals | Agent role, granted scopes, pending and granted approvals | Limits what the agent may do and when | No approval recorded before its request |
| Hidden facts | Ground truth the agent must discover, such as the received quantity | Makes the task a search for evidence rather than a lookup | Absent from every agent-facing tool result |
| Episode bookkeeping | Step count, action budget, log pointer, termination flags | Ends episodes consistently and supports replay | Counters start at zero; log empty at reset |
Worked state schema: a disputed supplier invoice (illustrative)
The example below is illustrative, built for this article, and not drawn from any company's records. A buyer's accounts payable team has received an invoice for 120 units against a purchase order for 120 units, but the goods receipt shows 100 units. The buyer has put the invoice under query with the supplier. The agent's task is to resolve the dispute within policy: confirm the received quantity, request the right correction from the supplier and move the invoice to the correct status.
Each row gives a field, its value at reset, and whether the agent can see it. Values are counts, codes and dates only.
| Field | Value at reset | Agent can see it | Purpose |
|---|---|---|---|
| invoice.id, lines | INV-1047; 1 line, item A, quantity 120 | Yes, via invoice lookup | The disputed record |
| po.lines | PO-5521; item A, quantity 120 | Yes, via order lookup | Reference for what was ordered |
| receipt.lines | GR-3308; item A, quantity 100 | Yes, via receipt lookup | Evidence that conflicts with the invoice |
| case.status | UQ (under query), first response sent on business day 2 | Yes | Constrains next legal statuses |
| case.reason, case.action | QTY (quantity incorrect); action not yet requested | Yes | The agent must choose the correct requested action |
| thread.messages | 2 messages: buyer query, supplier reply asking which quantity was received | Yes, via inbox | The agent must answer with evidence |
| clock | Business day 4, calendar version 2026-cal-1 | Yes, as current date only | Response windows and aging |
| policy.version, tolerance | ap-policy-v3; quantity tolerance 0 units | Policy text yes, version id no | Defines whether 20 units over is acceptable |
| permissions | Role ap-clerk: read all, write case status and messages, no payment release | Indirectly, via tool errors | Bounds the action space |
| hidden.truth | Received quantity 100; supplier will issue a partial credit if asked with receipt evidence | No | Grader target and simulator behavior |
Why borrow the dispute lifecycle from a published invoice response standard?
Do not invent the dispute lifecycle; reuse one that a standards body has already defined. The Peppol invoice response specification, published by OpenPEPPOL, sets a fixed list of statuses a buyer reports to a seller: message acknowledged, in process, under query, conditionally accepted, rejected, accepted, and paid, with partial payment marked by a clarification code. Statuses must advance in a defined order, though any may come first and any may be skipped, and under query may repeat before the case moves on.
The specification also defines which statuses need a clarification, and its code lists separate reasons from requested actions. Reason codes include quantity incorrect, prices incorrect and references incorrect; action codes include provide information, issue new invoice, credit fully and credit partially. Rejected and fully paid are final. That gives the environment a ready-made transition table and a set of invariants that a validator can check mechanically.
It also supplies a clock rule worth modeling. The specification says the seller should receive a first response within 3 working days, which is why the worked state records a business calendar rather than wall-clock time. Using a published lifecycle has a second benefit: an agent trained on it learns a process that real buyers and sellers recognize, not one invented for the benchmark.
Which checks must a state pass before an episode starts?
Run the validator on every initial state and after every step in reference runs. Each check is deterministic and produces a named failure, so a broken state is rejected before any agent sees it, rather than discovered later as an unexplained low score. The checks below follow from the field groups and the lifecycle rules.
At a glance
Which checks must a state pass before an episode starts
- Schema: every record parses against the declared types, and required fields are present.
- Referential integrity: invoice lines reference an existing order line and item; every receipt references an order.
- Lifecycle order: the case status history never moves backward in the specification's sequence, except that under query may repeat.
- Clarification rule: a status that requires a clarification carries at least one reason code or reason text.
- Finality: no status change follows a final status such as rejected or fully paid.
- Hidden-field isolation: no hidden fact appears in any tool result, document or message the agent can reach at reset.
- Solvability: a reference sequence of actions from this state reaches a state that satisfies the grader; tau2-bench derives its target database state by replaying reference actions on a fresh environment, and the same idea works here.
- Determinism: two resets from the same specification produce byte-identical full states and the same first observation.
What does a failing state look like, and what does the validator report?
This failing state is also illustrative. It differs from the valid one in four fields, and each difference is a common authoring error. The validator rejects it before an episode starts.
| Field | Faulty value | Validator result | Fix |
|---|---|---|---|
| case.status history | UQ, then AP, then UQ | Lifecycle order: status moved backward from accepted to under query | Remove the later query or model a new case |
| case.reason | Status UQ with no reason code or text | Clarification rule: under query requires a clarification | Add QTY with a short reason text |
| receipt lookup result | Tool returns the receipt with a note quoting the supplier's planned credit | Hidden-field isolation: simulator behavior leaked into an agent-facing result | Strip the note; keep the planned credit only in hidden state |
| po.lines | Line references item B while invoice and receipt reference item A | Referential integrity: invoice line has no matching order line | Correct the order line or change the task to an item mismatch |
Which environment assumptions change what the grader concludes?
Four assumptions change outcomes most often, and each must be recorded in the state rather than left to the harness. The first is the clock: if the business date moves with the host, a run on a Monday and a run on a Friday see different response windows. The second is the policy version: changing the quantity tolerance from zero to a small allowance turns a correct dispute into an unnecessary one.
The third is the counterparty simulator: if the supplier's behavior is a model, its version, prompt and sampling settings belong in the state, along with any memory it keeps between messages. The fourth is the permission profile: an agent allowed to release payments faces a different task from one that is not, even with identical records.
Treat a change to any of these as a new environment version. Report results against that version, and when two versions disagree, compare their state definitions field by field before blaming the agent.
Where do believable dispute histories come from?
Synthetic records satisfy the schema but rarely capture how disputes actually unfold: partial receipts, replies that answer a different question, credit notes that arrive after the invoice was accepted, and policies that changed mid-quarter. De-identified dispute histories from real accounts payable and receivable systems give a state generator the distributions that make episodes believable, and the business must have the right to license them before they are used.
Such histories are found per request, not pulled from a shelf: SourceX locates them at U.S. businesses once an environment builder has set out the scope, volume, language, format, timing and licensing terms on the request-data form, and it then takes the holding business through qualification, rights review, contracting, preparation and delivery.
Frequently asked questions
What is the difference between state and observation in an RL environment?
The state is the complete record the environment uses to decide what happens next and how an episode is judged. The observation is what the agent receives after reset or each step. Enterprise environments are usually partially observed, so the agent sees tool results and messages, not the full database.
Should hidden ground truth be part of the state?
Yes. Facts the agent must discover, such as the quantity actually received, belong in the state so transitions and the grader can use them. They must never appear in any tool result or message the agent can reach at reset, and a validator should check that.
Why record the clock and policy version in the state?
Deadlines, aging and response windows depend on the business date, and the same action can be allowed or forbidden depending on policy. If either is read from outside the state, two runs from the same start can diverge, and a grader cannot tell an agent error from an environment change.
What valid and invalid examples should a state schema include?
At least one valid initial state that passes every validator check and one or more failing states that each break a specific rule, such as a backward status change, a missing clarification, a leaked hidden field or a broken reference. Each failing example should name the check it fails and the fix.
Can a published business standard define the state machine?
Often, yes. The Peppol invoice response specification defines invoice statuses, their order, which ones need clarification and which are final, with separate code lists for reasons and requested actions. Reusing it gives an environment tested transition rules and makes the trained behavior recognizable to real businesses.
Sources
- Gymnasium's environment interface says an environment can be partially or fully observed, that reset returns an initial observation and that each step returns an observation from the declared observation space. Source
- OSWorld describes each task with a snapshot identifier, a config of setup steps that produce the initial state, and an evaluator. Source
- AgentDojo defines a suite's initial environment state in a data file that must parse as the environment's typed model. Source
- tau2-bench defines a task's initial state as initialization data and actions, and derives the target database end state by replaying reference actions on a fresh environment. Source
- The Peppol invoice response specification fixes seven status codes, their order and which are final, lets under query repeat, and expects a first response within 3 working days. Source
- Peppol's status reason code list includes quantity incorrect, prices incorrect and references incorrect. Source
- Peppol's status action code list includes provide information, issue new invoice, credit fully and credit partially. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.