AI uses for records
RL environment observation schema: keeping business context intact
By SourceX Editorial · Updated
10 sections
Short answer
An RL environment observation schema should preserve what a competent employee would know at that step: which case it is, its status and history as of now, open commitments, the rule version in force, the actions the role may take and where each value came from. Write it as a versioned field-level contract and test it for reset determinism, replay equality and answer leaks.
Grader rubric
RL environment observation
Key takeaways
- The observation holds only what the role could see at that step; outcomes known later go to hidden state for the grader.
- A field-level contract names each field's type, business meaning, source, as-of rule and the failure caused by its absence.
- Fill as_of from event time, join the rule version in force and record who set each status, person or automation.
- Version the contract like an API and never reuse a removed field name with a new meaning.
- Acceptance tests should pin seeds, schema version and policy version, and separate truncated episodes from completed cases.
What should an observation schema preserve about business context?
It should preserve what a competent employee would know at that exact moment of the task: which case they are looking at, its status and history up to now, the commitments and deadlines attached to it, the rule version in force, what their role lets them do next, and where every value came from. Anything outside that set is either hidden state the grader uses or noise.
Most environment builders start from a record export and flatten it into a text blob or a wide row. That keeps the values but loses the context around them: whether a date is a promise or a timestamp, whether a status was set by a person or by an automation, whether a refund limit applied to this customer last month or only today. An agent trained on the flattened view learns from a world in which those distinctions do not exist.
The fix is a written contract: one row per observation field, stating its type, its business meaning, the system and as-of rule that fill it, and what breaks if it is missing. The contract is versioned like an API, because every stored episode and every acceptance result depends on which version produced it.
Observation, hidden state or info: where does each field belong?
Put a field in the observation only if the person doing the job could see it at that step; put ground truth the grader needs in hidden state; put debugging and per-term reward detail in a separate info channel. Mixing these is an easy way for an environment to leak the answer quietly.
Gymnasium's API makes the split explicit. Every valid observation must sit inside the environment's declared observation_space, reset returns an initial observation plus an info dictionary, and step returns the next observation, a reward, separate terminated and truncated flags, and info. The documentation says info may carry variables that are hidden from observations and individual reward terms, which is exactly where grader-only business facts such as the eventual claim outcome belong.
DeepMind's dm_env interface adds a useful discipline for episodes: each time step carries a step_type of FIRST, MID or LAST, and the documentation stresses that the discount does not decide when a sequence ends. For business tasks this means the observation should say plainly whether the case is at its start, in progress or closed, rather than letting the agent infer it from a status string.
| Kind of value | Example from a service or claims workflow | Where it goes | Why |
|---|---|---|---|
| Visible to the role at this step | Ticket status, customer tier, open SLA deadline | Observation | A person in the role would act on it |
| Known only after the case closes | Final resolution code, whether the customer returned | Hidden state, grader only | Showing it hands the agent the answer |
| Visible only after a tool call | Inventory on hand, prior invoices for the account | Observation after the call, not before | Realism: the employee had to look it up |
| Diagnostic detail | Which reward terms fired, extraction warnings | Info channel | Useful for debugging, not for acting |
| Rule text and version | Refund policy v7 effective at case time | Observation (reference plus version) | Correct action depends on the rule in force |
| Personal identifiers | Name, phone, street address | Removed or replaced with stable tokens | Not needed for the decision and a privacy risk |
The field-level observation contract
Use the table below as the starting contract and delete rows your task does not need rather than adding free-text catch-alls. Each row names the failure you will see in training or evaluation if the field is missing, which gives reviewers a concrete reason to keep or cut it.
At a glance
The field-level observation contract
Field names are illustrative; keep your own system's naming if it is stable, and record the source field path so a second team can rebuild the observation from the raw export.
| Field | Type and constraint | Business context it preserves | Source and as-of rule | Failure if missing |
|---|---|---|---|---|
| obs_schema_version | String, semantic version, required | Which contract produced this observation | Set by the environment build | Episodes from different versions are pooled silently |
| step_type and step_index | Enum FIRST, MID, LAST plus integer | Where the case is in its life | Environment loop | Agent cannot tell a fresh case from a closed one |
| as_of | RFC 3339 date-time with offset | The moment the employee is standing in | Event time of the step, not export time | Future information leaks into the past |
| case_ref | Record type plus opaque stable token | Which object the work is about, linkable across tools | Source primary key, tokenized | Tool results cannot be tied back to the case |
| case_status | Enum from the source status list, with set_by person or automation | Who moved the case and to what | Status history as of as_of | Automated closures read as human decisions |
| case_history | Ordered list, each item actor role, action, time; capped window stated | Prior steps the employee would see in the record | Activity log filtered to events before as_of | Agent repeats work or contradicts earlier commitments |
| open_commitments | List of promise, due time, owner role | Deadlines and promises already made to the customer | SLA and task fields as of as_of | Agent optimizes speed and breaks promises |
| policy_context | Rule set id plus version plus effective_from | The rule actually in force for this case | Rule history joined on as_of | Agent is graded against rules that did not apply |
| allowed_actions | List of tool names enabled for this role and state | Authority limits and approvals required | Role permissions plus case state | Agent learns actions no employee could take |
| provenance | Per field source system, field path, extracted_at, transform id | Where each value came from | Extraction pipeline | Nobody can audit or rebuild the observation |
Which encoding choices keep context from leaking or vanishing?
Prefer typed, required fields over prose, and make every enumerated value come from the source system's own list. A validator should reject an observation that is missing a required field or carries an unknown status, instead of letting it through as an empty string.
JSON Schema 2020-12 covers most of this directly. The required keyword lists properties that must be present, enum restricts a value to a fixed set, the date-time format follows RFC 3339, readOnly marks values owned by another authority, and deprecated tells consumers to stop relying on a property that may later be removed. Gymnasium offers the same ideas inside an environment: Dict spaces for named fields, Text for strings such as mission details, and Sequence for a variable number of items such as a case history.
- Keep business meaning in field names and enums, not in position. A history item is an object with actor_role, action and time, not the third element of a tuple.
- Store tool results the way the tool returned them. The Model Context Protocol lets a tool declare an outputSchema; servers must return structuredContent that conforms to it and clients should validate it. Keep that structure in the observation rather than re-summarizing it.
- Treat a failed tool call as context, not an exception. MCP reports business logic and API failures inside the tool result with isError set to true; an observation that drops these teaches the agent that tools never refuse.
- State the history window. If case_history is capped at the last N events, put N in the contract and in the schema version so a reviewer knows what the agent could not see.
- Separate absence from zero. A null open_commitments list means the source had no SLA field; an empty list means the source had one and nothing was open.
How should the schema and its version history be kept?
Version the observation contract like a public API and keep an append-only history of every change, with the reason and the effect on stored episodes. Without that history, two training runs that look identical can be reading different worlds.
Semantic Versioning gives the bump rules: increment the major version for backward-incompatible changes, the minor version for backward-compatible additions, and the patch version for backward-compatible fixes. Protocol Buffers adds the rule that matters most for stored data: field numbers should never be reused, and a deleted field's number should be reserved so nobody gives it a new meaning later. Apply the same rule to field names in a JSON contract.
| Change | Example | Version bump | Effect on stored episodes |
|---|---|---|---|
| Add an optional field | Add channel to case_ref | Minor | Old episodes stay valid; field absent, not empty |
| Tighten a constraint | case_status enum loses a retired value | Major | Re-validate; episodes with the old value are migrated or quarantined |
| Change meaning, same name | as_of switches from export time to event time | Major, and rename the field | Never reuse the old name; old episodes keep the old meaning |
| Remove a field | Drop free-text notes after a leak review | Major; mark deprecated first | Reserve the name; replay tools must ignore it |
| Fix an extraction bug | Time zone offset was dropped on as_of | Patch | Regenerate affected episodes and log which ones changed |
| Change the history window | Last 20 events becomes last 50 | Minor if additive, recorded in notes | Acceptance results before and after are not comparable |
Five assumptions that flip an observation acceptance result
Five assumptions change pass and fail results even when the agent and the task text are identical: how reset seeds randomness, which clock fills as_of, which rule version is joined, how long the history window is, and whether the episode ended by termination or by a time limit. Fix each one in the contract before running acceptance.
Gymnasium's reset documentation is specific about seeding: passing an integer seed resets the random number generator, while passing no seed to an environment that already has one leaves it untouched. An acceptance suite that resets without an explicit seed is not measuring the same starting states twice. Likewise, Gymnasium separates terminated (the task reached a terminal state) from truncated (a condition outside the task, typically a time limit), and a business acceptance test should never score a truncated episode as a completed case.
- Reset determinism: two resets with the same seed and options produce identical first observations, byte for byte.
- Replay equality: replaying a stored action list from a stored seed reproduces every observation and the final hidden state.
- Leak test: no grader-only field, or a value derivable from one, appears in any observation before the step where the role could know it.
- Solvability: for every task in the suite, at least one sequence of allowed_actions reaches the success state, shown by a recorded reference episode.
- Completion invariants: at LAST, the hidden state satisfies the business rule for done, for example a closed case has a resolution code and no open commitments past due.
- Version pinning: every acceptance report names obs_schema_version, policy version and environment build.
Worked example (illustrative): a warranty claim review environment
Consider an illustrative environment in which an agent reviews warranty claims for a field service company. The agent's role can approve, deny or request more evidence; approvals above a set threshold require a supervisor. The observation at the second step shows case_ref as a tokenized claim id, case_status as submitted and set_by as person, a case_history of three prior events, one open commitment (respond within 5 business days), policy_context of warranty policy version 4 effective from the first of the quarter, and allowed_actions of request_evidence, approve and deny, but not escalate_override.
The final outcome (whether the repair was later found to be covered) sits in hidden state. The grader compares the agent's decision to that outcome and to the policy version in force, not to the latest policy.
During review the team found three problems, recorded in the version history: as_of had been filled from export time, so the history showed a technician visit that happened after the decision point; a status of auto-closed was indistinguishable from a person closing the claim; and free-text notes contained the adjuster's eventual decision. Each fix produced a version bump and a regeneration of affected episodes. All counts and versions here are illustrative.
| Version | Change | Reason | Episodes affected |
|---|---|---|---|
| 1.0.0 | Initial contract, 10 fields | First build | All, as generated |
| 2.0.0 | as_of moved from export time to event time | Leak test found future events in history | All regenerated |
| 2.1.0 | case_status gains set_by person or automation | Auto-closed claims read as human denials | Old episodes valid, field absent |
| 3.0.0 | Free-text notes removed, name reserved | Notes contained the final decision | All regenerated; notes never reused |
| 3.0.1 | Offset restored on as_of for one region | Time zone dropped during extraction | About 4 percent regenerated (illustrative) |
When hiding context is the right design
The exception to preserving context is information the employee had to go and get. If a claims reviewer had to open the parts catalog or call the dealer, put that behind a tool and let the observation show it only after the call; preloading it removes the very skill you want to train.
A second exception is long history. When a case has hundreds of events, a capped window or a structured summary field can be better than the full log, provided the summary is produced by a stated, versioned rule rather than by a model whose output nobody checks. Record which approach you used in the contract, because it changes what the agent could have known.
Who has to sign the contract besides an engineer?
The person who runs the process in the business should sign off the observation contract, because only they can say whether a field reflects what the role really sees and whether a rule version was really in force. An RL engineer can verify types, seeds and replay; a process owner verifies meaning.
Narrow the scope when the process owner cannot name the rule set that applied to historical cases, when status values were reused with different meanings over the years, or when the role's permissions cannot be reconstructed for the period covered. Those gaps cannot be fixed in the schema, and an environment built on them will grade the agent against a guess.
Records that already carry this context
An observation contract is only as good as the records behind it. Activity logs with actor and time, status histories, SLA fields and dated policy versions are what make as_of, case_history, open_commitments and policy_context fillable; a current-state export alone cannot fill them.
Between the businesses that keep such records and the AI teams that need them, SourceX handles sourcing, rights review, contracting, preparation and delivery. Datasets are sourced from U.S. businesses to a request rather than held in stock, so a buyer describes the scope, volume, format, timing and licensing terms needed, and a contract like the one above is a useful way to state that scope.
Frequently asked questions
What is an observation schema in an RL environment?
It is the definition of what the agent receives at each step: the fields, their types and constraints, and the rules that fill them. In Gymnasium it is expressed as the environment's observation_space, and every observation must be an element of that space. For business tasks, a written field-level contract adds the business meaning and source of each field.
Should the observation include the full record or only selected fields?
Only the fields a person in the role could see at that step, plus provenance. Values known only later, such as a final outcome, belong in hidden state for the grader. Including everything usually leaks the answer and teaches shortcuts.
How do I version an observation schema?
Treat it like an API. Bump the major version for incompatible changes, the minor version for compatible additions and the patch version for fixes, and never reuse a removed field name with a new meaning. Keep a dated history of each change and which stored episodes it affected.
How do I test that the observation does not leak the answer?
Run a leak test that checks every observation for grader-only fields or values derivable from them before the step where the role could know them. Also check that as_of uses event time and that history is filtered to events before that time. Failed checks should block the build, not just warn.
Can free-text notes go into the observation?
Sometimes, but they are a frequent source of leaks because people write later outcomes into earlier notes. If you keep them, filter them by timestamp, review a sample for future information and record the decision in the schema version history.
Sources
- Gymnasium environments declare an observation_space that contains every valid observation; reset returns an observation and info; step returns observation, reward, terminated, truncated and info; info may hold variables hidden from observations; an integer seed resets the RNG while no seed leaves an existing RNG unchanged. Source
- Gymnasium provides Dict spaces for named fields, Text for strings such as mission details and Sequence for a variable number of items. Source
- dm_env time steps carry a step_type of FIRST, MID or LAST, and the discount does not determine when a sequence ends. Source
- JSON Schema 2020-12 defines required, enum, the RFC 3339 date-time format, readOnly and deprecated. Source
- MCP tools may declare an outputSchema; servers must return conforming structuredContent, clients should validate it, and business logic errors are reported in results with isError true. Source
- Protocol Buffers field numbers should never be reused, and deleted field numbers should be reserved. Source
- Semantic Versioning increments major for incompatible changes, minor for backward-compatible additions and patch for backward-compatible fixes. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.