Insights
AI uses for records
531 guides · page 9 of 12
Policy change agent benchmark: testing agents when rules change mid-task
A policy change agent benchmark spec: ten task families, a task schema, grading, twin controls and pre-registration for rules that change mid-task. No results.
How to build a post-training data difficulty curriculum for business tasks
Build a post-training data difficulty curriculum for business tasks: a ladder of expert-validated prerequisites, and a test of whether ordering helps.
Post-training data generalization: did the model learn the workflow or one company's rules?
Test post-training data generalization by holding out whole companies, tagging shared steps versus local rules, and measuring policy bleed and regressions.
Post-training dataset format conversion without losing meaning
Post-training dataset format conversion: a transformation contract and source fidelity checks that keep business records' meaning intact in training examples.
Post-training dataset refresh triggers after a business change
Post-training dataset refresh triggers: map each business change to the examples it makes stale, then prove the refresh fixed the failure without regressions.
Post-training instruction fidelity: testing requirement retention
Post-training instruction fidelity for agents: a requirement tracing record and grading rubric to test whether training data fixes dropped requirements.
Post-training negative transfer: can new data make an agent worse?
Post-training negative transfer, tested: freeze the agent's old job, compare budget-matched arms over repeated trials and trace regressions to the data slice.
Preference data versus verifiable rewards for business workflows
Preference data versus verifiable rewards: a decision tree with matched business examples, grader shortcut tests and an illustrative invoice exception case.
Private benchmark access control: who can see the test set before launch
Private benchmark access control before launch: who sees which split and which scores, how to split by company and time, and when items can be released.
Process reward in business workflows: when step scores help agents and when they mislead
Process reward in business workflows: when step-level rewards help enterprise agents, when they reward shortcuts, and an ablation plan to tell the two apart.
Procurement approval delegation: can an agent find a valid delegate?
A procurement approval delegation dataset guide: seven tests for a valid delegate when an approver is away, a dated task schema and ten illustrative cases.
Production bug evidence packet: linking issue, patch and outcome
A production bug evidence packet links the error, issue, starting state, failing test, reviewed patch, deployment and measured outcome so a fix can be verified.
Professional association administrative data: membership exceptions as reasoning tasks
How professional association administrative data turns membership exceptions into policy and evidence reasoning tasks, with a rights-screened task brief.
RAG access control evaluation data: grading retrieval per user
RAG access control evaluation data: a task specification for grading retrieval per user, with scoped gold evidence, forbidden documents and abstention checks.
RAG answerability dataset: when the business archive cannot answer
A RAG answerability dataset for business questions: label tasks by asker, date and snapshot, code the missing evidence, and score answers and declines apart.
How should a RAG evaluation handle a document that changes the answer?
RAG corpus update regression testing: find the gold answers a new document invalidates, re-judge them on authority and dates, and score the run before release.
RAG document absence verification: testing a no-match claim
RAG document absence verification: how to test a claim that no matching record exists, with planted matches, exhaustive ground truth and boundary scoring.
RAG evidence completeness dataset: testing answers that need more than one chunk
How to build a RAG evidence completeness dataset: label the minimal evidence set per question, add insufficient variants and score group recall and abstention.
RAG metadata quality ablation: how to measure what better metadata buys
A RAG metadata quality ablation protocol: matched arms, real-work queries, authority and version metrics, paired statistics and a pre-registration checklist.
RAG source authority dataset: labeling relevant vs authoritative documents
How to build a RAG source authority dataset: label relevance and authority apart, add hard negatives, and score whether answers cite the controlling source.
RAG source citation location datasets: scoring the line or region cited
How to build a RAG source citation location dataset: anchor types per evidence form, a gold record schema and six tests for where a citation points.
RAG spreadsheet evidence retrieval: return the formula and its inputs
RAG spreadsheet evidence retrieval that returns the formula, its inputs and the workbook version, with a task packet, extraction tips and metrics.
RAG table retrieval training data: linking each row to its column definitions
RAG table retrieval training data: a schema that links each retrieved row to its versioned column definitions, with validator checks and hard negatives.
Retail catalog correction dataset: which open orders does a fix affect?
A retail catalog correction dataset that links each product attribute fix to the open orders it affects, with a benchmark brief, labels and a worked case.
Retail promotion evaluation holdout: a protocol for unseen promotions
A retail promotion evaluation holdout protocol: group tasks by promotion family, hold whole families out and measure the seen versus unseen gap. No results.
Retrieval benchmark entity ambiguity: two vendors, nearly one name
Retrieval benchmark entity ambiguity: how to test whether RAG finds the right vendor when two names nearly match, with a task set, worked case and scoring.
Retrieval chunk boundary evaluation: a protocol for rules cut off from their conditions
A retrieval chunk boundary evaluation protocol: annotate rule and condition spans, compare chunkers on paired questions and report uncertainty. No results.
Retrieval corpus evaluation leakage: when the RAG index holds the answers
Retrieval corpus evaluation leakage: find answer keys hiding in a RAG index with canary, duplicate and retrieval probe scans, and an exclusion plan to reuse.
Retrieval document version benchmark: finding the controlling version
Build a retrieval document version benchmark that rewards the version in force for each user and date, with authority labels, hard negatives and metrics.
Retrieval evidence disagreement dataset: retrieve both records first
Design a retrieval evidence disagreement dataset: require both conflicting records and the authority rule before an answer counts, with fields and metrics.
Retrieval evidence hierarchy for business search: which source wins
Build a retrieval evidence hierarchy for business search: a worksheet naming the controlling source, version, permission and abstain rule per question.
Retrieval hard negatives from business records: the wrong customer's file
Retrieval hard negatives from business records: when another customer's same-template document is a true negative, and a privacy reviewed collection spec.
Retrieval and stale knowledge articles: when the most clicked is outdated
Retrieval stale knowledge article data: label controlling versions, superseded articles, as-of dates and abstention to catch popular but outdated answers.
Revenue recognition evidence dataset: linking contract terms to delivery records
Build a revenue recognition evidence dataset: link each performance obligation to contract versions, delivery and acceptance records, invoices and entries.
Reward function for delayed outcomes: scoring an agent when success comes later
A reward function for delayed outcomes needs a defined window, a censoring rule and grader tests. Compare seven designs and the shortcut each one invites.
Reward hacking in business agents: seven ways to game a weak grader
Reward hacking in business agents: seven nonharmful shortcuts weak graders reward, a red team pack format, a worked billing case and the checks that stop them.
Reward normalization by task difficulty for refunds and disputes
Reward normalization by task difficulty for refund and dispute agents: why equal success rewards beat difficulty bonuses, plus a stratification plan to use.
RL environment API failure simulation: how to train agents on outages
RL environment API failure simulation: which outages, rate limits and ambiguous timeouts to inject, on what schedule, and how to score an agent's recovery.
RL environment constraint validation: test that the rules are enforced
RL environment constraint validation: a catalog of invalid actions with expected responses, enforcement modes and replayable acceptance tests for agent rules.
RL environment exception coverage: when the customer changes their mind
RL environment exception coverage for customer mind changes: a branch map keyed to trigger points, final-intent rewards and the reset checks to run first.
RL environment hidden information: testing agents on facts revealed later
RL environment hidden information, done right: a staged disclosure task template, triggers, leak tests and scoring for agents that must wait for facts.
RL environment irreversible actions: a sandbox ledger with approval gates
RL environment irreversible actions: how to simulate refunds, sends and deletions without moving real money, using a sandbox action ledger and approval gates.
RL environment multi-tenant isolation: many companies, no mixing
RL environment multi-tenant isolation: keep each company's state, tools, policies and graders apart, with a ten-probe test and a contamination checklist.
RL environment observation schema: keeping business context intact
RL environment observation schema guide: the business context to keep at each step, a field-level contract, version rules and tests that catch leaks.
RL environment realism audit: a checklist an operating expert completes
How to run an RL environment realism audit: ten dimensions an operating expert rates against real operations, evidence methods and a sign-off record.
RL environment reset reproducibility: can you replay yesterday's failure?
RL environment reset reproducibility, tested: restore the logged start state, replay recorded actions and log where runs diverge. Reset spec template included.
RL environment scenario freshness: when enterprise scenarios go stale
RL environment scenario freshness: how to tell when an enterprise environment is outdated, with a drift register tying policy and system changes to scenarios.
RL environment simulation fidelity: how much is enough for an agent task?
RL environment simulation fidelity, settled by test: build the task at three fidelity levels, run the same agents on each, keep the cheapest one that predicts.