Insights
AI uses for records
531 guides · page 5 of 12
Agent incomplete handoff failure: how to evaluate escalation evidence
An agent incomplete handoff failure escalates correctly but leaves the human without evidence. Build the case, evidence set and recovery test that catch it.
Agent instruction hierarchy in business datasets: customer words are evidence
Agent instruction hierarchy for a business dataset: a labeling spec that marks customer messages as evidence, not policy, with fields, an example and metrics.
How to require an agent judge to cite the evidence behind its verdict
Agent judge evidence citation in practice: an evidence anchored grading report format, code checks for every anchor, shortcut tests and a worked example.
How to measure agent memory consolidation loss: a fidelity protocol
How to measure agent memory consolidation loss: a protocol with loss types, matched conditions, metrics and a pre-registration checklist. No results reported.
How to test agent memory against contradictory updates from a customer
Agent memory contradictory updates: how to test an agent when a customer changes their mind twice, with a ground-truth timeline, field spec and scoring probes.
Agent memory cross-account contamination: how to test for leaks
How to test agent memory for cross-account contamination: marker facts, write and deletion checks, nine isolation test cases and a severity scale for leaks.
Agent memory deletion evaluation: testing that deleted facts stay gone
Agent memory deletion evaluation: trace a deleted fact through every store, then probe whether the agent still uses it. Protocol, probes and a worked case.
Agent memory evaluation dataset: what to keep after first contact
An agent memory evaluation dataset that tests what an agent keeps, updates, expires and forgets after a first customer conversation, with labels and probes.
Agent memory and negative knowledge: recording an unreliable source
Agent memory negative knowledge, explained: how to record that a source was unreliable, with a note schema, expiry rules and a reliability memory benchmark.
Agent memory permission boundaries: testing recall after access is revoked
Agent memory permission boundaries: a test set that revokes access, probes later sessions and scores leaks, over-forgetting and time to enforcement.
Agent memory stale facts benchmark: will your agent notice a fact went out of date?
An agent memory stale facts benchmark with dated evidence: change types, item fields, probes and scoring that show whether an agent notices outdated facts.
Agent misleading filename failure: testing for approval evidence
Agent misleading filename failure: an illustrative evaluation task with trap variants and a rubric that checks approval evidence, not a final-looking name.
How to catch an agent missing hard records in invoice reconciliation
How to score an agent missing hard records: count every in-scope invoice, score each difficulty stratum and fail silent omissions or false completion claims.
Agent outdated policy failure case: how to test for expired rules
An agent outdated policy failure case: how to test whether an agent follows an expired rule, with a dated failure packet and the records that prove it.
Agent pagination omission failure: summarizing page one as the archive
An agent pagination omission failure turns one page of results into a false summary of the whole archive. Grade it with this illustrative completeness case.
Agent partial refund failure: the refund was right, the order was wrong
Agent partial refund failure, worked as an illustrative case packet: grade processor, order, inventory and ticket state, not the reply, with test variants.
Agent policy compliance evidence report: proving which boundaries held
An agent policy compliance evidence report that proves which instruction, data, authority and approval boundaries held, with trace-anchored verdicts per run.
Agent policy loophole failure: testing agents that pass the rubric and break the rule
Agent policy loophole failure: how to test agents that satisfy a rubric while breaking a business constraint, with an invariant test template and worked case.
Who should approve changes to an agent's definition of success?
Agent reward change control: who approves edits to an agent's grader or success criteria, plus a change request form and a regression protocol to run first.
Agent rollback benchmark: how to measure a safe undo of unfinished work
How to design an agent rollback benchmark for business workflows: interrupt runs in a sandbox, then score residue, collateral damage and honest reporting.
Agent safety dataset benign hard cases: testing ordinary ambiguous requests
Build an agent safety dataset of benign hard cases: ordinary ambiguous business requests, reviewed answer keys, contrast twins and grading for refusals too.
How do you test whether an AI agent damages unrelated records?
Agent side effect failure analysis: declare what a task may change, diff the whole state, check invariants each step and score maintenance apart from success.
Agent source credibility challenge dataset: signed policy or suggestion?
Build an agent source credibility challenge dataset: authority labels, evidence fields and sandboxed tests that show whether an agent follows signed policy.
Agent spreadsheet formula failure case: testing a wrong-period reference
Agent spreadsheet formula failure case: how to test formulas that reference the wrong period, with boundary rows, a case record and linked evidence checks.
Agent stale inventory failure: testing a stock promise made from an old snapshot
An agent stale inventory failure case: how an agent promised stock from an old snapshot, which conditions failed, and a reusable packet for testing freshness.
Agent task scope expansion benchmark: testing for unrequested changes
Design an agent task scope expansion benchmark: seed temptations, record granted scope, diff state for forbidden changes and score containment apart.
Agent third-party tool trust data: treating tool output as evidence
Agent third party tool trust data: grade tool output as evidence, catch truncated, stale or mismatched results, and test it with ten sandboxed cases.
Agent trajectory action provenance: telling human from agent steps
Agent trajectory action provenance: an actor attribution schema with uncertain labels, evidence tiers, a worked refund episode and validator checks.
Agent trajectory branching decisions: three valid paths, one task
Agent trajectory branching decisions: record several valid paths for one task, link each branch to its evidence and judge equivalence with written rules.
Agent trajectory evidence attachments: proof that an agent finished
Agent trajectory evidence attachments ranked: why read-backs beat screenshots, a step-linked manifest with digests, and checks a second team can replay.
Agent trajectory expert disagreement: when two valid paths should both stay
Agent trajectory expert disagreement handled by protocol: replay both paths, verify outcomes, classify the split and keep every valid solution with its reasons.
Agent trajectory failed tool calls: labeling errors so they teach recovery
Agent trajectory failed tool calls, labeled step by step: capture the error the model saw, mark fault and recovery, and package steps for SFT or preference.
Agent trajectory human intervention labels: how to mark a rescue
Agent trajectory human intervention labels show when a person rescued the run. Use this rubric, field list and six-level severity ladder to mark it.
Agent trajectory interruption recovery: what resume data must keep
Agent trajectory interruption recovery: what trajectories must keep to train resumption, plus a collection protocol for cuts, gaps and duplicate side effects.
Agent trajectory outcome delay: labeling results that arrive next week
Agent trajectory outcome delay: how to label business results that arrive after the task ends, using observation windows, censoring rules and a linkage table.
Agent trajectory replay benchmark: can another team rerun your data?
An agent trajectory replay benchmark checks if an independent team can rebuild state and outcome from your trajectory dataset. Protocol and report template.
Agent trajectory shortcut detection: auditing false successes
Agent trajectory shortcut detection: nine invalid-success patterns, the trace signals they leave, and the state, hash and replay checks that expose them.
Agent trajectory state diff dataset or full snapshots: what to store
When an agent trajectory state diff dataset beats full snapshots: a storage and reconstruction comparison, diff format rules and a check a second team can run.
Agent trajectory task boundary detection: where tasks start and end
Agent trajectory task boundary detection: rules for where a business task starts and ends in a messy event log, with an annotation guide and agreement check.
Agent trajectory tool response preservation: why each action needs its result
Agent trajectory tool response preservation, shown with a paired example: what reviewers lose when tool results are stripped, and the fields that keep them.
Agent trust boundary metadata: how to label sources before an agent reads them
Agent trust boundary metadata, field by field: label each source's instruction authority, fact authority, readers and permitted use, then test it in a sandbox.
Agent unauthorized action refusal data: when the user lacks authority
Agent unauthorized action refusal data: how an agent should handle a clear task from a user without authority, with an evaluation task pack and twins.
Agent unit conversion failure case: right formula, wrong units
An agent unit conversion failure case: how a correct weight formula on mixed kilograms and pounds approved an overweight load, and a record to test for it.
Agent unsupported confirmation failure: it said saved but never checked
How to grade an agent unsupported confirmation failure, when an agent says a change was saved but never checked, with an illustrative failure packet.
Agent user intent correction safety: can the agent undo the old plan?
Agent user intent correction safety: test whether an agent reverses its old plan after a corrected request, with a sandboxed correction and approval packet.
Agent wrong document version failure: it read the contract but missed the amendment
Agent wrong document version failure: how to test an agent that quotes a contract accurately but misses its amendment, with a case record and scoring scheme.
Agent wrong evidence source case study: a right answer from a forbidden file
An agent wrong evidence source case study: how to score a correct answer taken from a forbidden source, with a ten-field failure record and severity rules.
AI evaluation severity weights: scoring a wrong payment above a bad summary
How to set business severity weights in an AI evaluation so a wrong payment counts more than a weak summary: tier rubric, worked example, gate and reporting.