This article covers July 27–August 2, 2026 in Asia/Taipei. W31 research converged on one point: task success captures only a fraction of agent reliability. Persistent policy, ambiguous situations, cross-signal diagnosis, and high-stakes dialogue are where deployment claims become fragile.

HANDBOOK.md: policy in context is not policy followed

HANDBOOK.md places a roughly 100-page corporate handbook inside agent environments and tests standing-instruction compliance across five enterprise domains. That resembles organizational work better than isolated questions: a rule can be far from the current action and compete with tool output and a local objective.

The contribution reframes long context as executable constraint following. Its synthetic environments and scoring do not transfer automatically to every company. In production, critical policy should become a deterministic pre-action gate rather than something the model is merely expected to remember.

ORCA-bench: coding skill is not on-call readiness

ORCA-bench combines Prometheus, Jaeger, OpenSearch, source code, and six days of telemetry in a microservice system with 1,079 root-cause tasks. Across five frontier agents, the best RCA accuracy was 25.3% on realistic Medium tasks and 10.0% on Hard; human rescoring and the LLM judge reached weighted Cohen's κ=0.90.

This is strong but bounded evidence. The public 50GB testbed is smaller and more stable than production, so the authors frame the gap as a lower bound on required engineering. SRE agents should begin with read-only evidence gathering, time-window validation, and ranked hypotheses; remediation should remain behind explicit approval.

PatientAgentBench: high-stakes dialogue needs more than answer grading

PatientAgentBench wraps a foundation model in a healthcare-tool sandbox, has it converse with a simulated patient, and evaluates six dimensions using more than one hundred clinician-grounded criteria. It addresses the field's bias toward medical knowledge questions instead of patient-facing dialogue and action.

Simulated patients and an LLM-as-a-Jury can still miss population differences and rare harms. The benchmark is useful for pre-deployment gap discovery, but cannot replace clinical validation, responsibility boundaries, and human escalation.

Research view

Together these studies move evaluation from “did it finish?” to “did it comply, find the right evidence, and stop at high-risk boundaries?” The next evaluation stack should jointly measure success, violations, irreversible actions, calibration, and intervention cost.

Watchlist

• HANDBOOK.md under policy updates, rule conflicts, and longer tasks.
• ORCA-bench extensions for remediation, incident collaboration, and changing systems.
• Real-patient, multilingual, and underserved-population validation for PatientAgentBench.
• Public harnesses, prompts, tool failures, and full traces.
• Abstention and deterministic policy gates entering enterprise acceptance tests.