AI Agent Frontier Research Weekly W34: The Harness Becomes a Trainable Capability Layer (ClawGym II, HarnessEval-W, Anything2Skill)

AI Agent Frontier Research Weekly W34: The Harness Becomes a Trainable Capability Layer (ClawGym II, HarnessEval-W, Anything2Skill)

中文 EN

This edition covers August 17–23, 2026 in Asia/Taipei. W34's research theme is that the harness is no longer merely an interface between model and tools. It is a capability layer that can be trained, evaluated, and equipped with procedural memory.

ClawGym II trains through an opaque harness boundary

ClawGym II isolates existing harnesses in temporary sandboxes, intercepts calls at the model-serving boundary, and reconstructs multi-turn trajectories as prefix trees for PPO or GRPO. The authors report that Qwen3-30A3B improves ClawGym-Bench Pass@1 by 9.98 points through OpenClaw and 14.81 through Claude Code, remaining stable over 200–400 optimization steps.

This is a meaningful systems direction, not yet a universal result. Independent work should reproduce sandbox cost, reward leakage, harness-specific distributions, and transfer to unseen tasks. Mix-harness training is especially important: it may reveal whether the policy learned a capability or memorized one interface.

HarnessEval-W agentifies world-model evaluation

HarnessEval-W wraps visual-world evaluation in an interactive harness instead of relying only on one-shot generation metrics; its Apache-2.0 code is public. For world models intended to support planning, evaluation must inspect exploration, tool use, state retention, and compounding error. Open questions include leaderboard coverage, human baselines, judge sensitivity, and generalization to unseen worlds.

Anything2Skill compiles searchable documents into executable procedures

Anything2Skill, resurfacing through OpenReview this week, extracts heterogeneous records into skill contracts with invocation conditions, contraindications, steps, constraints, evidence, and confidence, then manages them in a versioned SkillBank. The paper reports 98.85% and 94.10% success for RAG+skills on qsv and GitHub CLI. The result supports procedural memory over repeatedly rereading docs, but the domain set is narrow; erroneous-procedure updates, revocation, and authority remain unresolved.

Context is still load bearing

Plans Don't Persist also resurfaced in this week's OpenReview signals. Removing a plan reduced a Llama-3.1-70B plan signal 4.1× after one action-observation step, while naive plan eviction cut ALFWorld success by 34.7 points; probe-gated resurfacing did not restore it. This does not prove that a probe understands plans. It shows that summary and compaction policy are capability variables.

W34 judgment and watchlist

Agent research should version model, harness, context policy, skill registry, reward, and budget together. Watch ClawGym II cross-model reproduction; private sets and human baselines for HarnessEval-W; conflict/revocation in Anything2Skill; and causal, cost-aware tests of context compression.