AI Agent Frontier Research Weekly W29: Harnesses Learn—and Need Fair Tests (MemoHarness, E3, GPT-Red)

AI Agent Frontier Research Weekly W29: Harnesses Learn—and Need Fair Tests (MemoHarness, E3, GPT-Red)

中文 EN

This article covers 2026-07-13 through 2026-07-19 in Asia/Taipei time. The common question was how an external harness can learn from experience and scale effort to task complexity—without mistaking benchmark search for general capability.

MemoHarness rewrites control from execution experience

MemoHarness decomposes a harness into six editable dimensions, stores case diagnoses and global patterns, then retrieves experience to adapt the harness per test case. The authors report improvements over fixed harnesses on shell-agent, code-generation, and analytical-reasoning benchmarks, with selective transfer across suites and base models.

Its contribution is to make memory a control strategy rather than merely conversation content. Risks include extra context, poisoned experience, and incomplete component attribution. The paper explicitly leaves broader statistical robustness for future work, so this is evidence of feasibility—not a solved recipe for general self-improvement.

E3 estimates first and expands only after failure

Do AI Agents Know When a Task Is Simple? proposes Estimate–Execute–Expand and the Agent Cognitive Redundancy Ratio. On 121 deterministic edits in MSE-Bench, E3 matches the strongest baseline’s 100% success while reporting 85% lower cost, 91% fewer tokens, and 92% fewer inspected files. A real-model case grades patches with the project’s actual pytest suite.

This directly targets agents that audit a repository before a one-line edit. However, the main benchmark is a controlled simulator and the real-model sample is limited. The result validates execution-policy research; its percentages should not be projected onto large, ambiguous, or safety-critical work.

The counter-result: search is not learning

Rethinking the Evaluation of Harness Evolution for Agents argues that evolving on feedback from a public benchmark and reporting on that same benchmark can disguise test-time search and overfitting as harness progress. On Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6, automatic evolution did not consistently beat simple scaling baselines at matched budgets, and held-out transfer was limited.

This negative result matters. Future work should disclose search budgets, feedback rounds, held-out tasks, and cross-model results.

GPT-Red puts safety into a self-play loop

OpenAI introduced GPT-Red on July 15: an automated red team trained through self-play across prompt-injection environments involving browsers, files, email, and tool output. OpenAI reports broader success than human red-teamers against GPT-5.1 on an internal mirror and uses attack data to train GPT-5.6. The model, full data, and several environments remain unavailable, limiting external reproduction.

Watchlist

  • MemoHarness ablations, memory poisoning, and cross-organization isolation.
  • E3’s minimum-sufficient scope in monorepos, ambiguous tasks, and risky changes.
  • Pre-registered held-out sets and matched search budgets for harness evolution.
  • The GPT-Red preprint, third-party tests, and attack-distribution overfitting.
  • Joint reporting of success, tokens, wall time, tool failures, and handoffs.