Benchmark Radar Day 106: The Harness Is the Benchmark Now
Published:
Day 106 of Benchmark Radar. The scan fetched 1,111 items, published 496, and marked 283 as recommended.
Author: Koutian Wu; GitHub: ktwu01
Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.
The new “Knowing When to Intervene” framework treats deployment-time oversight as a causal question: at a particular point in an agent’s sequence of actions, would human review, model verification, or steering improve the final outcome compared with letting the agent continue? It differs from static benchmarks that score only completed runs, and it lets you evaluate the intervention policy itself instead of using overall task success as a proxy. Knowing When to Intervene: Causal Evaluation of AI Agents.
The new Flex-RSI benchmark lets an agent revise its own harness, the prompts, tools, memory, and control loop around the model, over repeated rounds under a fixed budget. BOXBOX and its PerfRace benchmark independently test harness revision during long-running software optimization, so mutable harnesses are now a recurring evaluation pressure. A fixed-harness benchmark can miss both the gains and the costs of that adaptation. Flex-RSI: Benchmarking Recursive Self-Improvement of Frontier Agents. BOXBOX: Recursive Harness Self-Improvement for Long-Running Performance Optimization.
Why this matters
If you are deciding where to place costly supervision in an agent workflow, the intervention framework separates an agent-selection decision from a monitoring and escalation decision. And if you are comparing agent systems that adapt during use, you need evaluations that score the evolving system under a budget instead of attributing all performance to the underlying model. The harness is now part of what gets measured.
Scan coverage
This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 167 of 496 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.
Data and briefing: 2026-11-09 archived snapshot.