Benchmark Radar Day 118: Static Benchmarks Cannot Grade Live Agents
Published:
Day 118 of Benchmark Radar. The scan fetched 1,285 items, published 536, and marked 270 as recommended.
Author: Koutian Wu; GitHub: ktwu01
Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.
The new DynamicMCPBench framework evaluates agents against live, stateful Model Context Protocol servers, which expose external tools to a model. It generates goals, executes them to record successful trajectories, and scores effects rather than only the final answer or a fixed tool list. A companion sandbox, MCPAgentBench, adds irrelevant tools as distractors, so the agent must pick the right tool instead of the only tool. Together they test tool use under changing execution conditions rather than static prompts alone. If you are choosing an evaluation for tool-using agents, these releases let you test whether an agent selects and executes tools correctly on your own servers, not merely whether it produces a plausible answer after an unseen sequence of actions. DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers. MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use.
The new GUI-360° dataset gives desktop agents 17,000 trajectories and 131,000 validated successful action steps, plus 1.09 million failed steps. Each step includes screenshots, accessibility metadata, goals, reasoning traces, and structured interface or API actions. The failed executions change the evaluation choice: the same resource can measure action grounding and inspect where workflows break, while also supplying material for future recovery training instead of treating only completed tasks as useful data. If you build agents for office applications, a benchmark that keeps its failures tells you more than one that only counts its wins. GUI-360° : A Comprehensive Dataset and Benchmark for Desktop Productivity Computer-Using Agents.
Why this matters
If you grade a tool-using agent on a fixed prompt, you may be certifying a system that breaks the moment the server state changes. And if you train a desktop agent only on successes, you never learn where its workflows fall apart. The final answer is not the execution, and the success log is not the whole story.
Scan coverage
This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 125 of 536 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.
Data and briefing: 2026-11-21 archived snapshot.
