Benchmark Radar Day 114: 27 of 35 Leaderboard Wins Are Statistical Noise
Published:
Day 114 of Benchmark Radar. The scan fetched 1,234 items, published 635, and marked 346 as recommended.
Author: Koutian Wu; GitHub: ktwu01
Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.
The new study “Beyond Leaderboard Ranks” reframes software-agent selection as an accuracy, cost, and uncertainty decision. Across 58,197 agent runs on seven benchmarks, 27 of 35 adjacent options on the accuracy-cost front had statistically unresolved accuracy differences, meaning their apparent ordering did not reliably distinguish performance. If you are buying or evaluating coding agents under a budget, a leaderboard winner may not justify its higher operating cost. Selection reports need confidence intervals and the cost of running enough evaluations to resolve meaningful differences, rather than rank and average accuracy alone. Beyond Leaderboard Ranks: Selecting Software Engineering Agents Under Budget Constraints.
EVIDENT is a new benchmark that tests whether a web-browsing agent’s answer is supported by evidence it actually observed, rather than merely checking task completion. Its 220 English and Korean tasks include human-curated records of what evidence must be verified, and separate metrics score completion, evidence use, and their combination. If you are selecting a graphical user interface agent for research, compliance, or fact-finding, this design can reveal agents that reach plausible answers without inspecting the necessary page content. It changes the evaluation target from successful output alone to an auditable connection between navigation and answer. EVIDENT: Evaluating Evidence Grounding in GUI Agents.
Why this matters
If you pick a coding agent from a leaderboard, you may be paying a premium for a rank that statistics cannot defend. And if you only check whether an agent finished the task, you cannot tell an agent that actually read the evidence from one that guessed well. The ranking is not the result, and the finished task is not the proof.
Scan coverage
This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 156 of 635 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.
Data and briefing: 2026-11-17 archived snapshot.

