Benchmark Radar Day 108: Your Leaderboard Can’t Tell Who Actually Wins

Day 108 of Benchmark Radar. The scan fetched 1,292 items, published 573, and marked 249 as recommended.

Author: Koutian Wu; GitHub: ktwu01

Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.

The new study “Agent Benchmarks Stop Ranking Before They Run Out of Tasks” re-scores released runs from eight benchmarks and finds that binary pass/fail scoring can hide useful differences. On SWE-bench Verified, reported gaps between neighboring top-ten agents are smaller than estimated test errors, so corrections can reorder them. The study attributes some apparent saturation to measurement rather than exhausted tasks. If you are selecting an agent from a tightly grouped leaderboard, small pass-rate differences may not support a stable choice. Agent Benchmarks Stop Ranking Before They Run Out of Tasks.

OpenRE-Bench is a new open-source comparison harness for multi-agent requirements engineering, where agents turn software needs into specifications. It supplies a shared contract for inputs, intermediate phase artifacts, and metrics; frameworks join through an adapter rather than changing the cases or scoring. Its cases are based on mined GitHub issues. If you are choosing or developing a requirements-engineering agent framework, this design separates framework behavior from differences in runtimes and bespoke metrics, making side-by-side tests easier to reproduce. OpenRE-Bench: A Benchmark Harness for Multi-Agent Requirements Engineering with Mined GitHub Issues.

Why this matters

If you pick an agent from a leaderboard, the ranking itself may be noise: the gap between first and fifth can be smaller than the error in the test. And if you build agents that turn requirements into specifications, you have had no shared way to compare frameworks until now. Measurement is the bottleneck, not the models.

Scan coverage

This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 152 of 573 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.

Data and briefing: 2026-11-11 archived snapshot.