Benchmark Radar Day 115: The Leaderboard Winner Is Partly Luck

1 minute read

Published:

Day 115 of Benchmark Radar. The scan fetched 1,554 items, published 747, and marked 288 as recommended.

Author: Koutian Wu; GitHub: ktwu01

Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.

The new study “SOTA or Luck?” formalizes the winner’s curse in LLM leaderboards: ranking noisy benchmark scores rewards sampling luck alongside genuine ability, so the displayed winner may not be the best underlying system. The authors develop a bootstrap correction and show that across 14 benchmarks from five families, the rank-1 adjustment ranges from about 0.09 to 3.86 percentage points. In their SWE-bench Verified snapshot, the tied leaders receive an adjustment of 0.90 percentage points. If you are picking a model from closely spaced leaderboard results, the gap at the top may be noise, and a corrected estimate or an uncertainty interval should inform the choice. SOTA or Luck? The Winner’s Curse in LLM Leaderboards.

ForecastGym is a new benchmark that records how evidence from four commercial web-search APIs changes before forecast resolution, enabling live but replayable tests of forecasting agents with fixed evidence histories. Each system replays the same frozen evidence offline instead of searching on its own, so differences in retrieved information no longer mix with differences in reasoning. If you are comparing forecasting models, search providers, or memory strategies, this design separates reasoning quality from retrieval luck and avoids retrospective searches that may expose later information. ForecastGym: A Live, Controlled, and Replayable Benchmark for Forecasting Agents.

Why this matters

If you pick a model from the top of a leaderboard, you may be buying sampling luck rather than the best system. And if you test a forecasting agent with live search, you cannot tell whether it reasoned well or simply retrieved better evidence. The rank is not the result, and the forecast is not the reasoning.

Scan coverage

This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 178 of 747 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.

Data and briefing: 2026-11-18 archived snapshot.