Benchmark Radar Day 105: The Ranking Is the Config, Not the Model

Day 105 of Benchmark Radar. The scan fetched 1,322 items, published 599, and marked 205 as recommended.

Author: Koutian Wu; GitHub: ktwu01

Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.

The newly released “The Ceiling Is Not the Model” study reports that output limits, decoding settings, and the serving stack changed local large language model scores enough to suppress measured capability, reverse a ranking, and create an apparent reasoning loop. Its five-arm design isolates each configuration effect and includes raw records and a triage protocol. The Ceiling Is Not the Model: How Harness Configuration Fabricates Local LLM Benchmark Results.

The new corpus-assay release scans large language model training collections for benchmark contamination using auditable n-grams, short exact text fragments, and attributes matches to individual benchmarks and test items. corpus-assay: benchmark contamination detection for LLM training corpora.

Why this matters

If you compare locally served models, the evaluation must vary and record token budgets, decoding, and serving settings before attributing score differences to model weights. Otherwise a deployment setting may determine the ranking. And if you are deciding whether benchmark scores reflect generalization rather than memorized test material, corpus-assay adds an inspection step that names exactly which evaluation items overlap a training collection instead of returning only a corpus-level warning.

Scan coverage

This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 123 of 599 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.

Data and briefing: 2026-11-08 archived snapshot.