Benchmark Radar Day 75: Did the Agent Change, or Did the Scorer?
Published:
Day 75 of Benchmark Radar. The scan fetched 1,338 items, published 616, and marked 281 as recommended.
Author: Koutian Wu; GitHub: ktwu01
Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.
TRACE uses paired runs and rescoring of unchanged trajectories to diagnose brittle evaluators. In its controlled suite, renaming tools reduced a scripted agent’s score even though its operations stayed identical. The result shows why a score regression needs an evaluator check before it is attributed to the agent. TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation.
A tabular-data clarification benchmark separates the policy for asking questions from the eventual solution. It also tracks oracle leakage: a simulated user supplying information that a real user would not provide. Success with an overly helpful simulator can overstate an assistant’s ability to resolve ambiguity. Conversational Task Disambiguation over Tabular Data: Leakage-Aware Formulation, Benchmark Suite, and Training.
Why this matters
Check the scorer and simulated user before treating a better score as better behavior. Both can change the measurement while the agent stays the same.
Scan coverage
This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 163 of 616 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.
Data and briefing: 2026-10-09 archived snapshot.