Benchmark Radar Day 117: The Judge Follows the Numbers, Not the Rubric
Published:
Day 117 of Benchmark Radar. The scan fetched 1,339 items, published 626, and marked 289 as recommended.
Author: Koutian Wu; GitHub: ktwu01
Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.
The new AgentJudgeBench release tests whether large language models used as automated judges can grade tool-calling workflows represented as dependency graphs. Its 3,808 instances span six graph structures and three difficulty levels, and compare judging with and without ground-truth workflow information. This targets structured execution rather than open-ended text preferences. A separate scale-reversal probe in today’s feed also tests whether judges follow rubric meaning rather than numerical conventions. If you use model-based graders for agent evaluations, ordinary agreement scores may not establish that the grader understands workflow dependencies or the rubric itself. AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling. Rubric Semantics or Numerical Priors? Auditing Scale-Direction Bias in LLM Judges.
The new TraceProbe benchmark runs agents in executable sandboxes, records normalized action traces, and replays tool actions to locate where safety risks entered and propagated. It scores both final outcomes and hidden rule violations, rather than treating an apparently acceptable endpoint as sufficient evidence of safe execution. If you evaluate agents that use files, tools, memory, retrieved content, or other agents, TraceProbe offers a way to distinguish a safe-looking result from an unsafe path to that result. That can change whether a security suite is selected for release gating, incident diagnosis, or comparison of safeguards. TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents.
Why this matters
If you grade agents with model judges, a high agreement score may hide that the judge never understood the workflow or the rubric. And if you test agent safety by the final answer alone, a clean endpoint can hide the unsafe steps that produced it. The score is not the understanding, and the finish is not the path.
Scan coverage
This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 165 of 626 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.
Data and briefing: 2026-11-20 archived snapshot.
