Benchmark Radar Day 103: Benchmarks Accumulate Maintenance Debt Like Software

Day 103 of Benchmark Radar. The scan fetched 1,627 items, published 831, and marked 444 as recommended.

Author: Koutian Wu; GitHub: ktwu01

Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.

“Benchmarks as Software” uses Terminal-Bench to define continuous validation for agent benchmarks. Its workflow combines community reports, task artifacts, and agent traces to find cases where valid new solutions fail old verifiers or infrastructure changes invalidate task assumptions. It treats the resulting gap between scores and intended capability as benchmark maintenance debt. Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks.

The Agent Operation Recovery Benchmark tests whether an agent can resume an uncertain business operation without changing the original target or executing it twice, while still allowing a genuinely authorized new request. It supplies 24 synthetic cases across six failure families, a deterministic simulator and scorer, and 96 recorded model episodes. Ordinary task-completion scores can miss duplicate execution and identity-binding failures. Agent Operation Recovery Benchmark: 24 Synthetic Scenarios and 96 Model Episodes (full96-v1).

Why this matters

Release-time validation is not enough when tasks, verifiers, and infrastructure keep moving. Recovery deserves the same discipline, since an agent resuming work must first show it will not run the same operation twice.

Scan coverage

This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 180 of 831 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.

Data and briefing: 2026-11-06 archived snapshot.