Benchmark Radar Day 99: Tested at 86% Attacks, Deployed at 2%

Day 99 of Benchmark Radar. The scan fetched 1,039 items, published 414, and marked 193 as recommended.

Author: Koutian Wu; GitHub: ktwu01

Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.

A new audit of hijacked-agent detectors found that attacks represented 16%–86% of test samples across 11 evaluations, none varied that prevalence, and six did not test data from a second benchmark or source. The paper re-evaluates detectors at a 2% attack rate and tests transfer between AgentDrift and AgentDojo. Performance on attack-heavy, single-source test sets may not describe behavior when attacks are rare. What Standard Evaluations Hide About Detectors of Hijacked LLM Agents: Rare Attacks, Unseen Tools and Benchmark Transfer.

“Simple Features Beat Most AI Control Monitors” reports that features requiring no code execution outranked 20 of 24 open-weight monitoring models on a standard backdoor-detection dataset. It attributes this to label leakage: humans wrote the honest solutions while older language models wrote the backdoors, and the two classes also used different problem types. A high score may reflect source style or task selection rather than detection of malicious behavior. Simple Features Beat Most AI Control Monitors.

Why this matters

Deployment-rate testing and cross-benchmark transfer should be explicit selection criteria for a detector. And before monitors are compared, simple metadata or style baselines should first have to fail at predicting the labels.

Scan coverage

This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 160 of 414 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.

Data and briefing: 2026-11-02 archived snapshot.