Benchmark Radar Day 109: 433 of 500 Tasks Were Placeholders
Published:
Day 109 of Benchmark Radar. The scan fetched 1,181 items, published 467, and marked 228 as recommended.
Author: Koutian Wu; GitHub: ktwu01
Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.
The new report “PrincipalBench v2: A Validity-First Benchmark for LLM Orchestrators in Multi-Agent Pipelines” audits its predecessor against the underlying code and data. It finds that 433 of 500 v1 tasks were placeholders without real questions or gold answers, every episode included a failure, letting an always-flag policy score perfectly, and only three distinct failures were tested. The report says v1 cannot support model comparisons. If you are choosing a model to coordinate worker agents, v1 scores should not drive that choice. PrincipalBench v2: A Validity-First Benchmark for LLM Orchestrators in Multi-Agent Pipelines.
Journeyman is a new benchmark that evaluates how an agent completes work rather than only whether it passes. Across eight seed-rebuilt simulated jobs, it scores ten process dimensions, including repeated futile actions, stopping behavior, acknowledgment of unreachable goals, and repetition of planted false information; simulation state can expose claimed actions that never occurred. If you are evaluating operational agents, Journeyman lets you distinguish systems with identical final outcomes but different efficiency, honesty, and failure handling. Journeyman: a process-quality benchmark for LLM agents.
Why this matters
If you pick an orchestrator from v1 scores, you are ranking models on placeholder tasks where a policy that always flags failure scores perfectly. And if you only score pass or fail, you cannot tell a careful agent from a wasteful one that happens to land on the right answer. The work itself is the signal, not the final checkmark.
Scan coverage
This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 131 of 500 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.
Data and briefing: 2026-11-13 archived snapshot.
