Benchmark Radar Day 100: Long Is Not Deep: Turn Count Measures the Model, Not the Task

Day 100 of Benchmark Radar. The scan fetched 1,225 items, published 604, and marked 327 as recommended.

Author: Koutian Wu; GitHub: ktwu01

Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.

“Is Your Long-Horizon Benchmark Really Deep?” separates task length from task structure. Instead of counting how many turns a model happens to take, it measures work as required operations and depth as the steps that must occur sequentially because each depends on the previous step’s output. The authors report that turn count largely reflects model habits and that model-assigned difficulty labels barely track pass rates. Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks.

The THReD benchmark evaluates an agent that converts electronic health record extracts into analysis-ready datasets, alongside the preprocessing tools it invokes. Its 141 tasks cover goal-level requests, failure recovery, requests requiring no action, schema variation, and prompt injection, malicious instructions embedded inside the input data. Proposed processing steps run only after user confirmation. Benchmark of THReD: Evaluating an LLM Agent and Its Preprocessing Tools for Turning Electronic Health Records into AI-Ready Datasets.

Why this matters

Genuinely sequential work should be distinguished from tasks made long by inefficient behavior. For clinical-data agents the parallel test is control, whether the agent abstains, recovers, and keeps a person in charge before execution.

Scan coverage

This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 176 of 604 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.

Data and briefing: 2026-11-03 archived snapshot.