Benchmark Radar Day 62: Benchmarks Built From Real Repositories
Published:
Day sixty-two of Benchmark Radar. The daily scan published 540 records and recommended 171, led by repo2bench, which turns Python repositories into coding-agent tasks. Scoreboard: 255 stars, 41 forks.
Author: Koutian Wu; GitHub: ktwu01
Benchmark Radar is live. Track new AI benchmarks, datasets, and leaderboards every day: open the dashboard or star it on GitHub. Read the paper: arXiv:2609.11115. Query the data: Hugging Face dataset.
The run fetched 1,256 items, kept 540, and recommended 171.
repo2bench converts Python repositories into coding-agent tasks mined from version history or created by removing functions. It admits a model-drafted task only after deterministic checks, including abstract syntax tree mutation testing, which alters code systematically to verify that the tests detect faults.
Era by Eon extends an enterprise-agent test after four leading models scored 22 to 25 of its original 27 questions. The added templates require agents to infer hidden facts from conflicting or indirect company records. Code generates each company and computes the exact answers without a language model.
The LongHorizon Orchestrator Benchmark isolates the decisions made by a robot’s vision-language orchestrator while keeping the low-level control policy fixed. It separately exercises instruction decomposition, visual completion checks, and memory of finished goals during multi-object manipulation.
Why this matters.
When the answer key comes from code and not from another model, a score can be rechecked by anyone.
Issues addressed
- daily snapshot: 1,256 fetched, 540 published, 171 recommended
- scoreboard: 255 of 1,000 stars, 41 forks
Day sixty-three: the scan continues, with the scoreboard at 255 stars.
Want to follow Benchmark Radar? Star the repo on GitHub for daily updates, or open the live dashboard to explore the scans. Read the paper: arXiv:2609.11115.