Benchmark Radar Day 92: A Classic Document Benchmark Scores Label Errors and Duplicates

Day 92 of Benchmark Radar. The scan fetched 1,048 items, published 464, and marked 211 as recommended.

Author: Koutian Wu; GitHub: ktwu01

Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.

A new audit of RVL-CDIP reports that about 10.5% of its 400,000 documents have incorrect labels and 35.2% of the test set has a duplicate or near-duplicate in training. Page identification codes alone support 69% accuracy, and removing them reduces one model’s score by 11 points. Headline accuracy on this dataset partly measures label errors, repeated documents, and source-specific identifiers. Data Quality in RVL-CDIP: A Systematic Audit.

The THReD Agent Benchmark evaluates an electronic health record preprocessing assistant with 141 tasks covering simple and multi-step transformations, failure recovery, requests where the agent should not act, schema changes, and prompt injection hidden in data. It uses synthetic health-record tables, expert reference pipelines, and outcome checks rather than patient data. Operational safety and recovery are tested alongside successful preprocessing. THReD Agent Benchmark: Evaluation of an LLM Agent for Preprocessing Electronic Health Records into AI-Ready Datasets.

Why this matters

A benchmark decision is only as good as the data under it. Deduplication, corrected labels, and identifier-masked tests belong before a model choice, not after a suspiciously high score.

Scan coverage

This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 123 of 464 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.

Data and briefing: 2026-10-26 archived snapshot.