Benchmark Radar Day 97: Every Reply Can Pass While the Conversation Fails

Day 97 of Benchmark Radar. The scan fetched 1,202 items, published 517, and marked 238 as recommended.

Author: Koutian Wu; GitHub: ktwu01

Benchmark Radar is live. Open the dashboard, star the repository, or read the paper. The Hugging Face dataset provides the data.

SAFE-CARE Bench specifies a release-gate protocol for patient-facing healthcare agents across multi-turn conversations. It tests whether risk accumulates across otherwise acceptable replies and whether escalation to human care happens in time, while declaring active capabilities so teams can reproduce the protocol on their own runtime. The unit of evaluation is the full conversational path, not the isolated answer. SAFE-CARE Bench: A Benchmark Specification and Evaluation Protocol for Safety, Guardrails, and User Experience in Multi-Turn Healthcare AI Agent Conversations.

MURPHY uses 1,500 multilingual and emoji prompts to audit whether inference servers report token identifiers and probability records correctly. It scores record integrity separately from instruction following, so a model can complete the requested task while its diagnostic output is still wrong. Task success is not treated as evidence that the telemetry is correct. bytkim/MURPHY-BENCHMARK.

Why this matters

Approving a healthcare agent means evaluating delayed escalation and the safety-versus-accuracy trade-off across turns. The same discipline applies to measurement itself, where the records a system reports about its own outputs need their own check.

Scan coverage

This is a keyword-filtered discovery feed, not a representative survey of AI research. The archived briefing used 113 of 517 captured evidence records. These findings summarize its cited releases; discovery dates do not necessarily equal original publication dates.

Data and briefing: 2026-10-31 archived snapshot.