Benchmark Radar Day 14: Feed Coverage, Briefing Reliability, and Production Q&A
Published:
The daily pipeline went from fragile to reliable. Day fourteen expanded feeds, hardened the briefing, and enabled Q&A in production.
Author: Koutian Wu; GitHub: ktwu01
What shipped
Expanded first-party benchmark feed coverage. More benchmark-specific RSS feeds were added to the collection pipeline. Curated feeds for AI news and updates were included.
Sponsor-bait suppression. GitHub resource-listing spam (repos that exist only to aggregate links for SEO) is now filtered out. These repos inflate counts without adding signal.
OpenAI briefing retry budget. The briefing API call now has enough retry budget to outlast TPM (tokens per minute) limits. Previously, a single rate limit error could kill the entire briefing.
Truncated output prevention. The OpenAI briefing output is now checked for truncation. If the response was cut off, it is rejected and retried instead of being published as-is.
Single daily run at 9AM SGT. The radar was switched from multiple daily runs to a single run at 9AM Singapore time. This simplifies the schedule and reduces API costs while maintaining reliability.
Daily Q&A enabled in production. The daily question-and-answer feature was enabled and required in production. Each day, the radar generates questions about the benchmark landscape and answers them from the collected data.
Collection method labeling. Source health rows are now labeled by their actual collection method (what the run did), not the generic “Radar ingest” label. This makes it possible to diagnose which part of the pipeline produced each row.
Backfill collection method. Snapshots recorded before collection method labeling existed were backfilled with the correct method.
Worktrees gitignore. The .worktrees/ directory was added to .gitignore.
Why it matters
The switch to a single daily run at 9AM SGT was an operational decision. Multiple runs per day were creating confusion about which run was “the” daily run. A single predictable run at a fixed time made the schedule auditable.
The Q&A feature in production was a milestone. The radar was no longer just collecting data; it was generating its own questions about the data and answering them. This is a step toward autonomous monitoring: the system notices things and reports them, instead of waiting for a human to ask.
Issues addressed
- #169: expand first-party benchmark feed coverage
- #170: suppress sponsor-bait spam
- #171: briefing retry budget for TPM limits
- #172: prevent truncated briefing output
- #175: enable daily Q&A in production
- #176: single daily run at 9AM SGT
- #177: label radar ingest sources by collection method
- Collection method backfill
Day fifteen: social media integration and daily post generation.