FIELD GUIDE · THE LEADERBOARD PROBLEM
The leaderboard can be real and still lie to you. Tests get stale, contaminated, saturated — then oversold at the launch event.
UPDATED 2026-09-21
TL;DR
01
Stanford CRFM's HELM framework publishes prompts, scenarios, metrics, and reproducible evaluation tooling — not just a leaderboard score. A score is easier to trust when outsiders can inspect how it was made.
02
HELM Capabilities explicitly weighs whether a scenario is saturated, how recent it is, and its quality when selecting capability evaluations. A once-useful benchmark can stop separating frontier systems as the field moves on.
03
SWE-bench Verified is a human-annotated subset built after human review of the original software-engineering benchmark; OpenAI says the verified set supersedes the original benchmark variants for its evaluation use. The questions and graders need auditing too.
04
OpenAI's GDPval evaluates models on economically valuable tasks drawn from 44 occupations rather than relying only on academic question sets. A real-world-task benchmark still samples — but it asks a more consequential question than trivia accuracy.
LATEST FROM AI NEWS PAGE
KEEP READING
NEXT FIELD GUIDE →WHY DOES PETER THIEL FEAR THE ANTICHRIST?A billionaire who builds surveillance software fears a world government promising protection. Douthat pressed him in 2025. Then came the secret lectures, the pope, the AI safetyists — and an Iran-war break with Trump.Peter Thiel made his fortune on the internet and keeps insisting the internet was not enough. Flying cars, Mars, AI, longevity and the meaning of progress — the other half of his worldview.
Washington is trying to choke off China's AI — chip bans, blacklists, and now Taiwan making smuggling a crime. Beijing just returned the favor, locking down its own AI exports. Meanwhile the models keep getting cheaper, better, and closer together. A field guide to the argument over whether the race protects anyone, or just speeds everything up.
Anthropic's own researchers ran an AI across fifteen $55 TV boxes and say small models are months from copying themselves onto your gadgets. Nobody has seen one in the wild — but the argument over who owns the problem is already here.
SHOW YOUR WORK
A benchmark is a measuring instrument. Measuring instruments can saturate, leak, and get optimized against.