Last month, three major AI labs released new models within the same week. What stood out was not the models themselves, but what came with them: not a single public benchmark number in sight. Instead, each lab published detailed descriptions of internal evaluation protocols, custom test suites, and qualitative assessments conducted by their own research teams.
This is not an accident. Over the past eighteen months, the industry has quietly moved away from the leaderboard culture that defined the early 2020s. Public benchmarks like MMLU, HumanEval, and GSM8K are still cited, but increasingly as footnotes rather than headline claims. The real competition is happening inside walled gardens, on tests the public cannot see and, in many cases, cannot verify.
If you are trying to choose between models for a product, invest in a company, or simply understand which system is actually better at a task you care about, the information you need is becoming harder to find. The shift toward internal evaluation makes comparison shopping nearly impossible and raises real questions about whose interests these evaluations serve.
From leaderboards to lock-in
The public benchmark era had obvious flaws. Scores could be gamed through overfitting, test sets leaked into training data, and labs cherry-picked the metrics that made their models look best. But at least the numbers were public. Researchers could replicate them, auditors could question them, and buyers could compare them.
Internal evaluation solves some of these problems and creates new ones. Labs argue that their private tests are harder, more realistic, and better aligned with what users actually need. They are almost certainly right about the first two. The third claim is harder to verify when the tests themselves are confidential.
"We moved away from public benchmarks because they stopped being predictive of real-world performance. Our internal suite is designed around tasks our enterprise customers actually run." — Research director at a major AI lab, speaking on condition of anonymity
The practical effect is a fragmentation of credibility. A score of 92 on a private reasoning test tells you almost nothing unless you know what the test measures, how it was constructed, and whether it was administered with the same constraints that apply to real use. None of that information is typically available.
What the new evaluations actually look like
The labs that have disclosed details describe multi-layered systems. There is usually a core set of automated tests covering reasoning, coding, mathematics, and safety. These are supplemented by human evaluation panels, red-team exercises, and longitudinal monitoring of model behavior in production-like environments.
Some labs have introduced what they call "capability cards" — structured documents that report performance across a range of tasks without collapsing everything into a single number. These are more informative than a leaderboard score, but they are still prepared by the same organization that built the model and stands to benefit from favorable results.
The role of third-party auditors
A small but growing number of organizations are positioning themselves as independent evaluators. Groups like the AI Safety Institute, MLCommons, and several university labs have begun publishing their own assessments of major models, often with access that the general public lacks.
These efforts are valuable but limited. Third-party evaluators rarely get the same level of access as internal teams. They may not see the latest model versions, the full range of capabilities, or the specific deployment contexts that matter most to buyers. And their funding often comes from the same companies they are evaluating, creating conflicts that are difficult to fully disclose.
"The question is not whether internal evaluation is better or worse than public benchmarks. The question is who gets to define what 'better' means."
— Signal AI AnalysisWhat buyers and builders should watch for
For practitioners choosing a model, the current landscape requires a different kind of diligence. Published benchmarks are still worth checking, but they should be treated as a floor, not a ceiling. The more useful signals are often found in release notes, research papers, and the specific claims labs make about their evaluation methodology.
- Look for detailed descriptions of test construction and task selection, not just final scores.
- Ask whether the evaluation was conducted under realistic constraints — latency limits, context window sizes, and tool access matter.
- Check whether the lab has published any negative results or acknowledged limitations alongside strengths.
- Consider whether the evaluation tasks map to your actual use case, or whether they measure something adjacent but not equivalent.
The shift away from public benchmarks is probably irreversible. The leading labs have too much to lose from transparent comparison and too much to gain from controlling the narrative around their own models. The best response is not to lament the loss of leaderboards, but to develop better habits for reading what remains.
