Artificial Analysis's new edition of the index widens the gap between open-source and closed-source models

Artificial Analysis launched the 5th edition of the Intelligence Index; Kimi K3.1 Pro fell from 4th place to 9th place, ranking behind eight closed-source models, and the gap between open-source models and the top of the list widened from 3 points to 12 points. The new edition of the index, for the first time, bases 60% of the weight on a private test set to prevent models from gaming the leaderboard (benchmaxxing) against public test questions; Grok 4.7's score also dropped by 12 points. AI researcher Nathan Lambert believes that leaderboard gaming is only part of the reason: much of today's training gains come from reinforcement learning environments similar to benchmark tasks, and staying ahead across multiple domains at the same time requires complex RL infrastructure, which not every lab has. The disadvantage of private test sets is that outsiders cannot check for errors in the prompts; the AA-Omniscience knowledge test has such problems, and Artificial Analysis revised its ambiguous and flawed questions for the new edition of the index.