Meta has denied allegations that it manipulated its latest AI models, Llama 4 Maverick and Llama 4 Scout, to achieve better benchmark scores while hiding their limitations.

Ahmad Al-Dahle, Meta’s vice president of generative AI, addressed the rumors in a post on X, saying they are “simply not true.” He rejected claims that the company trained the models on “test sets,” which are typically used for performance evaluation after training. This could result in misleading scores.

The rumors began circulating over the weekend on X and Reddit, reportedly originating from a post on a Chinese social media platform.

The individual claimed to have resigned from Meta due to its benchmarking practices. They noted inconsistent performance from Maverick and Scout, alongside Meta’s use of an unreleased version of Maverick for the LM Arena benchmark.

Al-Dahle acknowledged that users reported “mixed quality” when accessing the models through different cloud providers. He explained that the models were made available as soon as they were ready. Ongoing adjustments and bug fixes aim to resolve these issues.

🔗 Source: TechCrunch

🧠 Food for thought

A. AI benchmark optimization creates a credibility gap in the industry

The controversy around Meta’s AI benchmarking reveals a persistent challenge in the industry: benchmark scores often fail to reflect real-world capabilities.

When Google’s Gemini model topped a crucial AI benchmark with a score of 1344, surpassing OpenAI’s GPT-4o, researchers discovered its performance plummeted when controlling for simple factors like response formatting 1.

This pattern of benchmark optimization has created what experts call a “crisis in AI evaluation,” where companies optimize for leaderboard rankings rather than genuine improvements in reasoning and reliability 1.

The gap between benchmark performance and practical utility is further illustrated by the ARC-AGI-2 test, where leading models from OpenAI and Google scored dismally at 4% and 1.3% respectively on reasoning tasks, despite impressive results on standard benchmarks 2.

These discrepancies highlight why Meta’s benchmark controversy resonates within the AI community, as the industry’s reliance on potentially manipulable metrics undermines trust in reported AI capabilities.

B. The evolution of AI evaluation reflects growing industry maturity

The Meta benchmark controversy is part of a broader shift toward more comprehensive and meaningful AI evaluation methods as the field matures.

Stanford researchers recently developed new benchmarks specifically designed to measure AI bias and understanding more effectively, arguing that existing fairness benchmarks often yield misleading results as models can score well without demonstrating true fairness 3.

Recent Meta developments