Understanding Benchmarks
Every AI launch comes with benchmark scores. Learn what the major benchmarks actually measure, what they miss, and why a high score does not always mean a better model for your needs.
Benchmark Decoder Tool
Click any benchmark card to expand it. Use the category filters to narrow your view.
Only for that specific benchmark. A model scoring 2% higher on MMLU may score 5% lower on SWE-bench. The best model depends entirely on your task.
Scores depend heavily on evaluation methodology: prompt formatting, number of few-shot examples, temperature settings, and post-processing all affect results significantly.
Standardised tests measure narrow skills. Passing the bar exam tests legal knowledge recall, not legal judgment, client empathy, or courtroom strategy.
Some benchmarks are rigorously designed with anti-contamination measures. Others use publicly available test questions that may have leaked into training data. The provenance of the benchmark matters enormously.
What Benchmarks Miss Explorer
Six scenarios where benchmark scores are misleading. For each, identify why the benchmark failed to predict real-world performance.
How to Read a Benchmark Report
When a company publishes benchmark results, use this mental checklist to evaluate what the numbers actually mean:
Companies choose which benchmarks to report. Missing benchmarks may be ones where the model underperformed. Look for what is NOT shown.
Did they use the standard evaluation protocol, or a modified version? Small changes in prompting, temperature, or few-shot examples can significantly alter scores.
Is the comparison against the latest version of competitors, or older versions? A "we beat GPT-4o" claim means little if GPT-5.5 has been out for months.
If benchmark questions appeared in the model's training data, high scores reflect memorisation, not capability. Reputable reports address this concern explicitly.
A 1-2% improvement on a saturated benchmark is within noise margin. A 15% improvement on a challenging benchmark is genuinely significant. Scale matters.
Self-reported benchmarks are marketing until independently replicated. Check whether third-party evaluation platforms (like LMSYS Chatbot Arena or independent researchers) confirm the claims.
Benchmark Literacy Summary
A high benchmark score tells you what a model can do under ideal conditions. It says very little about how often it will do that correctly in your specific use case, with your data, under real-world constraints. The gap between "can" and "will reliably" is where most AI disappointments live.
When every top model scores above 95% on a benchmark (like HellaSwag or GSM8K), the benchmark no longer differentiates meaningfully. A 96% vs 95% difference is within noise margin. Look for benchmarks where there is still significant headroom — those reveal genuine capability gaps.
No public benchmark perfectly predicts how a model will perform on your specific workflow. The most reliable way to evaluate a model is to test it on your actual tasks, with your actual data, measuring the outcomes you actually care about. Public benchmarks are a useful starting point, not a final verdict.
Always look at the human baseline on any benchmark. On ARC, humans score around 85% while the best AI achieves 30-55% — revealing a massive real gap. On MMLU, top models match or exceed average human performance, but this tells you more about the test format than about genuine understanding. Context is everything.
Models can be optimised to score well on specific benchmarks without genuine improvement in underlying capability. This is called "teaching to the test." When a model shows dramatic improvement on one benchmark but not on related tasks, benchmark gaming should be suspected. Look for consistent improvement across multiple independent evaluations.