MMLU benchmark

MMLU (Massive Multitask Language Understanding; Hendrycks et al., ‘Measuring Massive Multitask Language Understanding’, ICLR 2021, arXiv:2009.03300) is a multiple-choice benchmark of 15,908 questions spanning 57 subjects, from elementary mathematics and US history to professional law and clinical medicine. A model is scored on raw accuracy against the correct answer choice, reported either as a single overall figure or broken out by subject group (humanities, social sciences, STEM, other). It became the dominant headline capability benchmark cited in model releases and technical reports through 2022-2024, and remains widely reported for cross-model comparability even as its usefulness as a differentiator has declined.

What MMLU actually measures

MMLU tests breadth of encoded factual and procedural knowledge under a fixed multiple-choice format, not open-ended reasoning, tool use, or task performance in a deployment setting. Because every question has exactly one correct option among four, a model can score well above random chance through partial knowledge, test-taking heuristics, or memorised answer patterns, without necessarily reflecting the same competence a free-response or applied task would reveal.

Known limitations: saturation and contamination

• Saturation: frontier models plateaued around 86-87% accuracy from GPT-4’s 86.4% (March 2023) onward, with little further separation between subsequent top models. A benchmark that most leading models already answer correctly on roughly seven of every eight questions has limited remaining headroom to distinguish current-generation capability, which is what drove development of harder successors.
• Contamination: because MMLU’s question-and-answer pairs have circulated publicly since 2021, they can appear verbatim or near-verbatim in later models’ pretraining data. Independent contamination audits applying lexical-overlap detection to MMLU questions have found measurable contamination rates that vary sharply by subject (low in some STEM categories, much higher in categories such as philosophy), with estimated accuracy inflation in the low single-digit percentage points depending on category. A high MMLU score for a given model is not on its own evidence the score is contamination-free.

Successors: MMLU-Pro and GPQA

MMLU-Pro (Wang et al., 2024) increases the answer-choice count from four to ten, filters out easy or ambiguous questions, and weights the item set toward reasoning-intensive problems, specifically to restore headroom and reduce the benefit of guessing. GPQA (Rein et al., 2024) takes a different approach: expert-written, graduate-level science questions designed to be resistant to being answered correctly by non-experts searching the open web, aimed at measuring genuinely hard-to-fake domain expertise. Neither retires MMLU; both are commonly reported alongside it in current model technical reports as complementary evidence of remaining headroom.

Why a research-administration audience meets MMLU

Research offices, library/IT procurement teams, and units evaluating AI writing or research-assistant tools frequently encounter MMLU scores in vendor sales materials and technical reports as a shorthand for “how capable is this model.” Because MMLU is saturated and open to contamination, a headline MMLU figure alone is weak evidence for a procurement or research-integrity decision; a defensible evaluation should also check whether the vendor reports newer, harder benchmarks (MMLU-Pro, GPQA) or a broader multi-metric framework such as HELM, which scores calibration, robustness, fairness and toxicity alongside accuracy rather than accuracy alone.

Frequently asked questions

Is MMLU the same as HELM?

No. MMLU is a single multiple-choice knowledge benchmark producing one accuracy figure. HELM is a broader evaluation framework that can run an MMLU-style task as one component among many scenarios, scored across multiple properties (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) rather than accuracy alone.

Why do vendors still cite MMLU if it is saturated?

It remains the most widely reported benchmark historically, so citing it lets a new model be compared against years of prior releases on a common scale, even though it no longer differentiates well among current frontier models.

Does a high MMLU score guarantee good performance on a specific institutional task?

No. MMLU measures multiple-choice knowledge recall across broad academic subjects; it does not test a model’s performance on a specific applied task such as literature summarisation, grant-text drafting, or compliance-document review, which is why procurement evaluations should test the actual intended use case rather than relying on a benchmark score alone.

References

• Hendrycks et al., ‘Measuring Massive Multitask Language Understanding’ (ICLR 2021; arXiv:2009.03300).
• Wang et al., ‘MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark’ (NeurIPS Datasets and Benchmarks Track, 2024).
• Rein et al., ‘GPQA: A Graduate-Level Google-Proof Q&A Benchmark’ (2024).