What Gets Graded: Model, Company, or Document

No, and the overlap is smaller than it looks. FLI grades nine companies and SaferAI grades twelve, but only five sit on both lists: Anthropic, OpenAI, Google DeepMind, Meta and xAI. DeepSeek, Mistral, Z.ai and Alibaba Cloud are graded by FLI and absent from SaferAI, which instead covers Microsoft, Amazon, NVIDIA, Cohere, Naver, G42 and Magic. AILuminate ranks neither list, because it ranks configured chatbots. If you need one table covering one set of vendors, you have to build it yourself and label which grader supplied which column.

Both, usually. Meta at 33 percent on SaferAI and D+ on the FLI Index is the clearest case in the current data. SaferAI read Meta’s published frontier safety framework and found it drafted roughly as thoroughly as Anthropic’s. FLI read Meta’s whole public record across risk assessment, current harms, safety frameworks, existential safety, governance and information sharing, and found most of those domains weak. A well-written framework document inside a company with a poor overall disclosure record produces exactly this pattern, and noticing it is more informative than either number alone.

No, and MLCommons says so in its own materials: “Performing well on the benchmark does not mean that your model is safe, simply that we have not identified critical safety weaknesses.” The benchmark is designed to have negative predictive power only. Four of its five grade bands are defined relative to a reference system rather than to an absolute violation rate, results carry considerable variance from prompt selection and automated evaluation, and the current suite is single-turn and English-and-French-first. A grade tells you a weakness was not found in that configuration under those prompts.

It depends on the question. If you are standing up a campus AI assistant, the object you care about is your own deployment, which is its own system-under-test: the vendor’s bare-model grade does not transfer to your configuration once you add guardrails, a retrieval layer and a system prompt, and AILuminate’s own SUT definition is the reason why. If the question is instead whether the vendor is an organisation you can rely on across a multi-year agreement, that is an FLI and SaferAI question. A sponsored programs or research security review asking about governance, disclosure and incident handling gets nothing useful from a hazard benchmark, and an IRB asking whether a study tool will hand participants unqualified medical or legal advice gets nothing useful from a company letter grade.

None of the three. All are voluntary third-party assessments by non-governmental organisations, and no regulator adopts their scores. What regulation does require, in places, is the underlying artefact rather than the grade: California SB 53 and the EU AI Act’s general-purpose AI obligations put weight on a developer publishing and maintaining its own framework and reporting. That makes SaferAI’s object, the framework document, the one that most closely tracks what a statute already expects to exist, without making SaferAI’s percentage a compliance finding.

Because it makes the comparison auditable. The tracker applies 65 criteria across four equally weighted dimensions, and attaches to each score a rationale quoting the company’s framework with page numbers, so an outside reader can check the scoring rather than trust it. That is only possible against a fixed public text. The cost of that choice is the obvious one, and SaferAI does not hide it: the tracker measures what a company has written down, not what it does.

Not directly, and it is the largest shared gap. A model can behave differently when it detects it is being evaluated, can underperform deliberately, or can be optimised against a known grader. CASRAI’s NIKOLAI dictionary carries a controlled list for exactly these conditions under its N5 evidence and evaluations track, the evaluation-validity threat element, naming evaluation awareness, sandbagging, alignment faking, metagaming or grader-gaming, and reward hacking. NIKOLAI is CASRAI’s own independent dictionary and is not endorsed by MLCommons, the Future of Life Institute or SaferAI; any alignment between its terms and theirs is a shadow mapping unless one of those organisations files a Mapping Declaration.