What Is AI Safety? A Plain-Language Guide

Last verified: September 20, 2026. “AI safety” is one of the most-searched terms in this whole subject area, and one of the least precisely defined in casual use. People type it looking for very different things: a plain definition, a list of who works on it, how it relates to AI ethics or AI alignment, or what a company means when it says its model “passed safety testing.” This page is the definitional starting point — it explains what the field actually covers, introduces the working vocabulary the rest of this site’s frontier-AI-safety coverage assumes you already know, and maps out how the work is organized in practice, with links to the deeper page on each piece.

What “AI Safety” Actually Means

AI safety is the field of research, engineering, and governance work aimed at preventing AI systems from causing harm. That harm can come from three different directions, and most confusion about the term comes from people picturing only one of them:

• Misuse — a person deliberately directs a capable AI system toward a harmful end: generating a bioweapon synthesis route, running a large-scale disinformation campaign, or automating cyberattacks.
• Accidents — the system causes harm without anyone intending it to, because it was deployed with a flaw, a gap in testing, or in a context its safeguards weren’t built for.
• Loss of control — a longer-horizon concern, mostly discussed in the context of highly capable future systems, that an AI system could end up pursuing goals or taking actions its operators did not intend and cannot correct or shut down.

Under that umbrella sits a wide range of concrete work: evaluating models for dangerous capabilities before release, building technical safeguards like content filters and monitoring systems, writing the internal policies that govern when a model is safe to deploy, and the external government and institutional oversight layered on top of all of it. “AI safety” is the field name; the rest of this site’s frontier-AI-safety cluster is mostly about the specific mechanisms inside it.

AI Safety vs. AI Security

The two terms get used interchangeably in casual conversation, but they answer different questions: AI safety is about what a system does — whether its own behavior or capabilities cause harm — while AI security is about what is done to a system, or what someone does with it once they’ve compromised it. A model that confidently generates a wrong medical dosage is a safety failure; a model whose weights get stolen off a company’s servers is a security failure. The two disciplines overlap in practice — a lab’s frontier safety framework typically requires both — but they are not the same job. For the full distinction, see AI Safety vs. AI Security.

The Working Vocabulary

Once you’re past the plain-language definition, the field runs on a small set of recurring concepts. Every deeper guide on this site — on a specific lab’s framework, a specific statute, a specific evaluator — assumes you already have these five terms straight. Note that these are working definitions, not statutory or universally agreed ones — usage varies somewhat by lab and by regulator, which is exactly the kind of variation CASRAI tracks via its own unendorsed NIKOLAI dictionary of frontier-AI-safety elements, rather than treating as settled.

• Thresholds — the specific capability level (e.g., a model that can meaningfully uplift someone attempting to build a bioweapon) that triggers extra scrutiny or restrictions before a lab will deploy a system. CASRAI tracks this concept, and the checkpoints tied to it, in NIKOLAI’s N3 (Thresholds & Checkpoints) track.
• Evaluations — the structured tests run against a model to find out whether it has crossed one of those thresholds, including who ran the test and what techniques they used to try to elicit the behavior. NIKOLAI’s N5 (Evidence & Evaluations) track is where CASRAI maps this vocabulary.
• Safeguards — the mitigations put in place once a risk is identified: monitoring, access restrictions, output filtering, or a decision not to deploy a capability at all. CASRAI tracks safeguard and mitigation terminology in NIKOLAI’s N6 (Mitigations & Security) track.
• Incident reporting — what counts as a safety incident, who has to be told, and on what clock. This varies significantly by statute and by lab policy — CASRAI maps the different definitions and deadlines in NIKOLAI’s N7 (Incidents) track.
• Accountability roles — the specific people or functions a framework designates as responsible for a safety decision, such as an accountable decision-maker who has to sign off before a risky deployment proceeds. CASRAI tracks these role definitions in NIKOLAI’s N10 (Assurance & Roles) track.

How AI Safety Work Is Actually Organized

In practice, the field runs on three layers that check and reference each other. None of the three is a single, universal rulebook — each is a family of documents and bodies that overlap in subject matter but differ in who wrote them and how much force they carry.

Frontier Labs’ Own Safety Frameworks

The major frontier AI developers each publish their own internal policy describing how they assess and manage catastrophic risk — Anthropic’s Responsible Scaling Policy (RSP), OpenAI’s Preparedness Framework, and Google DeepMind’s Frontier Safety Framework (FSF) chief among them. These are voluntary documents, not law, though several US states now require large developers to publish something that covers similar ground under a different statutory name. See RSP vs. Preparedness Framework vs. FSF for how the three compare, and Frontier AI Labs for the fuller roster of who publishes what.

Government Frameworks

Two government frameworks come up constantly in this space, and they work differently from each other. The NIST AI Risk Management Framework, released by the U.S. National Institute of Standards and Technology on January 26, 2023, is voluntary guidance organized around four functions — Govern, Map, Measure, and Manage — meant to help any organization incorporate trustworthiness considerations into how it builds and evaluates AI systems. The EU AI Act is the opposite in character: a binding statute that classifies AI systems into risk tiers — unacceptable, high, limited (transparency), and minimal — and attaches escalating legal obligations to each, with its high-risk system requirements phasing in through 2027 and 2028. See the EU AI Act high-risk compliance checklist for the deadline-by-deadline detail.

Independent Assessment Bodies

The third layer sits outside both the labs and the legislatures: government-run AI Safety Institutes and independent third-party evaluators who test models rather than write policy about them. The U.S. equivalent is the Center for AI Standards and Innovation (CAISI) — see what is CAISI — and a number of national AI Safety Institutes now coordinate through a shared network, covered in the AI Safety Institute network guide. Separately, private third-party evaluators conduct model testing under their own methodologies and independence standards, detailed in third-party AI evaluator standards.

Where NIKOLAI Fits In

Because the working vocabulary above — thresholds, evaluations, safeguards, incident reporting, accountability roles — is used somewhat differently by every lab, statute, and evaluator this page links to, comparing them directly is harder than the shared words suggest. CASRAI built NIKOLAI, its own frontier-AI-safety dictionary of elements, specifically to make those differences visible rather than to paper over them: 64 elements across 10 tracks (N1–N10), spanning exactly the five concepts covered above, each with a plain definition and links to where different organizations use the term. It is important to be precise about what NIKOLAI is not: it is an independent, unendorsed CASRAI reference work, not an official standard, and no lab, regulator, or evaluator has approved its mappings. Every crosswalk row in NIKOLAI is a “shadow mapping” — CASRAI’s own read of how a term is used — unless an organization has gone through NIKOLAI’s Mapping Declarations process to verify its identity and confirm how the term maps in its own usage. If you work at a lab, regulator, or evaluator and want your organization’s own usage reflected rather than CASRAI’s inferred mapping, that declaration process is the way to do it.

Frequently Asked Questions

What does AI safety mean, in one sentence?

AI safety is the field of work aimed at preventing AI systems from causing harm, whether through deliberate misuse, unintended accidents, or — in longer-horizon discussions — loss of human control over the system’s behavior.

Is AI safety the same thing as AI ethics?

No, though they overlap. AI ethics is the broader field examining fairness, bias, transparency, and the social impact of AI systems in ordinary use. AI safety is narrower and more focused on preventing serious, often catastrophic or irreversible harm, particularly from highly capable systems — it is closer to a risk-management and engineering discipline than a purely normative one.

What’s the difference between AI safety and AI alignment?

AI alignment is usually treated as a sub-problem within AI safety: specifically, the technical challenge of making an AI system actually pursue the goals its designers intended, rather than a proxy for them. AI safety is the broader field that includes alignment research alongside evaluation, governance, and incident response. See What Is AI Alignment? for the deeper explainer.

What is “frontier AI safety” specifically?

It’s the subset of AI safety work focused on frontier models — the most capable systems at or near the current state of the art, where the catastrophic-risk categories (like bio, cyber, or loss-of-control risk) are considered most plausible. See What Is a Frontier AI Model? for how that term itself is defined.

Who actually works on AI safety?

A mix of people inside frontier labs (safety researchers, red-teamers, policy staff), government bodies (AI Safety Institutes, standards agencies like NIST), independent nonprofits and evaluators, and academic researchers. See The Center for AI Safety and AI Red Teaming for two concrete examples of what that work looks like day to day.

Where did the modern AI safety field come from?

Its current shape traces substantially to a 2016 paper, “Concrete Problems in AI Safety”, which reframed safety away from speculative long-term scenarios and toward specific, empirically tractable engineering problems — a framing that heavily influenced how frontier labs structure their safety work today.