Building an Internal AI Safety Incident Response Program

An AI safety incident response program is the operational counterpart to a frontier AI risk assessment: the risk register says what could go wrong; the incident response program is what actually happens when something does. Building one from scratch is often treated as a copy-paste exercise from an existing security incident response plan. That produces a program that handles a data breach well and misses the incidents that matter most for AI safety — because the triggers, the people who need to sign off, and the external reporting duties are different.

How This Differs From a Generic Security Incident Response Program

A security incident response program is built around confidentiality, integrity, and availability: unauthorized access, data exfiltration, service disruption. Its triage logic asks what system was affected and how badly. An AI safety incident response program has to ask a different first question: did the model itself do something it should not have been able to do, independent of whether any system was breached.

Three differences matter structurally:

• The trigger can be behavioral, not just technical. A model that evades a safeguard, produces output consistent with a capability it was not supposed to have, or is used to generate content that materially increases a catastrophic risk is a reportable event under some jurisdictions even if no system was compromised and no data left the network.
• Severity classification needs a safety framework, not just a CVSS-style score. Triage has to map an observed event against the organization’s own risk taxonomy (its risk register, its responsible scaling or safety framework commitments) rather than only against confidentiality/integrity/availability impact.
• Some triggers create a legal reporting clock that a generic security plan doesn’t have. California’s SB 53 and New York’s RAISE Act both create statutory reporting deadlines for a defined class of “critical safety incident,” running from the moment of discovery — not from confirmation, not from remediation. A program that only starts its clock at confirmed-and-scoped, the way most security incident response plans do, will already be late.

In practice this means an AI safety incident response program should sit alongside the security incident response plan, share its escalation infrastructure where that makes sense (on-call rotations, ticketing, a war-room process), but run its own triage logic and its own reporting-obligation checklist.

Detection: Where These Incidents Actually Surface

Unlike a network intrusion, a safety incident rarely announces itself with an alert. The organizations most likely to catch one early are watching several different channels, because no single one is reliable on its own:

• Evaluation and red-team findings that surface during pre-deployment testing or ongoing capability evaluations — the same evaluations that feed a responsible scaling or preparedness-style framework.
• Production monitoring and anomaly detection on model outputs and tool-use patterns, distinct from standard application performance monitoring.
• User and customer reports, which require a route into the safety team that doesn’t dead-end in a general support queue.
• Internal employee escalation, including a route that doesn’t require going through the reporting employee’s own manager, since some of the most consequential incidents involve pressure to ship on schedule.
• Third-party and academic disclosure — a researcher or auditor reporting a finding, which needs a published or at least documented intake path so it doesn’t arrive as an unstructured email that sits unread.

The detection layer’s job is narrow: get a candidate event in front of triage fast, with enough of a timestamp and description to start the clock. It is not the detection layer’s job to decide whether the event is reportable — that decision belongs to triage, made against a written classification standard, not to whoever first noticed the event.

Triage and Severity Classification

Triage has one job: take a candidate event and produce a severity classification and an initial reportability determination, on a clock, using a written standard rather than case-by-case judgment. Three elements make that possible:

• A written severity taxonomy tied to the organization’s own risk register categories, not invented fresh for each incident. If the risk register names a catastrophic risk category, the incident taxonomy should use the same labels, so an incident review can trace directly back to the risk it materialized.
• A named triage owner, on-call, who can make the initial severity call within hours, not at the next scheduled meeting. This person does not need to be the final decision-maker on external reporting — that is a separate role, below — but they need the authority to escalate immediately when the facts fit a statutory trigger.
• A reportability checklist, not a judgment call, for each jurisdiction the organization is subject to. For SB 53, that means checking the event against the statute’s four defined categories of critical safety incident (unauthorized access to or exfiltration of model weights causing death or bodily injury, materialization of a catastrophic risk, loss of control causing death or bodily injury, or the model deceptively circumventing the developer’s own monitoring outside a deliberate evaluation, in a way that materially increases catastrophic risk). A checklist that names these four categories explicitly, rather than asking a triage engineer to reason about the statute from memory at 2am, is the difference between a 15-day deadline being met and missed.

Escalation and the Accountable Decision-Maker

A severity classification is not the same as a decision to report externally, and the two should not be made by the same person under time pressure. The structure that holds up under review has three layers:

• Triage classifies and starts the clock.
• A cross-functional review — safety, legal, and the relevant product or research lead — confirms the classification and drafts the external notification if one is required. This group should be pre-named, not assembled ad hoc, and should have a documented maximum response time.
• A single accountable decision-maker, senior enough to be named in the organization’s published safety framework, signs off on the final determination and the external report before it goes out. New York’s RAISE Act makes this explicit on the front end rather than only at incident time: it requires frontier developers to designate senior personnel responsible for compliance with their published safety and security protocol before deployment, which means the accountable decision-maker for an incident should already be named in that protocol, not chosen after the fact.

The reason to separate these roles is specifically to keep the person closest to an incident — who may have built the system in question, or be under commercial pressure to keep it running — from also being the person who decides whether it gets reported. That separation is the single most common gap in programs adapted from generic security incident response plans, where the on-call engineer often has effective sign-off authority by default.

External Reporting Triggers: SB 53 and the RAISE Act

Two statutes currently create a hard external clock for a defined class of AI safety incident, and a program has to track both separately because the definitions, deadlines, and recipients differ.

California SB 53 (the Transparency in Frontier Artificial Intelligence Act) requires a frontier developer to report a critical safety incident to the state Office of Emergency Services within 15 days of discovering it, or within 24 hours if the incident poses an imminent risk of death or serious physical injury, in which case it must also go to a law enforcement or public safety agency with jurisdiction. The obligation attaches to frontier developers — those training models above the statute’s compute threshold — with additional transparency-report and quarterly risk-assessment duties for large frontier developers above a $500 million revenue threshold. See our SB 53 guide for the full definition of critical safety incident and the transparency-report obligations that sit alongside reporting.

New York’s RAISE Act takes effect January 1, 2027, and runs on a faster clock: developers must report safety incidents to New York State authorities within 72 hours of determining an incident occurred, or 24 hours for an incident posing imminent risk of death or serious physical injury. Its reportable-incident list is broader in some respects than SB 53’s — it explicitly includes autonomous model behavior not requested by a user, theft of model weights, and control-system failures, in addition to the categories of critical harm the statute defines. It also requires the pre-deployment safety and security protocol described above, with a named senior compliance owner.

The practical consequence for a reporting-obligation checklist: an event can be reportable under one statute and not the other, on different clocks, to different recipients. A single “is this reportable” checkbox is not sufficient once an organization is subject to more than one of these regimes — the checklist needs a row per statute, each with its own trigger definition and deadline.

Documentation and Post-Incident Review

Every incident that reaches triage should leave a written record independent of whether it clears the bar for external reporting: what was detected, when, the severity classification and who made it, the reportability determination and who signed it, and any remediation taken. This record does two things a verbal or informal process cannot. First, it is what a large frontier developer’s quarterly risk assessment and published transparency report draw from under SB 53 — those documents are only as good as the underlying incident log. Second, it is what turns a single incident into an update to the risk register: a near-miss that didn’t require external reporting is still evidence that a risk the organization identified is more likely, or manifests differently, than assumed. A program that treats sub-threshold incidents as closed rather than as register updates is discarding its own best source of evidence.

Where NIKOLAI Fits

CASRAI’s NIKOLAI dictionary is not an incident response tool and does not replace any part of the program described above — it does not detect, triage, or report anything. What it provides is vocabulary: NIKOLAI’s Track N7 defines a shared set of terms and identifiers for incident types, discovery methods, and reporting-deadline concepts, drawn from published frontier-developer safety frameworks and the regulatory text itself. The practical use case is internal: when a triage checklist, a post-incident report, and a published transparency report each need to describe the same event using the same term for the same underlying concept — not three slightly different phrasings of “loss of control” that make it harder to search incident history later — a shared identifier is what keeps those documents aligned. NIKOLAI does not certify, audit, or endorse an organization’s program; it is a dictionary, and CASRAI states plainly that no lab, evaluator, or regulator has reviewed or endorsed its mappings.

Frequently Asked Questions

Does a near-miss that never affected a live system need to go through this program?

Yes. Both SB 53 and the RAISE Act define reportable incidents by what the model did or what was exposed — not by whether a downstream system was compromised. A capability evaluation or red-team exercise that surfaces a qualifying event should go through the same triage path as a production incident, even though it happened in a controlled setting. (Note: an evaluation specifically designed to elicit deceptive circumvention is carved out of that category under SB 53’s definition — the deceptive-technique trigger applies outside the context of such an evaluation.)

Can the security incident response team just add AI safety incidents to their existing process?

They can share infrastructure — on-call tooling, a war room process, a ticketing system — but the triage logic and the sign-off authority should not simply be folded in. A security team’s severity taxonomy is built around CIA-triad impact, which does not map cleanly onto categories like loss of control or catastrophic risk materialization. At minimum, the reportability checklist and the accountable decision-maker role need to be specific to AI safety incidents.

What’s the single most common gap in a first-draft program?

Starting the reporting clock at the wrong point. Both statutes start the clock at discovery or determination, not at the point an organization has finished investigating, scoped the impact, and drafted a report. A program that treats the statutory deadline as the deadline to finish the investigation, rather than the deadline to notify, will miss it.

Do SB 53 and the RAISE Act require the same accountable decision-maker?

The statutes don’t require a single named individual to hold both roles, but there is no structural reason to split them, and most organizations subject to both will find it simpler not to. What the RAISE Act adds is a pre-deployment requirement to name senior personnel responsible for compliance in the published safety and security protocol — which is a natural place to also name the incident accountable decision-maker, even though the statute does not require they be the same appointment.