Examples
Worked examples
- Is an instance
A contracted red team probes a deployed chatbot for ways to extract another user’s private data via crafted prompts.
- Is an instance
One model is used to automatically generate thousands of adversarial prompts to test another model’s refusal behaviour at scale.
Counter-examples
Looks similar, but isn't
- Not an instance
Running a model against a fixed benchmark like BIG-bench with a defined scoring rule is evaluation, not red-teaming, because it tests known capabilities against a known answer key rather than searching for unknown failure modes.
Editorial commentary
Red-teaming is an adversarial testing method: people (or, increasingly, other models) deliberately try to make an AI system produce harmful, unsafe, policy-violating, or otherwise undesired output, in order to surface failure modes before deployment. It is a practice borrowed directly from information-security and military exercise traditions, where a designated ‘red team’ attacks a system the way a real adversary would, rather than checking it against a known list of requirements.
How this differs from a model evaluation suite
A model evaluation suite runs a model against a fixed benchmark and a defined scoring rule — it measures known capabilities against a known answer key, reproducibly. Red-teaming is adaptive and open-ended: testers actively search for unknown failure modes the benchmark authors never anticipated, adjusting their approach as they learn what the model will and won’t do. A model can score well on every standard benchmark and still fail badly under red-teaming, because the two methods are looking for different things — one confirms known capabilities, the other hunts for the unknown.
How this differs from a safety case
An AI safety case is the structured, evidence-based argument that a system is acceptably safe to deploy. Red-team findings are typically one input into that argument’s evidence base — they show what was tried and what happened — not a substitute for the case itself, which also has to argue why the evidence supports a deployment decision.
Human and automated red-teaming
Early AI red-teaming relied on human testers probing a model manually. As models scaled, automated or ‘AI red-teaming’ emerged alongside it: using one model to generate large volumes of adversarial prompts against another, to cover far more of the input space than a human team could manage in the same time, at the cost of missing the creative, context-aware attacks a skilled human tester finds. Most serious red-teaming programs now combine both.
A common target for red-teaming exercises is prompt injection and jailbreak resistance — testers specifically try to get a deployed system to ignore its own instructions or leak information it shouldn’t, which is exactly the class of failure a fixed benchmark is least likely to catch.
References
- Ganguli et al., ‘Red Teaming Language Models to Reduce Harms’ (arXiv, 2022); Perez et al., ‘Red Teaming Language Models with Language Models’ (arXiv, 2022); OpenAI ‘GPT-4 System Card’ (2023).
Also known as
AI red team · adversarial AI testing
Machine-readable encodings
Use in your systems
<role vocab="credit"
vocab-identifier="https://casrai.org/dictionary/"
vocab-term="Red-teaming"
vocab-term-identifier="https://casrai.org/dictionary/term/red-teaming" />{
"@context": "https://schema.org",
"@type": "DefinedTerm",
"@id": "https://casrai.org/dictionary/term/red-teaming",
"name": "Red-teaming",
"identifier": "https://casrai.org/dictionary/term/red-teaming",
"description": "Adversarial testing in which people or models deliberately try to make an AI system produce harmful, unsafe, or policy-violating output, surfacing failure modes that fixed benchmarks don't test for.",
"inDefinedTermSet": "https://casrai.org/dictionary/domain/ai-ml-research-outputs#set",
"url": "https://casrai.org/dictionary/term/red-teaming",
"sameAs": [
"AI red team",
"adversarial AI testing"
],
"license": "https://creativecommons.org/licenses/by/4.0/",
"publisher": {
"@id": "https://casrai.org/#organization"
},
"author": {
"@id": "https://casrai.org/#editorial-team"
},
"datePublished": "2026-05-21T02:22:51",
"dateModified": "2026-08-23T23:49:52",
"inLanguage": "en-GB",
"isAccessibleForFree": true
}
