Written and maintained by CASRAI Editorial Board
Last updated
On this page: what Cohen’s kappa measures and why raw percent agreement overstates reliability; the formula, built from observed agreement minus chance-expected agreement; a fully worked 2×2 example computed step by step; the Landis & Koch interpretation bands, and why they are a convention rather than a validated scale; why Cohen’s kappa is specifically the two-rater case, with a direct handoff to Fleiss’ kappa for three or more raters; and the kappa paradox — the well-documented failure mode where high raw agreement still produces a low or even negative kappa.
What Cohen’s kappa measures
Cohen’s kappa (κ) is the standard chance-corrected agreement statistic for exactly two raters classifying the same set of subjects into unordered (nominal) categories — two qualitative coders assigning the same theme label to a passage, two radiologists calling a scan positive or negative, two abstract screeners marking include/exclude. Jacob Cohen introduced it in 1960 (“A Coefficient of Agreement for Nominal Scales,” Educational and Psychological Measurement 20(1), 37–46) specifically to answer a question raw percent agreement cannot: how much of the observed agreement is real, versus how much would two raters achieve just by guessing at their own typical response rates?
That correction matters because percent agreement alone is easy to inflate without either rater doing any genuine discrimination. Two screeners who both mark 95% of abstracts “exclude” without reading closely will agree on roughly 95% of cases even if neither is applying the inclusion criteria correctly. Kappa asks what fraction of the observed agreement sits above what that kind of guessing would already produce.
The formula
κ = (Po − Pe) / (1 − Pe)
Po is the observed proportion of agreement — the share of subjects both raters classified the same way. Pe is the proportion of agreement expected by chance, computed from the product of each rater’s own marginal totals: for each category, multiply the proportion of times Rater A used it by the proportion of times Rater B used it, then sum across categories. The denominator, 1 − Pe, is the maximum possible improvement over chance — so kappa expresses observed-minus-chance agreement as a fraction of the headroom actually available.
κ = 1 means perfect agreement. κ = 0 means the raters did no better than their own chance-guessing rates would predict. Negative values are possible and mean agreement worse than chance — usually a sign of a confused coding scheme, ambiguous category definitions, or one rater working from different underlying assumptions than the other, not literal perversity.
Worked example: a 2×2 agreement table
Two raters independently review the same 50 case files and classify each as showing a symptom “Present” or “Absent”:
| Rater B: Present | Rater B: Absent | Row total | |
|---|---|---|---|
| Rater A: Present | 20 | 5 | 25 |
| Rater A: Absent | 5 | 20 | 25 |
| Column total | 25 | 25 | 50 |
Step 1 — observed agreement. The raters agreed on 20 + 20 = 40 of the 50 cases: Po = 40/50 = 0.80.
Step 2 — expected agreement. Rater A said “Present” on 25/50 = 0.50 of cases and “Absent” on 0.50. Rater B’s marginals are identical: 0.50 and 0.50. Multiply and sum:
Pe = (0.50 × 0.50) + (0.50 × 0.50) = 0.25 + 0.25 = 0.50
Step 3 — kappa.
κ = (0.80 − 0.50) / (1 − 0.50) = 0.30 / 0.50 = 0.60
Note what this shows: 80% raw agreement, but only a kappa of 0.60, because half of that 80% would have been expected from chance alone given how often each rater used each category. This is the ordinary, well-behaved case — balanced marginals, no paradox. The paradox section below shows what happens when the marginals are skewed instead.
Interpreting the result: useful convention, not a validated scale
The interpretation bands most commonly cited come from Landis, J.R. & Koch, G.G., “The Measurement of Observer Agreement for Categorical Data,” Biometrics 33(1), 1977, 159–174:
| Kappa | Landis & Koch label |
|---|---|
| < 0.00 | Poor |
| 0.00–0.20 | Slight |
| 0.21–0.40 | Fair |
| 0.41–0.60 | Moderate |
| 0.61–0.80 | Substantial |
| 0.81–1.00 | Almost perfect |
The worked example above, κ = 0.60, lands right at the top of “moderate.”
These bands are cited constantly, and just as constantly over-trusted. Landis and Koch themselves described the divisions as clearly arbitrary, offered only as useful benchmarks for their own paper’s context — not as a derived or field-validated pass/fail threshold. Later methodologists have been blunter: Ludbrook (2002) called the approach lacking any sound theoretical basis and capable of being “positively misleading to investigators.” Treat the bands as a shared vocabulary for describing a kappa value in a results section, not as a standard your study either meets or fails. What counts as an acceptable kappa for a given purpose (screening two coders’ agreement in a pilot round versus certifying a diagnostic tool for clinical use) is a field- and stakes-specific judgment the bands don’t make for you.
The two-rater limit — when you need Fleiss’ kappa instead
Cohen’s kappa is defined for exactly two raters, full stop. Its expected-agreement term is built directly from two individual raters’ marginal distributions multiplied together, and there is no natural way to extend that construction to three, five, or twelve raters — whose marginals would you multiply, and in what order? If your design has three or more raters per subject, see CASRAI’s guide to Fleiss’ kappa for multiple raters, which reformulates the problem around per-subject category counts rather than individual rater identity, and also handles a rotating pool of raters where the specific people scoring each subject vary. One detail worth knowing before assuming Fleiss’ kappa is just “Cohen’s kappa with more raters”: in the two-rater case Fleiss’ kappa does not reduce to Cohen’s kappa — it reduces to a related but distinct statistic, Scott’s pi, because Fleiss’ kappa computes its chance term from pooled category rates rather than each rater’s individual marginals.
If your two categories are ordered rather than nominal — severity grades, Likert ratings, tumor stage — unweighted Cohen’s kappa treats every disagreement as equally bad, which understates agreement when raters are merely off by one adjacent category. Cohen’s own 1968 extension, weighted kappa, corrects for this with linear or quadratic penalty weights; see the weighted-kappa section of CASRAI’s guide to choosing an inter-rater reliability coefficient for how the two weighting schemes differ and when each applies.
The kappa paradox: why high agreement doesn’t guarantee a good kappa
Cohen’s kappa has a well-documented, frequently overlooked failure mode, formalized by Feinstein, A.R. & Cicchetti, D.V. in a pair of 1990 papers usually cited together as “the kappa paradox.” The mechanism is the expected-agreement term, Pe: when one category dominates the data — most patients are healthy, most abstracts get excluded, most items are coded “no issue” — Pe climbs toward 1 almost automatically, because both raters are very likely to land on the majority category regardless of any real judgment. That leaves almost no headroom (1 − Pe) for kappa to reward, so even a genuinely well-agreeing pair of raters can score a disappointing kappa.
A compact illustration makes the mechanism concrete. Two raters again classify 100 cases as “Present” or “Absent,” but this time the condition is rare:
| Rater B: Present | Rater B: Absent | Row total | |
|---|---|---|---|
| Rater A: Present | 1 | 2 | 3 |
| Rater A: Absent | 2 | 95 | 97 |
| Column total | 3 | 97 | 100 |
Raw agreement here is (1 + 95)/100 = 96% — higher than the first worked example above. But the marginals are 0.03/0.97 for both raters, so Pe = (0.03 × 0.03) + (0.97 × 0.97) = 0.0009 + 0.9409 = 0.9418, leaving very little headroom:
κ = (0.96 − 0.9418) / (1 − 0.9418) = 0.0182 / 0.0582 ≈ 0.31 — “fair” on the Landis & Koch scale, despite 96% raw agreement, and numerically worse than the balanced 80%-agreement example earlier on this page, which scored 0.60. The raters didn’t get worse; the prevalence changed. Feinstein and Cicchetti describe a second, related paradox: under unbalanced marginals, raters who actively disagree about how common the condition is can sometimes produce a higher kappa than raters who agree on its base rate, at the same level of raw agreement — the opposite of the intuitive direction.
The practical fix is not to abandon kappa but to never report it alone. Always report raw percent agreement alongside kappa — it’s the only figure that lets a reader diagnose a paradoxical result. When prevalence is genuinely skewed and this pattern is a live concern, Gwet’s AC1 is a purpose-built alternative that keeps kappa’s same (Po − Pe)/(1 − Pe) shape but redefines the chance term so it doesn’t inflate under skew — report it as a companion to kappa, decided in advance, not as a post-hoc swap once kappa disappoints. CASRAI’s guide to choosing an inter-rater reliability coefficient covers Gwet’s AC1, Krippendorff’s alpha, and the full coefficient-selection logic in depth.
Other limitations, and what to report alongside kappa
- Exactly two raters, full stop. Three or more requires Fleiss’ kappa (fixed number of raters per subject, pool can vary) or Krippendorff’s alpha (any number of raters, and the only common option that handles an incomplete rating matrix without discarding cases).
- Nominal categories only, unweighted. For ordered categories, use weighted kappa (linear or quadratic weights) rather than treating every disagreement as equally severe.
- Sensitive to category prevalence. See the paradox above — always pair kappa with raw percent agreement.
- Requires a complete rating matrix. Both raters must have scored every subject; a coder who dropped out partway or a case only one rater reached forces listwise deletion or a switch to Krippendorff’s alpha.
- Doesn’t distinguish two different disagreement structures that produce the same kappa value — a report should still show the full confusion matrix, not just the coefficient, so a reader can see whether disagreement is scattered or concentrated in one category pair.
Frequently asked questions
What counts as a “good” Cohen’s kappa?
There’s no universal cutoff. The Landis & Koch bands (above) are the most commonly cited convention, but the authors themselves called the divisions arbitrary, and what’s acceptable depends heavily on the stakes of the classification and the base rate of the categories involved — see the kappa paradox section for why the same raters can score very differently under different prevalence.
Can Cohen’s kappa be negative?
Yes. A negative kappa means the raters agreed less often than their own individual response rates would predict by chance, which usually signals a coding-scheme or definitional problem rather than raters who are somehow “worse than random” in any literal sense.
What’s the difference between Cohen’s kappa and Fleiss’ kappa?
Cohen’s kappa is defined for exactly two raters and computes chance agreement from each rater’s individual marginals. Fleiss’ kappa generalizes to three or more raters (with the rater pool allowed to vary by subject) and computes chance agreement from pooled category rates instead — which is also why Fleiss’ kappa doesn’t reduce to Cohen’s kappa in the two-rater case; it reduces to Scott’s pi. See CASRAI’s Fleiss’ kappa guide for the full formula and a worked multi-rater example.
Is Cohen’s kappa the same as percent agreement?
No, and confusing the two is the single most common misreading of an agreement statistic. Percent agreement is the raw share of matching classifications; Cohen’s kappa is that same figure adjusted for how much agreement the raters’ own response rates would have produced by chance alone. Two raters with 95% raw agreement can have a kappa anywhere from close to 1 down to near zero or negative, depending on category prevalence — always report both numbers together.
What software computes Cohen’s kappa?
SPSS (Crosstabs > Statistics > Kappa), R (the irr or psych packages), Stata (kap), and Python (scikit-learn‘s cohen_kappa_score) all compute it directly from a two-rater confusion matrix; most also report a standard error and confidence interval alongside the point estimate.
