Written and maintained by CASRAI Editorial Board
Last updated
Cochran’s Q test is the extension of McNemar’s test to three or more related dichotomous measurements — the same subjects (or matched units) scored on a binary outcome under three or more conditions, or at three or more time points, instead of just two. Where McNemar’s test can only ask “did this binary outcome change between two measurements,” Cochran’s Q asks the same question across a whole repeated-measures design at once, then leaves the question of which pairs of conditions differ to a post-hoc procedure — almost always a set of pairwise McNemar tests with a correction for multiple comparisons.
What Cochran’s Q Test Does
Cochran’s Q test evaluates whether the proportion of “successes” (however the binary outcome is coded — pass/fail, present/absent, yes/no, correct/incorrect) is the same across k related conditions, where k is 3 or more. “Related” means within-subject or matched: every subject contributes one binary score to every condition, the same way a repeated-measures ANOVA requires every subject to contribute a score under every condition — Cochran’s Q is that same logic applied to a dichotomous rather than continuous outcome.
The test was introduced by William G. Cochran in 1950 specifically to generalize the paired-comparison logic later formalized as McNemar’s test (1947) to more than two related samples. When k = 2, Cochran’s Q reduces exactly to McNemar’s chi-square statistic — they are the same test at that boundary, not two independently-derived procedures that happen to agree.
When to Use It — and When Not To
- Use it when: the same subjects (or matched blocks) are scored on a binary outcome under three or more related conditions — repeated time points, successive treatment phases, or the same cases judged against three or more criteria/raters.
- Use McNemar’s test instead when: there are only two related measurements. Cochran’s Q with k = 2 gives the identical result, but McNemar’s test is the more familiar and more directly interpretable tool for that specific case.
- Use the Friedman test instead when: the repeated outcome is ordinal or continuous-but-non-normal rather than strictly binary. Friedman ranks each subject’s scores across conditions; Cochran’s Q only has two possible values (0 or 1) to work with, so ranking does not apply.
- Use a repeated-measures ANOVA instead when: the outcome is continuous and its assumptions (normality, sphericity) hold. Cochran’s Q throws away information a continuous measure would give you — only use it because the outcome is genuinely dichotomous, not as a “safer” substitute for a continuous test.
- Use a chi-square test of independence instead when: the groups being compared are made up of different, independent subjects rather than the same subjects measured repeatedly — that is the ordinary unpaired case Cochran’s Q is not built for.
The Formula and Degrees of Freedom
Arrange the data as an N (subjects) × k (conditions) table of 0s and 1s. Let Cj be the column total for condition j (how many subjects scored 1 under that condition), let Ri be the row total for subject i (how many of the k conditions that subject scored 1 on), and let C̄ be the mean column total (the grand total of all 1s, divided by k). The test statistic is:
Q = k(k−1) × Σ(Cj − C̄)² ÷ ΣRi(k − Ri)
Under the null hypothesis that the success probability is equal across all k conditions, Q is approximately distributed as chi-square with df = k − 1. That approximation is the standard large-sample route; it weakens the same way the ordinary chi-square approximation does when the table is sparse (row totals bunched near 0 or near k, so that Ri(k−Ri) is small for most subjects). For small samples, exact distributions for Cochran’s Q exist (Patil’s 1975 algorithm and later refinements) and are worth reaching for instead of trusting the chi-square approximation on a thin dataset, the same way the exact binomial version of McNemar’s test replaces its chi-square approximation when the discordant total is small.
Assumptions
- Dichotomous outcome, common coding across conditions. Every score is a 0/1 (or equivalent two-level) value, and “1” has to mean the same thing in every condition — a passed inspection in condition A has to be comparable to a passed inspection in condition B.
- The same subjects (or genuinely matched blocks) contribute a score to every condition. Cochran’s Q needs a complete N × k table — a subject missing even one condition has no valid row and is ordinarily dropped listwise, which can matter a great deal if the missingness is not random with respect to the outcome.
- Subjects are independent of each other. The repeated structure is within-subject across conditions, not between subjects — one subject’s set of scores should not depend on another subject’s.
- A large enough, non-degenerate sample for the chi-square approximation. Rows where every condition came back the same (all-0 or all-1 across the board) contribute nothing to the statistic and effectively reduce the usable sample — a design with many such rows needs more subjects than the raw N suggests.
Worked Example: A Certification Check at Three Time Points
Illustrative example, not a real study. Fifteen employees take the same pass/fail safety-certification quiz at three points: immediately before a training session, immediately after it, and again at a three-month follow-up. “1” means passed; “0” means failed.
| Employee | Pre-training | Post-training | 3-month follow-up |
|---|---|---|---|
| 1 | 0 | 1 | 1 |
| 2 | 0 | 1 | 1 |
| 3 | 0 | 1 | 0 |
| 4 | 0 | 1 | 1 |
| 5 | 0 | 0 | 1 |
| 6 | 1 | 1 | 1 |
| 7 | 0 | 1 | 1 |
| 8 | 0 | 1 | 1 |
| 9 | 0 | 0 | 0 |
| 10 | 0 | 1 | 1 |
| 11 | 1 | 1 | 1 |
| 12 | 0 | 1 | 1 |
| 13 | 0 | 1 | 0 |
| 14 | 0 | 1 | 1 |
| 15 | 0 | 1 | 1 |
| Column total (Cj) | 2 | 13 | 12 |
Column totals: pre-training C₁ = 2, post-training C₂ = 13, follow-up C₃ = 12, out of N = 15. The grand total is 27, so the mean column total C̄ = 27 ÷ 3 = 9. Summing Ri(k−Ri) across all 15 subjects’ row totals gives 24 for the denominator, and k(k−1)Σ(Cj−C̄)² gives 3 × 2 × [(2−9)² + (13−9)² + (12−9)²] = 6 × (49+16+9) = 6 × 74 = 444 for the numerator.
Q = 444 ÷ 24 = 18.5, df = 2, p < .001. Pass rates are not equal across the three time points — but Q alone does not say where the difference lies. That is the post-hoc procedure’s job.
Post-Hoc Testing: Pairwise McNemar Tests with a Bonferroni Correction
A significant Cochran’s Q, like a significant Friedman test, only tells you that at least one condition differs from at least one other — it does not identify which pair. Because Cochran’s Q with exactly two conditions is McNemar’s test (see above), the natural post-hoc procedure is to run McNemar’s test — the exact binomial version where the discordant total is small, which it usually is in a three-comparison follow-up — on every pair of conditions, with the significance threshold adjusted for running multiple tests. A Bonferroni correction divides the family-wise alpha by the number of pairwise comparisons: with k = 3 conditions there are 3 pairs, so α = .05 ÷ 3 ≈ .0167 per comparison. (Some practitioners prefer a Benjamini-Hochberg false discovery rate correction instead of Bonferroni when there are many pairs to compare; either is defensible, but state which one you used.)
Applying that to the worked example, using the exact binomial version of McNemar’s test on each pair (b and c are the discordant counts — subjects who switched from fail to pass, or pass to fail, between that pair of conditions):
- Pre-training vs. post-training: b = 11 (fail→pass), c = 0 (pass→fail). Exact p ≈ .001 — significant even against the corrected threshold.
- Pre-training vs. follow-up: b = 10, c = 0. Exact p ≈ .002 — significant.
- Post-training vs. follow-up: b = 1, c = 2. Exact p = 1.0 — not significant.
The overall Q result is driven entirely by the jump from pre-training to the other two time points; pass rates did not change further between immediately-post and the three-month follow-up. That is a materially more useful conclusion than the omnibus Q statistic alone — and it is exactly the kind of claim a Q result on its own cannot support without the pairwise follow-up.
How to Report the Result
State the statistic, degrees of freedom, sample size, and which post-hoc correction was used:
“A Cochran’s Q test indicated that pass rates differed significantly across the three time points, Q(2, N = 15) = 18.50, p < .001. Post-hoc pairwise comparisons using the exact McNemar test with a Bonferroni correction (α = .0167) showed pass rates increased significantly from pre-training to post-training (p = .001) and from pre-training to follow-up (p = .002), with no significant change between post-training and follow-up (p = 1.0).”
Running It in Statistical Software
R has Cochran’s Q available through the rstatix package’s cochran_qtest() function (and a corresponding pairwise_mcnemar_test() for the post-hoc step), or the base-level DescTools package. SPSS runs it under Analyze > Nonparametric Tests > Related Samples (or the legacy k Related Samples dialog with Cochran’s Q selected). Python’s statsmodels library implements it as cochrans_q() in statsmodels.stats.contingency_tables. Whichever tool is used, confirm it is reporting the exact or chi-square-approximation version consistently with what is stated in the write-up — the two can diverge meaningfully on a small dataset.
Frequently Asked Questions
What is Cochran’s Q test used for?
Testing whether the proportion of a binary (yes/no) outcome differs across three or more related measurements of the same subjects — for example, pass/fail status at three time points, or agree/disagree ratings from the same reviewers across three or more items.
How is Cochran’s Q test different from McNemar’s test?
They are the same test at different scales. McNemar’s test handles exactly two related dichotomous measurements; Cochran’s Q handles three or more. When Cochran’s Q is applied to exactly two conditions, it produces the identical statistic as McNemar’s test.
What is the difference between Cochran’s Q test and the Friedman test?
Both are nonparametric tests for k ≥ 3 related samples, but they apply to different data types. Cochran’s Q requires a strictly dichotomous (0/1) outcome. The Friedman test is for ordinal or non-normal continuous outcomes, and works by ranking each subject’s scores across conditions — a step that is not meaningful for a variable with only two possible values.
What post-hoc test follows a significant Cochran’s Q?
Pairwise McNemar’s tests (the exact binomial version, if discordant counts are small) on every pair of conditions, with the significance threshold adjusted for multiple comparisons — most commonly a Bonferroni correction (family-wise α divided by the number of pairs).
What sample size does Cochran’s Q test need?
There is no single fixed minimum; it depends on how many subjects have row totals away from the extremes (all-0 or all-1 across every condition), since those subjects contribute nothing to the statistic. When the chi-square approximation looks unreliable on a small or sparse table, exact distributions for Cochran’s Q (via Patil’s algorithm or later refinements) are the more defensible choice.
Can Cochran’s Q test be used on ordinal or continuous data?
No — it requires a strictly dichotomous outcome. An ordinal or continuous repeated-measures outcome calls for the Friedman test or a repeated-measures ANOVA instead, depending on whether its assumptions hold.
