Written and maintained by CASRAI Editorial Board
Last updated
A permutation test answers a narrow, honest question: if the group labels in this dataset were meaningless — assigned at random rather than reflecting a real difference — how often would relabeling produce a result at least as extreme as the one actually observed? Instead of comparing a test statistic to a theoretical distribution (the normal, the t, the F) that assumes something about how the data were generated, a permutation test builds its own reference distribution directly from the data at hand, by shuffling the labels thousands of times and recomputing the statistic each time. That difference in approach is the whole appeal: a permutation test doesn’t need the sample to be large, doesn’t need it to look normal, and doesn’t need the two groups’ variances to match. What it needs instead — exchangeability — is a weaker, more defensible assumption that most randomized designs already satisfy by construction.
What a Permutation Test Actually Requires: Exchangeability, Not Normality
The condition a permutation test depends on is that, under the null hypothesis, the group labels are exchangeable: every possible way of dividing the pooled observations into groups of the same sizes is equally likely. That’s a statement about how the labels relate to the data, not a statement about the data’s shape. It holds automatically whenever the null hypothesis is that treatment assignment has no effect and assignment was itself random — which is exactly the situation in a randomized experiment. It does not require the underlying distribution to be normal, symmetric, or even the same shape across groups; it doesn’t require variances to be equal (the assumption a standard two-sample t-test leans on, and the one Welch’s correction exists to soften); and it places no lower bound on sample size beyond having enough observations to produce a meaningful set of permutations at all.
Exchangeability is weaker than the assumptions behind a parametric test, but it isn’t nothing. It can fail: in an observational (non-randomized) comparison where group membership correlates with some other variable that also drives the outcome, swapping labels no longer represents a genuinely equivalent re-draw under the null, and a permutation test inherits that confounding exactly the way any other test would. Randomization is what delivers exchangeability for free; without it, a permutation test tests the sharp null of “identical distributions,” not “identical treatment effect,” and that distinction is worth stating plainly rather than glossing over.
The Resampling Logic: A Worked Example
The mechanics are easiest to see on data small enough to enumerate by hand. Take an illustrative — not real-study — dataset: four observations in a treatment group and five in a control group, deliberately built with one large outlier to make a point about robustness:
- Group A (n = 4): 3, 4, 4, 5 — mean = 4.0
- Group B (n = 5): 9, 10, 11, 12, 40 — mean = 16.4
The observed difference in means is 4.0 − 16.4 = −12.4. Two ways to test whether that’s a real difference or noise:
The parametric route. A Welch’s t-test (the variance-unequal version, appropriate here since Group B’s single outlier inflates its variance far past Group A’s) gives t ≈ −2.09 on ≈4.04 degrees of freedom, for a two-sided p-value of about 0.104 — not significant at the conventional 0.05 threshold. The outlier is doing damage twice: it pulls Group B’s mean up, but it inflates Group B’s variance even more, and that inflated variance dominates the standard error the t-statistic divides by. The test ends up penalized for the very feature — one large value — that’s actually informative about a real between-group difference.
The permutation route. Pool all nine values and ask: of every possible way to split nine numbers into a group of 4 and a group of 5, how many splits produce a difference in means at least as extreme (in absolute value) as 12.4? There are exactly C(9,4) = 126 distinct ways to choose which four values form “Group A” (the remaining five automatically form “Group B”). Enumerating all 126 and recomputing the mean difference for each shows only 3 of the 126 relabelings produce a difference at least as extreme as the one observed — giving an exact two-sided p-value of 3/126 ≈ 0.024.
Same data, opposite conclusion at the 0.05 threshold. The permutation test isn’t more lenient — it’s answering with the actual structure of these nine numbers rather than a theoretical curve that assumes the outlier is just added noise to a normal distribution’s variance. That’s the practical payoff of “no distributional assumption”: it isn’t an abstract virtue, it changes which decision the data supports when a sample is small and one point doesn’t behave.
How Many Permutations Are Enough
With 9 observations split 4/5, enumerating all 126 relabelings exactly is trivial. That stops being true fast: 20 observations split 10/10 already has over 184,000 distinct splits, and the count grows combinatorially from there. Past a certain size, exhaustive enumeration is replaced with a Monte Carlo approximation — drawing a large random sample of relabelings instead of every possible one, then computing the p-value as (number of sampled permutations at least as extreme as observed, plus 1) ÷ (number of permutations sampled, plus 1). The “+1” in both numerator and denominator isn’t cosmetic: it accounts for the observed arrangement itself belonging to the null distribution being sampled from, and guarantees the p-value can never come back as exactly zero, which an uncorrected count/count ratio can produce with an unlucky (or too-small) random sample. SciPy’s scipy.stats.permutation_test defaults to 9,999 random resamples using exactly this adjusted formula, switching automatically to full exact enumeration whenever the true number of distinct permutations is smaller than that — as it is in the worked example above.
As a practical rule: a few thousand resamples is generally enough resolution to distinguish p = 0.05 from p = 0.01 reliably; if the analysis plan calls for a much smaller significance threshold (a stringent multiple-comparisons correction, for instance), the resample count needs to be large enough that the smallest achievable p-value (1 ÷ (resamples + 1)) is still comfortably below that threshold — a test run at 999 resamples cannot, by construction, ever report a p-value below 0.001.
When the Computational Cost Is Worth It
A permutation test costs more compute than looking up a value in a t-distribution — sometimes by orders of magnitude for large samples with exact enumeration, though Monte Carlo resampling keeps that cost bounded and predictable. That cost is worth paying, specifically, when:
- The sample is small and its shape is unknown or clearly non-normal. Below roughly n = 20–30 per group, the Central Limit Theorem’s justification for treating a t-statistic as approximately normal is thin, and a single outlier (as in the worked example) can distort a parametric test’s variance estimate more than it distorts the underlying signal.
- Group sizes or variances are unequal, and the imbalance itself might matter. Permutation tests handle heteroscedasticity and unequal n directly, by construction, rather than requiring a separate correction (Welch’s df adjustment, Levene’s test as a gatekeeper) bolted onto a parametric model built for the balanced case.
- The statistic of interest has no known theoretical sampling distribution. Plenty of useful statistics — a trimmed mean, a custom weighted effect size, the difference between two medians, a statistic from a multivariate or graph-structured comparison — simply don’t have a textbook parametric test built for them. A permutation test works for any statistic you can compute and recompute under relabeling; that generality is often the actual reason to reach for one, independent of sample size.
- The design is genuinely randomized. Exchangeability under the null is guaranteed by the randomization itself, which is exactly the condition a permutation test needs and the condition many parametric tests only approximate.
Conversely, skip it when a parametric test’s assumptions are actually well met (a reasonably large, roughly symmetric sample with comparable variances) — a t-test or ANOVA is faster to compute, more familiar to a reviewer, and will return essentially the same answer, so the extra machinery buys nothing.
Permutation Tests vs. the Alternatives
| Approach | Distributional assumption | Handles small/skewed samples | What it tests |
|---|---|---|---|
| Parametric test (t-test, ANOVA) | Normal population, often equal variances | Weak — degrades as n shrinks or shape departs from normal | Difference relative to a theoretical reference distribution |
| Rank-based test (Mann-Whitney, Wilcoxon, Kruskal-Wallis) | None on the raw values; assumes similarly-shaped distributions for a location-shift interpretation | Good — robust to outliers via ranking, though ranking discards magnitude information | Whether one group tends to rank higher, not the raw-value difference |
| Permutation test | Exchangeability of labels under the null (delivered by randomization) | Strong — works for essentially any sample size or shape, exactly for small n via enumeration | Whatever statistic you choose, against its own empirical null |
| Bootstrap | The sample is representative of the population (resamples with replacement from one sample) | Good, but answers a different question | Sampling variability of an estimate (confidence intervals), not a label-exchange null |
The bootstrap is worth distinguishing carefully, since it’s easy to conflate with permutation testing — both are resampling methods, but they resample differently and answer different questions. A permutation test resamples without replacement by reshuffling existing labels across the pooled data, to build a null distribution for a hypothesis test. A bootstrap resamples with replacement from within a single sample, to estimate the sampling variability of a statistic (typically for a confidence interval) — it doesn’t relabel anything and isn’t, by itself, testing a null hypothesis about group differences. Reach for a permutation test when the question is “could this observed difference plausibly arise from random assignment alone”; reach for a bootstrap when the question is “how much would this estimate wobble on a fresh sample.”
Running One: R and Python
Base R has no single built-in permutation-test function, but the logic is short enough to write directly with sample() or combn() for exact enumeration on small n; the coin package’s independence_test() and the dedicated perm package wrap the same logic with more test-statistic options and reporting. In Python, scipy.stats.permutation_test (SciPy 1.9+) takes an arbitrary statistic function, a data structure describing which axes/samples to permute, and a permutation_type argument ('independent' for a two-sample comparison like the worked example above, 'samples' for paired data, 'pairings' for correlation-style permutation of one variable against a fixed other) — it automatically switches between exact enumeration and Monte Carlo resampling based on how many distinct permutations actually exist for the given sample sizes.
Limitations Worth Knowing
A permutation test is exact and assumption-light for the hypothesis it actually tests — the sharp null that the two groups’ distributions are identical — not automatically for the narrower question a researcher often really wants answered, such as “do the means differ.” When variances differ sharply between groups (as in the worked example), a permutation test built on the raw mean difference can have different sensitivity to that variance difference than a test built to target means specifically; some implementations address this with a studentized (variance-standardized) test statistic rather than the raw difference in means. It also doesn’t fix confounding in non-randomized comparisons (exchangeability has to come from somewhere, and random assignment is the cleanest source), and — for very large exact-enumeration cases — the computational cost is real, which is precisely why Monte Carlo resampling with the adjusted p-value formula above exists as the practical default.
Frequently Asked Questions
Is a permutation test the same as a randomization test?
The terms are used near-interchangeably in most methods writing, though some authors reserve “randomization test” specifically for designs where the randomization was performed by the researcher (an actual randomized experiment) and use “permutation test” as the broader label covering the general resampling procedure regardless of how the original assignment happened.
Can a permutation test be used with more than two groups?
Yes — the same relabeling logic extends to a permutation-based ANOVA (permuting labels across three or more groups and recomputing an F-statistic or another between-group statistic each time), and to more complex designs including repeated-measures and factorial structures, though the permutation scheme has to respect the design’s actual exchangeability structure rather than treating every observation as freely swappable.
Does a small p-value from a permutation test mean the same thing as one from a t-test?
Both are answering “how surprising is this result under the null,” but the null and the reference distribution differ — a permutation p-value is exact relative to the empirical resampling distribution of this specific dataset, while a t-test p-value is exact only insofar as the underlying normality/variance assumptions actually hold. When those assumptions are well met, the two typically agree closely; the worked example above is a case built specifically to show them diverging.
Do I need special software, or can I compute one by hand?
For very small samples — as in the nine-observation example above — exact enumeration by hand or in a spreadsheet is genuinely feasible. Past a few dozen observations, exhaustive enumeration becomes computationally impractical and Monte Carlo resampling (via R’s coin/perm packages or Python’s scipy.stats.permutation_test) is the standard practical approach.
For the broader landscape of quantitative analysis choices this fits into, see the Research Methods & Statistics hub, and for the underlying inferential logic every one of these tests shares, see p-value and confidence interval.
