Bland-Altman Plots for Method Agreement: Why Correlation Isn’t the Right Tool

Two methods that measure the same thing rarely agree perfectly, and the most common way researchers check whether they do is the wrong tool for the job. A correlation coefficient — Pearson’s r most often — tests whether two sets of numbers move together, not whether they are close to each other. A new device can correlate at r = 0.99 with a reference standard and still read several units high on every single subject; Pearson’s r is blind to that offset by construction, because it measures how tightly points cluster around some straight line, not around the line of identity (where method A equals method B). The Bland-Altman plot, introduced by J. Martin Bland and Douglas Altman in a widely cited 1986 Lancet paper, answers the question correlation cannot: how much do two methods actually disagree, in the original measurement units, across the range where they are used?

Why a Correlation Coefficient Cannot Assess Agreement

Correlation and agreement are different questions. Correlation asks whether higher values of method A go with higher values of method B — a question about the strength and direction of a linear relationship. Agreement asks whether method A and method B give the same number for the same subject. A method that consistently reads exactly double the reference standard has a perfect Pearson correlation (r = 1.0) with that standard and essentially zero agreement with it: every single reading is wrong, in a completely predictable way. The same failure applies to a constant additive offset, which is the more common real-world case — a device that reads a fixed number of units high or low on every subject can still correlate near-perfectly with the reference it disagrees with.

Correlation has a second problem specific to method-comparison studies: it is sensitive to the range of values sampled. Widen the range of true values in your sample and r goes up, holding the actual measurement error constant, because Pearson’s r is a function of both the covariance between methods and the variance of the values measured. A method-comparison study that happens to sample a wide physiological range will report a reassuringly high r for the same underlying disagreement a narrower-range study would report as unimpressive. Neither number describes the disagreement directly; both describe how the sampled range interacts with it.

None of this means correlation and covariance-based statistics are never useful for measurement questions — CASRAI’s guide to the intraclass correlation coefficient (ICC) covers a properly agreement-sensitive, variance-partitioning alternative for reliability studies, and the two tools are complementary rather than competing: ICC gives a single reliability number with a confidence interval, while a Bland-Altman plot visualizes where and how two methods disagree across the measurement range. What Pearson’s r specifically cannot do is stand in for either one. See CASRAI’s guide to the correlation coefficient for what Pearson and Spearman correlation are actually built to measure.

What a Bland-Altman Plot Shows

The plot has an unusual pair of axes, and the axes are the whole method. For each subject, compute two numbers:

• The difference, method A minus method B — plotted on the y-axis.
• The mean of the two methods for that subject, (A + B) / 2 — plotted on the x-axis, standing in for the subject’s true value, since neither method alone is assumed to be it.

Three horizontal reference lines go on top of the resulting scatter:

• The bias line — the mean of all the differences. This is the average systematic disagreement between the two methods, in the original units.
• The lower and upper limits of agreement (LoA) — the bias plus and minus 1.96 times the standard deviation of the differences. Assuming the differences are approximately normally distributed, this is the interval expected to contain about 95% of the disagreements between the two methods for any individual subject.

A flat, randomly scattered band of points around a bias line close to zero, with a tight LoA interval, is what good agreement looks like on this plot. A bias line displaced from zero shows a systematic offset; a funnel shape widening at one end shows the disagreement is not constant across the measurement range (covered under proportional bias, below).

Constructing a Bland-Altman Plot From a Worked Dataset

The construction is four steps:

1. Compute the per-subject difference and mean — the two right-hand columns above.
2. Compute the bias — the mean of the 15 differences: −5.60 mmHg. Method B reads, on average, 5.6 mmHg higher than Method A.
3. Compute the standard deviation of the differences: 2.97 mmHg.
4. Compute the limits of agreement: bias ± 1.96 × SD = −5.60 ± 5.82, giving a lower LoA of −11.42 mmHg and an upper LoA of 0.22 mmHg.

Plotted, that means: for any individual subject, Method B’s reading is expected to fall somewhere between 11.4 mmHg above and 0.2 mmHg below Method A’s reading — a wide range built entirely from data whose Pearson correlation is r = 0.998. The same 15 pairs that produce a near-perfect correlation coefficient produce a systematic 5.6 mmHg bias and a nearly 12 mmHg spread of individual disagreement — the exact gap between “these move together” and “these agree” that Pearson’s r cannot see.

Confidence Intervals on the Bias and the Limits of Agreement

The bias and the two limits of agreement are estimates from a finite sample, and each carries its own standard error:

• SE of the bias = SD of differences ÷ √n
• SE of each limit of agreement ≈ SD of differences × √(3/n)

A 95% CI for each is the estimate ± t_{(n−1, 0.975)} × its SE. For the worked example (n = 15, so 14 degrees of freedom, t ≈ 2.145):

• Bias −5.60 mmHg, 95% CI [−7.25, −3.95]
• Lower LoA −11.42 mmHg, 95% CI [−14.27, −8.57]
• Upper LoA 0.22 mmHg, 95% CI [−2.63, 3.07]

Notice how wide these CIs are relative to the estimates themselves — that is what 15 pairs buys you. The limits of agreement are themselves estimates, not fixed boundaries, and a method-comparison study run on a small sample reports limits that could plausibly sit several mmHg away from the values in a repeat study. This worked example is deliberately small enough to keep the arithmetic checkable by hand; a real method-comparison study should use enough pairs that the CI on each limit of agreement is narrow enough to support the decision the study exists to make.

Detecting Proportional Bias

The fixed-width limits of agreement above assume the size of the disagreement is roughly constant across the measurement range. That assumption can fail: the higher the blood pressure, the more Method B might overshoot Method A, which is exactly what the worked dataset was built to also show. The test is a simple linear regression of the difference (A − B) on the mean of A and B: a slope significantly different from zero is proportional bias — the disagreement between the methods is itself a function of the value being measured, not a constant offset.

For the worked example, that regression gives a slope of −0.11 mmHg of additional disagreement per mmHg of mean blood pressure, with t = −6.15 on 13 degrees of freedom — clearly non-zero. In plain terms: Method B’s positive bias against Method A grows as blood pressure rises, which the single fixed bias-and-LoA summary above does not communicate on its own.

When proportional bias is present, a constant ±1.96 SD band misrepresents the data at both ends of the range — too wide where true disagreement is small, too narrow where it is large. Bland and Altman’s own follow-up methodology paper (Statistical Methods in Medical Research, 1999) sets out two standard fixes: regress the differences on the means and construct limits of agreement that widen or narrow with the mean instead of a single fixed band, or — when the scatter of differences grows roughly in proportion to the magnitude being measured, rather than as a fixed additive amount — log-transform both sets of measurements before running the analysis, then back-transform the resulting limits.

Reading the Plot: What Counts as Good Agreement

The limits of agreement are not a significance test, and treating them like one is the second most common Bland-Altman mistake after reaching for Pearson’s r in the first place. A CI that excludes zero for the bias tells you the two methods are statistically distinguishable — with enough subjects, almost any two methods will be, since a real-world offset of any nonzero size becomes detectable at a large enough sample size. That is a different question from whether the methods agree closely enough to be used interchangeably.

The right practice is to set a maximum allowable difference before looking at the data — a clinically or operationally defined threshold for how much two methods are allowed to disagree and still be considered interchangeable for the purpose at hand — and then check whether the observed limits of agreement fall inside it. A method with a highly statistically significant, tiny bias and a narrow LoA well inside the pre-specified threshold shows good practical agreement despite the significant test. A method with a non-significant bias but wide limits of agreement that exceed the threshold shows poor practical agreement despite the non-significant test. The pre-specified threshold, not the p-value, is what the plot is for.

Bland-Altman vs. the Other Agreement Statistics

Bland-Altman analysis is one member of a small family of agreement tools CASRAI covers, and picking the wrong one is a recurring error in method-comparison and reliability work:

• Two different measurement methods, one occasion (a new device against a reference standard, as in the worked example above) — Bland-Altman, as covered on this page.
• The same instrument, two occasions on the same subjects (test-retest reliability) — a different design question, covered on CASRAI’s guide to test-retest reliability, which also uses a Bland-Altman check to catch systematic drift between occasions specifically.
• Multiple raters scoring the same subjects — use inter-rater reliability coefficients (kappa for categorical ratings, ICC for continuous ones), not Bland-Altman, which is built for exactly two measurement series.
• A single reliability number with a confidence interval, for continuous data — the intraclass correlation coefficient, which is complementary to Bland-Altman rather than a substitute for it.
• How much change in a score is real, not measurement noise — CASRAI’s guide to minimal detectable change, which is numerically related to the limits of agreement: when the bias is zero, MDC95 and the half-width of the limits of agreement are the same quantity derived two different ways.

Broader background on what “agreement” and “validity” mean as distinct measurement-quality concepts is on CASRAI’s guides to types of validity in research, criterion validity, and accuracy vs. precision in measurement.

Frequently Asked Questions

Is a Bland-Altman plot the same as testing correlation?

No. A correlation coefficient tests whether two sets of measurements move together; a Bland-Altman plot tests whether they are close to each other. Two methods can be highly correlated and disagree substantially, as the worked example above shows directly (r = 0.998, bias −5.6 mmHg).

Can a Bland-Altman plot compare more than two methods?

Not directly. The original method is built for exactly two paired series. Comparing three or more methods against each other means either running a separate Bland-Altman plot for every pair, or using one of the extensions to Bland-Altman’s framework built specifically for more than two methods or raters — the classic two-line plot itself does not generalize past a pair.

What if the differences aren’t normally distributed?

The mean ± 1.96 SD form of the limits of agreement assumes the differences are approximately normal. Check with a histogram or Q–Q plot of the differences before trusting the limits. If the differences are skewed or the scatter grows with magnitude, a log transform before analysis (see the proportional-bias section above) or nonparametric, percentile-based limits are the standard alternatives.

What is a “maximum allowable difference,” and who sets it?

It is the largest disagreement between the two methods that the researcher considers acceptable for the two methods to be treated as interchangeable for a given purpose — a clinical or operational judgment, set before the data are collected, not a statistic the Bland-Altman analysis produces on its own. The plot tells you what the actual disagreement is; the threshold is a separate decision the study has to bring to the analysis.

References

• Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet 1986;1(8476):307–310. DOI 10.1016/S0140-6736(86)90837-8.
• Bland JM, Altman DG. Measuring agreement in method comparison studies. Statistical Methods in Medical Research 1999;8(2):135–160.