Written and maintained by CASRAI Editorial Board
Last updated
An ROC curve plots a classifier’s true-positive rate against its false-positive rate across every possible decision threshold, and the area under that curve (AUC) collapses the whole curve into one number: the probability that the model ranks a randomly chosen positive case higher than a randomly chosen negative case. Of the prediction-model evaluation metrics in this cluster, ROC/AUC is the most foundational and the most widely recognized outside statistics itself — it is usually the first number a reviewer or a manuscript asks for. That familiarity is also its risk: AUC gets quoted as if it were a single, complete verdict on a model, when it actually answers one specific question (can the model rank cases correctly?) and stays silent on several others that matter just as much in practice.
What an ROC curve actually plots
Every classification model that outputs a continuous score or probability, rather than a bare positive/negative label, needs a threshold to turn that score into a decision. Move the threshold and every diagnostic accuracy statistic built on the underlying 2×2 table — sensitivity, specificity, positive and negative predictive value — changes with it. The sensitivity vs. specificity guide builds that 2×2 table in full; ROC/AUC is what you get when you stop picking one threshold and instead look at all of them at once.
To build the curve: sweep the threshold from its most conservative setting (nothing is called positive) to its most liberal (everything is called positive). At each threshold, plot two coordinates:
- Y-axis — sensitivity (true-positive rate): the proportion of true positive cases the model correctly flags at that threshold.
- X-axis — 1 − specificity (false-positive rate): the proportion of true negative cases the model incorrectly flags at that threshold.
Connecting the resulting points across every threshold produces the receiver operating characteristic (ROC) curve — a name inherited from World War II radar-signal-detection research, where the same trade-off (missed targets vs. false alarms) first got formalized. A model with no discriminating ability at all traces the diagonal line from (0,0) to (1,1); a model that separates the two classes perfectly hugs the top-left corner, reaching 100% sensitivity at 0% false-positive rate. Every real model’s curve sits somewhere between those two extremes, and where it sits at any given point tells you the sensitivity/specificity trade-off available at that particular threshold.
What AUC means — and the interpretation to get right
The area under the ROC curve (AUC, sometimes written AUROC) compresses that entire curve into a single number between 0.5 and 1.0 (values below 0.5 mean the model is doing worse than chance, which typically indicates its predictions are inverted). The correct interpretation is a ranking probability, not a percentage of anything: AUC is the probability that a randomly selected positive case receives a higher predicted score than a randomly selected negative case, averaged across every possible threshold.
That framing matters because AUC is routinely misread as “percent of cases classified correctly,” which it is not. A model can have a high AUC while every individual threshold you might actually deploy produces a mediocre confusion matrix, because AUC averages performance across thresholds nobody would ever use in practice — a point the decision curve analysis guide covers in more depth when it comes to picking a single clinically meaningful cutoff. AUC also has a direct, useful equivalence: it equals the Mann-Whitney U statistic (rescaled), which is why AUC is sometimes introduced through nonparametric rank-sum testing rather than through the ROC curve at all.
Reading AUC values: conventions, not fixed rules
AUC = 0.5 means no discrimination — the model ranks cases no better than a coin flip. AUC = 1.0 means perfect discrimination — every positive case outranks every negative case. Between those anchors, one widely cited rule of thumb (Hosmer and Lemeshow, Applied Logistic Regression) offers rough verbal bands:
- 0.5–0.7: poor discrimination
- 0.7–0.8: acceptable discrimination
- 0.8–0.9: excellent discrimination
- >0.9: outstanding discrimination
Treat these as a shared vocabulary for talking about a number, not a pass/fail bar. What counts as an adequate AUC is genuinely field- and task-dependent: a 0.75 AUC screening tool competing against nothing (no existing test) can be clinically valuable, while a 0.90 AUC model in a domain with an existing 0.95 AUC standard of care is a regression. An AUC near 1.0 in a research setting can also be a warning sign rather than good news — it sometimes points to data leakage (a feature that encodes the outcome) rather than genuine predictive power, and is worth a second look before it is reported as a result.
Where AUC misleads, and what it doesn’t tell you
AUC’s biggest practical failure mode is class imbalance. Because the false-positive-rate axis is calculated against the entire negative pool, a rare outcome means that pool is very large, and a model can rack up a deceptively high AUC while still generating an overwhelming number of false positives in absolute terms whenever the outcome is genuinely uncommon (rare disease screening, fraud detection, adverse-event prediction). In that setting, a precision-recall curve — which weights performance against the positive class instead of the whole negative pool — is typically the more honest summary, and reporting AUC alongside it rather than instead of it is the safer default.
Two further gaps matter just as much and are easy to forget because AUC’s single number invites treating it as a complete verdict:
- AUC says nothing about calibration. AUC only checks whether cases are ranked correctly, and is mathematically invariant to any monotonic transformation of the predicted probabilities — a model can discriminate well (high AUC) while its actual probability outputs are badly miscalibrated (a predicted 20% risk that doesn’t correspond to a 20% observed event rate). The calibration plots guide covers how to check the numeric accuracy question AUC cannot answer.
- AUC says nothing about clinical or operational utility. A high AUC does not by itself mean acting on the model’s predictions produces a net benefit once the real costs of false positives and false negatives are weighed at the threshold a clinician or decision-maker would actually use. Two models with identical AUC can have very different net benefit at the one or two thresholds that matter in practice. The decision curve analysis guide is built specifically to answer that question.
How ROC/AUC relates to the other evaluation metrics in this cluster
ROC/AUC, diagnostic odds ratio, calibration plots, and decision curve analysis are frequently discussed together because they all evaluate a diagnostic or prediction model — but each answers a genuinely different question, and none substitutes for the others:
| Metric | Question it answers | What it misses |
|---|---|---|
| ROC curve / AUC | Across all thresholds, how well does the model rank positives above negatives? | Calibration; clinical utility; behavior at a single real-world threshold |
| Diagnostic odds ratio | At one chosen threshold, how much do the odds of a positive result differ between people with and without the condition? | Threshold-independence (it is genuinely threshold-dependent, unlike AUC); calibration; clinical utility |
| Calibration plots | Do the model’s predicted probabilities match observed outcome rates? | Ranking ability (a model can be well-calibrated and still discriminate poorly) |
| Decision curve analysis | Does acting on the model beat treating everyone or no one, at the thresholds decision-makers actually use? | Overall discrimination or calibration across thresholds nobody uses |
In practice, a complete evaluation of a diagnostic or prediction model typically reports more than one of these: ROC/AUC (or the DOR at a specific operating point) for discrimination, a calibration plot for probability accuracy, and decision curve analysis for whether any of it changes what a clinician should actually do.
Reporting ROC/AUC well
- Report a confidence interval around AUC, not just the point estimate — AUC computed on a small or imbalanced test set can carry a wide interval that a bare point estimate hides.
- State what the curve was computed on: internal cross-validation, a held-out internal test set, or external/temporal validation. AUC on the data a model was trained on is optimistic; external-validation AUC is the number that matters for generalizability.
- If the paper also reports a single sensitivity/specificity pair, state which threshold produced it and why that threshold was chosen — AUC alone doesn’t justify a specific operating point.
- Pair AUC with a precision-recall curve when the outcome is rare, and with calibration and/or decision curve analysis when the paper’s conclusion depends on clinical or operational impact rather than ranking ability alone.
Frequently asked questions
Is a higher AUC always better?
Higher AUC means better discrimination, but “better” is relative to context, not an absolute quality bar. A modest AUC can be genuinely useful when no comparable test exists; a very high AUC can be a red flag for data leakage rather than a genuine result. AUC also can’t tell you whether a model is well-calibrated or clinically useful — both require separate checks.
What counts as a “good” AUC?
There’s no universal cutoff. The Hosmer-Lemeshow verbal bands (0.7–0.8 acceptable, 0.8–0.9 excellent, >0.9 outstanding) are a widely used convention, not a fixed rule, and what’s adequate depends heavily on the field, the outcome’s rarity, and what the model is being compared against.
Can two models have the same AUC but perform very differently in practice?
Yes. AUC averages performance across every threshold, including ones nobody would use. Two models with identical AUC can behave very differently at the one or two thresholds an actual decision-maker would apply — which is exactly the gap decision curve analysis is designed to close.
Does a high AUC mean the model’s predicted probabilities are accurate?
No. AUC only measures ranking, and is unaffected by any transformation of the predicted probabilities that preserves their order. A model can rank cases well and still be badly miscalibrated; calibration has to be checked separately.
Why does AUC look inflated when the outcome is rare?
Because the false-positive-rate axis is measured against the entire negative pool, and when the outcome is rare that pool is very large. A model can post a high AUC while still producing a large absolute number of false positives. A precision-recall curve, which is scored against the positive class, is usually a more honest summary in that setting.
