Many classifiers output scores or probabilities. The ROC curve shows performance across all decision thresholds.
The ROC Curve
It plots:
- True positive rate (recall) on the vertical axis.
- False positive rate on the horizontal axis.
Each point is a threshold. A perfect classifier reaches the top-left corner; random guessing follows the diagonal.
AUC
The area under the ROC curve summarises performance in one number from 0.5 (random) to 1.0 (perfect). It's the probability that a random positive example is scored higher than a random negative one.
Strengths
- Threshold-independent.
- Useful for comparing models' ranking ability.
Limitations
- With heavy class imbalance, ROC AUC can look good even when precision is poor.
- It doesn't tell you which threshold to use.
- It ignores calibration.
Precision-Recall Curves
For rare positive classes — fraud, disease — precision-recall curves and average precision are often more informative.
Choosing a Threshold
Pick based on the costs of false positives and false negatives in your application, not just the curve.