AUC-ROC

The ROC (receiver operating characteristic) curve plots a classifier’s true-positive rate against its false-positive rate as you sweep the decision threshold from strict to lenient. AUC-ROC is the area under that curve, and it has a direct reading: the probability that the model ranks a randomly chosen positive example above a randomly chosen negative one (Google, Machine Learning Crash Course: ROC and AUC). A random ranking scores about 0.5; a perfect ranking scores 1.0. Scores below 0.5 are possible: on that evaluation set, the classifier ranks positives below negatives more often than above.

Its strength is that it does not depend on any one threshold. It tells you how well the scores separate the classes before you decide where to cut, which makes it a reasonable way to compare models on the same data. Choosing the threshold is a separate decision based on what a false positive and a false negative each cost you, and precision and recall or F1 at that threshold describe the result users will see.

The common trap is class imbalance. When negatives vastly outnumber positives, as with fraud, rare defects, or abuse detection, a model can show a high AUC-ROC while most of its positive predictions are still wrong, because a small false-positive rate applied to a huge pool of negatives is a large number of false alarms. Saito and Rehmsmeier showed that ROC plots can be visually misleading on imbalanced data and that precision-recall plots, which measure the share of true positives among positive predictions, give a more accurate picture of future performance (The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets). Report both when positives are rare. AI agent metrics beyond accuracy covers why a single headline number rarely describes production behavior.