ROC vs. Accuracy [duplicate]

Question

If you want to compare two learning algorithms, which metric is better to use in general: ROC or accuracy? I understand that in ROC, you get both the sensitivity and specificity?

Answer 1

The most important aspect of accuracy is usually absolute accuracy of predicted probabilities. This can be assessed using high-resolution calibration curves, e.g., using the loess nonparametric smoother on predicted vs. observed with suitable de-biasing to account for overfitting. Once you have accurate calibration, you can also assess predictive discrimination using e.g. the area under the ROC curve or $c$-index which don't require consideration of any cutpoints at all. With accurate calibration you can also make an accurate lift curve which can be a good basis for decision making without all the problems associated with up-front categorization of risk as used in classification.

Answer 2

Good question. The best metrics to use are in fact those defined at single thresholds - so Precision, Accuracy, Recall, Uplift (which is Precision / Prior). The reason why you ought to consider these measures more important than, say ROC, AUC or Gini, is that these measures aggregate over all possible thresholds. This means the threshold is still to be chosen, something that will be necessary for your final product.

AUC, ROC, and Gini will help you choose a threshold but not tell you how "good" your algorithm actually is. Think if these measures as corresponding to "the probability that your algorithm is good if you where to randomly pick a threshold", but of course in reality you don't randomly pick a threshold.

Now what you really want to do is relate some measure of accuracy, like precision, to the actual thing in the business you wish to optimize. This is usually something that has currency as a unit.

So I'll give an example, in Adtech, the important business measure is CPA, which is "Cost Per Acquisition" - in other words the cost in $ to get someone to convert (buy a product). Usually this measure directly corresponds with Precision, but NOT accuracy. In this business case Accuracy is meaningless as it rewards algorithms on how good they are at predicting Negatives as well as Positives, but in Adtech we just want to know how good we are at predicting the Positives. For example we are not interested in predicting whether someone does NOT want to buy a car, we are interested in if they DO want to buy a car so we can give them an advert.

In other businesses other measures might be important. In fraud detection, maybe Recall is the most important thing - i.e. how good are we at ensuring fraudsters do not slip through our net.

So the real answer to the question is the measure of "good" depends on what it is your trying to do - try to find a mathematical relationship between a measure of "good"ness and the business measure your trying to optimize. This is why AUC and Gini are not that useful because there cannot exist a mathematical correspondence between concrete business measures and these numbers because a threshold has not yet been chosen, without a threshold you don't have a product.

Hope that isn't too rambly