# ROC vs Accuracy

I've designed a 4 classifiers which perform pretty decently (all of them are above 90% in accuracy).

However, they don't have similar AUC for their respective ROC curves (obviously, it doesn't have to be).

If I were to use these classifiers in real-time data, which one do I choose based on the following result

Classifier A: Accuracy: 100%, AUC: 84%

Classifier B: Accuracy: 95%, AUC: 83%

Classifier C: Accuracy: 100%, AUC: 69%

Classifier D: Accuracy: 100%, AUC: 77%

-
I'm building a gesture recognition classifiers. Each feature vector is a 49 unit length long. Classifiers A and C are multinomial Naive Bayes Classifiers and Classifiers B and D are LDA classifiers. –  raul_w Jul 11 '12 at 22:15
ROC curve shows behaviour of a quantitative classifier on various cut-points. Accuracy varies for various cut-points. For what cut-points do you report your accuracies? –  ttnphns Jul 11 '12 at 22:47
Unless I am mistaken the answer so far seem to miss that the result does not seem possible. An accuracy of 100% for any cutoff should automatically result in an AUC of 1. So... what gives? Do you have a strange definition of accuracy? Do you really compare the same scenarios? –  Erik Jul 12 '12 at 7:52
I strongly agree with @Erik that 100 ACC <=> AUC=1. However, I suspect a rounding error here combined with a heavy class skew towards the "negative" class. –  steffen Jul 12 '12 at 11:16
show 1 more comment

One thing to keep in mind is that both accuracy and AUC are point estimates. Estimating confidence intervals for both makes comparisons more interpretable. However, it is more challenging to obtain confidence intervals for accuracy (depending on the resampling scheme).

One paper that discusses this is "Calculating confidence intervals for prediction error in microarray classification using resampling" by Jiang and colleagues.

Aliferis and colleagues (Factors Influencing the Statistical Power of Complex Data Analysis Protocols for Molecular Signature Development from Microarray Data) review the perspective that accuracy is an undesirable metric. I think Frank Harrell (coauthor on the above paper) also reviews this in his book "Regression Modeling Strategies".

-
I'll add to @julieth: I wouldn't get excited about any "accuracy" measure unless a high-resolution calibration plot accompanies it, and the plot (say, based on loess) is penalized for overfitting using the bootstrap. The R rms package makes this easy to do for some problems. –  Frank Harrell Jul 12 '12 at 10:48