ROC vs Accuracy I've designed a 4 classifiers which perform pretty decently (all of them are above 90% in accuracy).
However, they don't have similar AUC for their respective ROC curves (obviously, it doesn't have to be).
If I were to use these classifiers in real-time data, which one do I choose based on the following result
Classifier A: 
Accuracy: 100%,  AUC: 84%
Classifier B:
Accuracy: 95%,  AUC: 83%
Classifier C:
Accuracy: 100%,  AUC: 69%
Classifier D:
Accuracy: 100%,  AUC: 77%
 A: One thing to keep in mind is that both accuracy and AUC are point estimates. Estimating confidence intervals for both makes comparisons more interpretable. However, it is more challenging to obtain confidence intervals for accuracy (depending on the resampling scheme). 
One paper that discusses this is "Calculating confidence intervals for prediction error in microarray classification using resampling" by Jiang and colleagues. 
Aliferis and colleagues (Factors Influencing the Statistical Power of Complex Data Analysis Protocols for Molecular Signature Development from Microarray Data) review the perspective that accuracy is an undesirable metric. I think Frank Harrell (coauthor on the above paper) also reviews this in his book "Regression Modeling Strategies".
A: The AUC averages the performance over the whole range of classifier scores, starting from low coverage / low false positive rate and ending at high coverage / high false positive rate. This is not always the best way to compare performance because you may have a stronger emphasis on coverage rather than precision or vise versa. 
Once you plot the ROC curves and/or the precision-recall curves (for the relevant R functions see, e.g., this answer), you can compare the classifiers and select the one that provides better precision for a given high recall value (if these are your needs) or vise versa. This approach 
 will also provide you with the cutoff for accepting the selected classifier's predictions.
