With respectful deference to Dr. Harrell’s answer regarding proper scoring metrics like the Brier Score, if only the two choices of Accuracy and AUC ROC are given, the answer is it depends upon the data and the desired outcome measure.
• The Data: AUC ROC is prevalence-invariant; it will not vary from class imbalance. If your binary classification dataset is not balanced (nearly equal positive and negative examples), you won’t know from the AUC. Accuracy is a poor metric in the imbalanced case.
• The Question: If only concerned with correct prediction, accuracy is fine. If the real-word penalty for a ‘miss’ is similar to a ‘hit’, accuracy is fine. However, accuracy does not distinguish between errors and can overestimate the algorithm’s ability. AUC ROC gives a good comparison of two models, but is only a starting point as it represents ALL potential operating points, not the SINGLE operating point an algorithm would function at. Two algorithms with the same AUC ROC can each be superior at different operating thresholds if they do not dominate each other.
The Powers Paper Evaluation: From Precision, Recall, and F-Factor…is helpful. Updated link.
So, both are inadequate. However, in a balanced dataset, between two algorithms tested on the same data, AUC ROC is probably a better measure than accuracy.