Area under curve of ROC vs. overall accuracy I am a little bit confused about the Area Under Curve (AUC) of ROC and the overall accuracy.

*

*Will the AUC be proportional to the overall accuracy? In other words, when we have a larger overall accuracy will we definitely a get larger AUC? Or are they by definition positively correlated?


*If they are positively correlated, why do we bother reporting both of them in some publications?


*In real case, I performed some classification task and got the results as follows: classifier A got an accuracy 85% and AUC of 0.98 and classifier B got an accuracy of 93% and AUC of 0.92. Question is, which classifier is better? Is it possible to get results similar to these (or do my results indicate a bug in my implementation)?
 A: Is AUC really very useful metric?
I would say expected cost is more appropriate measure.
Then you would have a cost A for all False Positives and cost B for all False Negatives. It might easily be that other class is relative more expensive than other. Of course if you have costs for false classification in the various sub-groups then it would be even more powerful metric.
By plotting cut-off in the x-axis and expected cost on then y-axis you can see which cut-off point minimizes expected cost.
Formally you have a loss-function Loss(cut-off|data,cost) which you try to minimize.
A: Like all the answers have been posted: ROC and accuracy are fundamentally two different concepts.
Generally speaking, ROC describes the discriminative power of a classifier independent of class distribution and unequal prediction error costs (false positive and false negative cost).
Metric like accuracy is calculated based on the class distribution of test dataset or cross-validation, but this ratio may change when you apply the classifier to real life data, because the underlying class distribution has been changed or is unknown. On the other hand, TP rate and FP rate which are used to construct AUC will be not be affected by class distribution shifting.
A: While the two statistics measures are likely to be correlated, they measure different qualities of the classifier.
AUROC
The area under the curve (AUC) is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative example.  It measures the classifiers skill in ranking a set of patterns according to the degree to which they belong to the positive class, but without actually assigning patterns to classes.
The overall accuracy also depends on the ability of the classifier to rank patterns, but also on its ability to select a threshold in the ranking used to assign patterns to the positive class if above the threshold and to the negative class if below.
Thus the classifier with the higher AUROC statistic (all things being equal) is likely to also have a higher overall accuracy as the ranking of patterns (which AUROC measures) is beneficial to both AUROC and overall accuracy.  However, if one classifier ranks patterns well, but selects the threshold badly, it can have a high AUROC but a poor overall accuracy.
Practical Use
In practice, I like to collect the overall accuracy, the AUROC and if the classifier estimates the probability of class membership, the cross-entropy or predictive information.  Then I have a metric that measures its raw ability to perform a hard classification (assuming false-positive and false-negative misclassification costs are equal and the class frequencies in the sample are the same as those in operational use - a big assumption!), a metric that measures the ability to rank patterns and a metric that measures how well the ranking is calibrated as a probability.
For many tasks, the operational misclassification costs are unknown or variable, or the operational class frequencies are different to those in the training sample or are variable.  In that case, the overall accuracy is often fairly meaningless and the AUROC is a better indicator of performance and ideally we want a classifier that outputs well-calibrated probabilities, so that we can compensate for these issues in operational use.  Essentially which metric is important depends on the problem we are trying to solve.
A: AUC (based on ROC) and overall accuracy seems not the same concept.
Overall accuracy is based on one specific cutpoint, while ROC tries all of the cutpoint and plots the sensitivity and specificity.
So when we compare the overall accuracy, we are comparing the accuracy based on some cutpoint. The overall accuracy varies from different cutpoint.
