38
$\begingroup$

I am a little bit confused about the Area Under Curve (AUC) of ROC and the overall accuracy.

  1. Will the AUC be proportional to the overall accuracy? In other words, when we have a larger overall accuracy will we definitely a get larger AUC? Or are they by definition positively correlated?

  2. If they are positively correlated, why do we bother reporting both of them in some publications?

  3. In real case, I performed some classification task and got the results as follows: classifier A got an accuracy 85% and AUC of 0.98 and classifier B got an accuracy of 93% and AUC of 0.92. Question is, which classifier is better? Is it possible to get results similar to these (or do my results indicate a bug in my implementation)?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I found a paper may have some interest for some of you. google.co.uk/… $\endgroup$
    – Samo Jerom
    Commented Sep 2, 2013 at 9:53
  • $\begingroup$ Isn't AUC supposed to be less than the overall accuracy since we count for the false positive rate in the AUC measure while we don't in the accuracy??? $\endgroup$
    – Ali Sultan
    Commented Nov 26, 2014 at 5:03
  • $\begingroup$ ROC AUC is beneficial when the classes have different size. If 99% of objects are positive, an accuracy of 99% is obtainable by random sampling. Then the ROC AUC value will be much more meaningful. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Dec 4, 2014 at 22:07

4 Answers 4

Reset to default
33
$\begingroup$

AUC (based on ROC) and overall accuracy seems not the same concept.

Overall accuracy is based on one specific cutpoint, while ROC tries all of the cutpoint and plots the sensitivity and specificity. So when we compare the overall accuracy, we are comparing the accuracy based on some cutpoint. The overall accuracy varies from different cutpoint.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ Thanks very much for your reply! I understand that the overall accuracy is obtained from certain cut-point (or threshold value). However, there is one best cut-point, i.e., the one nearest to the top left corner. For example, my overall accuracy is calculated using this best cut-off point, and the AUC is for all the different cut-points. Then how to interpret this accuracy and AUC? For example, the performance of the two classifiers I mentioned above. $\endgroup$
    – Samo Jerom
    Commented Sep 1, 2013 at 19:08
  • 3
    $\begingroup$ Oh, I see. You are comparing the best overall accuracy and AUC. But they are still different concept again. The AUC is the P(predicted TRUE|actual TRUE) vs P(FALSE|FALSE), while the overall accuracy is the P=P(TRUE|TRUE)*P(actual TRUE) + P(FALSE|FALSE)*P(actual FALSE). So this depends on the proportion of the true value on your data set very much. In practice, it seems that the best overall accuracy is usually achieved when the cutpoint is near the P(actual TRUE). $\endgroup$
    – Vincent
    Commented Sep 1, 2013 at 23:25
  • $\begingroup$ So AUC and the best overall accuracy may be not consistent, depending on the proportion of the true value of your data set. In your case, it seems that one of the classifier is more focus on sensitivity while the other on specificity. And on your current data set, the P(TRUE) is not 50%. So sensitivity and specificity contribute to the overall accuracy by different weighted. In practice, the ROC can give us more information, and we would like to choose the classier case by case. For example, the spam classifier may be more focus on P(not spam|not spam) to prevent from missing important emails. $\endgroup$
    – Vincent
    Commented Sep 1, 2013 at 23:38
  • $\begingroup$ Thanks for your answer. It is much clearer now. But if someone has more want to discuss please post here. $\endgroup$
    – Samo Jerom
    Commented Sep 2, 2013 at 9:55
37
$\begingroup$

While the two statistics measures are likely to be correlated, they measure different qualities of the classifier.

AUROC

The area under the curve (AUC) is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative example. It measures the classifiers skill in ranking a set of patterns according to the degree to which they belong to the positive class, but without actually assigning patterns to classes.

The overall accuracy also depends on the ability of the classifier to rank patterns, but also on its ability to select a threshold in the ranking used to assign patterns to the positive class if above the threshold and to the negative class if below.

Thus the classifier with the higher AUROC statistic (all things being equal) is likely to also have a higher overall accuracy as the ranking of patterns (which AUROC measures) is beneficial to both AUROC and overall accuracy. However, if one classifier ranks patterns well, but selects the threshold badly, it can have a high AUROC but a poor overall accuracy.

Practical Use

In practice, I like to collect the overall accuracy, the AUROC and if the classifier estimates the probability of class membership, the cross-entropy or predictive information. Then I have a metric that measures its raw ability to perform a hard classification (assuming false-positive and false-negative misclassification costs are equal and the class frequencies in the sample are the same as those in operational use - a big assumption!), a metric that measures the ability to rank patterns and a metric that measures how well the ranking is calibrated as a probability.

For many tasks, the operational misclassification costs are unknown or variable, or the operational class frequencies are different to those in the training sample or are variable. In that case, the overall accuracy is often fairly meaningless and the AUROC is a better indicator of performance and ideally we want a classifier that outputs well-calibrated probabilities, so that we can compensate for these issues in operational use. Essentially which metric is important depends on the problem we are trying to solve.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Dikran, do you have a reference for your first paragraph ? $\endgroup$
    – Bunder
    Commented Nov 22, 2016 at 22:05
  • $\begingroup$ @Bunder not directly, the AUROC is the probability that a randomly selected +ve pattern will be ranked higher than a randomly selected -ve pattern (en.wikipedia.org/wiki/…) and hence is a measure of the quality of the ranking, as we would want that probability to be as high as possible. $\endgroup$
    – Dikran Marsupial
    Commented Nov 23, 2016 at 10:11
6
$\begingroup$

Is AUC really very useful metric?

I would say expected cost is more appropriate measure.

Then you would have a cost A for all False Positives and cost B for all False Negatives. It might easily be that other class is relative more expensive than other. Of course if you have costs for false classification in the various sub-groups then it would be even more powerful metric.

By plotting cut-off in the x-axis and expected cost on then y-axis you can see which cut-off point minimizes expected cost.

Formally you have a loss-function Loss(cut-off|data,cost) which you try to minimize.

Improve this answer
$\endgroup$
1
  • 4
    $\begingroup$ The expected cost can only be evaluated if you know the false-positive and false-negative costs, which are not necessary for calculation of the AUC, which is a good statistic to use if the costs are unknown or variable. $\endgroup$
    – Dikran Marsupial
    Commented Sep 13, 2013 at 11:42
6
$\begingroup$

Like all the answers have been posted: ROC and accuracy are fundamentally two different concepts.

Generally speaking, ROC describes the discriminative power of a classifier independent of class distribution and unequal prediction error costs (false positive and false negative cost).

Metric like accuracy is calculated based on the class distribution of test dataset or cross-validation, but this ratio may change when you apply the classifier to real life data, because the underlying class distribution has been changed or is unknown. On the other hand, TP rate and FP rate which are used to construct AUC will be not be affected by class distribution shifting.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.