2
$\begingroup$

I understand that using the area under the ROC curve is a useful error measurement for unbalanced data. What happens if we use it for balanced data?

$\endgroup$
  • $\begingroup$ Please check the paper by Davis, Jesse, and Mark Goadrich. "The relationship between Precision-Recall and ROC curves." Proceedings of the 23rd international conference on Machine learning. ACM, 2006. machinelearning.wustl.edu/mlpapers/paper_files/… which shall give a precise answer to your question $\endgroup$ – soufanom Feb 21 '15 at 4:00
2
$\begingroup$

ROC curves are insensitive to class balance, so they can be used in any setting. Area under the ROC curve is not the same as accuracy. Accuracy is determined based on a single contingency table, e.g. a single classification threshold. Area under the curve summarizes performance based on all thresholds and is therefore much more informative.

The problem with ROC curves is that, for highly unbalanced data, the differences between curves tend to be small (but present!). Precision-recall curves are better in that regard, since you can spot large differences between classifiers in unbalanced settings for which the difference in ROC space seems small. This is shown in figure 4 in this paper, a difference of 4.5% in ROC space corresponds to 25% in PR space.

$\endgroup$
1
$\begingroup$

The problem with unbalanced data is that the classifier can be unfairly weighted to favour the larger of the two groups. For example consider a medical test for a rare condition that always provides a negative result. There are no false positives, but the false negative rate is unacceptably high (i.e. there are never any true positives).

The use of TPR / FPR is such that we can have more information regarding the nuances of the accuracy of a classifier. The example is highly accurate (in the sense of proportion of correct diagnoses), but remains useless.

The ROC penalises such a situation by limiting the maximum achievable value along the vertical axis, and similar examples are possible to elucidate the equivalent along the horizontal axis. Any limitation along the axis will result in a reduced AUC.

As balanced data is not susceptible to the bias of classification we can use the naive definition of accuracy. The ROC (and particular the AUC) still provide additional insight in that it is a graphical representation of a value closely related to the Mann-Whitney U test which is representative of classification "overlap".

$\endgroup$
  • $\begingroup$ Hmm, so using the ROC with a balanced dataset is equal to say: classification accuracy? $\endgroup$ – user46925 Feb 20 '15 at 3:34
  • 1
    $\begingroup$ What is your use case? Using the AUC as a measure of classification accuracy when fitting a model to training data? I'm being pedantic, but the ROC is a graph whereas classification accuracy is a numerical value. The area below the ROC (AUC) is a numerical. $\endgroup$ – Arran Schlosberg Feb 20 '15 at 3:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy