7
$\begingroup$

I have always always understood the diagonal of the ROC plot to represent the performance of a "random" classifier (corresponding to an AUC of 0.5). Is this still the case for highly imbalanced problems? (e.g. 10 positives vs 1000 negatives).

If not, is there any way to estimate the ROC-AUC for a "random" classifier (fully blind to features and to the proportions of positives and negatives).

What if we test against a stratified random classifier? (i.e. one that guesses positives and negatives randomly but according to the proportion of positives and negatives).

What can be said about the metric Average Precision in the same scenarios? (random and stratified random).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

Area under the ROC curve is insensitive to class balance. Area under the PR curve, on the other hand, is highly influenced by this.

Improve this answer
$\endgroup$
9
  • 2
    $\begingroup$ PR curves plot recall as a function of precision. For a data set with $x\%$ positives, the precision of a random classifier will be $x\%$, regardless of the recall. Its area under the PR curve is therefore equal to $x\%$, directly proportional to the class imbalance. Whether or not this is desirable depends on why you are comparing classifiers. If you want to see the effect of class balance on your model's performance, area under the PR curve is a good measure. $\endgroup$
    – Marc Claesen
    Commented Jul 30, 2014 at 13:21
  • 1
    $\begingroup$ @user023472 a random classifier will always have AUC=0.5 regardless of balance. This is because in ROC curves both $x$ and $y$ axes are plotted from the data perspective rather than the model perspective (1-specificity for ROC versus precision in PR curves). Stratified random is a special case of random with the threshold in such a way that a certain fraction is labeled positive. This constitutes 1 point in ROC space, so you cannot compute AUC (AUC for a certain model is obtained by varying the positive prediction threshold on decision values). $\endgroup$
    – Marc Claesen
    Commented Jul 30, 2014 at 13:44
  • 1
    $\begingroup$ Much of your problem stems from the use of classification as opposed to risk prediction. $\endgroup$
    – Frank Harrell
    Commented Jul 30, 2014 at 14:41
  • 2
    $\begingroup$ @MarcClaesen Your statement "stratified random constitutes 1 point in ROC space" isn't correct. The same could be said for a uniform random classifier. A stratified random classifier could output scores with a Beta distribution fitted to the label information of the training data, for example. You would be able to get a curve, and not just a single point. So it does make sense to talk of the AUC of a stratified random classifier. The question is: what AUC can we expect for such a classifier, and if the comparison is useful (e.g. would it outperform a uniform random classifier?) $\endgroup$
    – Amelio Vazquez-Reina
    Commented Jul 30, 2014 at 15:33
  • 1
    $\begingroup$ The optimum optimality criterion is maximum likelihood, especially if penalized or using a Bayesian prior. The concordance probability ($c$-index; AUROC) is not something to maximize; neither is any quantity that makes up the ROC curve or any quantity that uses dichotomization. Optimum predictions will come from the use of maximum likelihood. The $c$-index is not sensitive enough to choose from among competing models. $\endgroup$
    – Frank Harrell
    Commented Jul 30, 2014 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.