Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have always always understood the diagonal of the ROC plot to represent the performance of a "random" classifier (corresponding to an AUC of 0.5). Is this still the case for highly imbalanced problems? (e.g. 10 positives vs 1000 negatives).

If not, is there any way to estimate the ROC-AUC for a "random" classifier (fully blind to features and to the proportions of positives and negatives).

What if we test against a stratified random classifier? (i.e. one that guesses positives and negatives randomly but according to the proportion of positives and negatives).

What can be said about the metric Average Precision in the same scenarios? (random and stratified random).

share|improve this question

1 Answer 1

Area under the ROC curve is insensitive to class balance. Area under the PR curve, on the other hand, is highly influenced by this.

share|improve this answer
    
When you say "highly influenced" by class imbalance, what do you mean exactly? I have read e.g. here that PR is usually preferred over ROC in highly imbalanced problems. Is that sensitivity desirable then? How does AveP change as a function of class imbalance (in random classification)? –  user023472 Jul 30 at 13:17
2  
PR curves plot recall as a function of precision. For a data set with $x\%$ positives, the precision of a random classifier will be $x\%$, regardless of the recall. Its area under the PR curve is therefore equal to $x\%$, directly proportional to the class imbalance. Whether or not this is desirable depends on why you are comparing classifiers. If you want to see the effect of class balance on your model's performance, area under the PR curve is a good measure. –  Marc Claesen Jul 30 at 13:21
1  
@user023472 a random classifier will always have AUC=0.5 regardless of balance. This is because in ROC curves both $x$ and $y$ axes are plotted from the data perspective rather than the model perspective (1-specificity for ROC versus precision in PR curves). Stratified random is a special case of random with the threshold in such a way that a certain fraction is labeled positive. This constitutes 1 point in ROC space, so you cannot compute AUC (AUC for a certain model is obtained by varying the positive prediction threshold on decision values). –  Marc Claesen Jul 30 at 13:44
1  
Much of your problem stems from the use of classification as opposed to risk prediction. –  Frank Harrell Jul 30 at 14:41
2  
@MarcClaesen Your statement "stratified random constitutes 1 point in ROC space" isn't correct. The same could be said for a uniform random classifier. A stratified random classifier could output scores with a Beta distribution fitted to the label information of the training data, for example. You would be able to get a curve, and not just a single point. So it does make sense to talk of the AUC of a stratified random classifier. The question is: what AUC can we expect for such a classifier, and if the comparison is useful (e.g. would it outperform a uniform random classifier?) –  user023472 Jul 30 at 15:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.