7
$\begingroup$

I have a binary classification scenario with a dataset that is unbalanced (much more negatives than positives). When I train a classifier on this dataset I get a Precision-Recall AUC of 0.7.

Then I under-sampled the dataset to make it balanced. Then I trained the classifier on this balanced dataset and I got a PR-AUC of 0.9.

My question: is it correct to use PR-AUC and not ROC-AUC here? Because as I know PR-AUC is highly influenced by the class-imbalance, and now I'm afraid that I got a high PR-AUC with the balanced dataset because of the connection between PR-AUC and class-imbalance.

In other words, did under-sampling the dataset bias the result? Or was it really that under-sampling the dataset was a good thing to increase the classification performance?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
8
$\begingroup$

You can't compare PR-AUC values based on differently balanced data. You can use ROC-AUC for that, though, since that does not depend on class balance.

The larger the fraction of positives in the data set, the larger the area under the PR curve will be for a given model. By increasing the fraction of positives in the data, you artificially inflate PR-AUC (which may or may not be additional to an improved model, you cannot measure).

A random model has PR-AUC equal to the fraction of positives, since it's precision is always equal to the fraction of positives regardless of the recall. For ROC curves the AUC of a random model is 50%, independent of class balance. If you want to assess models under varying levels of class balance, I suggest using ROC-AUC instead of PR-AUC.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ I'm not sure if the PR-AUC area will really have this expected value. The Average Precision will have this value. But due to the odd shape of the PR-Curve, it will at least have really high variance. If one actually hacks it in a way that the AUC is defined at all, because there is no precision at recall 0. I agree that the ROC AUC is much more well-behaved to analyze. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Oct 28, 2014 at 21:44
  • $\begingroup$ @Anony-Mousse strictly speaking the value is indeed undefined at recall 0. Under similar assumptions as those that lead to 50% ROC-AUC for a random model, the expected value for PR-AUC can be derived since the influence of the undefined point can be neglected (with some hand waiving). $\endgroup$
    – Marc Claesen
    Commented Oct 28, 2014 at 21:50
  • $\begingroup$ I'm more concerned about e.g. small data sets or extreme unbalancedness. Positives first make the curve start at 1; negatvies at 0. On unbalanced data, the curves drop fast, but raise slowly. If you do the non-linear interpolation, the possible curves don't obviously sum up to a constant (if you do linear or constant interpolation, things are much easier; the latter is AveP). $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Oct 28, 2014 at 22:07
  • $\begingroup$ @Anony-Mousse absolutely. One of the assumptions that is made to derive the expected ROC-AUC for a random model (50%) is infinite sample size. This also takes care of the concerns you mention for PR curves, regardless of class balance. $\endgroup$
    – Marc Claesen
    Commented Oct 28, 2014 at 22:12
  • $\begingroup$ @MarcClaesen can you please explain to me in precision-recall formulas why this sentence is true: "The larger the fraction of positives in the data set, the larger the area under the PR curve will be for a given model"? $\endgroup$
    – Jack Twain
    Commented Oct 29, 2014 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.