6
$\begingroup$

I asked a question earlier about comparing models using Precision-Recall AUC. One of the answers included the following statement: "The larger the fraction of positives in the data set, the larger the area under the PR curve will be for a given model". However I couldn't understand why is it true.

Here are the formulas to compute precision and recall:

$Recall = \frac{TP}{TP+FN}$

$Precision = \frac{TP}{TP+FP}$

It doesn't make sense to me because when we have an imbalanced dataset (too many negatives and few positives) then the model will make a low False Positive (FP) [since it will tend probably to classify most samples as negatives] and so the precision will be high. When the dataset is balanced, then it will make a high False Positive and so the precision will be low.

Am I missing something?

$\endgroup$
  • 1
    $\begingroup$ The expected precision of a random result is #positives/#total. If you increase the rate of positives, the expected value even for a random result increases. $\endgroup$ – Anony-Mousse Oct 29 '14 at 11:59
  • $\begingroup$ @Anony-Mousse can you please elaborate more on why the random model PR AUC is the fraction of the positives? Also can you please give me a paper that could potentially include and explain everything you said here? I need to reference it in my master thesis :) $\endgroup$ – Jack Twain Oct 31 '14 at 9:36
  • $\begingroup$ Sorry, I don't have references at hand. It's probably too basic to find an article on this. The derivation of the expected precision is statistics 101: hypergeometric distribution. For referencing, consider citing Encyclopedia of Machine Learning, which includes short articles on various basic topics. $\endgroup$ – Anony-Mousse Oct 31 '14 at 10:11
3
$\begingroup$

Remember that PR curves visualize a model's performance over the entire operating range, not just where its classification threshold happens to be. Your reasoning in the final paragraph seems to be based on the model's classification of test instances, rather than their ranking.

PR curves are not computed based on predicted labels (positive/negative) but rather from the model's ranking of test instances based on decision values. Decision values can generally be considered in $\mathbb{R}$, though for some models these are probabilities (for instance logistic regression).

Non-random model

For a real model, when more positives are added to the data set from which PR curves are computed, the observed precision of the model for a given level of recall can never go down$^*$. This follows readily from the way precision is calculated ($\frac{TP}{TP+FP}$, adding positives increases $TP$ and leaves $FP$ unchanged so the precision increases). In other words, the new PR curve (computed with a higher fraction of positives) necessarily dominates the original one for the same model.

$^*$assuming the added positives are distributed in the same way as the 'original' ones, so the model's recall as a function of its decision value remains the same.

Random model

As Anony-Mousse stated, a random result will have expected precision equal to the fraction of positives in the data set for any recall.

The recall of a random model is directly linked to the fraction of data it assigns to be positive $f_{pos}$, particularly: $$recall = \frac{n_{pos}^{(pred)}}{n_{pos}^{(truth)}} = \frac{f_{pos}\times n_{total}}{n_{pos}^{(truth)}},$$

where $n_{pos}^{(pred)}$ is some fraction of the total data ($f_{pos}$ is based on the decision threshold). The expected precision of a random model is always equal to the fraction of positives in the data set. This follows directly from the definition of precision, namely the fraction of true positives in all positive predictions. In a random model the predictions are unrelated to the true label, so the expected value of its precision is by definition equal to the fraction of positives in the data.

As such, the expected PR curve of a random model is essentially a horizontal line. This line spans the entire recall range (e.g. width 1) and has height equal to the expected precision (equal to the fraction of positives) so the associated AUC is equal to the fraction of positives.

Conclusion

As a result, increasing the fraction of positives inflates the area under the PR curve. You cannot compare PR curves (nor their AUC) computed at different levels of class balance. ROC curves do not exhibit such behaviour.

$\endgroup$
  • $\begingroup$ can you please elaborate more on why the random model PR AUC is the fraction of the positives? Also can you please give me a paper that could potentially include and explain everything you said here? I need to reference it in my master thesis :) $\endgroup$ – Jack Twain Oct 30 '14 at 20:13
  • $\begingroup$ @JackTwain done. $\endgroup$ – Marc Claesen Oct 31 '14 at 10:11
  • $\begingroup$ and paper to reference in master thesis ;) $\endgroup$ – Jack Twain Oct 31 '14 at 10:14
  • $\begingroup$ I don't know any paper about this as it is a fairly basic result based on the definition of PR curves. $\endgroup$ – Marc Claesen Oct 31 '14 at 10:27
  • $\begingroup$ So can we say this in other words: by having the precision of the random classifier to be #positives/#total we are actually saying that the probability of the classifier to be 'precise' is #positives/#total. Is my way correct? But hmmm I'm still struggling with the idea of having the recall always constant. $\endgroup$ – Jack Twain Oct 31 '14 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.