Why AUC-PR increases when the number of positives increase?

Question

I asked a question earlier about comparing models using Precision-Recall AUC. One of the answers included the following statement: "The larger the fraction of positives in the data set, the larger the area under the PR curve will be for a given model". However I couldn't understand why is it true.

Here are the formulas to compute precision and recall:

$Recall = \frac{TP}{TP+FN}$

$Precision = \frac{TP}{TP+FP}$

It doesn't make sense to me because when we have an imbalanced dataset (too many negatives and few positives) then the model will make a low False Positive (FP) [since it will tend probably to classify most samples as negatives] and so the precision will be high. When the dataset is balanced, then it will make a high False Positive and so the precision will be low.

Am I missing something?

Answer 1

Remember that PR curves visualize a model's performance over the entire operating range, not just where its classification threshold happens to be. Your reasoning in the final paragraph seems to be based on the model's classification of test instances, rather than their ranking.

PR curves are not computed based on predicted labels (positive/negative) but rather from the model's ranking of test instances based on decision values. Decision values can generally be considered in $\mathbb{R}$, though for some models these are probabilities (for instance logistic regression).

Non-random model

For a real model, when more positives are added to the data set from which PR curves are computed, the observed precision of the model for a given level of recall can never go down$^*$. This follows readily from the way precision is calculated ($\frac{TP}{TP+FP}$, adding positives increases $TP$ and leaves $FP$ unchanged so the precision increases). In other words, the new PR curve (computed with a higher fraction of positives) necessarily dominates the original one for the same model.

$^*$assuming the added positives are distributed in the same way as the 'original' ones, so the model's recall as a function of its decision value remains the same.

Random model

As Anony-Mousse stated, a random result will have expected precision equal to the fraction of positives in the data set for any recall.

The recall of a random model is directly linked to the fraction of data it assigns to be positive $f_{pos}$, particularly: $$recall = \frac{n_{pos}^{(pred)}}{n_{pos}^{(truth)}} = \frac{f_{pos}\times n_{total}}{n_{pos}^{(truth)}},$$

where $n_{pos}^{(pred)}$ is some fraction of the total data ($f_{pos}$ is based on the decision threshold). The expected precision of a random model is always equal to the fraction of positives in the data set. This follows directly from the definition of precision, namely the fraction of true positives in all positive predictions. In a random model the predictions are unrelated to the true label, so the expected value of its precision is by definition equal to the fraction of positives in the data.

As such, the expected PR curve of a random model is essentially a horizontal line. This line spans the entire recall range (e.g. width 1) and has height equal to the expected precision (equal to the fraction of positives) so the associated AUC is equal to the fraction of positives.

Conclusion

As a result, increasing the fraction of positives inflates the area under the PR curve. You cannot compare PR curves (nor their AUC) computed at different levels of class balance. ROC curves do not exhibit such behaviour.

Answer 2

precision $$ prec = \frac{TP}{TP+FP} $$ and in case when we add positive values prec is not deceasing (so we find at least previous TP number of ones. When we find them more, the prec is increasing, like: $$ \frac{TP+1}{TP+1+FP} \geqslant \frac{TP}{TP+FP} $$ (is strictly greater if FP > 0) FP is not changing because we don't add zeros.

recall $$ recall = \frac{TP}{TP+FN} $$ when we are adding ones, statistically some of them will be found as TP and some undiscovered as FN. The ratio of this classification probability is proportional to ratio of areas TP and FN, so if we add k ones then: $$ k\frac{TP}{TP+FN} $$ are classified as TP and $$ k\frac{FN}{TP+FN} $$ are classified as FN

So let compute new recall

$$ \frac{k\frac{TP}{TP+FN} + TP}{k\frac{TP}{TP+FN}+TP+k\frac{FN}{TP+FN}+FN} $$ $$ \frac{k\frac{TP}{TP+FN} + TP\frac{TP+FN}{TP+FN}}{k+TP+FN} $$ $$ \frac{TP(\frac{k}{TP+FN} + \frac{TP+FN}{TP+FN})}{k+TP+FN} $$ $$ \frac{TP(\frac{k+TP+FN}{TP+FN})}{k+TP+FN} $$ $$ \frac{TP}{TP+FN} $$ so recall does not change