Why is ROC insensitive to class distributions?

Question

I am confused over why ROC is invariance under class distribution described in the paper An Introduction to ROC analysis. I cannot understand the example on why the proportion of positive to negative classes in a test set does not affect the ROC curves.

Also to quote from this post, it says that:

To show this, first let's start with a very nice way to define precision, recall and specificity. Assume you have a "positive" class called 1 and a "negative" class called 0. $\hat{Y}$ is your estimate of the true class label $Y$. Then: $$ \begin{aligned} &\text{Precision} &= P(Y = 1 | \hat{Y} = 1) \\ &\text{Recall} = \text{Sensitivity} &= P(\hat{Y} = 1 | Y = 1) \\ &\text{Specificity} &= P(\hat{Y} = 0 | Y = 0) \end{aligned} $$ The key thing to note is that sensitivity/recall and specificity, which make up the ROC curve, are probabilities conditioned on the true class label. Therefore, they will be the same regardless of what $P(Y = 1)$ is.

I cannot reconcile these few concepts together, likely due to a gap in statistical rigour. I would highly appreciate someone to give me a more detailed example on why the above is true.

---

To be more specific, can someone explain the above quote? In particular what does it mean to be conditioned on P(Y=1)? What is this P referring to? And why does conditioning on this implies that ROC is insensitive to class distribution. To add on, I did read through almost every post related to this question, but don't see consensus in whether ROC curve is sensitive or insensitive to class imbalance.

The posts I read: I know it's quite a lot, I even managed to implement ROC curve using pure python code without an issue. But it seems that even if I can implement it, I still do not fully understand it.

Interpretation of ROC

• Wikipedia has an extensive explanation of the probability behind ROC
• Probabilistic Perspective of AUC
• AUC - Insider's Guide to the Theory and Applications
• Safe Handling Instructions for Probabilistic Classification.
• c-statistics.
• SIIM Melanoma ROC
• An Introduction to ROC analysis

Pros and Cons of AUROC

• The Relationship Between Precision-Recall and ROC Curves
• Drawbacks of AUROC.
• ROC vs precision-and-recall curves
• ROC vs Precision-recall curves on imbalanced dataset
• on why AUC can be misleading

---

Latest Understanding 21st September 2021:

As Professor Frank Harrell has mentioned in the post below, I reinforce my understanding by further saying:

Y takes on 0 and 1, and the area under the ROC graph (call this value $a$), in a simplified manner, signifies that if you take randomly a positive sample, and a negative sample, your probability of positive sample being ranked higher (read: higher probability) than the negative sample is $a$.

Now with his analogy, the teacher is negative sample, and soccer star is positive sample, so now you conditioned on Y = 0, and Y = 1. Once you conditioned on say, $Y=0$, (Specificity/TNR or 1-FPR) then your sample space effectively reduces from the whole population of the samples, to only $Y=0$, From this, I intuitively think that $Y=1$ does not play a part and hence does not influence the FPR in any way. Similar concept can be applied to TPR. As a result, neither TPR nor FPR depends on the whole sample space (the whole distribution of the test (?) set), and as a result will not be influenced under class distributional changes in the testset (?).

TODO: To reason why precision depends on class distribution.

Answer 1

Since all points on an ROC curve condition on Y, the distribution of Y is necessarily irrelevant for the points. This also points out why ROC curves should not be used except in a retrospective case-control study where samples are taken from Y=0 and Y=1 observations. For prospectively observed data where we sample based on X or take completely random samples, it is not logical to use a representation that disrespects how the samples arose. See https://www.fharrell.com/post/addvalue/

Answer 2

What I am answering

I felt your key statement was:

I cannot reconcile these few concepts together, likely due to a gap in statistical rigour.

so my answer if based around addressing the difference between mathematical and statistical implications of class imbalance on AUCROC.

Recap of AUCROC

The AUCROC is calculated based on the area under the curve for the receiver operators characteristics curve. This curve plots 1-sensitivity vs specificity for a series of thresholds (e.g. each actualised value in the dataset).

Sensitivity/recall is the ratio of TP to all actual positives $TP/(TP+FN)$ or $TP/Cases$. There is no consideration for the actual negatives in the calculation of sensitivity.

Specificity is the ratio of true negatives to all actual negatives $TN/(TN+FP)$ or $TN/Controls$. There is no consideration for the actual positive group in the calculation of specificity

Mathematical and Statistical Interpretation

Since the AUCROC is directly calculated from these two metrics and neither metric takes into consideration the other group there is no mathematical link between group balance and the expected AUCROC.

However, it is critical to note that 'expected' has a precise statistical meaning, in the form of the value you would expect the metric to converge to over a very very (read infinite) long run experiment. The critical thing about statistics is that we not only consider the long range expected value, but also consider the short range variability/reliability/confidence of an actual result based on finite sampling.

The confidence that we have in a actual realised result is proportional to $\pm \frac{\sigma}{\sqrt{n}} $ where $\sigma$ is the standard deviation of the data and $n$ is the total number of samples. If $n_1>>n_2$ then $\sqrt{n_1}>\sqrt{n_2}$. The points in the ROC are displaced by errors in both specificity and sensitivity so the area under that curve is a composite of those errors and the combined impact on overall confidence is proportional to $$\pm\sqrt{ (\frac{\sigma_1}{\sqrt{n_1}})^2 + (\frac{\sigma_2}{\sqrt{n_2}})^2}$$.

If $n_1 \sim n_2$ then group prevalence will be balanced and neither group will skew the confidence in the calculated result. If $n_1>>n_2$ then the confidence will be more limited by the low prevalence group.

Summary

The expected long range AUCROC value is not influenced by class prevalence, but statistical confidence is dragged down by low prevalence classes.

Answer 3

In classification problems the model output is probability. Different problems have different threshold boundaries. For example when deciding between a dog and a cat, 50% makes sense but when we talk about probability to have a heart attack all probabilities will be much much lower. AUC solves it  by checking the $FPR$ & $TPR$ for many (as much as possible) thresholds between 0 and 1.

AUC only cares about the ranking of the model i.e. if model rank the ones higher than the zeros. 

Let's examine the component of AUC:

$TPR = \frac{TP}{P} $

and

$FPR = \frac{FP}{N} $ 

Let's take as an example $TPR$ (will be similar to $FPR$).

We calculate $TPR$ for every threshold and for each example. for each example $TP$ is a function of $Y, \hat{Y}, threshold$ - this is not affected by the ratio of positive and negative.

Now, the total number of $TP$ is affected by the total number of $P$ but the $TPR$ should remain the same. Because, if we have more $P$, we will also have more $TP$ at the same ratio for the given threshold.

To conclude, changing the number of $P$ should not affect the $TPR$ for a given threshold. The same is for $FPR$ and $N$, and thus the ratio between positive and negative should not change the ROC curve.

Answer 4

Compared to the others, my answer is focused on understanding how you use ROC and AUC in Data Science cases. If you need the mathematical / statistical part, my answer won't help you.

Basically, ROC curve shows false positive (FP) RATE and true positive (TP) RATE for each threshold of the model (score you decided as being the limit between classification '1' and '0'). So at the start, if your threshold is 1 (max possible score for your model), you classify everything as 0 and then there's 0% FP and 0% TP. If threshold is 0 (min possible score for your model), everything is classified as 1 and so your TP and FP rates are 100%. Using a threshold strictly between 0 and 1, you'll have FP and TP rates between 0% and 100%.

Since this curve is representing Rates obtained at each possible threshold, if you print ROC Curve for your test set, it's totally independant from the training set. It only shows how much FP and TP you have, compared to the maximum you can have in the set.

Let's take an easy example : You have a test set with 100 '0' and 10 '1'. Having found 5 of the 10 '1', but misclassifying 30 '0' as '1' to achieve that, you obtain for your curve

x = FP_Rate = 30/100 = 0.3
y = TP Rate = 5/10 = 0.5

Imagine now your dataset is balanced and you have 50 '0' and 50 '1'. If you still find half of the ones (25 '1') misclassifying 30% of your zeros (15 '0'), you'll still find x=0.3 ; y=0.5 for your curve.

The only matter with ROC Curve is the percentage of FP compared to the percentage of TP, wether the model is balanced or not.

--- Edit after comment question :

This depends how you use AUC (Area under ROC curve, what you might call ROC metric). AUC measures the performance of 1 model on 1 set. So if you apply it on Train, it'll measure how your model (built on Train) performs on Train (you often do it to compare AUC_Train and AUC_Test and see if you overfit). AUC has nothing to do with how your model is built, it just evaluates the result of 1 model applied on 1 certain set. Wether the set is Train or Test, when you calculate AUC, it's just "The set in which you test your model performance". So this makes no difference.

Also, if you want a probabilistic way to understand AUC : If you have a 0.8 AUC, it means that if you take one random '1' row and one random '0' row and apply your trained model on them, the probability of having a higher score for your '1' row than for your '0' is 0.8

You then understand how AUC=0.5 means the model is a random classifier.