12
$\begingroup$

I am confused over why ROC is invariance under class distribution described in the paper An Introduction to ROC analysis. I cannot understand the example on why the proportion of positive to negative classes in a test set does not affect the ROC curves.

Also to quote from this post, it says that:

To show this, first let's start with a very nice way to define precision, recall and specificity. Assume you have a "positive" class called 1 and a "negative" class called 0. $\hat{Y}$ is your estimate of the true class label $Y$. Then: $$ \begin{aligned} &\text{Precision} &= P(Y = 1 | \hat{Y} = 1) \\ &\text{Recall} = \text{Sensitivity} &= P(\hat{Y} = 1 | Y = 1) \\ &\text{Specificity} &= P(\hat{Y} = 0 | Y = 0) \end{aligned} $$ The key thing to note is that sensitivity/recall and specificity, which make up the ROC curve, are probabilities conditioned on the true class label. Therefore, they will be the same regardless of what $P(Y = 1)$ is.

I cannot reconcile these few concepts together, likely due to a gap in statistical rigour. I would highly appreciate someone to give me a more detailed example on why the above is true.


To be more specific, can someone explain the above quote? In particular what does it mean to be conditioned on P(Y=1)? What is this P referring to? And why does conditioning on this implies that ROC is insensitive to class distribution. To add on, I did read through almost every post related to this question, but don't see consensus in whether ROC curve is sensitive or insensitive to class imbalance.

The posts I read: I know it's quite a lot, I even managed to implement ROC curve using pure python code without an issue. But it seems that even if I can implement it, I still do not fully understand it.

Interpretation of ROC

Pros and Cons of AUROC


Latest Understanding 21st September 2021:

As Professor Frank Harrell has mentioned in the post below, I reinforce my understanding by further saying:

Y takes on 0 and 1, and the area under the ROC graph (call this value $a$), in a simplified manner, signifies that if you take randomly a positive sample, and a negative sample, your probability of positive sample being ranked higher (read: higher probability) than the negative sample is $a$.

Now with his analogy, the teacher is negative sample, and soccer star is positive sample, so now you conditioned on Y = 0, and Y = 1. Once you conditioned on say, $Y=0$, (Specificity/TNR or 1-FPR) then your sample space effectively reduces from the whole population of the samples, to only $Y=0$, From this, I intuitively think that $Y=1$ does not play a part and hence does not influence the FPR in any way. Similar concept can be applied to TPR. As a result, neither TPR nor FPR depends on the whole sample space (the whole distribution of the test (?) set), and as a result will not be influenced under class distributional changes in the testset (?).

TODO: To reason why precision depends on class distribution.

$\endgroup$
2
  • 1
    $\begingroup$ It is only true that it is not directly mathematically affected. It is however indirectly statistically affected because the statistical power of the lower prevalence group is weaker and so will skew the confidence of the respective contributions of each group. The long run average will have the same expected TP and FP rate, but short run repeated small experiments would give estimates will vary a lot more for low prevalence groups than for high prevalence groups, which will have a knock on effect on the reliability of the AUCROC. $\endgroup$
    – ReneBt
    Sep 20 '21 at 8:43
  • $\begingroup$ @ReneBt Do you mind illustrating the concepts mathematically? I cannot seem to derive the idea above. $\endgroup$
    – nan
    Sep 20 '21 at 8:47
6
$\begingroup$

Since all points on an ROC curve condition on Y, the distribution of Y is necessarily irrelevant for the points. This also points out why ROC curves should not be used except in a retrospective case-control study where samples are taken from Y=0 and Y=1 observations. For prospectively observed data where we sample based on X or take completely random samples, it is not logical to use a representation that disrespects how the samples arose. See https://www.fharrell.com/post/addvalue/

$\endgroup$
4
  • 3
    $\begingroup$ Thanks Professor, I’ve also referred to your articles on why ROC is less than an ideal metric under certain cases. However, May I know if you can point me to a right direction when you mention that distribution of Y is irrelevant in this case? This point is precisely where my understanding break down, simply because I am not thinking probabilistically enough. $\endgroup$
    – nan
    Sep 20 '21 at 14:39
  • 2
    $\begingroup$ When you condition on something, you make its distribution irrelevant. Suppose you randomly choose one soccer star and one teacher from the world population, and compare them. The fact that there are many times more teachers than soccer stars in the world cannot be relevant to any comparison you make between the two people. $\endgroup$ Sep 21 '21 at 11:41
  • $\begingroup$ Thanks Professor, I have edited my post to show my latest understanding, based on your comment. $\endgroup$
    – nan
    Sep 21 '21 at 13:33
  • $\begingroup$ @FrankHarrell It is not uncommon for me to see discussions about how ROCAUC is a poor performance metric where there is class imbalance, and precision-recall AUC would be preferred. Setting aside your usual arguments for log loss or Brier score over ROCAUC, what is going on to make people believe that? $\endgroup$
    – Dave
    Sep 21 '21 at 21:24
5
$\begingroup$

In classification problems the model output is probability. Different problems have different threshold boundaries. For example when deciding between a dog and a cat, 50% makes sense but when we talk about probability to have a heart attack all probabilities will be much much lower. AUC solves it  by checking the $FPR$ & $TPR$ for many (as much as possible) thresholds between 0 and 1.

AUC only cares about the ranking of the model i.e. if model rank the ones higher than the zeros. 

Let's examine the component of AUC:

$TPR = \frac{TP}{P} $

and

$FPR = \frac{FP}{N} $ 

Let's take as an example $TPR$ (will be similar to $FPR$).

We calculate $TPR$ for every threshold and for each example. for each example $TP$ is a function of $Y, \hat{Y}, threshold$ - this is not affected by the ratio of positive and negative.

Now, the total number of $TP$ is affected by the total number of $P$ but the $TPR$ should remain the same. Because, if we have more $P$, we will also have more $TP$ at the same ratio for the given threshold.

To conclude, changing the number of $P$ should not affect the $TPR$ for a given threshold. The same is for $FPR$ and $N$, and thus the ratio between positive and negative should not change the ROC curve.

$\endgroup$
5
$\begingroup$

What I am answering

I felt your key statement was:

I cannot reconcile these few concepts together, likely due to a gap in statistical rigour.

so my answer if based around addressing the difference between mathematical and statistical implications of class imbalance on AUCROC.

Recap of AUCROC

The AUCROC is calculated based on the area under the curve for the receiver operators characteristics curve. This curve plots 1-sensitivity vs specificity for a series of thresholds (e.g. each actualised value in the dataset).

Sensitivity/recall is the ratio of TP to all actual positives $TP/(TP+FN)$ or $TP/Cases$. There is no consideration for the actual negatives in the calculation of sensitivity.

Specificity is the ratio of true negatives to all actual negatives $TN/(TN+FP)$ or $TN/Controls$. There is no consideration for the actual positive group in the calculation of specificity

Mathematical and Statistical Interpretation

Since the AUCROC is directly calculated from these two metrics and neither metric takes into consideration the other group there is no mathematical link between group balance and the expected AUCROC.

However, it is critical to note that 'expected' has a precise statistical meaning, in the form of the value you would expect the metric to converge to over a very very (read infinite) long run experiment. The critical thing about statistics is that we not only consider the long range expected value, but also consider the short range variability/reliability/confidence of an actual result based on finite sampling.

The confidence that we have in a actual realised result is proportional to $\pm \frac{\sigma}{\sqrt{n}} $ where $\sigma$ is the standard deviation of the data and $n$ is the total number of samples. If $n_1>>n_2$ then $\sqrt{n_1}>\sqrt{n_2}$. The points in the ROC are displaced by errors in both specificity and sensitivity so the area under that curve is a composite of those errors and the combined impact on overall confidence is proportional to $$\pm\sqrt{ (\frac{\sigma_1}{\sqrt{n_1}})^2 + (\frac{\sigma_2}{\sqrt{n_2}})^2}$$.

If $n_1 \sim n_2$ then group prevalence will be balanced and neither group will skew the confidence in the calculated result. If $n_1>>n_2$ then the confidence will be more limited by the low prevalence group.

Summary

The expected long range AUCROC value is not influenced by class prevalence, but statistical confidence is dragged down by low prevalence classes.

$\endgroup$
6
  • 1
    $\begingroup$ I don't see what the Z formula has to do with ROC. $\endgroup$ Sep 20 '21 at 11:39
  • $\begingroup$ Can you explain a bit more why you're using pooled variance? $\endgroup$
    – ZachTurn
    Sep 20 '21 at 16:00
  • $\begingroup$ @ZachTurn because the points on ROC will have error along both x and y axes and the error in the area will depend on both the x and y errors. $\endgroup$
    – ReneBt
    Sep 21 '21 at 4:50
  • $\begingroup$ @FrankHarrell, fair point that is a restrictive inclusion that is unnecessary $\endgroup$
    – ReneBt
    Sep 21 '21 at 4:54
  • $\begingroup$ This is not the correct formula for the standard error of AUROC. $\endgroup$ Sep 22 '21 at 11:14
2
$\begingroup$

Compared to the others, my answer is focused on understanding how you use ROC and AUC in Data Science cases. If you need the mathematical / statistical part, my answer won't help you.

Basically, ROC curve shows false positive (FP) RATE and true positive (TP) RATE for each threshold of the model (score you decided as being the limit between classification '1' and '0'). So at the start, if your threshold is 1 (max possible score for your model), you classify everything as 0 and then there's 0% FP and 0% TP. If threshold is 0 (min possible score for your model), everything is classified as 1 and so your TP and FP rates are 100%. Using a threshold strictly between 0 and 1, you'll have FP and TP rates between 0% and 100%.

Since this curve is representing Rates obtained at each possible threshold, if you print ROC Curve for your test set, it's totally independant from the training set. It only shows how much FP and TP you have, compared to the maximum you can have in the set.

Let's take an easy example : You have a test set with 100 '0' and 10 '1'. Having found 5 of the 10 '1', but misclassifying 30 '0' as '1' to achieve that, you obtain for your curve

x = FP_Rate = 30/100 = 0.3
y = TP Rate = 5/10 = 0.5

Imagine now your dataset is balanced and you have 50 '0' and 50 '1'. If you still find half of the ones (25 '1') misclassifying 30% of your zeros (15 '0'), you'll still find x=0.3 ; y=0.5 for your curve.

The only matter with ROC Curve is the percentage of FP compared to the percentage of TP, wether the model is balanced or not.

--- Edit after comment question :

This depends how you use AUC (Area under ROC curve, what you might call ROC metric). AUC measures the performance of 1 model on 1 set. So if you apply it on Train, it'll measure how your model (built on Train) performs on Train (you often do it to compare AUC_Train and AUC_Test and see if you overfit). AUC has nothing to do with how your model is built, it just evaluates the result of 1 model applied on 1 certain set. Wether the set is Train or Test, when you calculate AUC, it's just "The set in which you test your model performance". So this makes no difference.

Also, if you want a probabilistic way to understand AUC : If you have a 0.8 AUC, it means that if you take one random '1' row and one random '0' row and apply your trained model on them, the probability of having a higher score for your '1' row than for your '0' is 0.8

You then understand how AUC=0.5 means the model is a random classifier.

$\endgroup$
3
  • $\begingroup$ Thanks, in your example, are we able to represent your idea using the probabilistic interpretation I mentioned? So when the ROC metric is insensitive to class proportions in test set, but this does not mean it is insensitive to class proportions in train set? $\endgroup$
    – nan
    Sep 20 '21 at 8:43
  • $\begingroup$ I answer editting my post (new paragraph at the end) since I had not enough space in comments $\endgroup$
    – Adept
    Sep 20 '21 at 9:15
  • $\begingroup$ This answer is completely at odds with how decisions are made. See fharrell.com/post/backwards-probs - the ROC curve is appropriate for retrospectively sampled data (e.g. case-control study) but not for forward prediction. $\endgroup$ Sep 22 '21 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.