Timeline for Why is ROC insensitive to class distributions?
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 20, 2023 at 22:38 | comment | added | Frank Harrell | The concordance probability ($c$-index; AUROC), since it conditions on $Y$, i.e., takes random observation with $Y=0$ and compares predictions to a random observations with $Y=1$, is completely free of any $Y$-prevalence issue. Covariate shift is a different issue, as it is with all prediction methods. And lowering the spread of the distribution of a covariate will make all discrimination indexes go down. | |
| May 20, 2023 at 18:52 | comment | added | Antonios Sarikas | @FrankHarrell So does this mean that we don't expect AUC-ROC of a model to change when applied to new data that experience covariate shift or label shift? | |
| Sep 25, 2021 at 4:11 | vote | accept | nan | ||
| Sep 21, 2021 at 21:24 | comment | added | Dave | @FrankHarrell It is not uncommon for me to see discussions about how ROCAUC is a poor performance metric where there is class imbalance, and precision-recall AUC would be preferred. Setting aside your usual arguments for log loss or Brier score over ROCAUC, what is going on to make people believe that? | |
| Sep 21, 2021 at 13:33 | comment | added | nan | Thanks Professor, I have edited my post to show my latest understanding, based on your comment. | |
| Sep 21, 2021 at 11:41 | comment | added | Frank Harrell | When you condition on something, you make its distribution irrelevant. Suppose you randomly choose one soccer star and one teacher from the world population, and compare them. The fact that there are many times more teachers than soccer stars in the world cannot be relevant to any comparison you make between the two people. | |
| Sep 20, 2021 at 18:38 | history | edited | Henry | CC BY-SA 4.0 |
link that works
|
| Sep 20, 2021 at 14:39 | comment | added | nan | Thanks Professor, I’ve also referred to your articles on why ROC is less than an ideal metric under certain cases. However, May I know if you can point me to a right direction when you mention that distribution of Y is irrelevant in this case? This point is precisely where my understanding break down, simply because I am not thinking probabilistically enough. | |
| Sep 20, 2021 at 11:42 | history | answered | Frank Harrell | CC BY-SA 4.0 |