AUC ROC Threshold Setting in heavy imbalance [duplicate]

Question

I am doing binary logistic regression on a dataset with very heavy class imbalance. Class 1 is only 1% of data. When I train logistic regressor without class weights I get ROC AUC Score of 0.6269. Which is decent. However, when I see my confusion matrix I see that my model never predicted any 1's at all. So why is my AUC so high? I though AUC is meant to be a good measure for such a case.

Confusion matrix
 Predicted      0    All
True                   
0          32109  32109
1           1223   1223
All        33332  33332

I know Confusion matrix makes the probability threshold 0.5, so is score saying there is some threshold for which model will give higher recall? How can I get this threshold?

          Class  precision    recall  f1-score   support

           0       0.96      1.00      0.98     32109
           1       0.00      0.00      0.00      1223

I only care about precision and recall of class 1.

Answer 1

AUC measure the area under the ROC curve, wich is built by taking (TPR,FPR) for all the possible thresholds. See here for the whole definition and construction : What does AUC stand for and what is it?

It is a good metric for textbook problems with separable classes and when your decision has no practical impact. It let you look at multiple thresholds at once. In general, it is not a good metric if you want only one threshold, ie. to make a decision. It is especially bad with imbalanced data set as you have seen. See Frank Harrell answer to the question mentionned above.

There might be a bigger problem with your approach : with imbalanced data set the class are probably not separable. So it doesn't make sense to use a classifier in the ML sense. You might want to use a probabilistic approach to model your output, then use a good metric to choose how you set a threshold and you separate your classes. That metric would heavily depend on what you are trying to model and practical implication associated to your decision. See Frank Harrel blog post: https://www.fharrell.com/post/classification/