1
$\begingroup$

I've trained a logistic regression using a small number of predictors - pseudo R-squared is only 0.1 but I have significant terms and a nice low p value for the model. However, even on its own training data, the AUC for the model is only 0.28:

ROC curve of the model

I thought this was impossible, and my only intuition for what's going on is that the class imbalance (only 5% of the observations are in the positive class) means that the no information rate is pretty high. I still think my model should beat random guessing, at least on its own training data.

I assumed this was a coding error but I think I've ruled that out now, so can anyone explain to me how this is possible?!

I've looked for other discussions on here, most seem to focus on AUC < 0.5 on the test set which I can can more easily understand (e.g. here and here). This one came the closest, demonstrating that a predictor that's really just noise can come out below 0.5 - but I think my model has found some real relationships in the data...

Improve this question
$\endgroup$
7
  • $\begingroup$ Switch the labels. :) $\endgroup$
    – usεr11852
    Commented Dec 20, 2021 at 20:02
  • $\begingroup$ Thanks @usεr11852 but I've tried flipping the labels, respecifying the target variables and other coding errors but I just keep reproducing this result. $\endgroup$
    – Tom Wagstaff
    Commented Dec 20, 2021 at 20:10
  • $\begingroup$ Thanks @Sycorax, I'd get that if my terms were all insignificant, but given I've got significant terms and a decent p value for the model overall, seems like there's some signal in the features. I realise I haven't shared enough to rule out a code error, but I've satisfied myself it's unlikely. $\endgroup$
    – Tom Wagstaff
    Commented Dec 20, 2021 at 20:12
  • 1
    $\begingroup$ @usεr11852 you were totally right - the labels were still flipped in the final plotting command, the one bit of code I didn't check - silly me, thanks for your help $\endgroup$
    – Tom Wagstaff
    Commented Dec 21, 2021 at 11:39
  • 1
    $\begingroup$ No worries Tom, it has happened to all of us in the past too. :) (I will write it as an answer so the question isn't unanswered) $\endgroup$
    – usεr11852
    Commented Dec 21, 2021 at 11:42

2 Answers 2

Reset to default
2
$\begingroup$

AUC-ROC can be below 0.5 but when it is substantially below 0.5 as in the case shown here (~0.28) there is a good chance that the labels are flipped/reversed at some point in our modelling pipeline. Such a low AUC-ROC score would suggest we are consistently bad, it can happen, but usually we are just bad!

Improve this answer
$\endgroup$
0
$\begingroup$

It turned out this was a coding error, here's the correct ROC curve: enter image description here with AUC of 0.72.

Moral of the story: if you see this, triple- and quadruple-check your code, because it is impossible.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.