2
$\begingroup$

everyone!

I am a newbie on machine learning, and I am now interested on classification modeling.

I used logistic regression, linear discriminant analysis (LDA), and naive Bayes on my notebook DataCamp Certification - Travel Insurance as a tool to exploratory data analysis. A thing that caught my attention is the difference between AUC of the three models, more specific between logistic regression and the other tow techniques. I use the Yardstick to calculate this measure.

Well, I am trying to apply this same strategy on Titanic - Machine Learning from Disaster. When I model on my notebook the data with logistic regression, I was surprised by a AUC very low (0.1464) despite my score of 0.7584 on the competition.

It is important to say that some terms on the first notebook show a low p.value. Differently, the second notebook, all terms show p.value < 0.05.

So, I would like to understand this situation because it is worries me a lot!

Thank you all for your time!

Improve this question
$\endgroup$
13
  • $\begingroup$ 0. Welcome to CV.SE. 1. Please note that values of AUC-ROC below 0.50 suggest that the label during evaluation are likely inverted. $\endgroup$
    – usεr11852
    Apr 21, 2022 at 0:37
  • 2
    $\begingroup$ An AUC less than 0.5 means the model could do better by just switching the predicted labels. Are you sure there is no bug in your code which may code positive responses as 0 and negative responses as 1? If your outcome is a factor, this can happen easily. $\endgroup$
    – Demetri Pananos
    Apr 21, 2022 at 2:59
  • $\begingroup$ @Gregory Oliveira Your links are broken. $\endgroup$
    – frank
    Apr 21, 2022 at 7:12
  • $\begingroup$ @frank I changed them. I try to send you directly to the cells that I want. $\endgroup$
    – GregOliveira
    Apr 21, 2022 at 10:13
  • $\begingroup$ I'll inspect again, but I had tried to look at this issue of changing labels before. If successful, I'll let you know here. Thank you! $\endgroup$
    – GregOliveira
    Apr 21, 2022 at 10:16

1 Answer 1

Reset to default
0
$\begingroup$

It is unintended behaviour by yardstick::roc_auc; if we used another library to compute AUC-ROC, e.g. MLmetrics::AUC(y_true=data_select_pred$Survived, y_pred=data_select_pred$ProbSurvived), we would get an AUC-ROC value of ~0.8536 that matches 1 minus the value you currently get. In general, AUC-ROC values significantly below 0.5 are very often related to coding error/label inversion. The CV.SE thread on: Can AUC-ROC be between 0-0.5? explores this further.

Please consider opening an issue about this in yardstick git repo. There is a similar issue mentioned in September 2021 which I suspect affects you too.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I will document the situation and report to them! Thanks for your support. :-) $\endgroup$
    – GregOliveira
    Apr 21, 2022 at 14:46
  • $\begingroup$ Cool, I am glad I could help! $\endgroup$
    – usεr11852
    Apr 21, 2022 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.