I am using logistic regression (PROC LOGISTIC) and for both of my two models, the Hosmer and Lemeshow Test is significant. I also computed AUC :
AUC(model 1) = 0.583 and AUC(model 2) = 0.604.
How can I choose one of them ?
9
-
1$\begingroup$ You reject the model if the goodness of fit is not good enough. That means your models are not good fit. $\endgroup$– SmallChessCommented Mar 31, 2017 at 13:25
-
$\begingroup$ Hosmer Lemeshow goodness of fit test has a glitch in it, for larger set of data >1000 observations ? (I have 1650000 observations) $\endgroup$– user44677Commented Mar 31, 2017 at 13:28
-
$\begingroup$ in this case don't worry the test just look at AUC. $\endgroup$– SmallChessCommented Mar 31, 2017 at 13:31
-
3$\begingroup$ The Hosmer-Lemeshow test is notoriously underpowered. Therefore, if you reject the null, that often indicates gross lack-of-fit. I wouldn't use the model. This is consistent with the very bad AUC values (even for the "better" model). As a sidebar: always be cautious with this test, because it can depend heavily on the arbitrary choice of the number of groups chosen. $\endgroup$– GoF_LogisticCommented Mar 31, 2017 at 13:55
-
2$\begingroup$ Hard to say. Maybe look to see whether there are non-linear effects (polynomial terms or interactions perhaps) and see whether that improves the fit, because a failed Hosmer-Lemeshow test often relates to a misspecified model. The low AUC could mean model misspecification or just predictors that are not very strongly associated with the outcome. Do you have other data you can bring in? $\endgroup$– GoF_LogisticCommented Mar 31, 2017 at 15:12
|
Show 4 more comments
1 Answer
$\begingroup$
$\endgroup$
2
Model 2 has the higher area under the response curve. So it therefore appears to be slightly better.
answered Mar 31, 2017 at 13:26
-
1$\begingroup$ A significant Hosmer-Lemeshow test does not necessarily mean that a predictive model is not useful or suspect ? $\endgroup$ Commented Mar 31, 2017 at 13:31
-
1$\begingroup$ This is consistent with the response I gave in my answer. $\endgroup$ Commented Mar 31, 2017 at 13:32