# Hosmer-Lemeshow vs AIC for logistic regression

If the Hosmer-Lemeshow indicates a lack of fit but the AIC is the lowest among all the models....should you still use the model?

If I delete a variable, the Hosmer-Lemeshow statistic is not significant (which means there is no gross lack of fit). But the AIC increases.

Edit: I think in general, if the AIC's of different models are close (i.e. $<2$) to each other then they are basically the same. But the AIC's are much different. This seems to indicate that the one with the lowest AIC is the one I should use even though the Hosmer-Lemeshow test indicates otherwise.

Also maybe the H-L test only applies for large samples? It has low power for small sample sizes (my sample size is ~300). But if I am getting a significant result... This means that even with low power I am getting a rejection.

Would it make a difference if I used AICc versus AIC? How do you get AICc's in SAS? I know there could be problems with multiplicity. But a priori I hypothesize that the variables have an effect on the outcome.

Edit2: I think I should use the model with one fewer variable and the higher AIC with non-significant H-L. The reason is because two of the variables are correlated with each other. So getting rid of one makes sense.

• Take into account that all your models may be junk. – user88 Nov 22 '11 at 8:20
• @mbq: How does this help? – Thomas Nov 22 '11 at 14:15
• Well, that even in a group of not-significant models there is one with best AIC. Anyway, please do not use answers to extend your question. – user88 Nov 23 '11 at 7:05

The Hosmer-Lemeshow test is to some extent obsolete because it requires arbitrary binning of predicted probabilities and does not possess excellent power to detect lack of calibration. It also does not fully penalize for extreme overfitting of the model. Better methods are available such as Hosmer, D. W.; Hosmer, T.; le Cessie, S. & Lemeshow, S. A comparison of goodness-of-fit tests for the logistic regression model. Statistics in Medicine, 1997, 16, 965-980. Their new measure is implemented in the R rms package. More importantly, this kind of assessment just addresses overall model calibration (agreement between predicted and observed) and does not address lack of fit such as improperly transforming a predictor. For that matter, neither does AIC unless you use AIC to compare two models where one is more flexible than the other being tested. I think you are interested in predictive discrimination, for which a generalized $R^2$ measure, supplemented by $c$-index (ROC area) may be more appropriate.
• I'm not sure you have studied all of the above. Generally we choose a model that has competitive predictive discrimination, then validate that the index of discrimination is not good just because of overfitting, then validate the calibration of the model. The last step is best done using a high-resolution smooth nonparametric calibration curve. All of these things are implemented in the R rms package. And avoid comparing AIC of many models which is just another way of using $P$-values to select variables. If you are comparing only 2 pre-specified models you're OK. – Frank Harrell Nov 23 '11 at 5:21