14
$\begingroup$

If the Hosmer-Lemeshow indicates a lack of fit but the AIC is the lowest among all the models....should you still use the model?

If I delete a variable, the Hosmer-Lemeshow statistic is not significant (which means there is no gross lack of fit). But the AIC increases.

Edit: I think in general, if the AIC's of different models are close (i.e. $<2$) to each other then they are basically the same. But the AIC's are much different. This seems to indicate that the one with the lowest AIC is the one I should use even though the Hosmer-Lemeshow test indicates otherwise.

Also maybe the H-L test only applies for large samples? It has low power for small sample sizes (my sample size is ~300). But if I am getting a significant result... This means that even with low power I am getting a rejection.

Would it make a difference if I used AICc versus AIC? How do you get AICc's in SAS? I know there could be problems with multiplicity. But a priori I hypothesize that the variables have an effect on the outcome.

Any comments?

Edit2: I think I should use the model with one fewer variable and the higher AIC with non-significant H-L. The reason is because two of the variables are correlated with each other. So getting rid of one makes sense.

Improve this question
$\endgroup$
4
  • $\begingroup$ Take into account that all your models may be junk. $\endgroup$
    – user88
    Commented Nov 22, 2011 at 8:20
  • $\begingroup$ @mbq: How does this help? $\endgroup$
    – Thomas
    Commented Nov 22, 2011 at 14:15
  • 2
    $\begingroup$ Well, that even in a group of not-significant models there is one with best AIC. Anyway, please do not use answers to extend your question. $\endgroup$
    – user88
    Commented Nov 23, 2011 at 7:05
  • $\begingroup$ Hosmer-Lemeshow is considered obsolete: stats.stackexchange.com/questions/273966/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented May 14, 2020 at 11:31

1 Answer 1

Reset to default
19
$\begingroup$

The Hosmer-Lemeshow test is to some extent obsolete because it requires arbitrary binning of predicted probabilities and does not possess excellent power to detect lack of calibration. It also does not fully penalize for extreme overfitting of the model. Better methods are available such as Hosmer, D. W.; Hosmer, T.; le Cessie, S. & Lemeshow, S. A comparison of goodness-of-fit tests for the logistic regression model. Statistics in Medicine, 1997, 16, 965-980. Their new measure is implemented in the R rms package. More importantly, this kind of assessment just addresses overall model calibration (agreement between predicted and observed) and does not address lack of fit such as improperly transforming a predictor. For that matter, neither does AIC unless you use AIC to compare two models where one is more flexible than the other being tested. I think you are interested in predictive discrimination, for which a generalized $R^2$ measure, supplemented by $c$-index (ROC area) may be more appropriate.

Improve this answer
$\endgroup$
5
  • $\begingroup$ So would using the likelihood ratio test be better for assessing goodness of fit of the model with lowest AIC? Because this test shows that there is no lack of fit. $\endgroup$
    – Thomas
    Commented Nov 22, 2011 at 14:43
  • $\begingroup$ Looking at AICs of more than 2 models will result in some selection bias/overfitting. AIC does not explicitly assess goodness of fit except in the context I gave above. The best way to assess fit is to demonstrate good calibration using a continuous smooth nonparametric calibration plot, and showing little evidence for more complex components that might have made the model predict better. $\endgroup$
    – Frank Harrell
    Commented Nov 23, 2011 at 3:20
  • $\begingroup$ Assuming I don't have access to any of those tools. Model A which has a non-significant H-L test also has one less variable than Model B which has a significant H-L test. I am comparing only these two models. Model A has the lowest AIC and model B has a much higher AIC. $\endgroup$
    – Thomas
    Commented Nov 23, 2011 at 3:27
  • $\begingroup$ I meant Model B has the lowest AIC and Model A has a much higher AIC. $\endgroup$
    – Thomas
    Commented Nov 23, 2011 at 3:34
  • 3
    $\begingroup$ I'm not sure you have studied all of the above. Generally we choose a model that has competitive predictive discrimination, then validate that the index of discrimination is not good just because of overfitting, then validate the calibration of the model. The last step is best done using a high-resolution smooth nonparametric calibration curve. All of these things are implemented in the R rms package. And avoid comparing AIC of many models which is just another way of using $P$-values to select variables. If you are comparing only 2 pre-specified models you're OK. $\endgroup$
    – Frank Harrell
    Commented Nov 23, 2011 at 5:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.