Goodness of fit test for predictive model with small sample size

Question

I am new to building the prediction model and would like to hear from you regarding the goodness of fit test (model calibration). Between Akaike information criteria (AIC), Hosmer-Lemeshow Test, Likelihood Ratio Test, and/or Pearson Residual Test, which goodness of fit test should I use if my sample size is small (N=60)?

Answer 1

I'm not familliar with the Pearson Residual Test so I will exclude it.

It has been recommended here that Hosmer-Lemeshow not be used assess calibration of logistic regression or other probabilistic models due to low statistical power and uninformative p value with respect to the type/extent of miscalibration. A better way to assess calibration would be to compute the bootstrapped calibration slope and intercept using the Efron-Gong Bootstrap.

The remaining tests (AIC -- though this is not really a test per se, Likelihood Ratio Test) are not my favourite either. They are meant for model comparison and I don't see how they can really assess improvement. For example, I can construct a candidate model and reference model which when put through the likelihood ratio test would yield a rejection of the null. However, rejection of the null tells you nothing about if the candidate model is better (because, for example, the effect of any new predictors could be relatively small making the new prediction similar).