0
$\begingroup$

I am new to building the prediction model and would like to hear from you regarding the goodness of fit test (model calibration). Between Akaike information criteria (AIC), Hosmer-Lemeshow Test, Likelihood Ratio Test, and/or Pearson Residual Test, which goodness of fit test should I use if my sample size is small (N=60)?

$\endgroup$
1
$\begingroup$

I'm not familliar with the Pearson Residual Test so I will exclude it.

It has been recommended here that Hosmer-Lemeshow not be used assess calibration of logistic regression or other probabilistic models due to low statistical power and uninformative p value with respect to the type/extent of miscalibration. A better way to assess calibration would be to compute the bootstrapped calibration slope and intercept using the Efron-Gong Bootstrap.

The remaining tests (AIC -- though this is not really a test per se, Likelihood Ratio Test) are not my favourite either. They are meant for model comparison and I don't see how they can really assess improvement. For example, I can construct a candidate model and reference model which when put through the likelihood ratio test would yield a rejection of the null. However, rejection of the null tells you nothing about if the candidate model is better (because, for example, the effect of any new predictors could be relatively small making the new prediction similar).

$\endgroup$
2
  • $\begingroup$ Hi Demetri, thank you so much for the feedback. Can you pls elaborate on the "compute the bootstrapped calibration slope and intercept using the Efron-Gong Bootstrap" part? (Preferably an R code would be greatly appreciated). $\endgroup$
    – R Beginner
    Nov 24 '21 at 2:48
  • $\begingroup$ Check out the rms package. That package has lots of functions to do that sort of thing. $\endgroup$ Nov 24 '21 at 6:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.