2
$\begingroup$

Can you just eyeball the Hosmer-Lemeshow test (e.g. just look at observed and expected values in each group and see if they agree more or less)? Even if the Hosmer-Lemeshow statistic is significant...can I still say the model fits the data well for my purposes just by observing the observed and expected values in each group?

$\endgroup$
1
$\begingroup$

Yes. Plot the calibration curve(s) and draw your conclusions from the agreements with the 45° line corresponding to perfect calibration, keeping in mind the distribution of observed and predicted risks. No big deal if your model doesn't calibrate well in high risk ranges if you don't have a lot of cases out there in practice, especially if the event of interest is rare and/or low risk to begin with. One handy tip: if your curves follow a J/U-shape (like a hockey stick — you'll know it when you see it) just cube the probability estimates that your model spits out. This often will straighten out the curves and improve calibration substantially. You might have to experiment with the specific transformation but I've found cubing usually works well.

In my experience, the H-L test is rubbish, and I wouldn't rely on it as an assessment of model calibration, at all. I've managed to get H-L p-values of $10^{-300}$ to near 1 using the same classifier and dataset just by varying the sample size, even though the calibration was nearly perfect visually in all cases.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.