I am referring to the question and its answers: How to compare (probability) predictive ability of models developed from logistic regression? by @Clark Chong and answers/comments by @Frank Harrell. and to the question Degrees of freedom of $\chi^2$ in Hosmer-Lemeshow test and the comments.
I have read the paper D.W. Hosmer, T. Hosmer, S. Le Cessie, S. Lemeshow, "A comparison of goodness-of-fit tests for the Logistic regression model", Statistics in Medicine, Vol. 16, 965-980 (1997).
After reading I was confused because the question I referred to asks explicitly for "(probability) predictive ability", which is in my opinion not the same as what the goodness-of-fit-tests in the paper supra aim at:
As most of us know, the logistic regression assumes an S-shaped link between the explanatory variables and the probability of success, the functional form for the S-shape is
$P(y=1|_{x_i})=\frac{1}{1+e^{-(\beta_0+\sum_i \beta_i x_i)}}$
Without pretending that there are no shortcomings with the Hosmer–Lemeshow test, I think that we have to distinguish between tests for the (a) '(probability) predictive ability' and (b) 'goodness-of-fit'.
The former's goal is to test whether the probabilities are well predicted, while the goodness-of-fit tests test whether the S-shaped function above is the 'right' function. More formally:
- tests for 'probability predictive ability tests' have a $H_0$ stating that the success probabilities are well predicted by the model;
- while for goodness-of-fit tests $H_0$ is (see Hosmer et. al. ) that the S-shaped functional form supra is the correct one. Hosmer et al. perform simulations where they find the power to detect two types of deviations from the null namely that the link function is wrong or that the exponent in the denominator is not linear.
Obviously, if the above function has the 'right' functional form (so if the tests concludes that we can accept $H_0$ for the goodness-of-fit test), then the predicted probabilities will be fine, ...
First remark
...however, accepting the $H_0$ is a weak conclusion as explained in What follows if we fail to reject the null hypothesis?.
First question
The most important question/remark that I have is that if the goodness-of-fit $H_0$ is rejected, then the conclusion of the test is that the functional form was not the 'right' one, however, does this imply that the probabilities are not well predicted ?
Second question
Furthermore, I want to point to the conclusions of Hosmer et. al; (I cite from the abstract):
''An examination of the performance of the tests when the correct model has a quadratic term but a model containing only the linear term has been fit shows that the Pearson chi-square, the unweighted sum-of-squares,the Hosmer-Lemeshow decile of risk, the smoothed residual sum-of-squares and Stukel’s score test,have power exceeding 50 per cent to detect moderate departures from linearity when the sample size is 100 and have power over 90 per cent for these same alternatives for samples of size 500. All tests had no power when the correct model had an interaction between a dichotomous and continuous covariate but only the continuous covariate model was fit. Power to detect an incorrectly specifed link was poor for samples of size 100. For samples of size 500 Stukel's score test had the best power but it only exceeded 50 per cent to detect an asymmetric link function. The power of the unweighted sum-of-squares test to detect an incorrectly specifed link function was slightly less than Stukel’s score test ''
Can I conclude from this which test has more power or that Hosmer–Lemeshow has less power (to detect these specific anomalies) ?
Second remark
The paper by Hosmer et. al. that I referred to supra, compute (simulate) the power to detect specific anomalies (the power can only be computed if an $H_1$ is specified). This does at my opinion not imply that these results can be generalised to ''all possible alternatives $H_1$'' ?