Applicability of the Hosmer-Lemeshow test

Question

Description

To test the goodness of fit of modelled event probabilities $\bar{P}=\{P_{1},...,P_{g}\}\in(0,1)^{G}$ for binomially distributed $G\in\mathbb{N}^{+}$ grades, I’ve applied the HL (Hosmer-Lemeshow) test as follows:

Note that for $G$ grades, $\bar{P}$ is initially based on a multivariate logistic regression and is monotonically increasing.

1. Let $P_{g}$ be the modelled event probability, $N_{g}\in\mathbb{N}^{+}$ the number of observations and $O_{g}\in\mathbb{N}^{+}$ the number of outcomes (naturally $O_{g}\leq N_{g}$) for grade $g\in\{1,…,G\}$

2. For each grade $g$ sample $M\in\mathbb{N}^{+}$ times $\hat{O}_{g}\sim B(N_{g},P_{g})$ and check (either visually or based on e.g. total variation distance) whether the HL test statistic $$H = \sum^{G}_{g=1}\Big(\frac{(O_{g}-N_{g}P_{g})^{2}}{N_{g}P_{g}}+\frac{(N_{g}-O_{g}-(N_{g}-N_{g}P_{g}))^{2}}{N_{g}-N_{g}P_{g}}\Big)$$ follows a $\chi^{2}$ distribution with $G$, $G-1$ or $G-2$ degrees of freedom for large enough $M$

3. Compute the p-value of $H$ based on $\bar{O}=\{O_{1},...,O_{g}\}$, $\bar{N}=\{N_{1},...,N_{g}\}$ and $\bar{P}$ using a $\chi^{2}$ distribution with $G$, $G-1$ or $G-2$ degrees of freedom depending on step 2

Question

Sampling in step 2 demonstrates that $H$ follows a $\chi^{2}$ distribution with $G$ degrees of freedom when $M\rightarrow\infty$ – which I expect as $H$ follows the sum of $G$ squared standard normal random variables – however, literature states that the expected degrees of freedom is either $G-2$ or $G-1$ in some special cases (Wikipedia and Surjanovic & Loughin, Oct/23 – most recent article I found containing all necessary references). I believe this is because the above test is different from the HL test; however, after setting $\bar{P}$ such that it follows the curve of a logistic regression according to $$P_{g}=\frac{1}{1+e^{-(g-\mu)/s}}$$ with $G=10$ and, location- and scale parameters $\mu=5.5$ and $s=1$, $G$ degrees of freedom still applies. How does the test above deviate from the HL test (as described on the Wikipedia page)?

Answer 1

For any binary sample which can be reasonably grouped, a general goodness of fit test can be defined using the same definitions of "observed" and "expected" event frequencies, their deviance, and the expected sum of deviance when the O and E events are expected to be equal.

The Hosmer-Lemeshow is just such a GoF test, but it takes no account of "grades". The groupings are created based on the predicted probabilities. The suggested number of groups by Hosmer and Lemeshow is 9, i.e. "deciles" of the fitted probabilities. I would say the HL-test can be reasonably generalized to consider other quantiles. But creating groups according to fitted probabilities is, in my opinion, a defining characteristic.

Other subdivisions of the sample to be verified is simply a goodness of fit test.