0
$\begingroup$

Description

To test the goodness of fit of modelled event probabilities $\bar{P}=\{P_{1},...,P_{g}\}\in(0,1)^{G}$ for binomially distributed $G\in\mathbb{N}^{+}$ grades, I’ve applied the HL (Hosmer-Lemeshow) test as follows:

Note that for $G$ grades, $\bar{P}$ is initially based on a multivariate logistic regression and is monotonically increasing.

  1. Let $P_{g}$ be the modelled event probability, $N_{g}\in\mathbb{N}^{+}$ the number of observations and $O_{g}\in\mathbb{N}^{+}$ the number of outcomes (naturally $O_{g}\leq N_{g}$) for grade $g\in\{1,…,G\}$

  2. For each grade $g$ sample $M\in\mathbb{N}^{+}$ times $\hat{O}_{g}\sim B(N_{g},P_{g})$ and check (either visually or based on e.g. total variation distance) whether the HL test statistic $$H = \sum^{G}_{g=1}\Big(\frac{(O_{g}-N_{g}P_{g})^{2}}{N_{g}P_{g}}+\frac{(N_{g}-O_{g}-(N_{g}-N_{g}P_{g}))^{2}}{N_{g}-N_{g}P_{g}}\Big)$$ follows a $\chi^{2}$ distribution with $G$, $G-1$ or $G-2$ degrees of freedom for large enough $M$

  3. Compute the p-value of $H$ based on $\bar{O}=\{O_{1},...,O_{g}\}$, $\bar{N}=\{N_{1},...,N_{g}\}$ and $\bar{P}$ using a $\chi^{2}$ distribution with $G$, $G-1$ or $G-2$ degrees of freedom depending on step 2

Question

Sampling in step 2 demonstrates that $H$ follows a $\chi^{2}$ distribution with $G$ degrees of freedom when $M\rightarrow\infty$ – which I expect as $H$ follows the sum of $G$ squared standard normal random variables – however, literature states that the expected degrees of freedom is either $G-2$ or $G-1$ in some special cases (Wikipedia and Surjanovic & Loughin, Oct/23 – most recent article I found containing all necessary references). I believe this is because the above test is different from the HL test; however, after setting $\bar{P}$ such that it follows the curve of a logistic regression according to $$P_{g}=\frac{1}{1+e^{-(g-\mu)/s}}$$ with $G=10$ and, location- and scale parameters $\mu=5.5$ and $s=1$, $G$ degrees of freedom still applies. How does the test above deviate from the HL test (as described on the Wikipedia page)?

$\endgroup$
1

1 Answer 1

3
$\begingroup$

For any binary sample which can be reasonably grouped, a general goodness of fit test can be defined using the same definitions of "observed" and "expected" event frequencies, their deviance, and the expected sum of deviance when the O and E events are expected to be equal.

The Hosmer-Lemeshow is just such a GoF test, but it takes no account of "grades". The groupings are created based on the predicted probabilities. The suggested number of groups by Hosmer and Lemeshow is 9, i.e. "deciles" of the fitted probabilities. I would say the HL-test can be reasonably generalized to consider other quantiles. But creating groups according to fitted probabilities is, in my opinion, a defining characteristic.

Other subdivisions of the sample to be verified is simply a goodness of fit test.

$\endgroup$
2
  • $\begingroup$ Thanks! In hindsight, my question was unclear, but it drove me to rebuild a HL test and concluded that if every sampled data set is fitted by a logistic regression, the HL test statistic follows $G-2$ degrees of freedom, however, if these probabilities are fixed (even when based on a logistic regression-like curve ) and multiple data sets are sampled based on these probabilities, the HL test statistic follows $G$ degrees of freedom even when using the characteristic deciles and/or groups according to the ("fitted") - here I concluded my question was unclear based on your answer - probabilities $\endgroup$
    – CorrieElba
    Commented Feb 23 at 10:16
  • 1
    $\begingroup$ @CorrieElba if G is any arbtrary grouping, then you just have adapted a goodness of fit test. It is a valid thing to do, but it is not the HL test per se. $\endgroup$
    – AdamO
    Commented Feb 29 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.