0
$\begingroup$

I have extracted predicted probabilities (logistic model) from a graph according to the nine classes of a certain variable (I don't own the model). I need to compare the predicted probabilities, that tend to be clustered (same value) according to the value of this variable, against the observed ones to perform a calibration. Since my population is approximately composed of 62 subjects, the Hosmer-Lemeshow test sometimes fails to create the standard 10 bins.

Is bin number reduction a good choice ? Do I have any alternative test in this case ?

If you have other tests to suggest, please provide me any reference to the relative R package.

Thanks in advance

$\endgroup$

1 Answer 1

0
$\begingroup$

Firstly, 9 class labels spread out over n=62 likely suggests a class imbalance with some classes having many more members than others. However, if you already have the observed and predicted classes, then run interrater agreement using a 9x9 table for the two "raters": observed and predicted. With probabilities however, you could get away from the 9-class format and rather recode the observed and the predicted probabilities into quartile values (1,2,3,4) based on probability cutpoints of 0.25, 0.5, and 0.75. This would give you two new variables: $qobs$ and $qpred$ which contain values of 1,2,3,4. Then run interrater agreement on the 4x4 table to determine whether the two "raters" agree or not.

I'm a little concerned that the counts in the diagonal of a 9x9 table will be too sparse (cell count <5). Thus, collapsing probabilities into a 4x4 table based on quartiles would yield more counts in the 4 diagonal cells -- where you want most the counts to accumulate (agreement between obs and pred).

$\endgroup$
2
  • $\begingroup$ I apologize if I'm missing something, but isn't: - in the first part of your answer, what you described, a Pearson chi-squared goodness of fit test? - in the second part, a H-L test with 4 bins ? $\endgroup$
    – vixxovs
    Jan 21 at 21:22
  • $\begingroup$ No, the 9x9 table analysis mentioned is an interrater agreement test for tables, based on the Kappa statistic. It's totally different from a chi-squared contingency table analysis, which is based on a chi-squared statistic. $\endgroup$
    – wjktrs
    Jan 21 at 23:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.