Assessing model fit in logistic regression with multiple imputation

Question

I'm wondering if there is any established method for assessing model fit in logistic regression conducted with multiple imputed datasets. To the best of my knowledge, there are two primary approaches for assessing model fit in logistic regression, but I'm not sure if these are possible with multiple imputation despite a lot of searching:

1. Calculate a form of Pseudo R2. I haven't been able to find any established method for either Nagelkerke R2 or McFadden's R2.

2. Examine the predictive accuracy of the model using a confusion matrix. This doesn't seem to work because the imputed datasets can't be split into training and testing data (in this case, they are 5 imputed datasets that need to be pooled after analysis).

Is there simply no way to examine the model fit of a logistic regression performed using multiple imputation? I want to address missing data, but doing so appears to come at the cost of not knowing the goodness-of-fit for my model. I can calculate Nagelkerke R2 for a model run using a single imputed dataset to get an idea at least, but I don't think that will fly for publication.

Answer 1

Section 3.9 of Frank Harrell's Regression Modeling Strategies discusses this matter in some detail, with code.

A suggested approach, based on a paper by Chang and Meng (Statistica Sinica 32: 1489–1514, 2022), is to stack all the imputations, run a single model, and get the likelihood-ratio $\chi^2$ for that full model. Get a rough estimate of a correct $\chi^2$ value by dividing by the number of imputation sets, then "multiply by a discounting factor to take imputation into account" based on information from the individual model fits.

Harrell shows a logistic regression example with code. Summary tables in that Section show the (corrected) pseudo-$R^2$ values that you seek.

Answer 2

In addition to the answer by EdM, if you want to evaluate out-of-sample prediction error, it is not hard to combine multiple imputation with train-test splitting or cross-validation. Then, you should treat multiple imputation similarly to other model-building steps, in the sense that you first create train-test splits or cross-validation folds, then learn the imputation model on the training data/folds and impute the test cases/fold with the model learned on the training data. In R, the mice package allows for train-test splitting by specifying the ignore argument in the mice call (if you use cross-validation, you should manually loop over the folds). Accordingly, you can calculate your performance measures on each imputed data set (ignoring the cases whose outcome is unobserved), and assuming they are all proportions, use the pooling rules for proportions (see for example the R-package miceafter).