Multiple Imputations, Predictive Modeling

Question

I am trying to build different predictive models using electronic health records. As they have missing values (between 0.5-18% missing values in each feature) I executed multiple imputation using MICE (the R package mice), taking into account van Buurens et al. recommendations and instructions. As the final step in MI is to pool the results (combining inferences from imputed data sets) and as I want to use different learning algorithms in a cross-fold validation set up, following question arises:

Can I use each imputation (having for example 5) seperately in order to build a model, cross-validate it and afterwards (having 5 different modeling results) average the obtained measures (accuracy, sensitiviy, etc.) and calculate the standard deviation in order to obtain one valid, representative result?

I would be very thankful if someone could help me out here!

Kind regards

Answer 1

A pooling step based on Rubin's rules will be unsuited to RF, SVM, etc, because these methods typically don't provide standard errors for parameters of interest. Rubin's rules therefore can't be applied apart from the trivial aspect of simply averaging the parameters (accuracy, sensitivity, etc) across all imputed datasets.

I would recommend the approach in this paper as an alternative https://www.jstor.org/stable/2291746. The suggested procedure (roughly) is:

1. bootstrap one dataset
2. impute missing values in this dataset
3. use the imputed dataset to build your model and obtain your parameter estimates. In your case accuracy, sensitivity, etc
4. repeat steps 1-3 B times. B should be on the order usual for bootstrapping, not for imputing. So B=1000 or so - not 5! - depending on size of data, uncertainty
5. pool your parameter estimates using normal bootstrap rules, rather than Rubin's rules.

You then have a reasonable estimate not only of the parameters of interest, but also their uncertainty. This approach really is only feasible if you have fairly small datasets or large computing power because imputing 1000 datasets takes time.

I'm not entirely confident of the statistical validity of this approach, and I know little about your data. But it should give you an idea of the centrality and uncertainty of parameters in the presence of missing data.