1
$\begingroup$

I am trying to build different predictive models using electronic health records. As they have missing values (between 0.5-18% missing values in each feature) I executed multiple imputation using MICE (the R package mice), taking into account van Buurens et al. recommendations and instructions. As the final step in MI is to pool the results (combining inferences from imputed data sets) and as I want to use different learning algorithms in a cross-fold validation set up, following question arises:

Can I use each imputation (having for example 5) seperately in order to build a model, cross-validate it and afterwards (having 5 different modeling results) average the obtained measures (accuracy, sensitiviy, etc.) and calculate the standard deviation in order to obtain one valid, representative result?

I would be very thankful if someone could help me out here!

Kind regards

$\endgroup$
  • $\begingroup$ Welcome to stats.SE! Please take a moment to view our tour. The sort answer is, yes you can (I am also not entirely sure if the idea is statistically sound). Will it convey the information in a meaningful and impactful way? It depends on the customer of your data. $\endgroup$ – Tavrock Mar 2 '17 at 21:24
  • $\begingroup$ Thank you for the reply! The overall goal is research-based and just to point out that building a model with xx%±xx accuracy is feasable. I am just not sure if the result will be statistically faulty because of not using the provided pooling methods, which seem not be able to work with RF,SVM,NB,DT, etc. in a CV-setup. $\endgroup$ – Andreas-Philipp H. Mar 2 '17 at 21:34
  • $\begingroup$ Isn't it the way how Multiple Imputation works ? As far as I know, there are 3 steps to an MI procedure: 1) Impute, 2) Analyze, 3) Pool. First, you create m number of imputed datasets, then you apply analysis on these datasets separately, and finally you pool the results of the analysis into a final output. $\endgroup$ – Ahmedov Mar 9 '17 at 14:51
1
$\begingroup$

A pooling step based on Rubin's rules will be unsuited to RF, SVM, etc, because these methods typically don't provide standard errors for parameters of interest. Rubin's rules therefore can't be applied apart from the trivial aspect of simply averaging the parameters (accuracy, sensitivity, etc) across all imputed datasets.

I would recommend the approach in this paper as an alternative https://www.jstor.org/stable/2291746. The suggested procedure (roughly) is:

  1. bootstrap one dataset
  2. impute missing values in this dataset
  3. use the imputed dataset to build your model and obtain your parameter estimates. In your case accuracy, sensitivity, etc
  4. repeat steps 1-3 B times. B should be on the order usual for bootstrapping, not for imputing. So B=1000 or so - not 5! - depending on size of data, uncertainty
  5. pool your parameter estimates using normal bootstrap rules, rather than Rubin's rules.

You then have a reasonable estimate not only of the parameters of interest, but also their uncertainty. This approach really is only feasible if you have fairly small datasets or large computing power because imputing 1000 datasets takes time.

I'm not entirely confident of the statistical validity of this approach, and I know little about your data. But it should give you an idea of the centrality and uncertainty of parameters in the presence of missing data.

$\endgroup$
  • $\begingroup$ This is an excellent question. In the frequentist world we don't have principled unique solutions for such problems, and we need more simulation studies to help us tune the algorithms and optimally order the steps. Contrast that with Bayesian modeling where full models including handling missings as unknown parameters provides a linear path forwards. $\endgroup$ – Frank Harrell Feb 4 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.