Stepwise regression modeling using multiply imputed data sets

Question

After multiply imputing data, it is natural to estimate regression models on the data. When multiple predictors are available, sometimes stepwise regression is used for model building (forward inclusion or backward elimination of covariates). My question is how to use a similar procedure on multiply imputed data sets (e.g. using the function mice in R). At first glance, this appears to be difficult. An AIC criterion as in the stepcould be used for exaple on each multiply imputed data set. But it seems hard to pool data sets after stepwise regression in the end. Maybe there is some procedure available to do this.

Answer 1

Avoid stepwise regression due to the many issues indicated in many other threads.

If you search Google.Scholar for methods that are more appropriate for variable selection (e.g. cross-validation approaches) + "multiple imputation" you will find many suggestions. For example:

• Heymans, M. W., van Buuren, S., Knol, D. L., van Mechelen, W., & de Vet, H. C. (2007). Variable selection under multiple imputation using the bootstrap in a prognostic study. BMC Medical Research Methodology, 7(1), 33.
• Chen, Q., & Wang, S. (2013). Variable selection for multiply‐imputed data with application to dioxin exposure study. Statistics in medicine, 32(21), 3646-3659.
• Sabbe, N., Thas, O., & Ottoy, J. P. (2013). EMLasso: logistic lasso with missing data. Statistics in medicine, 32(18), 3143-3157.

Answer 2

I wish I had seen this question earlier. I've worked on an R package called glmmplus which addresses just this problem - variable selection when there is extensive missing data in the predictors, for both linear and linear mixed effects models. You can get it from github and the best I have for documentation are these slides I used at JSM 2014. I didn't really receive the enthusiasm I was hoping for, but I keep finding it useful myself when dealing with messy survey data.

The main idea is to base model-selection on Wald statistics instead of likelihood-based ones like AIC. Once you make that sacrifice, you can use Rubin's rules and have a principled approach to dealing with missing values. What's not so principled is the stepwise selection. These days you're not cool if you're doing backward elimination, but I haven't found a way to do a LASSO with lots of missing values in the predictors (though it's been a while since I looked). Please let me know if you try out the package.

Answer 3

One way to pool the stepwise analyses of individual imputations is to identify the predictors that most frequently appear in them.

For example, names(coef()) applied to output from stepAIC() (MASS package) applied to a linear regression gives the predictor names retained in the final stepwise regression. Add up, over all the individual imputations analyzed this way, how often each predictor appears. That quickly shows which predictors are the best candidates for your further model building.

I'm assuming that you have a good reason for reducing the number of variables in your model. Stepwise selection loses information; for application to prediction, a full model even including "non-significant" predictors may be preferable.