# Stepwise regression modeling using multiply imputed data sets

After multiply imputing data, it is natural to estimate regression models on the data. When multiple predictors are available, sometimes stepwise regression is used for model building (forward inclusion or backward elimination of covariates). My question is how to use a similar procedure on multiply imputed data sets (e.g. using the function mice in R). At first glance, this appears to be difficult. An AIC criterion as in the stepcould be used for exaple on each multiply imputed data set. But it seems hard to pool data sets after stepwise regression in the end. Maybe there is some procedure available to do this.

• I'd think your predict model should be selected based on the observed data and use the model to predict the missing values during multiple imputation as well as the pooling analysis. – David Z Aug 4 '14 at 19:05

Avoid stepwise regression due to the many issues indicated in many other threads.

If you search Google.Scholar for methods that are more appropriate for variable selection (e.g. cross-validation approaches) + "multiple imputation" you will find many suggestions. For example:

• Heymans, M. W., van Buuren, S., Knol, D. L., van Mechelen, W., & de Vet, H. C. (2007). Variable selection under multiple imputation using the bootstrap in a prognostic study. BMC Medical Research Methodology, 7(1), 33.
• Chen, Q., & Wang, S. (2013). Variable selection for multiply‐imputed data with application to dioxin exposure study. Statistics in medicine, 32(21), 3646-3659.
• Sabbe, N., Thas, O., & Ottoy, J. P. (2013). EMLasso: logistic lasso with missing data. Statistics in medicine, 32(18), 3143-3157.

One way to pool the stepwise analyses of individual imputations is to identify the predictors that most frequently appear in them.

For example, names(coef()) applied to output from stepAIC() (MASS package) applied to a linear regression gives the predictor names retained in the final stepwise regression. Add up, over all the individual imputations analyzed this way, how often each predictor appears. That quickly shows which predictors are the best candidates for your further model building.

I'm assuming that you have a good reason for reducing the number of variables in your model. Stepwise selection loses information; for application to prediction, a full model even including "non-significant" predictors may be preferable.

• Good answer, thanks! Actually I am interested in finding a number of meaningful predictors for a concept ('service quality') from a larger pool of possible predictors. I would like to avoid to overlook some significant predictors due to suppression effects or colinearity, which is why I decided to stepwise include or exclude covariates from the model. Please correct me if this procedure is not optimal. Also I am currently considering to summarize some correlated covariates using factor analysis first. – tomka Aug 6 '14 at 12:26
• It's not clear that stepwise selection will help avoid co-linearity; might make it worse by removing truly useful predictors that just happen to be less related to "service quality" in your particular data set. If you are building a model to predict future "service quality" from future data on covariates, err on the side of keeping too many covariates, even those that aren't "significant." If you're instead trying to infer an explanation of what produces "service quality," you will in any event have to deal with explaining the roles of the co-linear predictors. – EdM Aug 6 '14 at 17:33

I wish I had seen this question earlier. I've worked on an R package called glmmplus which addresses just this problem - variable selection when there is extensive missing data in the predictors, for both linear and linear mixed effects models. You can get it from github and the best I have for documentation are these slides I used at JSM 2014. I didn't really receive the enthusiasm I was hoping for, but I keep finding it useful myself when dealing with messy survey data.

The main idea is to base model-selection on Wald statistics instead of likelihood-based ones like AIC. Once you make that sacrifice, you can use Rubin's rules and have a principled approach to dealing with missing values. What's not so principled is the stepwise selection. These days you're not cool if you're doing backward elimination, but I haven't found a way to do a LASSO with lots of missing values in the predictors (though it's been a while since I looked). Please let me know if you try out the package.