Multiple imputation and model selection

Question

Multiple imputation is fairly straightforward when you have an a priori linear model that you want to estimate. However, things seem to be a bit trickier when you actually want to do some model selection (e.g. find the "best" set of predictor variables from a larger set of candidate variables - I am thinking specifically of LASSO and fractional polynomials using R).

One idea would be to fit the model in the original data with missing values, and then re-estimate this model in MI datasets and combine estimates as you normally would. However, this seems problematic since you are expecting bias (or else why do the MI in the first place?), which could lead to selecting a "wrong" model from the start.

Another idea would be to go through whatever model selection process you are using in each MI dataset - but how would you then combine results if they include different sets of variables?

One thought I had was to stack a set of MI datasets and analyze them as one large dataset that you would then use to fit a single, "best" model, and include a random effect to account for the fact you are using repeated measures for each observation.

Does this sound reasonable? Or perhaps incredibly naive? Any pointers on this issue (model selection with multiple imputation) would be greatly appreciated.

Answer 1

There are many things you could do to select variables from multiply imputed data, but not all yield appropriate estimates. See Wood et al (2008) Stat Med for a comparison of various possibilities.

I have found the following two-step procedure useful in practice.

1. Apply your preferred variable selection method independently to each of the $m$ imputed data sets. You will end up with $m$ different models. For each variable, count the number of times it appears in the model. Select those variables that appear in at least half of the $m$ models.
2. Use the p-value of the Wald statistic or of the likelihood ratio test as calculated from the $m$ multiply-imputed data sets as the criterion for further stepwise model selection.

The pre-selection step 1 is included to reduce the amount of computation. See https://stefvanbuuren.name/fimd/sec-stepwise.html (section 5.4.2) for a code example of the two-step method in R using mice(). In Stata, you can perform Step 2 (on all variables) with mim:stepwise.

Answer 2

It is straightforward: You can apply standard MI combining rules - but effects of variables which are not supported throughout imputed datasets will be less pronounced. For example, if a variable is not selected in a specific imputed dataset its estimate (incl. variance) is zero and this has to be reflected in the estimates used when using multiple imputation. You can consider bootstrapping to construct confidence intervals to incorporate model selection uncertainty, have a look at this recent publication which addresses all questions: http://www.sciencedirect.com/science/article/pii/S016794731300073X

I would avoid using pragmatic approaches such as selecting a variable if it is selected in m/2 datasets or sth similar, because inference is not clear and more complicated than it looks at first glance.

Answer 3

I was having the same problem.

My choice was the so-called "multiple imputation lasso". Basically it combines all imputed datasets together and adopts the concept of group lasso: every candidate variable would generate m dummy variables. Each dummy variable corresponds to a imputed dataset.

Then all the m dummy variables are grouped. you would either discard a candidate variable's m dummy variables in all imputed datasets or keep them in all imputed datasets.

So the lasso regression is actually fit on all imputed datasets jointly.

Check the paper:

Chen, Q. & Wang, S. (2013). "Variable selection for multiply-imputed data with application to dioxin exposure study," Statistics in Medicine, 32:3646-59.

And a relevant R program

Answer 4

I've been facing a similar problem -- I've got a dataset in which I knew from the start that I wanted to include all variables (I was interested in the coefficients more than the prediction), but I didn't know a priori what interactions should be specified.

My approach was to write out a set of candidate models, perform multiple imputations, estimate the multiple models, and simply save and average the AIC's from each model. The model specification with the lowest average-of-AIC's was selected.

I thought about adding a correction wherein I penalize between-imputation variance in AIC. On reflection however, this seemed pointless.

The approach seemed straightforward enough to me, but I invented it myself, and I'm no celebrated statistician. Before using it, you may wish to wait until people either correct me (which would be welcome!) or upvote this answer.