1
$\begingroup$

I am building an imputation model and I am wondering, If I have to be careful because of multicollinearity of my predictor variables. I found the following in the literature:

Van Buuren (2012, p. 127) writes

For datasets containing hundreds or thousands of variables, using all predictors may not be feasible (because of multicollinearity and computational problems) to include all these variables.

Allison (2012) writes in a general statement about collinear predictors (not in the context of imputation)

But so long as the collinear variables are only used as control variables, and they are not collinear with your variables of interest, there’s no problem. The coefficients of the variables of interest are not affected, and the performance of the control variables as controls is not impaired.

I am confused, because for me it seems like the two statements are saying the opposite. Therefore, I have the question:

Is multicollinearity problematic for imputation models?

$\endgroup$
3
$\begingroup$

The way I read van Buuren's background on imputation, and this part specifically, is that for multiple imputation models the goal is to use as much information as you have in order to obtain the estimates required to complete any missing data. This is grounded on the idea that multiple imputation is founded on the missing at random assumption. Consequently, overfitting the imputation models - i.e. making it more dependent on your data - is less of/not a problem: any available associations, small or large are used. If these associations are estimated with large error, it will be reflected in the variance across imputation sets and taken into account when properly pooling analysis results. On the other hand, if the estimation error of these 'overfitted' errors is relatively small, it may improve the estimates by reducing variance across imputation sets.

I feel this reasoning applies to multicollinearity as well. I assume van Buuren supports this, but I think the van Buuren statement you quote is more related to the practical implications of modelfitting under circumstances of high multicollinearity: your computer might start spewing pink smoke, while not being able to fit the model at all!

Even if so, I feel there are no obvious reasons to state multicollinearity is more of a problem than for example overfitting: it's just about using as much information as is available. The only difference is that in the case of multicollinearity you might want to complete this with it's just about using as much information as is available and your are able to fit.

So if the computer throws an error, you might have to 'dumb down' your model. However obtaining replacement values through a less multicollinearity sensitive model might also be an option.

Sidenote: if you have a set of perfectly multicollinear variables, they're probably identical, or completely dependent on one and other, and on or the other does not provide additional information for estimating replacement values. In that case removal of one of the variables is probably indicated.

$\endgroup$
  • 1
    $\begingroup$ The way I've understood multiple imputation is that we first assume some information about missingness is 'captured' in other variables (the MAR assumption). The technique for imputation is based on this idea by its use of the associations in your data (plusminus some randomness) to predict replacement values for missing values. (for me at least) It is really hard to conceptualize which other variables are related, but basically, all other (available) variables might be. To accommodate all of these possible variables, you'll almost always have to overfit. ctd... $\endgroup$ – IWS May 10 '17 at 12:43
  • 1
    $\begingroup$ ctd. If this leads to unstable estimates, this will not be incorporated into each imputation set separately, but instead each imputation set will have (very) different replacement values allocated to the missing spots. This way, the variance across imputation sets will probably increase. Pooling according to Rubin's rule takes this into account and will lower the certainty (i.e. increase the standard error of you estimates, penalize test-statistics) accordingly. ctd... $\endgroup$ – IWS May 10 '17 at 12:47
  • 1
    $\begingroup$ ctd. So as long as you have enough imputation sets to capture this variance across imputation datasets, overfitting itself would not result in very large bias. On the other hand, model misspecification, missing predictors or missing values which are (partly) MNAR (missing not at random), is not solved and missing data biases may exist (even) after imputation. $\endgroup$ – IWS May 10 '17 at 12:49
  • 1
    $\begingroup$ "increase the standard error of your estimates ... would not result in very large bias" - I am wondering, if standard errors and bias of the estimates could be reduced by an exclusion of overfitting predictors. Or in other words: are there scenarios where it hurts to include more variables? "It is really hard to conceptualize which other variables are related" - I definitely agree, however there are ways to exclude bad predictors from your model. More concretely, right now I am playing around with variable selection techniques like the lasso and random forests to build imputation models. $\endgroup$ – JSP May 10 '17 at 13:07
  • 1
    $\begingroup$ Right now I am comparing the performance of different selection methods within a simulation with real data. It is definitely the case that different selection methods result in better/worse imputation performances (bias of estimates etc.). The study is not focusing on the overfitting of these models though. Maybe I am going to do some checks about it. I will let you know, if I do that. Thank you very much for your help anyway! $\endgroup$ – JSP May 10 '17 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.