How to reduce multicollinearity between predictors

I would like to run an prediction model and have a set of continuous independent variables. They are all important but highly correlated. How can I effectively reduce collinearity and still use these variables in my prediction model?

• It depends on the nature of the model: could you be more specific? – whuber Jul 5 '18 at 11:41
• Outcome: 3 categories (mild, mod, severe) - model is ordered logistic regression. Predictors: continuous measurement of lab values and a few demographics. The lab values are correlated to each other. – Lucy Jul 5 '18 at 18:53
• Could you share the evidence you have that collinearity will create any problems with prediction? Are you sure you really want to focus on prediction as opposed to explanation or coefficient estimation? – whuber Jul 5 '18 at 20:41
• If we have collinearity among predictors, one or both can have an inflated p-value, thus my selection method (keep all variables jointly significant at p<0.05) will be biased. – Lucy Jul 6 '18 at 2:32
• The problem lies with your selection method, which misuses the p-value, rather than with the collinearity. – whuber Jul 7 '18 at 1:48

You can reduce multicollinearity using PCA. There are lots of questions/answers about how to implement PCA. This method allows you to group similar covariates into independent "Principal Components" which can give insight into the relative relatedness of your covariates.

Also, check into Variance Inflation Factor (VIF) protocols. There are ways to use stepwise VIF reduction to rid yourself of highly collinear variables in the dataset. However, if you need to keep every covariate for some reason, a clustering approach like PCA or PLS would do the trick.

• It will do the trick provided the response doesn't depend on the components you throw away! – whuber Jul 7 '18 at 1:49
• That is a good point. You always run that risk when doing any sort of variable-reduction techniques. Generally, I would stick with keeping my correlated covariates, running Principal Components Regression with an oblique transformation in my PCA routine to flex the independence criterion. – ERT Jul 7 '18 at 1:55

If you are solely interested in prediction, then you probably don't need to reduce the collinearity among variables. You can test this with the perturbpackage in R.

Collinearity will increase the variance of the parameter estimates, but the predictions won't be affected; this is sort of the nature of collinearity: When you have collinear variables, two (or more) different ways of combining them will give nearly identical results. That matters a lot for explanation, but not for prediction.

Also, as @whuber pointed out in a comment, selecting variables by

keep[ing] all variables jointly significant at p<0.05

is problematic, regardless of collinearity and regardless of whether you are interested in prediction, explanation or both.