Is it possible to use an ensemble of regression predictions to avoid issues of multicolinearity?

Question

I am using a regression approach to make predictions using a variety of variables. However, some of my variables are pretty collinear (with a Pearson's r > 0.75), so I can't include them all in the regression. Furthermore, I am finding that the model predictions are pretty sensitive to which variables I include as predictors. One approach I am considering is to use an ensemble of regression models, where each model is based on a different combination of predictor variables so that no two are collinear. My thinking is that this might even help me estimate upper and lower limits of my predictions. However, I can't find any examples of studies that have done this (from any field), so perhaps there is a reason why one would not want to take such an approach?

Some background on why I am trying to do: I have a dataset of (continuous) child nutrition outcomes from multiple countries, and the outcomes were observed across a wide range of geographic conditions as well as meteorological conditions (some children were observed during drought years, others during normal years). I am trying to make a model of child nutrition outcomes under drought based on a variety of geographic factors, such as rainfall levels, GDP, national imports, and irrigation, and then use the model to predict how drought would affect child nutrition based on all of those factors. I am using regression because I am interested in having an interaction term between each geographic variable and a categorical variable for whether or not drought is occurring, and regression seems to be the best way to model this specific interaction. Furthermore, I am controlling for a variety of other variables that aren't geographic, such as the child's birth order, which also lends itself to a regression approach.

What I'd really like to know is: Has this sort of approach been done before? If so, where can I find literature on how to do it right or at least examples of studies that took this approach? If not, why not? Are there certain reasons why one would not want to use an ensemble of regression predictions?

Answer 1

I have not seen that approach used before personally. It would be a large amount of work to fit all the possible models, and if you were to exclude some combinations for reasons other than collinearity, then you would need to justify the choice.

Once you have fitted them then you would need to develop code which is capable of feeding in the values you are interested in to the respective models and from there somehow combining the prediction.... and that raises the question of how you combine the predictions.

Do you weight all models equally? Or do models with a better R-squared deserve a greater say? What about models which are more sensible from a mechanistic view point (i.e. are grounded in known theory)? Mean or median? How do you combine your errors?

It would be hard to make a completely defensible choice simply because their is a lot of subjectivity in this.

Instead, what I would recommend given your objective, would be to use some form of regularised regression (i.e. ridge, lasso or elastic net regression), as these techniques are specifically developed to deal with multi-collinearity by constraining the coefficients associated with each variable. You do need to estimate the cost parameter, but this can be done using cross-validation.

If you are using R then the caret package can do this all for you in a couple simple lines which is very helpful and will give you an objective model choice based on prediction error which sounds like it would be in line with the goal of your study.

If anything is unclear, feel free to ask for clarification!

Answer 2

I see no reason that this procedure would outperform a standard multiple linear model that includes all the predictors in the model (notwithstanding their multi-collinearity). Inclusion of collinear predictors makes some individual coefficient estimators highly variable, but it does not detract from the model in an overall predictive sense. At the moment it is unclear in your question why you have decided you cannot include all predictors; this type of belief is often based on misunderstanding what regression is doing, or a false belief that results can be interpreted causally if collinearity is absent.

I have not heard of the approach you are suggesting in the literature, but my hunch is that it would not be a good procedure, and would be inferior to standard linear regression. (I'm happy to be proved wrong on that, but it would be surprising to me.) The proposed procedure involves randomly producing sets of predictors based on exclusion of collinear pairs. This excludes available predictor variables, which would be likely to reduce model performance. If you decide to pursue this avenue of analysis, you would need to specify your procedure for producing these sets, and then derive the properties of the resulting model.

Answer 3

The approach you are describing sounds like complete subset regression (CSR) by Elliott, Antoni, Gargano and Timmermann, see here. If you are behing a paywall, here is a working paper version.

In CSR you first estimate all models with k-parameters. If you have a total of K parameters, this means you estimate n = K! / ((K - k)!k!) models where k <= K.

Then, you get the prediction for each of the n models and average them, for instance using the simple mean.