Weighting of multiple linear regressions in an ensemble

Question

If I have a continuous dependent variable and N continuous predictors, and I fit all possible regressions with zero up to N variables, how should I weight those regressions for prediction? One possibility is to give all the weight to the regression with the smallest AICc (corrected Akaike information criterion), and another is to use weights proportional to exp(-0.5*AICc), as described at https://www.brianomeara.info/tutorials/aic/ . What other approaches should be considered?

Answer 1

There are dozens of methods that have been proposed. We discuss and evaluate them in some length in

Dormann, et al. Model averaging in ecology: a review of Bayesian, information‐theoretic, and tactical approaches for predictive inference. Ecological Monographs 88, no. 4 (2018): 485-504.

Some specific comments to your question:

• If you give all the weight to the regression with the smallest AICc, this is AIC model selection. We discuss in the Dormann et al. that and why model averaging often outperforms model selection.
• About how to weight the average: AIC is an option, but it is relatively aggressive (meaning it puts a lot of weight on few models). Tactical approaches that split weights more evenly (e.g. via CV error ) or even weights often outperformed exp(-AIC) weights. The underlying reason is that exp(-AIC) basically acts like a Posterior model weight, so like a modified exp(-BIC) weight, and this makes only sense if the true model is in the set (M-closed). I know that B&A state that AIC weights can be used also if the true model is not in the set, but the point is that if all models in the set are biased, and you put too much weight on one model, you can't improve over this bias, while an average of the models can (and AIC will tend to put all weight on one model asymptotically).

Answer 2

Since you ask "What other approaches should be considered?" we'll have to describe various methods.

First, Regression by leaps and bounds is a technique for considering all possible regressions. In some sense it is a "brute force" model selection method with some optimization to allow a large number of related linear models to be fit quickly. Based on your comment:

I fit all possible regressions with zero up to N variables

Leaps and bounds may be exactly what you are looking for, or it may not.

Second, there are other model selection techniques are based on the idea of finding the single best model are forward selection, backwards selection, etc. However, I won't discuss these because they are strictly inferior to feature selection via the LASSO. Essentially, a LASSO is a linear model with an l1 regularization penalty. The l1 regularization penalty encourages individual parameters to be exactly zero if they do not contribute significantly to reducing loss; parameters which are exactly zero can be considered to be "removed" from the model. Least angle regression in particular is a very efficient algorithm for the LASSO, and is the most efficient way I know of for doing variable selection on a very large number of variables.

Third and finally, if you have a variety of different models and want to form an ensemble model by taking a weighted average, you have two choices. You can either come up with some a priori scheme for weighting them, such as equal weights or some function of their goodness-of-fit, or you fit a new separate model which picks optimal values for those weights. This second strategy is called stacking. The easiest and most common way to stack models is to treat the outputs (predictions) of all other models as the inputs of a linear regression model and fit it to the dependent variable.

$$ y = \beta_0 + \beta_1 h_1(X) + ... + \beta_N h_N(X)$$

This regression will set a coefficient $\beta_i$ close to zero if $h_i(X)$ is uncorrelated with $y$, and a large coefficient when $h_i(X)$ is highly correlated with $y$. In this way, it not only selects weights for you, but it chooses the optimal weights in a certain sense. l1/l2 regularization can be a used to make the stacking model more robust, and it's not uncommon to remove the intercept term $\beta_0$ so that we give a true average over the constituent models in the ensemble.

If you suspect that your constituent models are of very different quality I would definitely recommend stacking over any a priori weighting scheme. On the other hand, if you suspect your models are very close in quality, a simple arithmetic mean is known to work well, as demonstrated by random forest.