Suppose I'm trying to predict $Y$ (a real number) and I have $n$ experts with guesses $Y_1,...Y_n$. Each prediction is a reasonable guess as to the value of Y in itself (hence the name "expert"), but we should be able to come up with a better prediction for $Y$ by combining the experts' guesses. We could take our guess $Y_o$ to be


but this isn't optimal if some experts are better than others.

I have a few questions related to this topic:

  1. Is this equal weights method used much? Are there any other simple methods? Can you refer me to (preferably relatively basic) information on the topic?
  2. I've seen model averaging discussed in this as a possible alternative to other methods of model selection for linear regression which may fail, for example, if there is multicollinearity. That's led me to wonder whether the following solution might be justifiable:

We want (perhaps out of naïveté) to regress $Y$ against predictors $X_1$, $X_2$ and $W$. Suppose, in fact, that $X_1$ and $X_2$ themselves are predictors of $Y$. Using multilinear regression for $Y$ with all three variables is problematic because, being predictors of $Y$, $X_1$ and $X_2$ exhibit multicollinearity, but what about creating two multilinear regression models

$$Y=\alpha_o + \alpha_1X_1 + \alpha_2W$$


$$Y = \beta_o + \beta_1X_2 + \beta_2W$$

and then using some average-like combination of these two regression-based predictors to predict $Y$? What weights might we use? Again, is there any literature on this topic?

EDIT: Since I posted this, I've read about stacked regression, as described here. Does the situation I've described sound suitable for applying stacked regression?

  • 1
    $\begingroup$ Note that multicolinearity is not a problem for predictive purposes as it just means there is more than one optimal model. you can write the least squares estimate of $\beta$ as $\hat{\beta}=X^{+}Y$ where $X^{+}$ is the moore penrose pseudo inverse of $X$. this reduces to the usual least squares estimate when $X$ has full column rank but still exists uniquely in case of multicolinearity. it resolves the non uniqueness of the original least squares estimates by taking the beta with the smallest sum of squares and corresponds to limiting ridge regression as penalty goes to zero. $\endgroup$ – probabilityislogic Mar 23 '12 at 23:04
  • $\begingroup$ IF (and this is a big if) the $Y_i$ are unbiased estimators of a parameter, then any weighted average $\hat{\theta} =w_1Y_1+...+w_nY_n$ if $w_1+...+w_n=1$ will be unbiased and this includes the average that you showed. What is good depends on what loss function you want to minimize. If you know the variance $\sigma_i^2$ of each observation, the observations are independent, and a squared error loss makes sense, then the average with $w_i= 1/\sigma_i^2$ has lowest mean squared error. $\endgroup$ – AlaskaRon Dec 21 '15 at 2:31

Just to help keep perspective, it is useful to remember that model averaging is a complex and hard to interpret process that does not surpass a global penalized model in terms of predictive accuracy. That is, if all the models are part of the same family so that a global all-inclusive model can be formed. So if you fit a more saturated model (e.g., include main effects of all potential predictors and expand them with regression splines so as to not assume linearity) but you use penalized maximum likelihood estimation or Bayesian estimation so as to avoid overfitting, you will get a very competitive result that is easier to interpret, as you can estimate the partial effect of any one variable by simply looking at its partial coefficients. Model simplification (also called pre-conditioning) can also be of help here.

  • $\begingroup$ Ive been wondering if penalised methods such as ridge regression are basically doing model averaging over all possible sub-models within the same regression class. my main intuition for this is that a nodel averaged estimates are shrunk toward zero in the orthogonal case, as they are in ridge regression. Do you know of any theoretical work in this area? $\endgroup$ – probabilityislogic Mar 23 '12 at 23:18
  • $\begingroup$ I wish I did. I don't think the two are formally the same in any way but that appropriate shrinking of full models is a much easy way to obtain predictions that are as good as work-intensive and hard to interpret averages of multiple models. $\endgroup$ – Frank Harrell Mar 26 '12 at 13:23

Concerning model averaging I would use other metrics such as harmonic mean for example. Harmonic mean as it offers interesting properties when averages.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.