If I estimate a collection of models predicting $Y$ by $\hat{Y}$, which methods are out there to combine these forecasts? Which methods work well/best (and why?) to improve prediction accuracy? My interest is of theoretical nature, for frequentist and bayesian approaches alike.

I am aware that this question is very open, but I want to gain an overview. Consequently, references to further sources or survey papers, book chapters, ... are also highly appreciated!

Edit: Bagging, boosting and stacking in machine learning was poposed as an answer to this question. I was very thankful for the link, and I recommend everyone interested in this question to read the post and its answers if they haven't done so already. However, the post does not answer my question: I am interested in specific methods for model averaging and bagging. The aforementioned post elaborates on the differences between the concepts of 'boosting', 'bagging', and 'stacking' rather than giving explicit different implementations. (e.g., model averaging use weights for each model. What I want to enquire with this post is the ways in which these weights can be obtained.)

  • 1
    $\begingroup$ As you have already mentioned in your title I would start with looking at methods for bagging but also check out product of expert models and the Bayesian committee machine. $\endgroup$ Commented Feb 6, 2016 at 18:53
  • $\begingroup$ Thanks! Do you know a good source for me to read up details? :) $\endgroup$
    – Jeremias K
    Commented Feb 6, 2016 at 18:57
  • 1
    $\begingroup$ Possible duplicate of Bagging, boosting and stacking in machine learning $\endgroup$
    – Tim
    Commented Feb 6, 2016 at 18:59
  • 1
    $\begingroup$ There's an overview paper on model averaging in the Journal of Economic Surveys, see here. It be of interest to you to check out Clements and Hendry (1999) who present a theory of forecasting non-stationary time series. Their taxonomy of forecast errors will help you on the issue of improving, or at least understanding, forecast performance. As far as I know, they've worked quite a bit on developing the theory of forecasting. See recent papers also. $\endgroup$ Commented Feb 6, 2016 at 21:36
  • 1
    $\begingroup$ Oh snap, that's a sweet overview! Have to look into it in more detail tomorrow. Hendry is always a good read, too - thank you! $\endgroup$
    – Jeremias K
    Commented Feb 6, 2016 at 21:50


Browse other questions tagged or ask your own question.