Approaches to average forecast in machine learning? I have researched some approaches so far. My situation is:
I have 9 different models, all targeting on the same time-series variable.
Now I want to combine these 9 forecasts to estimate a better, combined forecast.
Approaches I have researched so far:


*

*Equally weighted averaging, very naive, but unfortunately also very hard to beat...


*

*Regression approaches, minimizing R^2

*Weighting according the historical MSE, MAE, RMSE..

*Bayesian model averaging
Do you know more references / methods for this setup?
 A: I actually have written a little paper on forecast combinations (Kolassa, 2016, International Journal of Forecasting). You already have a very good grasp of the prevalent ways of combining forecasts.
I looked at weighting different forecasts by information criteria (specifically, Akaike weights, derived from the Akaike Information Criterion, AIC), as well as some other approaches, like the simple mean or median, or trimming or winsorizing.
My conclusion was that it's really hard to beat simple averages. And judging from what I have seen since in the academic literature, I'd say this still holds true, unless you have some very specific data. Claeskens et al. (2016, International Journal of Forecasting) offer some additional explanation for the good performance of unweighted averages - the idea being that if you try to estimate "optimal" weights based on data, you have an additional source of variation, and typically the improvement in average accuracy will not be enough to outweigh this.
One other thing seems to be common forecaster folklore: make sure the constituent methods are dissimilar. It's typically better to average a judgmental forecast, a neural network forecast and an exponential smoothing forecast than just three different exponential smoothing forecasts.
You may want to look through the references in these papers, and/or see who cited them. Or search for "forecast combination" in the International Journal of Forecasting.
