I've been following Kaggle competitions for a long time and I come to realize that many winning strategies involve using at least one of the "big threes": bagging, boosting and stacking.

For regressions, rather than focusing on building one best possible regression model, building multiple regression models such as (Generalized) linear regression, random forest, KNN, NN, and SVM regression models and blending the results into one in a reasonable way seems to out-perform each individual method a lot of times.

Of course, a solid understanding of each method is the key and an intuitive story can be told based on a linear regression model, but I'm wondering if this has become the state of art methodology in order to achieve the best possible results.

  • $\begingroup$ In some cases, Neural Network well define beat the "classic" way of doing regression. For exemple, in How much did it rain II. But it is definitely a black box. $\endgroup$ – YCR Jan 18 '16 at 12:51
  • $\begingroup$ @YCR I agree it's a blackbox. While at work, I built some awesome machine learning model and tried to explain to business people or someone who's not familiar with the model, the conversation usually ends up being like this: I built an awesome Machine Learning model, It works like magic, but I can't tell you an interesting story. $\endgroup$ – Maxareo Jan 19 '16 at 14:11

It is well-known, at least from the late 1960', that if you take several forecasts and average them, then the resulting aggregate forecast in many cases will outperform the individual forecasts. Bagging, boosting and stacking are all based exactly on this idea. So yes, if your aim is purely prediction then in most cases this is the best you can do. What is problematic about this method is that it is a black-box approach that returns the result but does not help you to understand and interpret it. Obviously, it is also more computationally intensive than any other method since you have to compute few forecasts instead of single one.

† This concerns about any predictions in general, but it is often described in forecasting literature.

Winkler, RL. and Makridakis, S. (1983). The Combination of Forecasts. J. R. Statis. Soc. A. 146(2), 150-157.

Makridakis, S. and Winkler, R.L. (1983). Averages of Forecasts: Some Empirical Results. Management Science, 29(9) 987-996.

Clemen, R.T. (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5, 559-583.

Bates, J.M. and Granger, C.W. (1969). The combination of forecasts. Or, 451-468.

Makridakis, S. and Hibon, M. (2000). The M3-Competition: results, conclusions and implications. International journal of forecasting, 16(4), 451-476.

Reid, D.J. (1968). Combining three estimates of gross domestic product. Economica, 431-444.

Makridakis, S., Spiliotis, E., and Assimakopoulos, V. (2018). The M4 Competition: Results, findings, conclusion and way forward. International Journal of Forecasting.

  • 1
    $\begingroup$ The link in the daggered footnote does not seem to work for me? $\endgroup$ – Silverfish Dec 10 '15 at 22:46
  • $\begingroup$ @Silverfish thanks, fixed. The link was of minor importance but still, if it doesn't work it's useless. $\endgroup$ – Tim Dec 11 '15 at 7:37

Arthur (1994) has a nice short paper/thought experiment that is well-known in the complexity literature.

One of the conclusions there is that agents cannot select better predictive models (even if they have a "forest" of these) under non-equilibrium conditions. For example, if the question is applied to stock market performance, the setting of Arthur (1994) might be applicable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.