Let's assume we've found 100 independent variables that can predict y. Each of those independent variable are close to uncorrelated and they are all curve fitted. Using any single one to predict y equates to a large probability of failed out of sample predictive performance. But if we combine all of them via some sort of ensemble averaging, the final prediction would be more robust. The logic here is similar to diversifying across each independent variables predictiveness since it's highly unlikely that all of them will fail together. Does this logic make sense? Are there any literature in predictive modelling that discuss this subject more?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ Perhaps, you could use ensemble methods. Also, feel free to take a look at my answer on model averaging - you might find it helpful, as it contains additional information. $\endgroup$– Aleksandr BlekhCommented Mar 5, 2015 at 7:39
-
$\begingroup$ Thanks for the note. I was more curious in whether curve fitting, a bad practice, can be partially alleviated when we combine them into an ensemble. From my experience, if individual models themselves do not have predictive ability out of sample, their aggregate performance taken together usually don't improve forecasting ability. I may be wrong though. $\endgroup$– user1234440Commented Mar 5, 2015 at 10:20
-
$\begingroup$ You're welcome. I don't have practical experience with ensemble methods so far, so couldn't be of much more help. I'm sure other people will share their opinion on this topic, which I find quite interesting. I will add more information as answer to simplify formatting. $\endgroup$– Aleksandr BlekhCommented Mar 5, 2015 at 10:59
-
$\begingroup$ Awarding the bounty to my answer is much appreciated! :-) $\endgroup$– Aleksandr BlekhCommented Mar 13, 2015 at 8:17
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
In addition to my comments above and links within, I'd like to share a bit more information, which is hopefully relevant and helpful. It seems that ensemble methods, such as ensemble Bayesian model averaging (EBMA), indeed improve predictive ability of individual models as well as offer certain other benefits. For example, Montgomery and Hollenbach (2012) write:
Yet, combining forecasts, and ensemble methods in particular, have been shown to substantially reduce prediction error in two important ways. First, across subject domains, ensemble predictions are usually more accurate than any individual component model. Second, they are significantly less likely to make dramatically incorrect predictions (Bates and Granger 1969; Armstrong 2001; Raftery et al. 2005). Combining forecasts not only reduces reliance on single data sources and methodologies (which lowers the likelihood of dramatic errors), but also allows for the incorporation of more information than any one model is likely to include in isolation.
A paper by Singh, Mishra and Ruskauf (2010) provides an interesting comparison of a subset of model averaging techniques, which include three types: frequentist, Bayesian and information theory-based. Finishing on a practical note, I would like to share a page from a popular Python machine learning library scikit-learn, dedicated to several frequentist ensemble methods.
References
Montgomery, J. M., & Hollenbach, F. (2012). Improving predictions using ensemble Bayesian model averaging. [Working paper] Retrieved from http://pages.wustl.edu/montgomery/ebma
Singh, A., Mishra, S., & Ruskauff, G. (2010). Model averaging techniques for quantifying conceptual model uncertainty. Ground Water, 48(5), 701-715.
answered Mar 5, 2015 at 11:25