When to use multiple models for prediction?

Question

This is a fairly general question:

I have typically found that using multiple different models outperforms one model when trying to predict a time series out of sample. Are there any good papers that demonstrate that the combination of models will outperform a single model? Are there any best-practices around combining multiple models?

Some references:

• Hui Zoua, Yuhong Yang "Combining time series models for forecasting" International Journal of Forecasting 20 (2004) 69– 84

Answer 1

Sometimes this kind of models are called an ensemble. For example this page gives a nice overview how it works. Also the references mentioned there are very useful.

Answer 2

The final year of the NetFlix Prize competition (2009) seemed to me to have sharply changed the general community-wide presumption against combining multiple learning algorithms.

For instance, my formal training (university courses) and later on-the-job oversight/mentoring taught us to avoid algorithm combination unless we had an explicit reason to do so--and "to improve resolution of my current algorithm", wasn't really deemed a good reason. (Others might have a different experience--of course i'm inferring a community-wide view based solely on my own experience, though my experience in coding poorly-performing ML algorithms is substantial.)

Still, there were a few "patterns" in which combining algorithms in one way or another was accepted, and actually improved performance. For me, the most frequent example involved some ML algorithm configured in machine mode (assigning a class label to each data point) and in which there were more than two classes (usually many more). When for instance, using a supervised-learning algorithm to resolve four classes, and we would see excellent separation except for let's say Class III versus Class IV. So out of those six decision boundaries, only one resolved below the required threshold. Particularly when classes III and IV together accounted for a small percent of the data, adding an additional algorithm optimized just on the resolution of those two classes, was a fairly common solution to this analytical problem type. (Usually that 'blind spot' was an inherent limitation of the primary algorithm--e.g., it was a linear classifier and the III/IV decision boundary was non-linear.

In other words, when we had a reliable algorithm suited to the processing environment (which was usually streaming data) and that performed within the spec except for a single blind spot that caused it to fail to resolve two (or more) classes that accounted for a small fraction of the data, then it was always better to 'bolt-on' another specialized algorithm to catch what the main algorithm was systematically missing.

Lastly, on this topic, i would like to recommend highly Chapter 17, Combining Multiple Learners, in Introduction to Machine Learning, 2d, by Ethem Alpaydin, MIT Press, 2010. Note that this is the second edition published a few months ago; the first edition was published in 2004 and i doubt it has the same coverage of this topic. (Actually i recommend the entire text, but that chapter in particular since it relates to Shane's Question.)

In 25 pages, the author summarizes probably every ML algorithm-combination scheme whose utility has been demonstrated in the academic literature or practice--e.g., bagging, boosting, mixture of experts, stacked generalization, cascading, voting, error-correcdting,....

Answer 3

Following up on Peter's response on ensemble methods:

• This is covered in "The Elements of Statistical Learning" (see page 288, for example).
• Witten and Frank "Data Mining: Practical Machine Learning Tools and Techniques" covers this in section 7.5, including a discussion of Bagging, Randomization, Boosting, Additive regression, Additive logistic regression, Option trees, Logistic model trees, and Stacking.
• This is covered in Chapter 14 of Christopher M. Bishop "Pattern Recognition and Machine Learning", including Bayesian Model Averaging, Boosting, Committees, Tree-based Models, and Conditional Mixture Models.

Answer 4

The most spectacular example is the Netflix challenge, which made really boosted blending popularity.

Answer 5

Here's a slightly out of left-field answer, that only touches the "best-practices around combining multiple models" part of your question. This is basically exactly my honours thesis, except that I'm dealing with complex, highly non-linear models that exhibit chaos and noise - climate models. This isn't likely to be broadly applicable to many fields, but might be useful in ecology or econometrics.

Until fairly recently in the climate modelling community, models were largely just smashed together in an unweighted average (usually after bias correction involving removing the model mean for part or all of the sample period). This is basically what the IPCC did for the 4th assessment report (4AR), and previous reports.

This is more or less an example of the "truth plus error" school of ensemble combination, where it is tacitly or explicitly assumed that observational series (eg. global temperature, local precipitation, etc) is true, and that if you take enough samples (eg. model runs), the noise in the model runs will cancel (see (1)).

More recently, methods for combining models based on performance weighting have been used. Because climate models are so noisy, and have so many variables and parameters, the only ways of assessing the performance (that I know of) are by taking the covariance, or by taking the MSE between the model output and the observed time series. Models can then be combined by weighting the mean based on that measure. There's a good overview of this in (2).

One assumption behind this method of combining simulations is assumption that the models are all reasonably independent - if some were highly dependent, they would bias the mean. This assumption was reasonably fair for the dataset used for 4AR (CMIP3, as this data set was made up of few model runs from many modelling groups (on the other hand, code is shared in the modelling community, so there may still be some interdependence. For an interesting look at this, see (3)). The dataset for the next assessment report, CMIP5, does not have this somewhat fortuitous attribute - some modelling teams will be submitting a few runs, while some will be submitting hundreds. Ensembles coming from different teams may be produced by initial condition peturbation, or by changes to the model physics and parametrisation. Also, this super ensemble isn't sampled in any systematic way - it's just who ever brings data is accepted (within reason). This is known in the field as an "ensemble of opportunity". There's a fair chance that using an unweighted mean on such an ensemble is going to git you some major bias toward the models with more runs (since even though there are hundreds of runs, there are likely a much smaller number of truly independent runs).

My supervisor has a paper in review at the moment describing a process of model combination involving performance AND independence weighting. There is a conference paper abstract available (4), I'll post the link to the paper when it's published (slow process, don't hold your breath). Basically, this paper describes a process that involves taking the covariance of model errors (model-obs), and weighting down models that have high covariance with all other models, (ie. models with highly dependent errors). The model error variance is computed as well, and used as the performance-weighting component.

It's also worth noting that climate modelling is obviously hugely impacted by the vagaries of numerical modelling in general. There's a thing called a "laugh test" - if you end up with a model run that implies that global mean temperatures will be +20°C by 2050, you just throw it out, because it's clearly not physically relevant. Obviously this kind of test is fairly subjective. I haven't required it yet, but I expect to in the near future.

That's my understanding of the state model combination in my field at the moment. Obviously I'm still learning, so if I hit on anything special, I'll come back and update this answer.

(1) Tebaldi, C. & Knutti, R., 2007. The use of the multi-model ensemble in probabilistic climate projections. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 365(1857), pp.2053–2075.

(2) Knutti, R. et al., 2010. IPCC Expert Meeting on Assessing and Combining Multi Model Climate Projections.

(3) Masson, D. & Knutti, R., 2011. Climate model genealogy. Geophys. Res. Lett, 38(8), p.L08703.

(4) Abramowitz, G. & Bishop, C., 2010. Defining and weighting for model dependence in ensemble prediction. In AGU Fall Meeting Abstracts. p. 07.