13
$\begingroup$

This is a fairly general question:

I have typically found that using multiple different models outperforms one model when trying to predict a time series out of sample. Are there any good papers that demonstrate that the combination of models will outperform a single model? Are there any best-practices around combining multiple models?

Some references:

$\endgroup$
  • $\begingroup$ i added what i think is a pretty good source (unfortunately a Textbook) w/ annotation, in light of your comment/question below my answer. I edited my original answer, so it appears at the end. $\endgroup$ – doug Aug 4 '10 at 7:07
8
$\begingroup$

Sometimes this kind of models are called an ensemble. For example this page gives a nice overview how it works. Also the references mentioned there are very useful.

| cite | improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ Indeed, blending is one of the possible ensemble techniques. In particular, there are two when you combine the same sort of classifier, boosting (like Adaboost) and bagging (like Random Forest), and blending, where you combine different classifiers (what was Shane's question about). $\endgroup$ – user88 Jul 23 '10 at 17:10
  • 3
    $\begingroup$ For blending, this paper from the netflix competition is worth reading: the-ensemble.com/content/feature-weighted-linear-stacking. $\endgroup$ – Shane Jul 23 '10 at 17:42
  • 2
    $\begingroup$ IT is fun that meteorologist also use the word "ensemble" but not for combination: they use it for an ensemble of prediction (like scenario) obtained by perturbation of the initial conditions of the numerical model. $\endgroup$ – robin girard Jul 24 '10 at 6:46
  • 1
    $\begingroup$ @mbq in fact they call themselves forecaster and they use statistic quite a lot ... $\endgroup$ – robin girard Jul 26 '10 at 6:46
  • 1
    $\begingroup$ @robin I know, this is just why it's called "ensemble" not a set or something like this. $\endgroup$ – user88 Jul 26 '10 at 7:46
11
$\begingroup$

The final year of the NetFlix Prize competition (2009) seemed to me to have sharply changed the general community-wide presumption against combining multiple learning algorithms.

For instance, my formal training (university courses) and later on-the-job oversight/mentoring taught us to avoid algorithm combination unless we had an explicit reason to do so--and "to improve resolution of my current algorithm", wasn't really deemed a good reason. (Others might have a different experience--of course i'm inferring a community-wide view based solely on my own experience, though my experience in coding poorly-performing ML algorithms is substantial.)

Still, there were a few "patterns" in which combining algorithms in one way or another was accepted, and actually improved performance. For me, the most frequent example involved some ML algorithm configured in machine mode (assigning a class label to each data point) and in which there were more than two classes (usually many more). When for instance, using a supervised-learning algorithm to resolve four classes, and we would see excellent separation except for let's say Class III versus Class IV. So out of those six decision boundaries, only one resolved below the required threshold. Particularly when classes III and IV together accounted for a small percent of the data, adding an additional algorithm optimized just on the resolution of those two classes, was a fairly common solution to this analytical problem type. (Usually that 'blind spot' was an inherent limitation of the primary algorithm--e.g., it was a linear classifier and the III/IV decision boundary was non-linear.

In other words, when we had a reliable algorithm suited to the processing environment (which was usually streaming data) and that performed within the spec except for a single blind spot that caused it to fail to resolve two (or more) classes that accounted for a small fraction of the data, then it was always better to 'bolt-on' another specialized algorithm to catch what the main algorithm was systematically missing.

Lastly, on this topic, i would like to recommend highly Chapter 17, Combining Multiple Learners, in Introduction to Machine Learning, 2d, by Ethem Alpaydin, MIT Press, 2010. Note that this is the second edition published a few months ago; the first edition was published in 2004 and i doubt it has the same coverage of this topic. (Actually i recommend the entire text, but that chapter in particular since it relates to Shane's Question.)

In 25 pages, the author summarizes probably every ML algorithm-combination scheme whose utility has been demonstrated in the academic literature or practice--e.g., bagging, boosting, mixture of experts, stacked generalization, cascading, voting, error-correcdting,....

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ this is great information. Do you know of any papers that cover this? $\endgroup$ – Shane Jul 29 '10 at 19:39
  • $\begingroup$ (didn't get a notifier from SA of your comment) Well, i wasn't referring to any papers when i wrote that, rather just informally summing pieces of my experience relevant to your Question. I'll look through my files and see what i have that is relevant though. $\endgroup$ – doug Aug 2 '10 at 18:32
4
$\begingroup$

Following up on Peter's response on ensemble methods:

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

The most spectacular example is the Netflix challenge, which made really boosted blending popularity.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

Here's a slightly out of left-field answer, that only touches the "best-practices around combining multiple models" part of your question. This is basically exactly my honours thesis, except that I'm dealing with complex, highly non-linear models that exhibit chaos and noise - climate models. This isn't likely to be broadly applicable to many fields, but might be useful in ecology or econometrics.

Until fairly recently in the climate modelling community, models were largely just smashed together in an unweighted average (usually after bias correction involving removing the model mean for part or all of the sample period). This is basically what the IPCC did for the 4th assessment report (4AR), and previous reports.

This is more or less an example of the "truth plus error" school of ensemble combination, where it is tacitly or explicitly assumed that observational series (eg. global temperature, local precipitation, etc) is true, and that if you take enough samples (eg. model runs), the noise in the model runs will cancel (see (1)).

More recently, methods for combining models based on performance weighting have been used. Because climate models are so noisy, and have so many variables and parameters, the only ways of assessing the performance (that I know of) are by taking the covariance, or by taking the MSE between the model output and the observed time series. Models can then be combined by weighting the mean based on that measure. There's a good overview of this in (2).

One assumption behind this method of combining simulations is assumption that the models are all reasonably independent - if some were highly dependent, they would bias the mean. This assumption was reasonably fair for the dataset used for 4AR (CMIP3, as this data set was made up of few model runs from many modelling groups (on the other hand, code is shared in the modelling community, so there may still be some interdependence. For an interesting look at this, see (3)). The dataset for the next assessment report, CMIP5, does not have this somewhat fortuitous attribute - some modelling teams will be submitting a few runs, while some will be submitting hundreds. Ensembles coming from different teams may be produced by initial condition peturbation, or by changes to the model physics and parametrisation. Also, this super ensemble isn't sampled in any systematic way - it's just who ever brings data is accepted (within reason). This is known in the field as an "ensemble of opportunity". There's a fair chance that using an unweighted mean on such an ensemble is going to git you some major bias toward the models with more runs (since even though there are hundreds of runs, there are likely a much smaller number of truly independent runs).

My supervisor has a paper in review at the moment describing a process of model combination involving performance AND independence weighting. There is a conference paper abstract available (4), I'll post the link to the paper when it's published (slow process, don't hold your breath). Basically, this paper describes a process that involves taking the covariance of model errors (model-obs), and weighting down models that have high covariance with all other models, (ie. models with highly dependent errors). The model error variance is computed as well, and used as the performance-weighting component.

It's also worth noting that climate modelling is obviously hugely impacted by the vagaries of numerical modelling in general. There's a thing called a "laugh test" - if you end up with a model run that implies that global mean temperatures will be +20°C by 2050, you just throw it out, because it's clearly not physically relevant. Obviously this kind of test is fairly subjective. I haven't required it yet, but I expect to in the near future.

That's my understanding of the state model combination in my field at the moment. Obviously I'm still learning, so if I hit on anything special, I'll come back and update this answer.

(1) Tebaldi, C. & Knutti, R., 2007. The use of the multi-model ensemble in probabilistic climate projections. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 365(1857), pp.2053–2075.

(2) Knutti, R. et al., 2010. IPCC Expert Meeting on Assessing and Combining Multi Model Climate Projections.

(3) Masson, D. & Knutti, R., 2011. Climate model genealogy. Geophys. Res. Lett, 38(8), p.L08703.

(4) Abramowitz, G. & Bishop, C., 2010. Defining and weighting for model dependence in ensemble prediction. In AGU Fall Meeting Abstracts. p. 07.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ first paragraph you mean "complex, highly non-linear models" right? For my area of work (non-climate area), I often find that using different models doesn't generally lead to greatly different predictions. However, just getting the computing up and running for combining models in a non ad-hoc way is a massive barrier for us. I would expect more accurate predictions, but I don't have the time to combine the models and accurately estimate the error in those predictions. $\endgroup$ – probabilityislogic Mar 23 '12 at 11:48
  • $\begingroup$ Thanks, fixed. I have access to acomputing cluster, so computing power isn't a huge problem, but yeah, even getting the one model I'm using set up properly is a pain, and it's already written. Which is why CMIP exists, so people don't have to go through all that trouble each time. Would be interested ti know what field you're in, if you're doing similar stuff. $\endgroup$ – naught101 Mar 24 '12 at 2:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.