Ensembles - few questions on approach with multiple models

Question

I'm looking for some general-high level understanding of how I should apply ensemble techniques. I understand that some models can be already thought of as ensembles - such as random forests or some other sort/variation of bagged tree approach.

However, the part I need more guidance on is when I have multiple models. For example if I am trying to classify a binary event and I run:

• random forest
• naive bayes
• logistic regression
• gbm
• svm
• neural network
• KNN

I get a bunch of different predictions of 1 or 0. Some of these models might be pretty accurate such as gbm and random forest, while others are not as accurate such as the naive bayes. If I take a voted average of 7 of these models or maybe 3 or 5 I might end up doing slightly better than just the output from Random Forest or GBM alone.

However, I'm looking for a more 'scientific' approach to this process. Yes - I know at this point you need to 'check' on your test set any approach to see how things go. However, I feel like I get to this point and I'm somewhat blind to what I should actually be doing.

• Should I be running a decision tree of each of the 7 predictions above (0s and 1s) to predict the true target and then running this final model on the test set? Logistic regression instead?
• Same thing but instead of using the 1s and 0s use the probability values obtained from predictions instead? Does this part even make sense as don't models interpret the probability in different scales?
• How exactly would I 'weight' the different models I guess more so in the event that this is a regression problem than classification? For example if I was in a high dimensions I wouldn't expect the KNN classifier to work that great.
• I read somewhere that you don't need to worry about 'overfitting' when developing the individual models here because of the concept of generating an ensemble is to help by taking advantage of the different tendencies of different models to predict something.

Thank you for your help. I'm looking for a bit more conceptual direction rather than any mathematical proofs at this point.

Answer 1

In general, with ensembles you have a number of well known possibilities:

• Model averaging: this is what you do with your majority vote over $n$ models. There exist weighted model combinations as well (e.g. weight by individual performance of model).
• Bagging: take a subset of samples and/or features for each model, thereby train each model type multiple times. This will likely give you more than $n$ models that you can use to combine predictions of.
• Boosting: more complicated: uses subsets of samples for each model. For all chosen sample, initially assigns equal weights to all samples, then iteratively trains multiple models using those weight. Increases weight in case samples is classified wrong and decreases weight in case sample in classified right - with models concentrating more on weighted samples during training (e.g. larger error), the focus is always on "those samples that are currently difficult to classify). The final prediction is again the average of all individual models. Using boosting, you can combine multiple weak and unstable models into one strong and relatively stable one. This might be what you want to achieve - but keep in mind that this requires you to use a modified training procedure.
• For completeness: random forest: use subset of samples for each model and subset of features in each split (similar to bagging).
• Model stacking: based on the predictions = outputs of your $n$ models you could train another model. This model tries to predict the actual output using the $n$ outputs of previous models as input (therefore the name "stacking"). This is what you mentioned in your 1st and 2nd bullet point. You could use both binary (0/1) output and continuous probability output, but from a gut feeling I'd say continuous output might be more successful as it transports more information.

Technically, you can use all those over multiple model types and even combine multiple approaches - so this should give you some options.

The "don't worry about overfitting of individual models" you mention in the last bullet point is only partially true. Yes, ensembles have an overall tendency to decrease variance (=decrease overfitting), but under the hood this relies on e.g. the individual models being unstable to a certain extent. For example, with a random forest, each individual tree usually is unstable. The overall instability leads to different models using many different ways of classification (and thereby each overfitting in a different way), and those different forms of overfitting ideally cancel each other out with the overall prediction - to some extent. This is also one of the reasons for using a subset of samples and features for training each individual model: this causes the models to be different. But in general, for most setups each model should try to overfit as least as possible. A counterexample to ensembles reducing variance would be using some deterministic model (to make one up: nonlinear logistics regression with fixed seeds) that will a) overfit your samples and b) always converge to (nearly) the same solution. Using multiple such models will naturally not counter your overfitting.