# Ensembles - few questions on approach with multiple models

I'm looking for some general-high level understanding of how I should apply ensemble techniques. I understand that some models can be already thought of as ensembles - such as random forests or some other sort/variation of bagged tree approach.

However, the part I need more guidance on is when I have multiple models. For example if I am trying to classify a binary event and I run:

• random forest
• naive bayes
• logistic regression
• gbm
• svm
• neural network
• KNN

I get a bunch of different predictions of 1 or 0. Some of these models might be pretty accurate such as gbm and random forest, while others are not as accurate such as the naive bayes. If I take a voted average of 7 of these models or maybe 3 or 5 I might end up doing slightly better than just the output from Random Forest or GBM alone.

However, I'm looking for a more 'scientific' approach to this process. Yes - I know at this point you need to 'check' on your test set any approach to see how things go. However, I feel like I get to this point and I'm somewhat blind to what I should actually be doing.

• Should I be running a decision tree of each of the 7 predictions above (0s and 1s) to predict the true target and then running this final model on the test set? Logistic regression instead?
• Same thing but instead of using the 1s and 0s use the probability values obtained from predictions instead? Does this part even make sense as don't models interpret the probability in different scales?
• How exactly would I 'weight' the different models I guess more so in the event that this is a regression problem than classification? For example if I was in a high dimensions I wouldn't expect the KNN classifier to work that great.
• I read somewhere that you don't need to worry about 'overfitting' when developing the individual models here because of the concept of generating an ensemble is to help by taking advantage of the different tendencies of different models to predict something.

Thank you for your help. I'm looking for a bit more conceptual direction rather than any mathematical proofs at this point.

• Model averaging: this is what you do with your majority vote over $n$ models. There exist weighted model combinations as well (e.g. weight by individual performance of model).
• Bagging: take a subset of samples and/or features for each model, thereby train each model type multiple times. This will likely give you more than $n$ models that you can use to combine predictions of.
• Model stacking: based on the predictions = outputs of your $n$ models you could train another model. This model tries to predict the actual output using the $n$ outputs of previous models as input (therefore the name "stacking"). This is what you mentioned in your 1st and 2nd bullet point. You could use both binary (0/1) output and continuous probability output, but from a gut feeling I'd say continuous output might be more successful as it transports more information.