Ensemble models in R I have a clinical dataset (1400 cases) and I applied 4 data mining techniques (ANN, Decision Tree, SVM, Logistic Regression) to predict the binary outcome (Yes, No).
Now, I want to improve prediction accuracy through ensemble methods.
What are the criteria to choose which model can be combined with another model?
And how can that be done in R? Can I use the "caret" package?
 A: Let reverse some answers to your questions:


*

*Yes this can be done in R.

*You can use the caret package to compare models, but for automatically build an ensemble you can use the package caretEnsemble. Read the vignettes first!

*Creating ensembles is as much art as it is science if you want to do it manually. But it gives you more control over what is happening. It all depends a bit on which kind of assembling you want to do.
Voting ensembles: combine the outcome of multiple predictions and have a majority vote. I.e. if you have the predictions of 3 models, and 2 models predict a 1 and 1 model a 0, the outcome is 1.
Averaging: Average the outcome of multiple models by taking the mean.
With both voting and averaging, less highly correlated models work better than highly correlated models. But even highly correlated models might improve the final answer, so it bears checking out. 
There are more methods, but starting with these is a good way to see what is going on.
A very good guide to this is written by mlwave. There is more information there also about stacking and blending. 
A: To answer in sequence :
1) You should note that the predicted values of the models in the ensemble should be disparate( may be check their Root mean square error values). Ideally, for a successful ensemble, you should combine a set of weak learners or a set of weak learners + strong learners. This helps the ensemble to learn more complex patterns in data. At each iteration , do check on cross validation to avoid overfitting.
2) You can either use the models from caret package. Also, there is caretEnsemble and caretStack packages. Please read up on these.
